# Presentation: 127413

## The Software Engineering Side of Computational Physics.

- PPPL Colloquium
- Jan. 29, 2003
- D. McCune

- _[http://w3.pppl.gov/~dmccune/papers-reports/colloquium_2003.htm](http://w3.pppl.gov/~dmccune/papers-reports/colloquium_2003.htm)_

## The Software Engineering Side of Computational Physics.

*(PPPL Colloquium, Jan 29, 2003, by D. McCune).*
_**Abstract:**_
- Software-driven computation is playing an ever greater role in all fields of physics,
- due in large part to the rapidly increasing power and falling cost of computers.
- Whether the aim is analysis of new experimental results, or the exploration of the
- implications of new theoretical ideas, computational physics is, increasingly, the
- tool of choice.  Large computational software systems have evolved to meet these
- challenges.  When these systems become large enough, they can no longer be
- supported in the traditional way by individual theoretical or experimental
- physicists.  New specialists– often, teams of physicists, computational scientists,
- and engineers– emerge to support the continued development of these
- computational software systems, further enhancing their increasingly critical role in
- physics research.  PPPL has considerable experience in such processes.  Success
- requires a cross-disciplinary approach.  In this colloquium, the author’s view of
- PPPL’s experience is explored, with emphasis on the aspects of the process which
- are not strictly physics.

## Outline

- Motivation for Computational Physics.
- Growth of Computational Physics.
- Falling cost of computational hardware.
- The Software Engineering Challenge.
- Large software systems.
- Multi-disciplinary requirement.
- Summary and Conclusions.

## Motivation for Computational Physics

- Study and analyze experimental results.
- Explore implications of theory.
- Reconcile theory and experiment.
- Advance the state of scientific knowledge.

## Computational Plasma Physics Application: TRANSP

- Analyze tokamak experiments.
- Use measurements directly whenever possible.
- Simulate the rest.
- Compare additional data using diagnostic simulations.

- Diagnostic Data

- Simulation

- Comparison

- Experiment

- Knowledge

## Use of TRANSP: Examples

- M. C. Zarnstorff et al., “Bootstrap Current in TFTR”, Phys. Rev. Lett. 60, 1306 (1988).
- R. V. Budny et al., “A Standard DT Supershot Simulation”, Nuclear Fusion 34, 1247 (1994).
- Following two slides from APS review talk “Confinement and Heating of DT Plasmas in TFTR” by M. Bell:
- _[http://w3.pppl.gov/tftr/info/aps9903/GP01_104-BellM.pdf](http://w3.pppl.gov/tftr/info/aps9903/GP01_104-BellM.pdf)_

## Slide 7

## Slide 8

## More Computational Plasma Physics Links

- _[http://w3.pppl.gov/CEMM](http://w3.pppl.gov/CEMM)_
- _[http://w3.pppl.gov/theory/proj_gksim.html](http://w3.pppl.gov/theory/proj_gksim.html)_
- _[http://gs2.sourceforge.net](http://gs2.sourceforge.net/)_
- _[http://web.gat.com/theory](http://web.gat.com/theory)_
- _[http://www.fusiongrid.org](http://www.fusiongrid.org/)_
- _[http://w3.pppl.gov/NTCC](http://w3.pppl.gov/NTCC)_
- _[http://w3.pppl.gov/TOPDAC](http://w3.pppl.gov/TOPDAC)_

## Computational Physics: a Growth Industry

- Growth in all fields of physics.
- Examples: DOE SciDAC, plasma physics, high energy physics.
- Spurred by amazing improvements in the price/performance of computers.
- Estimated factor of 100,000 over 25 years.
- Moore’s Law operation over decades.

## Growth of Computational Physics: Examples.

- FY-2002: $15,000,000 for DOE SciDAC.
- FESAC ISOFS subcommittee recommends:
- 15 year Fusion Simulation Project (FSP).
- $20,000,000/year.
- PPPL Computational Plasma Physics Group (CPPG) employs 10 full time staff.
- Enormous collaborations in HENP...

## The Particle Physics Data Grid Collaboratory Project (PPDG)

**The Particle Physics Data Grid Collaboratory Project (PPDG)**
- _[http://www.ppdg.net/docs/presentations/mount_scidac_jan02.ppt](http://www.ppdg.net/docs/presentations/mount_scidac_jan02.ppt)_

## Growth Driven by Advances in Computing Hardware

- Moore’s Law:  price/performance of computers doubles every 18 months.
- Price/performance improvement over the course of 25 years: approximately 100,000.

## Computers, Then and Now...

- 1980

- 2002

## Computers, Then and Now...

- _1980: DEC-PDP10._
- Cost: $1,600,000.*
- Memory: 9 Mbytes.
- Average TRANSP run: 1 to 3 days.
- Footprint: one PPPL Computer Center.

- _2002: Linux laptop._
- Cost: <$3000.00. (Dell).
- Memory: 500 Mbytes.
- Average TRANSP run:  1 to 3 hours.
- Footprint: one lap.

- *1980 dollars (priv. comm.: Paul Funk).

## A User’s Comments on TRANSP (1979 PLT shot #13600) 

**<*=*><*=*><*=*><*=*><*=*><*=*><*=*><*=*><*=*>**
**D= 4-DEC-80  BY:  ROB GOLDSTON     RUN # 1528**
**THE CODE HAS BEEN RUNNING FOR 2HRS AND 20MINS**
**SO FAR AND IT HAS ONLY DONE 100 MSEC OF OH.**
**AND IT IS ONLY DEALING WITH 1 SPECIES PLASMA.**
****SIGH****

- Reference: VMS TR1:[TRANSP.INF.PLT.79]1528TR.INF

## Today, CPU Speed Does Not Limit TRANSP Operations.

- 8 dual processor 1.7ghz Petrel nodes shared by GA & PPPL before 2002 APS.
- Capacity for *16 simultaneous runs* never fully utilized.
- Ability to use TRANSP now limited by human factors:
- Time to prepare input data.
- Time to make sense of simulation results.

## No “Moore’s Law” for Human Beings

- Price/performance improvement of 100,000 in 25 years: *hardware*, not *software*.
- No commensurate improvement in human capabilities:
- Time needed to learn new software.
- Time needed to understand simulation results.
- Time needed to write new code.

## The Computational Physics Software Engineering Challenge:

- Deliver the benefits of reduced cost computation, by making applications:
- More powerful.
- Easier to use.
- Less expensive to develop.
- Less expensive to maintain.
- Job requires multi-disciplinary approach.

## Limitations of Traditional “Single Author” Software Model

- Not scalable to *large* software systems.
- If author is a research physicist:
- Software engineering requirements clash with “publish or perish” imperative:
- Documentation.
- User Support.
- Maintenance.
- Difficult to span multiple physics domains.
- Authors are mortal human beings.

## Multi-Disciplinary Requirement

- Management of *large* software systems requires more than just physics expertise.
- Organization of (say) 1,000,000 line codes.
- Documentation; code development collaboration.
- User interface; visualization of results.
- Software development process.
- Portability, maintainability, reusability, etc.

## Software Development Productivity

- Katrina Maxwell, et al., “_[Software Development Productivity of European Space, Military and Industrial Applications](http://www.datamax-france.com/ieee96.pdf)_”, IEEE Trans. Software Eng., Vol. 22, No. 10, pp. 706-718 (1996).
- Study included 9 large fortran projects.
- Observed productivity range: 265-380 source lines of code per (144 hour) man-month.

## TRANSP Source Code Statistics:

- cd $CODESYSDIR/source
- rm –f mega.source
- foreach dir ( `ls` )
-   if ( $dir != “CVS” ) then
-     cat $dir/*.* >> mega.source
-   endif
- end
- wc –l mega.source

- Result: 1,569,653 source lines – McCune’s laptop, 24-Jan-2003.
- Filtered, to omit comments & doc files:  761,643 lines of code.
- Also found: 190 executable programs, 162 subroutine libraries.

## Nominal IEEE-96 “Replacement Cost” of TRANSP

**[# of source lines not counting comments]**

**12.0*[productivity, # of source lines per month]**

- = (approximately) **180 man-years**.

**x [approximate FY-2003 PPPL Labor Rates]**

- = (approximately) **$36,000,000 (!)**.

## Actual Costs Reduced via Automation

- ~25% of TRANSP source lines written from specifications by “code-generators”:
- TRANSP COMMON & namelist subsystems.
- TRANSP input data acquisition subsystem.
- TRANSP output data generator subsystem.
- Monte Carlo fast ion package i/o interface.
- Code generators improve reliability.
- Code generators written in C, f90, script...

## More Custom Automation Tools:

- Portable makefile generator:
- Build 162 libraries and 190 executables.
- Debug executables with fine-grained selection of debug-compiled components.
- Precision converter for fortran-77 legacy code.
- Recently used for REAL*8 conversion of TRANSP.

## Automation a Good Design Choice When...

**CP DA ****APLASM(IG)****  */1.0/ Atomic Wt of Plasma**

- Common Block
- Declarations:
-    **real*8 ****aplasm(mig)**
**   ****common/comaaa/...**
**  ****>  ****aplasm****,...**

- Default Value Specification

- Namelist Sort & Read

- Checkpoint/Restart File

- Documentation

**Specification Line:**

***...******there are ******numerous****** objects for which a ******concise****** specification***
*** ******maps reliably into ******multiple****** system components.***

## Automated Code Maintenance is Software Engineering to...

- Enhance functionality at reduced cost.
- In TRANSP, code generators automate:
- Insertion of input data channels.
- Creation of namelist and global variables.
- Creation of output profiles and scalars.
- Makefiles for new subroutine libraries and executable programs.
- Checkpoint/restart, etc., etc.

## Another Software Engineering Function: System Design

- Success of TRANSP as a *complete* *system*:
- End-to-end functionality.
- Data & namelist preparation tools.
- Job submission & monitoring.
- Output data visualization.
- Programs and APIs for output data post-processing.

## User Interface Design

- Automation can also help user productivity.
- Escape the “tyranny of forced interaction*”.
- Most interactive tools should allow automation via scripting or similar device.
- Design exercise: imagine program execution without any user attached.
- An old PPPL example: UREAD.

***Kernighan’s phrase, Jan-15 Colloquium**

## RPLOT Session

## UREAD: Scripting & Automation for Fortran Programs

- Program prints menu or instructions.
- Program sends prompt for user input.
- Program parses user’s reply.

- User reads instructions.
- User types response.
*or *--
- Script matches prompt.
- Script supplies response.
- Script is programmable.

## Sample UREAD Script.

**; *RPLOT* MAIN:  ENTER RUNID or 0 TO QUIT:$**
**‘****runid’       ****; “runid” is a script variable.**
**; RPLOT:MAIN  ENTER OPTION NUMBER:$**
**2             ****; requests a scalar plot.**
**; SUB:TPLT2D >> ENTER OPTION NUMBER:$**
**7             ****; requests a “multiplot”**
**; SUB:TPLT2D >> ENTER PACKAGE ID:$**
**xneut         ****; neutrons vs. time.**
**; GRAOPT: ENTER ONE LETTER OPCODE (C/A/...):$**
**Q Q Q         ****; exits program.**

**Invocation: ****rplot @script /runid=37065X09**

- UREAD Scripting language is programmable: support
- for variables, looping, branching, sub-procedures, etc.

## Basic Architecture for UREAD/SGLIB Applications.

**UREAD/SGLIB executable**

**Interactive User**
**  —****or—**
**Script Input.**

**Graphical**
**Output**
**(SGLIB)**

**Data**
**Output**
**(Ufiles, etc.)**

## UREAD History

- Archaic: 1st version on TK4010 screen attached to PDP-DEC-10 (Fortran-IV!).
- Developed for PLT/PDX data preparation and TRANSP output viewing tools.
- Mainstay of TFTR data analysis.
- 1000s of UREAD scripts were written.
- Still in wide use, >20 years later.

## Possible Architecture for Future Interactive Applications.

**Future application code**

**Script Input**

**Graphical**
**Output**
**(ElVis on**
**Internet)**

**Data**
**Output**
**(HDF-5, etc.)**

**Optional**** GUI script**
**generator (Java,**
**IDL, Python/Tk, ...)**

**Command interface**

## Non-Physics Components of Computational Physics Applications

- Automation.
- System design.
- User interface.
*Some examples have been shown*.

## More Non-Physics Components

- Parallelization.
- Algorithms.
- Use of networks:
- Data, e.g. MDS+.
- Grids, e.g. _[Globus](http://www.globus.org/)_.
- Grid-ftp; data streaming.
- Web monitoring.
- _[Collaboration](http://www.fusiongrid.org/)_.

- Visualization.
- Modernization:
- Modern programming languages.
- Portability, reusability, maintainability.
- Code sharing.
- _[Modules Library](http://w3.pppl.gov/NTCC)__._
- Documentation.

## PPPL Computational Plasma Physics Group (CPPG)

- Multi-disciplinary approach to computational plasma physics software.
- S. Jardin & D. McCune, co-heads.
- R. Andre, J. Chen, S. Ethier, E. Feibush, S. Klasky, K. Indireshkumar, C. Ludescher, A. Pletzer, L. Randerson, R. Samtaney.

## Summary and Conclusions

- Computational physics (CP) is useful.
- Low cost of computers gives CP a big opportunity.
- CP applications bound to grow.
- Management of large CP applications requires a lot more than physics expertise.
- Fusion Simulation Project: FESAC recommends: *$20,000,000/year for 15 years*.

- _[http://w3.pppl.gov/~dmccune/papers-reports/colloquium_2003.htm](http://w3.pppl.gov/~dmccune/papers-reports/colloquium_2003.htm)_