Fusion:
The App of our Eyes
David Keyes, Reporter and Questioner
On behalf of your CS and Applied Math colleagues
ISOFS Workshop
San Diego, CA18 September 2002

What we heard yesterday about yourproblem characteristics•

Multiple time scales-•Multiple spatial scales
-••
•
•

•

Range of about fourteen orders of magnitude
Reynolds numbers of 10**8

Linear ill conditioning
Complex geometry and severe anisotropy
Coupled physics, with essential nonlinearities
Variable fidelity models, some with remaining
closure questions
Ambition for predictability and design
ISOFS Workshop

Our pat answers•

Multiple time scales-•Multiple spatial scales-•••••Range of about fourteen orders of magnitude
Reynolds numbers of 10**8

Linear ill conditioning
Complex geometry and severe anisotropy
Coupled physics, with essential nonlinearities
Variable fidelity models, some with remaining
closure questions
Ambition for predictability and design
ISOFS Workshop

More problem characteristics•

Nonlocal operators•

Physical instabilities•

High dimensional independent variable space•

High dimensional dependent variable space•

Mixed dimension code components•

Mixed continuum/particle models, perhaps on
different co-located meshes

ISOFS Workshop

Our confidence•••••

You are devilishly clever
You are foolishly courageous
You are heroically energetic
You care about science, not computing stunts
Your science is as worthy of our own research
investments as any we can find …
Scientific ripeness- Methodological ripeness- Justification of our efforts- Visibility and external impact-ISOFS Workshop

Our fears•

You are wedded to what has worked up until now•

You regard your own specialty code as the base towhich others should adapt theirs•

You think that there is a magic CS solution thatprovides generality of application and ease of use
without counting the cost in performance•

You really do like F90 J•

You think 4,096 processors is like 64 processors,
only more so
ISOFS Workshop

Our issues••

High performance-

Single-node performance-

Parallel scalability

Algorithmic optimality-

Discretization complexity-

Solution complexity•

Verification of the numerics•

Extensibility (to advances in both models and methods)•Reusability (in other problems)•Portability (to other machines)•Ultimate functionality
ISOFS Workshop

Where we would enjoy helping•

Performance engineering-•

Complex geometry-•

Transferring physical quantities between “natural” representations in different code
components

Interpretation of results-•

Optimal schemes adapted to physics of interest, using all physics available

Coupling issues-•

Automated mesh and discretization refinement

Advanced solvers-•

High order, solution-specific schemes

Solution-based adaptivity-•

Unstructured and multicomponent meshes

Advanced discretizations-•

Cache and distributed memory friendly implementations

Convergence, conservation, visualization, advanced post-processing, data mining

Going after the ultimate ends
-Sensitivity, stability, design, control, parameter identification, data assimilation, experimental
validation, computational steering, scientific discovery
ISOFS Workshop

Why work with us?
•

Logically innermost kernels (e.g., linear algebra) are often themost computationally complex (in terms of memory accesses orflops or both) in the entire code — should be designed from theinside out by experts with right “handles” presented to users•

Many widely used methods and software libraries are quite
behind the times, algorithmically•Today’s components do not “talk to” each other very well•

Mixing and matching procedures too often requires mapping
data between different storage structures (taxes memory andmemory bandwidth)•Many fusion issues are generic to computational science andsolutions can be leveraged
ISOFS Workshop

The power of algorithms
••

•

•

Since the dawn of digital computers, algorithmic advances have matchedarchitectural advances in increasing the power available to end user
scientists
For instance, for a system of N linear equations derived from linear finiteelements applied to a 3D elliptic PDE:
Algorithm

Operation Complexity

Banded Gauss Elimination

O(N7/3)Gauss SeidelO(N5/3 log(N))

Optimal SOR

O(N4/3 log(N))

CG/MILU

O(N7/6 log(N))

F-cycle MGO(N)If N=106 (e.g., a 3D brick with 100 nodes on each side), then the net
improvement from banded elimination to multigrid is a factor of 108
For this problem, this is equivalent to replacing a 1 Mflop/s computer witha 100 Tflop/s computer
ISOFS Workshop

A model some of us like…

•Lab-university collaborations to develop reusable software“solutions” and partner with application groups•

For FY2002, 51 new projects at $57M/year total•-

Approximately one-third for applications-A third for integrated software infrastructure centers (ISICs)-

A third for grid infrastructure and collaboratories

Multi-Tflop/s IBM SP platforms “Seaborg” at NERSC (#3 inlatest “Top 500”) and “Cheetah” at ORNL available forSciDAC
ISOFS Workshop

Sample Gantt chart for ISICs
Each color module represents an algorithmic research idea on its way to becoming part of a supported
community software tool. At any moment (vertical time slice), TOPS has work underway at multiple levels.
While some codes are in applications already, they are being improved in functionality and performance aspart of the TOPS research agenda.

Dissemination
Applications Integratione.g.,PETSc

Hardened Codes
Research Implementations

e.g., ASPIN

Algorithmic Development
timeISOFS Workshop

It’s 2002; do you know what your solver is up to?
Has your solver not been updated in the past fiveyears?
Is your solver running at 1-10% of machine peak?
Do you spend more time in your solver than in yourphysics?
Is your discretization or model fidelity limited by thesolver?
Is your time stepping limited by stability?
Are you running loops around your analysis code?
Do you care how sensitive to parameters your results
are?
If the answer to any of these questions is “yes”, you are a potential customer!
ISOFS Workshop

Have you thought recently about your discretization?
Have you considered updating it in the past five years?
Have you evaluated spatial and temporal convergenceorder experimentally for a typical case?
Do you lack solution-based error controls?
Is is difficult for you to vary scheme order and meshtype to follow up on scientific hypotheses ornumerical hunches?
Is is difficult to combine methods, like Eulerian fields
and Lagrangian particles or fronts?
Do you spend more time setting up your problem thanrunning it?
If the answer to any of these questions is “yes”, you are a potential customer!
ISOFS Workshop

CS/AppMath project goals/success metrics
We will have succeeded if (you) users —••

••
•
•

•

Understand range of algorithmic options and their tradeoffs (e.g.,memory vs. time, memory bandwidth vs. flop/s, communication vs.
flops, inner iteration work vs. outer)
Can try all reasonable options from different sources easily without
recoding or extensive recompilation
Know how your discretizations are converging
Know how your solvers are performing
Spend more time in your physics than in meshing and solving
Are intelligently driving algorithmic research, and publishing joint
papers with CS and AppMath researchers
Can simulate truly new physics, as limits are steadily pushed back(finer meshes, higher fidelity models, complex coupling, etc.)
ISOFS Workshop

Interacting with SciDAC ISICs
Applications
APDEC

TSTT

TOPS

SDMPERCCCA

Indicates “dependence on”
ISOFS Workshop

SSOne view of interacting with ISICs
Applications
APDEC

TSTT

SDM

TOPS

PERC, CCASS
Indicates “dependence on”
ISOFS Workshop

Expectations ISICs have of users•••••

Be willing to experiment with novel algorithmic choices –optimality is rarely achieved beyond model problems without
interplay between physics and algorithmics!
Adopt flexible, extensible programming styles in whichalgorithmic and data structures are not hardwired
Be willing to let us play with the real code you care about, but bewilling, as well to abstract out relevant compact tests
Be willing to make concrete requests, to understand that requests
must be prioritized, and to work with us in addressing the highpriority requests
If possible, profile, profile, profile before seeking help

ISOFS Workshop

What we believe•

Many of us came to work on solvers through interests inapplications•

What we believe about …
-

applications-

users-

solvers-

legacy codes-

software… will impact how comfortable you are collaborating with us

ISOFS Workshop

What we believe about apps•

Solution of a system of PDEs
is rarely a goal in itself-

PDEs are solved to derivevarious outputs from
specified inputs-

Actual goal ischaracterization of aresponse surface or a design
or control strategy-

ÞTogether with analysis,
sensitivities and stability areoften desired•

No general purpose PDE solver
can anticipate all needs---

Why we have national
laboratories, not numericallibraries for PDEs today
A PDE solver improves withuser interaction
Pace of algorithmic
development is very rapid

Þ Extensibility is important

Software tools for PDE
solution should also support
related follow-on desires
ISOFS Workshop

What we believe about users•

Solvers are used by people ofvarying numerical backgrounds- Some expect MATLAB-likedefaults- Others want to control
everything, e.g., evenvarying the type of smootherand number of smoothingson different levels of amultigrid algorithmÞ Multilayered software design isimportant•Users’ demand for resolution isvirtually insatiable- Relieving resolution
requirements with modeling(e.g., turbulence closures,homogenization) only defers
the demand for resolution tothe next level- Validating such models
requires high resolution

Þ Processor scalability andalgorithmic scalability(optimality) are critical

ISOFS Workshop

What we believe about legacy code•

Porting to a scalable framework • Legacy solvers may be limiting
resolution, accuracy, anddoes not mean starting fromgenerality of modeling overall
scratch- Replacing the solver may- High-value meshing and“solve” several other issues
physics routines in original- However, pieces of thelanguages can belegacy solver may have value
substantially preserved
as part of a preconditioner- Partitioning, reordering andmapping onto distributed
Þ Solver toolkits should include
data structures (that we may“shells” for callbacks to highprovide) adds code but littlevalue legacy routines
runtimeÞ Distributions should include
code samples exemplifying“separation of concerns”
ISOFS Workshop

What we believe about solvers•

Solvers are employed as partof a larger code----Þ

Solver library is not only
library to be linked
Solvers may be called in
multiple, nested places
Solvers typically makecallbacks
Solvers should beswappable

Solver threads must notinterfere with othercomponent threads, including
other active instances ofthemselves•

Solvers are employed in many
ways over the life cycle of an
applications code-

During development andupgrading, robustness (of thesolver) and verbose diagnostics
are important-

During production, solvers are
streamlined for performanceÞ Tunability is important

ISOFS Workshop

What we believe about numerical software•

A continuous operator may appear ina discrete code in many different
instances-

Optimal algorithms tend to behierarchical and nested iterative-

Processor-scalable algorithms tend
to be domain-decomposed andconcurrent iterative-

Majority of progress towards
desired highly resolved, highfidelity result occurs through costeffective low resolution, low
fidelity parallel efficient stages•

Hardware changes many times
over the life cycle of a software
package-Processors, memory, andnetworks evolve annually-

Machines are replaced every
3-5 years at major DOE
centers-Codes persist for decadesÞ Portability is criticalÞ Operator abstractions and recurrence
are important

ISOFS Workshop

Exciting time for enabling technologies
SciDAC application groups have been chartered to build new andimproved COMMUNITY CODES. Such codes, such as NWCHEM,
consume hundreds of person-years of development, run at hundredsof installations, are given large fractions of community computeresources for decades, and acquire an “authority” that can enable orlimit what is done and accepted as science in their respectivecommunities. Except at the beginning, it is difficult to promote majoralgorithmic ideas in such codes, since change is expensive andsometimes resisted.
ISIC groups have a chance, due to the interdependence built into theSciDAC program structure, to simultaneously influence many of thesecodes, by delivering software incorporating optimal algorithms that
may be reused across many applications. Improvements driven by
one application will be available to all. While you are building
community codes, this is our chance to build a CODE COMMUNITY!
ISOFS Workshop

Related URLs•

Personal homepage: papers, talks, etc.
http://www.math.odu.edu/~keyes•

SciDAC initiative
http://www.science.doe.gov/scidac

ISOFS Workshop

Summary of Roundtables

ISOFS Workshop

Interpretation & Algorithms Roundtable•

Points of greatest needSpecialized meshing and discretization capability- Specialized solver capability-•

Both driven largely by anisotropy
General numerical PDE techniques may not apply
due to failure to respect the highly anisotropictransport along versus across magnetic flux surfaces- Fundamental; basis of magnetic confinement-•

Consequences
Small errors lead to artificially enhanced transport- Anisotropy hard for standard scalable solvers-

ISOFS Workshop

Interpretation & Algorithms Roundtable•Additional points of need
General meshing tools for toroidal and topologicallytoroidal geometries, fully structured and hybrid
structured-unstructured (in poloidal plane)- Sophisticated interpolation tools to move fields fromone discretized representation to another, co-locatedin same domain on different meshes-ü Especially including experimentally derived data, for validation

Interpolation tools to change dependent variablesfrom one physical model to another (a fundamental
part of specialized fusion/MHD framework??)- Hybrid continuum-particle solvers-ISOFS Workshop

Interpretation & Algorithms Roundtable•

Additional points of need, cont.
-

Fast curl-curl solvers-

Stiff MOL solvers for integration of either
compressive Alfven waves in the poloidal plane (CFLratio of 10-100) or both compressive and shear Alfven
waves (CFL ratio of 1000), to follow slower
dynamically relevant timescales more efficiently-

Homogenization and related upscaling techniques-Operator-adaptive multilevel methods for ill
conditioned linear systems
ISOFS Workshop

Interpretation & Algorithms Roundtable•

Additional points of need, cont.
High-order hyperbolic conservation law integrators(for v.grad) that include field effects- Nonlinearly consistent iterative methods for coupledphysics, with essential “two-way” finite amplitudenonlinearities-ü physics-based preconditioning for Newton-type methods,(harnessing “solvers” in current component codes as part of overall
operator-split preconditioner)-Methods for design/optimization subject to satisfying
the high-dimensional constraints coming from thesystem of governing equations
ISOFS Workshop

MHD state of art for gridding/discretization•••••

Not dynamic; good approximation to flux surfaces are known
in advanceIll-posed to grid based only upon field lines (e.g., using someBrackbill-Saltzman variational scheme), since the field lines
do not perfectly close
Hybrid h-p schemes provide sufficient accuracy to “not sweat
the small stuff”
High order FE methods are “good enough” to not require
exact discrete satisfaction of div(B)=0, etc.
Useful scheme for high-order hyperbolic solvers and for(logically) Cartesian AMR would be field- and body-fitted
coordinates in the poloidal planes and (possibly twisted)cylindrical extensions in the toroidal direction
ISOFS Workshop

We did not hear too much about but wish
to promote thoughts about…
•

Large-scale eigensolvers, stability analyses•

Sensitivity analyses•

Visualization•

Automated data mining for physical features•

Boundary and body control of plasmas throughformal optimization techniques•

Parameter identification•

Assimilating experimental data into simulations•

Computational steering
ISOFS Workshop