FUSION SIMULATION PROJECT
Integrated Simulation & Optimization of Fusion Systems
Final report of the FESAC ISOFS Subcommittee

ISOFS (Integrated Simulation & Optimization of Fusion Systems)
Subcommittee Members:
Jill Dahlburg, General Atomics (Chair)
James Corones, Krell Institute, (Vice-Chair)
Donald Batchelor, Oak Ridge National Laboratory
Randall Bramley, Indiana University
Martin Greenwald, Massachusetts Institute of Technology
Stephen Jardin, Princeton Plasma Physics Laboratory
Sergei Krasheninnikov, University of California - San Diego
Alan Laub, University of California - Davis
Jean-Noel Leboeuf, University of California - Los Angeles
John Lindl, Lawrence Livermore National Laboratory
William Lokke, Lawrence Livermore National Laboratory
Marshall Rosenbluth, University of California - San Diego
David Ross, University of Texas - Austin
Dalton Schnack, Science Applications International Corporation

EXECUTIVE SUMMARY

Fusion is potentially an inexhaustible energy source whose exploitation requires a basic

understanding of high-temperature plasmas. The development of a science-based predictive
capability for fusion-relevant plasmas is a challenge central to fusion energy science, in which
numerical modeling has played a vital role for more than four decades. A combination of the very
wide range in temporal and spatial scales, extreme anisotropy, the importance of geometric detail, andthe requirement of causality that makes it impossible to parallelize over time, makes this problem
one of the most challenging in computational physics. Sophisticated computational models are underdevelopment for many individual features of magnetically confined plasmas and increases in the scopeand reliability of feasible simulations have been enabled by increased scientific understanding andimprovements in computer technology. However, full predictive modeling of fusion plasmas will
require qualitative improvements and innovations to enable cross coupling of a wider variety ofphysical processes and to allow solution over a larger range of space and time scales. The exponential
growth of computer speed, coupled with the high cost of large-scale experimental facilities, makes anintegrated fusion simulation initiative a timely and cost-effective opportunity.
Worldwide progress in laboratory fusion experiments provides the basis for a recent FESAC
recommendation to proceed with a burning plasma experiment (see FESAC Review of Burning
Plasma Physics Report, September 2001). Such an experiment, at the frontier of the physics ofcomplex systems, would be a huge step in establishing the potential of magnetic fusion energy tocontribute to the world’s energy security. An integrated simulation capability would dramaticallyenhance the utilization of such a facility and lead to optimization of toroidal fusion plasmas ingeneral. This science-based predictive capability, which was cited in the FESAC integrated planningdocument (IPPA, 2000), represents a significant opportunity for the DOE Office of Science tofurther the understanding of fusion plasmas to a level unparalleled worldwide.
The ISOFS Subcommittee recommends that a major initiative be undertaken, referred to here
as the Fusion Simulation Project (FSP). The purpose of the initiative is to make a significant
advance within five years toward the ultimate objective of fusion simulation: to predict reliably the
behavior of plasma discharges in a toroidal magnetic fusion device on all relevant time and spacescales. By its very nature in enabling more comprehensive modeling, the FSP will lead to a wealth of
insights not realizable previously, with new understanding in areas as diverse as wall interaction
phenomena, the effects of turbulence on long time confinement, and implications of plasma self
heating in advanced tokamak operating regimes. The long-term goal is in essence the capability forcarrying out ‘virtual experiments’ of a burning magnetically confined plasma, implying predictive
capability over many energy-confinement times, faithful representations of the salient physics
processes of the plasma, and inclusion of the interactions with the external world. Since confidence
in the ability to predict is ultimately based on code performance against experimental data, avigorous and ongoing validation regime must also be a critical element of this project.
The characteristics of fusion plasmas make the goal extremely challenging. These characteristics
include the presence of multiple time scales, ranging over fourteen orders of magnitude, and multiple
spatial scales, ranging over eight orders of magnitude. The linear algebraic systems that must be
solved are often ill-conditioned. The computational domains are geometrically complex, and thesolutions severely anisotropic. In many cases, the physics approximations are not completely
understood, and hence the simulation equations are unclear. The underlying physics is coupled withessential nonlinearities. Taken in isolation, approaches have been developed or are underinvestigation for each of these challenges. However, an integrated simulation for fusion plasmas will
present all of these features simultaneously.

FESAC ISOFS Subcommittee Final Report2

Success of this project will require coordinated and focused advances in fusion physics (to further
develop the underlying models and elucidate their mathematical basis), applied mathematics (tofurther develop suitable algorithms for solving the mathematical models on the appropriatecomputer architecture, and to define frameworks within which these algorithms may be easily
assembled and tested), and computer science (to provide an architecture for integrated codedevelopment and use, and to provide analysis and communication tools appropriate for remotecollaboration). Strong collaborations, forged across these disciplines and among fusion scientists
working in different topical areas, will be an essential element of the program. In addition, the Fusion
Sim ula ti on Projec t will require s ign ifica nt impr ovem ents in compu tational and network
infrastructure, including enhancements to shared resources as well as to local or topical computing
centers. Because of the complexity of the FSP, the planning process should continue into CY2003.
We recommend a staged approach: beginning with clarification of the physics issues, accompanied byefforts to address algorithmic issues and followed by clarification of architectural issues.
The necessary core expertise for the FSP is resident in several units within the DOE Office of Science.
Primary among these are the ongoing fusion experimental and theoretical research and development
activities within the Office of Fusion Energy Sciences, the applied mathematics development
activities within the Office of Advanced Scientific Computing, the recently developed SciDAC
initiative, and materials science research in the Office of Basic Energy Sciences.
To achieve its goals, the FSP is envisioned as proceeding through three five-year phases in which
successively more complex and disparate phenomena will be integrated. During the first five years,
the project will concentrate on specific physics integration issues that are expected to deliversignificant scientific insights in their own right, but are also prototypical of the integration issues facedby the whole initiative. Each Focused Integration Initiative (FII) will concentrate on developing a
predictive modeling capability for a specific programmatically important scientific problem and will
begin to develop and gain experience with relevant mathematical tools, new algorithms, andcomputational frameworks. During the second five-year period the project will undertake larger andmore comprehensive integration activities and take them to the next level of development. During
the final five-year period, the focus will be on comprehensive integration. There will be links among
all the physics components of the project. To provide a tradeoff between computational efficiency
and physical fidelity there will be multiple levels of description of many of the physical processes.
Verification and validation are critical components of the FSP. To succeed, an integral feature of thisinitiative must be an intensive and continual close coupling between the simulation efforts andexperiments. The phenomena in magnetic fusion devices, the equations describing them, and theinteractions among the various critical phenomena are sufficiently complex that developing the mosteffective approximations and establishing that the models have the required accuracy can only be
accomplished by continual iteration and testing against experimental data.
Funding for the FSP must be at a level adequate to accomplish the project goals. The successful
NNSA Accelerated Strategic Computing Initiative (ASCI) Level 1 University Centers Program,funded at $25M/yr, provides an appropriate example of the level of resources required. A preliminary
assessment of the challenges and complexity of possible FIIs indicates that they would be comparable
to that of each of the five ASCI University Level 1 Center Programs. We further estimate thatfour-five such FIIs will be required to cover all the critical science areas which must eventually go into
the final integrated simulation code. Further refinement of the costs and timelines will be carried out
as the FSP is developed. Through the course of the project, we envision that funding would beapproximately equally allocated between the DOE OFES and OASCR research elements. Because
this initiative rests entirely on a progressing science base, and will for successful execution attract andretain junior researchers committed to the goals of fusion energy sciences, it is paramount that FSP
funing be new rather than redirected from present critical areas.
FESAC ISOFS Subcommittee Final Report3

TABLE OF CONTENTS
Executive Summary
I.

Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5

II.

Overview and Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7
a. Goals: 5,10,15 Year Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7
b. The Focused Integrated Initiative (FII) Approach . . . . . . . . . . . . . . . . . . . . .8
c. FSP Project Size and Scale . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8
d. Infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9
e. Due Attention to Governance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10
f. The Need to Continue the Planning Process . . . . . . . . . . . . . . . . . . . . . . . .11

III. The FSP Scientific Elements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12
a. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12
b. Focused Integration Initiatives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14
c. FII Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16
d. Insights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19
e. Computational Science . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .20
f. Verification and Validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .26
g. Infrastructure Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .29
IV.

The FSP Path . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .33
a. Project Roadmap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .33
b. The FSP and other Office of Science Activities . . . . . . . . . . . . . . . . . . . . . .35
c. Roadmap Implementation: Project Governance & Management . . . . . . . . .36

V.

Summar y . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .39

VI. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .41
Attachment: ISOFS Charge Letter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .42
Appendix. Overview of Fusion Science for the FSP . . . . . . . . . . . . . . . . . . . . .Vol 2

FESAC ISOFS Subcommittee Final Report4

I. BACKGROUNDI

n February 2002, the DOE Office of Science asked the Fusion Energy Sciences Advisory
Committee (FESAC) to assist in defining a major new initiative to be sponsored jointly by the Officeof Fusion Energy Sciences (OFES) and the Office of Advanced Scientific Computing Research(OASCR). The goal of this initiative, the Fusion Simulation Project (FSP), is to create acomprehensive set of theoretical fusion models, an architecture for bringing together the disparate
physics models, combined with the algorithms andcomputational infrastructure that enable themodels to work together. The required funding level for the FSP is expected to be on the order of$100M spread over five to six years. A FESAC ISOFS (Integrated Simulation and Optimization of
Fusion Systems) Subcommittee, with members from the fusion, applied mathematics and computerscience communities, was constituted to generate a plan for moving forward with the FSP. The
ISOFS Subcommittee membership is listed on the cover page of this document. The 2002 timelineof the ISOFS Subcommittee is shown in Fig.I.1.

ISOFS Interim Report:
see http://www.isofs.info
FESAC ISOFS
Charge Letter
IssuedFeb

ISOFS
Workshopand Meeting
Apr

June

ISOFS
Meeting
Aug

ISOFS Final Report
ISOFS
Workshopand Meeting
Oct

Snowmass
Fusion Summer Study

Figure I.1. FESAC ISOFS Subcommittee activities timeline for 2002. ISOFS Workshop presentations and discussion may befound at: http://www.isofs.info .

Impetus and fundamental interest for the FSP initiative primarily comes from the goal to develop an
attractive fusion energy source. The fossil fuels that underpin the United States economy cannot be
relied upon to carry our nation into the 22nd Century. Oil and gas are non-renewable resources
feeding a rapidly growing global energy appetite. There is also the threat of global climate change dueto the burning of fossil fuels.
In the summer of 2002, fusion physicists met at the Snowmass Fusion Summer Study to plan the nextstage of research towards the ultimate goal of fusion energy. The 2002 Snowmass Development
Pathway Subgroup discussed the major next step plasma physics facilities in the fusion International
Portfolio Approach that are required for this goal. These include advanced tokamak and nontokamak physics facilities, a burning plasma facility(s), a Fusion Plasma Simulator (FPS), and a strong
core program. In particular, the FPS is envisioned to be an integrated research tool that contains
comprehensive coupled self-consistent models of all important plasma phenomena that would be
used to guide experiments and be updated with ongoing experimental results. Most importantly, the

FESAC ISOFS Subcommittee Final Report5

FPS would serve as an intellectual integrator of physics phenomena in advanced tokamak
configurations, advanced stellarators and tokamak burning plasma experiments. It would integrate
the underlying fusion plasma science with the Innovative Confinement Concepts, thereby
accelerating progress. Development of a facilities class FPS capability is estimated to be a fifteenyear, $400M activity. The FSP that is discussed in detail in this report is a first five-year stage of theultimate FPS. The need for this kind of integrated simulation capability is recognized in the
preliminary report of the FESAC Development Path Subcommittee charged with identifying therequirements for the start of operation of a fusion energy demonstration power plant in 35 years.
The workshops, meetings, and regular correspondence of the ISOFS Subcommittee resulted in thevision for the FSP that is described in this report. This final report of the ISOFS Subcommittee
provides the response to the FESAC ISOFS charge letter of February 22, 2002; a copy of the letter isin the report Attachment. The report Appendix, an overview of frontier fusion science, addresses
aspects of the charge and also provides a self-contained reference summary of fusion science in thecontext of this initiative. Responses to particular questions contained in the ISOFS charge letter are
as follows:• What is the current status of integrated computational modeling and simulation?
Appendix Section VIII with additional detail in Appendix Sections III-VII.
• What should be the vision for integrated simulation of toroidal confinement fusion
systems?
Sections IIa,b, IIIb,c, and Appendix Section IB and IX.
• What new theory and applied mathematics are required for simulation and optimizationof fusion systems?
Section IIIe, and Appendix Sections III-VII and IX.
• What computer science is required for simulation and optimization of fusion systems?
Section IIIe.
• What are the computational infrastructure needs for integrated simulation of fusion
systems?
Sections IId and IIIg.
• How should integrated simulation codes be validated, and how can they best be used toenable new scientific insights?
Sections IIId,f, and Appendix Section I, IX, and X.
We note that this document contains a refined response to the first two charges above, building upon
the initial response in the July 12, 2002 ISOFS interim report.
The FSP computational undertaking represents a significant opportunity and a significant challenge
to fusion research, which has always been at the forefront of advanced scientific computing.
Integrating fusion computer codes for full-system fusion simulations will require even greater researchcollaboration among fusion physicists, and applied mathematicians and computer scientists
dedicated to putting fusion energy on the power grid. Creating the computational resources tosimulate fusion will do more than substantially advance fundamental science. We will give ourselves
the ability to see our energy future, and then build it.
FESAC ISOFS Subcommittee Final Report6II. OVERVIEW AND RECOMMENDATIONSThe Fusion Simulation Project (FSP) described in this document is designed to provide an

integrated simulation and modeling capability for magnetic fusion confinement systems. The FSP
is the detailed response to findings of the FESAC Integrated Program Planning Activity (IPPA 2000),
which identified the requirement for enhanced simulation for predicting the performance of externally controlled confinement systems. It is recognized that this goal can only be met through extensive and sustained collaborations between fusion scientists, and applied mathematicians and computer scientists. We note that these challenges are coming at a time of increasing opportunity between
these groups, recognized in large measure by the DOE Office of Science SciDAC projects, and that
the FSP will be able to further the momentum well-fostered by SciDAC. Hence,
We recommend that a major initiative be undertaken, here referred to as the Fusion Simulation
Project (FSP), to create a comprehensive set of theoretical fusion models, combined with thealgorithms required to realize them and an architecture and computational infrastructure thatenable them to work together.
The purpose of the FSP is to make a significant advance toward the ultimate objective of fusion
simulation: to predict in detail the behavior of any discharge in a toroidal magnetic fusion device on
all important time and space scales. This is in essence the capability for carrying out ‘virtual
experiments’ of burning, magnetically confined plasmas. This requires faithful representations of thesalient physical processes individually and their interactions with the external world (sources, control
systems and bounding surfaces), leading to a predictive capability over many energy-confinement
times,
a. GOALS: 5,10,15 Y EAR OVERVIEW
The goal of the FSP is to produce a comprehensive fusion simulation tool (the Fusion Plasma
Simulator) by the year 2020. This tool will play an essential role in the development path for fusionenergy. It will effectively serve as an intellectual integrator of physics phenomena in advanced
tokamak configurations, advanced stellarators and tokamak burning plasma experiments. In order toachieve this overarching goal, the project will proceed through three five-year phases in which
successively more complex and disparate phenomena will be integrated together. We describe thisprocess briefly here and in more detail in Section III.
During the first five years, the project will concentrate on specific high-profile physics integration
issues that are considered to be the most critical, and are also prototypical of the integration issues
faced by the whole initiative. We expect to gain new scientific insights during this period. We will
also develop the mathematical frameworks for the project and gain experience with computationalframeworks and new algorithms.
During the second five-year period the project will undertake larger and more comprehensive
integration activities and bring them to the next level of development. The mathematical framework
will be expanded to include a wider range of integrated phenomena and will become standardized forthe project. The project will develop a unified computational framework that aids in managing theincreasing complexity, as well as integrating such aspects as advanced graphics and user interface.
New algorithms will continue to be developed and refined as needed.

FESAC ISOFS Subcommittee Final Report7During the final five-year phase, the focus will be on comprehensive integration. There will be links
among all the physics components of the project. To provide a tradeoff between computational
efficiency and physical fidelity there will be multiple levels of description of many of the physicalprocesses. The simulation capabilities will be extensively exercised, and comprehensive comparisons
between the simulation and experiment will take place. The capability will be used to guideexperiments and be updated with ongoing experimental results.
b. THE FOCUSED INTEGRATED INITIATIVE (FII) APPROACH
Fusion computations at varying degrees of integration have already led to significant insights
pertaining to the physics mechanisms underlying the performance of plasmas confined in a range oftoroidal magnetic configurations. We expect that the FSP will lead to new surprises coming from
more comprehensive models and emerging from enhanced synergy between theory, experiments andmodeling. This integration initiative provides a tremendous opportunity to garner new insights byaddition of new physics to the plasma models and by enabling more comprehensive models throughintegration.
In order to realize integration from the beginning of the project, we recommend that the FSP
commence with programmatic teams, subsets of the full FSP that we term Focused Integration
Initiatives (FIIs). We describe the FIIs in detail in Section IIIb. The goal of each FII team is thesolution of a compelling problem in fusion science physics that requires integrated simulation. The
FIIs should be multi-disciplinary and multi-institutional, and by their research should integrate
subsets of the full breadth of fusion fundamentals and applications of varying complexity using
selected algorithms and interoperable software. The traditional modeling elements that structure ourunderstanding of fusion plasmas include: plasma sources; turbulence; extended MHD; 1.5D (oneand one-half dimensional) transport; and fusion materials. Each FII should cut across and integrate
two or more of these traditional elements, to provide physics integration both spatially andtemporally, with a guiding focus of a single overarching scientific question or topic that satisfies thecriterion of importance to the fusion program. The community will be invited to define overarching
FII themes through the proposal process.
As we envision it, each FII will focus on achieving predictive modeling capability for the particularfusion science problem it has elected to address. In order to develop a critical mass of research withadequate intellectual vibrancy, and to encourage development path risk and opportunity, the FSP
should be initially comprised of 4-5 FII units. Primary to each of the FII activities must be
verification of the accuracy of the new integrated model developed within the FII, and validation of
the model with experimental data. Verification and validation — critical components for the FIIs —imply non-trivial supporting access to experiments, experimental data and diagnostics. We thus
recommend close coupling of each FII research team with relevant experiments, and that thedevelopment of a reliable experimental predictive capability should be a substantive part of each FII.
c. FSP P ROJECT SIZE AND SCALE
We strongly recommend that within the five-year time frame specifically considered to be the FSP,
the initiative should be carried out at a scale such that certain computational goals can be achieved:
1) Robust computational modules are developed in each of the selected FII areas
representing the state-of-the-art in physics content, numerical methods, andcomputational science methods, enabling efficient incorporation into the integration
framework.
FESAC ISOFS Subcommittee Final Report82) Approaches are developed for the fundamental problems of disparate time or space scales,
and coupling of models of processes having different dimensionalities.
3) An initial inter-operable code capability that allows for three-dimensional geometry isavailable for widespread testing as a research tool.
4) The effectiveness of the integration approach is demonstrated by application tointerpreting experimental data, and testing the validity of various physics models.
This initiative rests entirely on a progressing science base. Therefore it is paramount that FSP
funding be new rather than redirected from present, critical areas. We fully support the assessmentof the importance of the core fusion program that was stated in the September 2002 Burning Plasma
Strategy Report: ‘The core program is … essential to the successful and full exploitation of theburning plasma program. Predictions on the confinement, stability properties and dynamics ofplasmas in the burning regime have all come from the intense experimental, modeling andtheoretical efforts of the core program. The underpinnings of any burning plasma experiment
therefore fundamentally rests on the foundation of knowledge that has come from the core program.
Moving forward with a burning plasma experiment requires experimental scientists, engineers, andtheorists and computational scientists from this core to design experiments and interpret the results.’
Further, funding for the FSP must be at a level adequate to accomplish the FSP goals. To derive anadequate funding profile that will enable a critical mass of research, we use the successful $25M/year
DOE Accelerated Strategic Computing Initiative (ASCI) Level 1 University Centers Program as an
example. In that program, each strategic element or Center is receiving $4-5M/year for each of 5-10
years. This indicates that each FSP FII team should be initiated with funding of about $4-5M/year,
and that the FSP will require approximately $20M/year for each year of the project. Through thecourse of the project, funding should be approximately equally allocated from the OFES and OASCR
research elements.
d. INFRASTRUCTURE
The FSP will be integrated into broader fusion science and simulation activities. For example, accessto experiments, experimental data and diagnostics are critical to the success of the initiative. Likewise,
reliable access to suitable computing facilities will be required, including data storage andnetworking, and also collaborative tools. None of these items are included in the budget for the FSP
as envisioned in this report. However we do stress here the need to supply a variety ofcomputational platforms. Computational infrastructure that includes platforms of extremely highcapability, and also high performance networks, will certainly be needed to achieve the goals of the
FSP. The FSP will push the available envelopes of both sustained performance and long-distanceresearcher collaboration from the outset and as the project moves forward. The graphic below (Fig.
II.1) illustrates fusion simulation performance projections in the context of past accomplishments.
Storage and networking needs for the future simulation activities can be deduced from the increasing
memory requirements necessary for the simulations.

FESAC ISOFS Subcommittee Final Report9Memory (Bytes)

1016

Burning Plasma
Integrated Simulation

Virtual Disruption

1014

Virtual Edge
Earth Simulator (2002)

1012

NERSC (2002)

GK Full Torus(w/ electron dynamics)
GK Full Torus(adiabatic electrons)

GK Flux TubeNERSC (1997)

10 10

NERSC
(1995)

1010

1012

10 14

1016

Computational Speed (Flops)
Figure II.1. Computational requirements for fusion simulations.

Further, it is essential to realize that many aspects of the project will require readily available
computational cycles in real time for program development and debugging. To assure high levels of
productivity by researchers, the latter are profoundly critical for success along with the ultra-scale
simulation capability.
e. DUE ATTENTION TO GOVERNANCE
To bring these disparate components together will require the dedicated skills of many accomplishedphysicists, applied mathematicians and computer scientists. There is no doubt that the sociology of
the FSP will be a challenge. On the one hand, a strong fusion physics effort is required, involving a
number of institutions and the relevant theory, simulation, and experimental communities, each ofwhich will bring a required degree of intellectual independence. On the other hand, setting
priorities and a considerable amount of central direction will be essential, for the reason that the FSP
must be a coordinated, goal driven activity. Even more challenging will be effective integration of
first-rate computer scientists and applied mathematicians as full partners with fusion physicists in thisventure. The issue of project governance includes also the establishment of an effective cooperative
arrangement between and within the two sponsor entities, OFES and OASCR, and clear delineationsof working relationships with other initiatives and activities in the DOE such as the Office of Science
SciDAC, the OFES fusion experiments, and OASCR computing resources. Harmonizing all of theseelements, particularly in light of robust institutional competition (which is a strength of the DOE
laboratory system), will require innovative, flexible management in the field and at headquarters.

FESAC ISOFS Subcommittee Final Report10Early success will require planning, leadership, and likely new management approaches. Should
success be achieved, the FSP could provide a template for other large-scale cross-disciplinary
computational initiatives for the future. It is our view that the next paradigm shift in problem
solving ability from large scale computation may be fueled by this and other comparable majorcollaborative computational projects that are now ongoing (e.g., the Community Climate Systems
Model [see, e.g., Kiehl, 23 May, 2002, http://www.isofs.info]). The sustained effort that may berequired to coordinate this project to a successful outcome is well balanced by this potential. Boththe investigation of the new fusion science that will be enabled, and also the possible outcomesachievable from the novel investigation of high-end computational science paradigms, well justify asubstantial degree of thoughtful planning from the outset.
f. THE NEED TO CONTINUE THE PLANNING PROCESSAs noted above, two workshops on the FSP were held in 2002, which brought together fusion
scientists, applied mathematicians and computer scientists with an interest in the FSP. The September
2002 meeting had two major goals: obtaining technical input for this report and establishing andenhancing contact among the participating communities. The level of intellectual energy andenthusiasm for the FSP activity was very high at the September meeting, and more than 100
technical researchers participated. The stage is clearly set for broad participation from all relevant
sectors. While the current report addresses the relevant strategic technical issues, further thinking
must be done to develop a working program plan. Because of the complexity of the FSP, thisplanning process is staged: first clarification of the physics issues; next clarification of the algorithmic
issues; and, finally clarification of architectural issues. This planning process is ongoing andoverlapping in time, and we expect that it will continue through the life of the FSP.
For success, we believe that the FSP planning process that has now begun should continue during
2003. Several sorts of activities should be considered:• focused technical workshops that continue to broaden participation among fusion
physicists and applied mathematicians and computer scientists;• small working groups that begin to clarify and define the software architecture, including
documenting requirements;• venues for the clarification of needed collaborative tools; continued integration of the outputs of the above by the ISOFS Subcommittee — or whichever future organization DOE
decides to enfranchise in this role — into a detailed planning document that will lead to asuitable FSP proposal call;• as technical planning becomes more refined, activities that provide more accurate budget
estimates for the duration of the FSP; and,• attention to new and ongoing international activities in these areas with a goal of fosteringcollaboration where feasible.

FESAC ISOFS Subcommittee Final Report11III. THE FSP SCIENTIFIC ELEMENTS
a. INTRODUCTION

C

entral to the understanding of fusion plasmas are fusion experiments. A toroidal fusion
experiment, for example the tokamak shown in Fig.III.1a, consists of an inner plasma of ionized gas
confined by a configuration of magnetic fields.

Magnetic flux
surfaces, ρ = const.

Toroidal angle φIntegrated Control
Current Profile
Control

Magnetic axis

Optimized Resistive
Wall Mode Control

separatrix

Disruption Detection,
Correction, Mitigation

Poloidal angle θFigure III.1a. Cutaway view of an advanced tokamak, DIII-D.

Figure III.1b. Key plasma and magnetic regions of a typical tokamak plasma, shown as a computed cross-section.

FESAC ISOFS Subcommittee Final Report
12Plasma containment results from the formation of closed, nested magnetic flux surfaces and thetendency of the individual plasma particles, ions and electrons, to move along magnetic field lines andthus remain close to the flux surfaces. Loss of confinement or transport results from the drifting ofparticles across these surfaces or from the breakup of the surfaces themselves; see Fig.III.1b. The wallsof the device form a vacuum chamber, which is in turn surrounded by the main magnetic coils andthe various devices for diagnosing the plasma behavior and for injecting particles, energy, andmomentum. Ultimately, the balance of these sources with a wide range of loss mechanisms, together
with large-scale instabilities that can disrupt the plasma, determine the performance of the machine.
These processes and the models used to describe them are overviewed in some detail in the Appendix
to this report.
It is widely recognized that the complexity of the dynamics of fusion experimental systems is suchthat the development of computational models to understand their behavior is critical. Numericalmodeling activities in magnetic fusion research are providing important physics understanding androutinely stretching the limits of available computational resources. However, crosscutting issues
crucial to the further development of these models require a qualitative change in approach. In
particular, there are two fundamental issues that have to date inhibited the integration of different
fusion physics areas: the coupling of phenomena at disparate time scales and the necessity of coupling
models of different spatial dimensionality.

Typical Time Scales in a next step experimentwith B = 10 T, R = 2 m, ne = 1014 cm-3, T = 10 keV
SAWTOOTH CRASHELECTRON TRANSIT

ωLH-1 Ω -1
Ωce
ci
10-10
10-8
-1

TURBULENCE

τA

10-6

ISLAND GROWTH

10-4

Neglect displacement
Single frequency
current, average overand prescribed
plasma background gyroangle, (some)with electrons
RF Codeswave-heating andcurrent-drive

ENERGY CONFINEMENT

10-2

CURRENT DIFFUSION

100

102

104 SEC.

Neglect displacement
current, integrate overvelocity space, neglect
electron inertia

Gyrokinetics Codes

Neglect displacement
current, integrate overvelocity space, averageover surfaces, neglect
ion & electron inertia
Extended MHD Codes Transport Codes

turbulent transport

device scale stability

Figure III.2. Summary of four major fusion timescales.

FESAC ISOFS Subcommittee Final Report13

discharge time-scaleA summary of the theoretically defined time scales in experimental fusion devices, and the numericalmodeling presently used to investigate physics phenomena in these various regimes, is given in
Fig.III.2. (See also Appendix Secs. IB and VII.) The ability to understand and predict the dynamics
of high temperature fusion-relevant plasmas, in these regimes, and in the more integrated systems thatwill be required for further advances, is a formidable physics challenge that is central to the goals of
the fusion energy sciences.
Because of the complexities, the goal of establishing a predictive simulation capability for the
integrated simulation and optimization of magnetic fusion systems will require an unprecedented
degree of collaboration and cooperation across many diverse areas in science and technology. For
example, modern tokamaks are hot enough for the individual ions and electrons that comprise theplasma to be virtually collisionless in the direction parallel to the magnetic field, yet their ensemble
can exhibit fluid-like behavior on relatively long time scales. The development of computationallytractable mathematical models that can accurately and efficiently capture simultaneously kinetic andfluid effects and describe their evolution and interaction on experimentally relevant time scales isnecessary for obtaining a true predictive capability. Such an effort will require coordinated andfocused advances in fusion physics (to further develop the underlying models andelucidate their mathematical basis), applied mathematics (to further develop suitable algorithms forsolving the mathematical models on the appropriate computer architecture, and to define frameworks
within which these algorithms may be easily assembled and tested), and computer science (to provide
an architecture for integrated code development and use, and to provide analysis and communication
tools appropriate for remote collaboration). We emphasize that we view fusion physics, appliedmathematics and computer science as fundamental to the FSP, and that healthy, focused, and sufficiently
funded programs in these areas are essential to the success of the initiative.
b. FOCUSED INTEGRATION INITIATIVES
The large scale of fusion integrated simulation ultimately called for in the 2020 Fusion Plasma
Simulator is unprecedented. At this time it is impossible to define precisely the technical details by
which this capability will be achieved. However, we can define a program structure that will promotea variety of technical approaches to integrated simulation while retaining the desired focus on fusion
science technical results. By encouraging diverse approaches to integration we will maximize thecreativity of the scientific community, and we expect that one or two of these initial approaches will
eventually emerge to become adopted throughout the initiative.
First, we outline the important aspects of FSP integration. As background and as noted above, two
fundamental issues are common to many fusion physics integration areas: coupling of phenomena at
disparate time and spatial scales, and coupling of models of different spatial dimensionality. To solvethese generic problems and achieve the integration we are seeking, strong collaboration and advancesin physics, applied mathematics, and computer science will be required. Seamless disciplinary
collaboration will be an essential element of the program. To this end, constitutive elements of the
FSP must be both large enough to encompass a critical mass of multidisciplinary researchers and also
small enough to enable team environments.
Further, an intensive and continual close coupling between the calculations and fusion-relevant
experiments must be a central feature of this initiative. Phenomena in magnetic fusion devices, theequations describing them, and their mutual interactions, are all sufficiently complex that developing
the most effective approximations and establishing when the models have the desired accuracy can
only be accomplished by continual iteration and testing of the models with experimental data. A
FESAC ISOFS Subcommittee Final Report14continual process of testing and iteration is required to advance both modeling and thecharacterization of experimental results.
From these objectives flow a number of requirements that the integration design must satisfy:•

It must be extensible.
- Easy connections can be made early in the project while more difficult ones, for example
those involving very disparate time-scales, can be added as techniques are developed.
- Its architecture must permit continuous improvements and additions.

•

It must be flexible.
- Only the needed physics modules required for a given study should be interconnected.
- It must be robust to changes in physics paradigms. For example a traditional diffusivetransport model will be inadequate if non-local effects turn out to be essential.
- It must be interpretive as well as predictive. That is, it must be possible to make use of bothexperimental information such as profiles, and predicted information such as source rates,
to interpret other needed quantities such as transport coefficients.
- It must support choice in appropriate level of description for any of the modules in aparticular study. It must allow for three dimensional (3D) effects but also be capable oflower level one dimensional (1D) and two dimensional (2D) models where appropriate.

•

It must support collaborative research.
- It should interface well with experimental databases and provide appropriate tools such assynthetic diagnostics to facilitate understanding of output.
- It must include protocols for effective communication among geographically andscientifically diverse participants.

•

It must complement existing research.
- The project must provide value to the individuals involved in basic physics research, who
may themselves be doing large-scale computation.
- It must not impose significant overhead (computational or human) on the use anddevelopment of the separate physics modules. It must provide needed services so as to beof value even to the user of a single module.

Above all, the integrated capability must technically enable fusion science.
- It must promote the development of the physics modules and their validation andverification through experimental comparison, beginning in the near term.
- It must facilitate study of mutual physics interactions presently modeled in separate codes
as such interconnections become appropriate.
- It must increase significantly the depth and breadth of fusion physics compared to today’stransport codes, incrementally, as better modules become available.
To achieve these goals, we believe it is necessary from the beginning to organize the project aroundmajor subsets of the whole integration problem, which pieces we term Focused Integration Initiatives(FIIs). The goal of an FII is the solution of an overarching problem in fusion physics that requires
integrated simulation. The community will be invited to define the FIIs through the proposal
process. As we envision it, each FII will focus on achieving a predictive modeling capability for aparticular scientific problem. The FIIs will be implemented by multi-disciplinary, multi-institutional
teams. The participants will be free to define a unique technical approach for each FII. Specifictechnical areas that should be addressed include:
FESAC ISOFS Subcommittee Final Report
15

Mathematical Models: Development of mathematical models to be included in the integrated
simulation, including their underlying theoretical basis and ranges of physical validity.
Algorithms: Development of the appropriate algorithms for solving the equations of themathematical models, including consistency, stability and convergence properties, theformulation and implementation of realistic boundary conditions, and performance on
advanced computer architectures.
Frameworks: The definition and development of software tools specific to the physicalmodels, mathematical models and algorithms that will enable rapid prototyping on
a variety of architectures.
Performance: The development of tools to analyze and predict the performance of themodels and algorithms on emerging architectures.
Verification and Validation: The definition and development of software to enable validationof integrated models with other models, and with experimental data.
Data Manipulation, Storage, and Analysis: The development of tools for the efficient storage
and analysis of data produced by the integrated simulations.
Collaboration: The definition and development of tools that enable remote collaboration andproject management.
Each FII should include approximately equal contributions from fusion physics, and fromcomputational physics and computer science. The details of the management structure can be
uniquely defined within each individual FII, although each FII will interface with other FIIs in the
FSP by means of an overall coordinating body. It is likely that certain individual researchers will
actively participate in and contribute to several FIIs.
Initially, the selected FIIs will likely pursue a variety of approaches to integrated simulation. Someapproaches will work better than others, and it seems inevitable that in time a consensus will emergethat one or two architectures should be adopted throughout the FSP. At a decision point on this
topic, it is expected that all project participants should be fully enabled to continue in the FSP.
c. FII E XAMPLES
In order to clarify the concept of the FIIs, we here provide some candidate examples. We emphasizethat these are not to be thought of as exclusive, since the actual FIIs will be defined by the community through further planning activities as well as through the proposal and peer review processes. We
consider four candidate FIIs, as shown in Fig. III.3.

FESAC ISOFS Subcommittee Final Report
16

Focused Integration Initiatives are built from Fundamentals of varying
complexity with selected algorithms using interoperable software

Sources

Turbulence

X-MHD

1 1/2 D
Transport

Materials

Plasma Edge
Turbulence on Transport Timescale
Global Stability
Whole Device Modeling

Figure III.3: Focused Integration Initiatives cut across all the traditional fusion disciplines.

1. FII Example: The Plasma Edge
Background: The boundary or edge-plasma of a fusion device plays a vital role in device operation.
The edge plasma system extends from the top of the pedestal to a few microns inside the confinement
device surface. The edge is generally considered to be the region where substantial
multidimensional variations can occur in the plasma, neutral particle, and magnetic equilibrium
quantities. In addition, owing to the lower plasma temperature and proximity to material surfaces,
neutral gases, sputtered impurities, and atomic line-radiation can become important components.
There is thus a rich variety of physics and a wealth of potential interactions that can take place in thisregion.
Comparative purpose: Four plasma edge elements are thought to be key to successful operation of an
MFE fusion device: (1) predicting conditions and properties of the pedestal energy transport barrier
just inside the magnetic separatrix; (2) understanding plasma/wall interactions for particle recycling
and wall lifetime from high energy fluxes; (3) controlling tritium inventory including co-deposition;and (4) controlling wall impurity production and transport into the plasma. All of these elements arebeing encountered to some extent now in long pulse discharges in operational devices, and they will
be encountered fully in a burning plasma experimental device in the ten year timeframe. A numberof models of varying sophistication exist to describe these processes. Some models already provide alevel of coupling, e.g., hydrogen transport, recycling neutrals, impurity sputtering, and impurity
transport codes. However, many of the constituent models need improvement, and more inclusive
couplings are required to self-consistently predict the edge-plasma behavior. See Fig.III.1b for a
pictorial of a tokamak cross section, and refer to the Appendix for details regarding the edge plasma
region.
FESAC ISOFS Subcommittee Final Report17Overarching theme: An early overarching issue for the edge region could be work toward a goodunderstanding of what controls the suppression of plasma turbulence to produce a transport barrierin the pedestal region (#1 above), and its associated impact on plasma profiles and stability. The
ability to predict the behavior of the edge pedestal barrier is essential for projecting the net fusion
output of MFE devices. Presently, the key parameter that is believed to control the core fusion
output is the plasma temperature at the top of the pedestal; this parameter is now either
extrapolated from existing experiments, or assumed. Subsequent edge plasma modeling could focus
on including detailed models of plasma-wall interactions, and also look to couple with core physics
inside the pedestal region.
2. FII Example: Turbulence on Transport Timescales
Background: The nature of the problem to be considered in an FII on this topic can be summarized
as follows: the ‘anomalous’ transport of mass, energy, and angular momentum in toroidal MFE
devices is dominated by fluxes driven by plasma turbulence. Further, while there is a significant
disparity of scales, especially timescales, this is a highly coupled system.
Comparative purpose: An objective of an FII in this area would be to bridge the range of temporaland spatial scales so as to compute the full system self-consistently, as opposed to just computing 3D
fine-scale turbulence with fixed background profiles, or computing 1D transport with highly reduced
theoretical or empirical models of the turbulent fluxes, as is often done at present.
Overarching theme: A single overarching science issue and goal is the self-consistent calculation ofcore temperature and density profiles from first-principles physics. An initial (easier) focus could beto determine steady-state confinement. Subsequent time evolution on the transport timescale isconceptually no more difficult but is more computationally demanding. The achievement of thesteady-state goal would, as a side benefit, enable optimization studies. Important issues likesimulation of both steady-state and time-dependent versions of internal transport barriers are subsetsof this overall goal.
3. FII Example: Global Stability
Background: Global stability issues play a central role in determining the optimal operating regimeof fusion devices, and in describing their time evolution. It is well known that under some operating
conditions, an experimental discharge can spontaneously transform from a symmetrical stable system
exhibiting good confinement into one that exhibits symmetry-breaking oscillations and poorconfinement or becomes unstable and disruptively quenches.
Comparative purpose: At relatively low temperatures, global stability dynamics is well described asa resistive magnetofluid. Solutions of this model are complicated by a wide separation of space andtime scales, and by the inherent high degree of anisotropy that occurs in a toroidally confinedmagnetized plasma. At the higher temperatures that occur in modern tokamaks, kinetic effects bothparallel and perpendicular to the magnetic field introduce important physical processes that can affect
the global magnetohydrodynamic (MHD) evolution of the plasma. Presently, mathematical andcomputational models that include some kinetic effects while retaining the computational
tractability of the fluid model are collectively called extended MHD. While good progress has been
made to date many of the approximations are adapted for the problems and resources at hand — andare not prescribed from first principles.

FESAC ISOFS Subcommittee Final Report18Overarching theme: The first-principles coupling of the transport and kinetic turbulence models to
address the issues of global fusion plasma stability at all relevant temperatures and densities andinclusive of regions extending beyond the central core is a formidable problem requiring integrated
modeling as envisioned by the FSP. The overarching science issue of this FII could be the predictive
calculation of the onset and evolution of global symmetry-breaking events such as sawtoothoscillations, neoclassical tearing modes, disruptions, edge localized modes, and resistive wall modes,
as perhaps triggered by edge plasma effects. A goal for this FII could be the development of a robust
predictive capability for fusion device optimization.
4. FII Example: Whole Device Modeling
Background: The distinguishing feature of a whole device modeling FII would be that from the outset it would provide a model of the entire device for the whole discharge timescale. Through its
ability to allow understanding of such coupled effects, a whole, or integrated device model should
connect theory to experiment, facilitate model validation, allow offline development and explorationof new operating regimes, and amplify the knowledge which can be extracted from experimental
results. Such capabilities are increasingly crucial to the development of economically attractive fusion
reactors, maximizing the efficient use of experiments, and accelerating design of new or optimized
devices with high-confidence validated models.
Comparative purpose: Because of the scope of whole device modeling, existing models are atpresent necessarily very simple. The state-of-the-art of whole-device complete-shot modeling isrepresented by an array of 1D transport codes, described in the Appendix. The 1D codes have many
features that would be required for a final product whole device model. They employ a formal
separation of time-scales between the rapid (Alfven) time on which 2D magnetic equilibria areestablished, and the much slower time on which heat, particles, and angular momentum, aretransported as 1D surface functions across the magnetic surfaces. They also incorporate many
features of a truly integrated device model (IDM): a hierarchy of models to describe particular aspectsof physics, with trade-offs between speed and accuracy; connection to experimental databases; and,predictive and interpretive modes.
Overarching theme: In essence, whole device modeling is a quintessentially integrated activity. It isenvisioned that simple models for all relevant aspects of a whole fusion experimental device would
exist in the model, and would be capable of being replaced by more complete and accurate models as
they become available and/ or as warranted by the application. Problems to which a whole device
modeling capability could be applied include global validation with experiment, development of new
or improved experimental diagnostics, or simulation of a proposed new machine on transport
timescales. It should also be possible for a whole device modeling code to serve as the 1D transport
solver throughout the development of any of the new couplings in other FIIs. From thisperspective, an FII initiative in the whole device modeling area would naturally overlap with other
FIIs.
d. INSIGHTS
In the past, computational modeling has contributed greatly to insights regarding the behavior ofmagnetically confined plasmas. We fully expect that the FSP will lead to new surprises coming from
more comprehensive models and emerging from enhanced synergy between theory, experiments andmodeling. As we start on the road to burning plasmas, some areas ripe for integration have been
identified above as FIIs. These include edge physics, turbulence on transport time scales, and global
stability, with contributions to the understanding of major and minor disruptions, plasma control,and effective rf heating mechanisms, among others.
FESAC ISOFS Subcommittee Final Report19

By their very nature of enabling more comprehensive modeling, the FIIs will lead to insights notrealizable previously. Regarding edge physics, overall transport and confinement are apparently
determined by the height of the temperature pedestal at the plasma edge. It is expected that coupled
and complex models of particle and heat transport, neutral and impurity fluxes, and edge
gradient-induced MHD instabilities and turbulence in a single computational edge framework will
pin down which of these mechanisms — either by itself or combined with another — regulates thepedestal height. Similarly, turbulence on transport time scales is a daunting physics andcomputational task. It is nevertheless deemed feasible at several levels, each exploiting separation of
space and time scales appropriately. The result of high-confidence integrated modeling of turbulence
and transport might be the discovery computationally of new favorable operating modes, with theultimate outcome being the determination of transport from first principles. With respect to global
stability, integration will facilitate extensions to MHD computations beyond the conventional ideal
and resistive models, and may provide a way to control MHD activity that is as effective nonlinearly
as it is linearly for realistic toroidal plasmas. Moreover, the inclusion of minority ion species with
non-Maxwellian populations will enable extended MHD models to take on burning plasmas.
Perhaps the greatest innovation afforded by integrated modeling will be realized for burning plasmastudies. It is well established that the grand challenge in the world fusion program is a burning
plasma experiment. Such an experiment is a necessary predecessor to a practical power demo plant
because, by its very nature, a burning plasma presents a new category of technical issues. With selfheating as the dominant plasma heating mechanism, new plasma processes and effects will arise. The
high flux of energetic particles will impact the plasma and produce a rich source of wall interaction
phenomena. Most importantly, all of these effects will be strongly coupled and must be understoodand managed in an integrated fashion to insure the stability and success of the experiment. The
ultimate Fusion Plasma Simulator will be targeted to model these processes and their consequences,
thereby providing the essential insights to guide experimental programs, optimize machine design,
provide information for fusion demo devices, and deepen our understanding of the science.
e. COMPUTATIONAL SCIENCE
1. Overview of Computational Mathematics Opportunities
Fundamental to the mandate of a program in integrated simulation of fusion systems is thatsimulation with any subset of components becomes routine. Bringing interacting components to astate of self-consistency, and then performing experimental computational science by studying the
behavior of the resulting integrated system as internal parameters or external forcings are varied,
implies a multiplicity of nests of iteration over the components. In this environment, ‘brute force’techniques for the individual topical analyses making up the inner loops of the integrated simulation
have untenable costs in computational complexity and storage. Among the opportunities presented
by the FSP are those of developing optimal discretizations and optimal solution techniques for fusion
systems, and of insuring that all known techniques of potential value are propagated into the fusion
context from related fields in computational physics and computational mathematics.
To appreciate the importance of optimal discretizations, namely discretizations that adapt to resolve
the most physics for the memory available, or all of the required physics in the least memory, one
need only consider the ‘curse’ of dimensionality. A doubling of resolution in one dimension of thesix-dimensional phase-space for the Boltzmann equation, which is at the heart of much of fusion
simulation, requires a 64-fold increase in the amount of memory, assuming that enhanced resolution
is propagated in a uniform way throughout phase space. An optimal discretization will tune the
FESAC ISOFS Subcommittee Final Report — 11/8/02
20

discretization locally to achieve a global error bound at minimum cost. This can be achieved via a
gridding technique or by means of an approximation technique built on the grid, or (preferably) both.
As another example, using an optimal iterative method for a sparse matrix solve as compared with aclassical direct method is equivalent, in the cost of solving a Poisson problem on a cube with 100
degrees of freedom on a side, to replacing a 1 Mflop/s computer with a 100 Tflop/s computer — andmuch cheaper than the hardware-only solution even if some rewriting of data structures is required.
Back-of-envelope scenarios for these and many other fusion-relevant numerical problems emphasize
the infeasibility of stepping from departmental clusters to terascale systems without a concurrent
research program in optimal algorithms for massive fusion simulations, and they underline theproverb: ‘I would rather have today’s algorithms on yesterday’s computers, than vice versa.’
Each of the topical areas in the FSP individually present characteristics that are extremely
challenging. These include the presence of multiple time scales, ranging over fourteen orders ofmagnitude, and multiple spatial scales, ranging over eight orders of magnitude. In many cases, theunderlying physics is often coupled with essential nonlinearities, and hence reasonable simulation
equation closures are the subject of research. Once closures are decided, the solutions are severely
anisotropic and the computational domains are often geometrically complex, resulting among otherissues in linear algebraic systems that can be sparse and ill-conditioned.
Taken in isolation, there are approaches that have been developed for most numerical challenges as
they have arisen in fusion and in other application areas. Such approaches include stiff integrators tohandle problems with a wide range of time scales, adaptive meshing to optimally place resolution
where it is needed most, optimal order linear solvers, physics-based preconditioners, and sensitivity
analysis tools. However, an integrated simulation will present all of these features simultaneously, as
well as additional troublesome characteristics, such as nonlocal operators, inherent physicalinstabilities that must be resolved numerically, and which cannot be confused with potential
numerical instabilities, high (i.e. greater than three) dimensionality for both dependent andindependent variable spaces, mixed dimensional code components, and mixed continuum-particle
models, based on different but physically co-located meshes.
While the combination of problem characteristics for integrated simulation of a fusion plasmapresents the applied mathematics and computational science communities with possibly theirgreatest challenge yet, it also presents these communities with a magnificent opportunity to explore
new methodologies on problems of visibility, usefulness, and external impact. The FIIs will clearlyrequire new modes of thinking and operation for the physicists, applied mathematicians, andcomputer scientists involved. For example, it is unlikely that a single speciality code can provide thebase to which all others should adapt, and ultimately what has worked until now may have to becompletely re-thought and redesigned. Furthermore, no transformations arranged by computerscience tools alone, such as a peer-to-peer software framework to couple existing codes through their
inputs and outputs, will be able to provide generality of application, ease of use, and acceptablecomputational performance. New algorithms, especially new discretizations and new physicsadapted multilevel preconditioners, will undoubtedly be required.
Research opportunities within an FII that will be shared by the applied mathematics, computational
science and fusion science communities include:

FESAC ISOFS Subcommittee Final Report21

1. Meshing: New methods for dealing with complex geometries via unstructured andmulticomponent meshes. This includes general meshing tools for tori and topologicallytoroidal geometries, using both fully structured and hybrid structured-unstructured
meshes in the poloidal planes. Recent developments in mesh generation, including
capabilities for generating hybrid and embedded-boundary Cartesian meshes, will comeinto play in this research.
2. Discretization: Advanced discretizations of differential or integral operators using highorder or solution-specific schemes. The extreme anisotropy present in many situations of
relevance to fusion science implies that PDE discretizations must be designed that respect
the orders of magnitude differences in transport along vs. across magnetic flux surfaces.
3. Local refinement techniques: Locally refined meshing and discretization techniques thatmight be determined either adaptively or statically. While tokamaks have relatively fixed
and well-defined geometries, the solution isosurfaces have dynamically convoluted andfolding geometries.
4. Linear, nonlinear and conservation law solver technology: The FSP will require optimalorder solvers for linear and nonlinear systems, hybrid continuum-particle solvers, fastcurl-curl solvers, and stiff method-of-lines solvers (for integrating compressive Alfven
waves in the poloidal field or both compressional and shear Alfven waves to follow
slower dynamically relevant timescales more efficiently). Multilevel methods will need tobe adapted to the afore-mentioned anisotropy.
Also required are hyperbolic
conservation law integrators, and nonlinearly consistent iterative methods for coupledphysics with essential two-way finite amplitude nonlinearities.
5. Transfer of field or particle data between representations: An FII will require techniques
for handling the coupling of code components by identifying natural representations thatallow transfer of physical quantities. For example, in order for a PIC (particle-in-cell)code and a finite element code to interchange data requires more than unit conversion,
and will be one area which cannot be accomplished efficiently and accurately without
involving computer scientists, and will be required to translate a field representation fromone discretization to another, possibly co-located in the same domain.
6. Data management, interpretation and visualization: Interpretation of results entails
multiple numerical and computer science research issues: checks for conservation anddiscrete satisfaction of continuous properties, visualization, advanced post-processing, anddata mining.
Following success on the direct problem of multiphysics simulation in the early years of theinitiative, collaborative work with applied mathematicians would pursue sensitivity analysis, stability,
design and control of experiments, parameter identification, data assimilation, experimental
validation, and computational steering. These ends must be considered in the early stages of software
design, however, to ensure that there is a path to the ultimate goal of scientific discovery and thecomputational optimization of a full burning plasma device.
Achieving FSP goals involves issues that are generic to a wide range of emerging computational
science problems involving other fields of physics and engineering. Solutions can likely be leveraged
across FIIs and from other similar activities such as the DOE SciDAC program. Capabilities whichshould be expected from the collaboration include understanding a range of algorithmic andmodeling options and their tradeoffs (memory versus time, interprocess communications versus
redundant computations, etc.). The simulator should be able to try a range of reasonable options
from different sources easily without recoding or even recompiling. Error estimates should be
FESAC ISOFS Subcommittee Final Report22automatically provided from meshing, discretization, and iterative methods, and performance
feedback provided from solvers. The collaboration should eventually help code users to spend moretime pushing back the limits of physics understanding, with less time spent in coding and developing mesh generation tools and solvers. On the other hand, the code users in an FII must be willingto experiment with novel algorithms and software methodologies. These collaborations must begin
early in the planning phases of an FII, with agreement on achieving the research goals of all stakeholders.
2. Overview of Computer Science Issues
Any FII must provide a software methodology and framework for designing, building, maintaining,and validating the software needed for integrated simulations. A first step an FII must make is to
identify the architectures needed, defined by the IEEE 610.2 spec as “the structure of the components,
their relationships, and the principles and guidelines governing their design and evolution over time.”
At least three major models are possible for fully integrated architectures starting from the current setof individual topical codes:
1. Peer-to-peer model: Existing codes are adapted to communicate directly with each other,but otherwise operate as separate processes. This is the approach used in systems like
University of Utah’s SciRun, Purdue’s PUNCH project, and Indiana University’s XCAT.
This is a distributed software components model, and maximizes the ability for
individual codes to continue to be developed independently, at the cost of large file or datatransfers over the network during a simulation.
2. Single executable model: Existing codes are subsumed as procedures in a single new
executable. This might be done by starting with an existing code (e.g., transport orextended MHD) and then adding on other capabilities step by step. Another approach isto refactor existing codes by decomposing them into constituent parts and rebuild a new
single code systematically designed from the ground up.
3. Hybrid model: Some existing codes are integrated together into single executables, butthen they interoperate as peers with other FSP modules. The codes integrated together
might be topical codes that require intimate coupling because of data exchange
requirements, while the separate modules have less intensive communication
requirements.
Any software architecture proposed for an FII must define clearly what the functional modules andcomponents are, indicate how interfaces between modules are defined, and provide a work flow
model for how a user will ultimately build simulations from the modules. The definition ofmodules should follow from the chosen intellectual and mathematical integrated framework, butshould also reflect two counterbalancing forces. First, physicists need to continue the full spectrumof the fundamental physics in their individual topics areas, all of which are in a rapid state ofdevelopment. This implies that some upgrade path is needed that allows the scientists involved tocontinue running and developing their individual codes during the development of the FSP. In thelimiting case, this would argue for a full peer-to-peer model. The second force is the need (driven bylimited resources) to identify shared modules usable by multiple topics codes: meshing algorithms andlinear and nonlinear solvers are possible candidates identified in the applied mathematics
section. In the limiting case, this would argue for a fully refactored system with single executable.
More generally, the components and modules in an FII architecture need to be defined at multiple
FESAC ISOFS Subcommittee Final Report23levels of granularity. At the highest level, general functional modules should be identified, whichmight consist of (modified) existing physics codes for transport, MHD, sources, etc. At a finer
grained level, the functions/routines from which to build codes should be identified, e.g., a toolbox
of solvers, meshers, discretizers, and data converters.
An FII will also have a data architecture, the model of all data needed to support the research: the typesof data and data objects, how they are described and defined, and their relationships. Verification andvalidation are critical components for the FSP framework, and supporting access to experiments,
experimental data and diagnostics implies the need for well-defined data systems. Metadata (“dataabout data”) mechanisms will be needed; an integrated simulation may span multiple geographically
distributed machines as well as multiple codes, and tracking the results of a simulation will need datasystems which can locate all related outputs and allow multiple users to attach annotations to the data.
Access policies and mechanisms will also need to be defined: which users get access to which data;
who has write versus read permissions; and, what if any security protocols are required to protect the
integrity of the data.
An architecture is implemented as a computational framework. The computational framework
includes how modules are linked together, the ‘run-time system’ which provides communications andcontrol between modules, and lifecycle control (starting, stopping, killing parts of the computation).
A computational framework might take the form of a problem-solving environment (PSE), which is thefull set of utilities and tools needed to set and solve a range of problems from a particular domain. A
PSE often includes a graphical user interface, a way of describing problems in the language natural tothe problem domain, and specialized post-run analyses that hide complexity from the user.
3. Computer systems issues an FII must address
The central goal for an FII is the complete integrated simulation of an overarching fusion physics
problem, with the eventual goals of predictive simulation of a burning plasma and parameter
optimization that can lead to more efficient magnetic confinement devices. Accomplishing thesegoals will require addressing several important but straightforward computer science issues. From asoftware engineering point of view, an excellent proposal will include the following items.
Requirements analysis: A process is needed to identify the components and capabilities necessary toaccomplish both the short and long term goals. Both envisioned scenarios and more formal ‘use cases’
could be helpful in deriving the requirements. A project like the Fusion Plasma Simulator will have
evolving requirements over its lifespan, and the process used for changing requirements needs to be
specified.
Sample requirements could include:
1. Computer languages: Will the computational framework be required to support
multiple computer languages, or will all components be required to have an interface ina single language?
2. Code ownership: Will the components be “owned” by their creators, or be communityowned, or be required to be open-source? Will commercial software be used, and if sowhat licensing will be necessary to assure long-term viability of the proposed FII?
3. Platform dependence: Will components be required to run on particular operating
systems and hardware platforms? How are those chosen and what support will beneeded for porting and testing between platforms if more than one is chosen?
FESAC ISOFS Subcommittee Final Report24

4. Performance requirements: An integrated simulation system will not run faster than theslowest of its components. What are the performance requirements, and how will they be
expressed?
Survey of existing systems: Many frameworks, systems, and architectures are currently underdevelopment and being used for high-performance scientific computing. Examples include
CACTUS for computational astrophysics, the DOE’s Common Component Architecture, Argonne’s
PETSc libraries of linear and nonlinear solvers, the National Transport Code Consortium, theUniversity of Utah’s SciRun framework, the Community Climate Modeling System. Identificationof what can be utilized from these and other similar projects, ranging from design to codes, and what
if any deficiencies they have for the FSP, will enable leveraging these existing code bases.
Basic software maintenance: An FII will likely span multiple laboratories, developers, andgeographically distributed sites. The code development as well as the final product will be shared, so
formal systems for software development and maintenance are required. Some version control
system like RCS, CVS, BitKeeper, or SCCS can help in coordinating distributed development, andkeeping archival versions of previous releases. Some formal bug tracking tools should be used since
the software is likely to be under rapid parallel development by separated code teams. A framework
for unit and regression testing will allow automated testing and notification of stakeholders in the FSPof potential problems from updates. Software maintenance is a critical infrastructure for successful
development and deployment.
Development path: A migration plan must be provided that indicates how to move from the current
standalone codes in different topical areas to an integrated physics framework. This path needs toreflect the requirements of participating code users to continue producing research with their codes
during the development of an FII framework. Software engineering research has shown that there istypically a 50% higher cost to develop components to the high quality standards needed for re-use
and sharing; however, once a core of usable and useful components is available they can raise
expectations and draw in other developers.
Flexibility and extensibility: Each FII should have a plan for tracking and using software utilities andcomponents developed by other FIIs. Eventual interoperability and shorter-term shared module
development need to be identified and exploited whenever possible. The problem domain that the
FSP framework handles must be explicitly stated.
Data models: A description of the data that needs to be shared or communicated between componentsof the architecture at runtime must be provided. How is the data described programmatically (e.g.,using HDF5 descriptors or XML schema), and how is the data model extended to unforeseen futuredata interactions? In addition to static information about data objects and how they are defined,
each FII needs to provide estimates of how often interacting components need to exchange data, thesizes of the data objects in those interchanges, and what if any data mediators (for interpolation, unit
conversion, coordinate transformation, etc.) are required. Related to this is a networking
requirements analysis, describing what must be transferred over local or wide area networks, thenetwork capability of the participating sites, and what parts of an integrated simulation will require
special high-speed connections, quality of service guarantees, or special security protocols.
While the items above are basic requirements for any FII proposal, additional desiderata might
include:FESAC ISOFS Subcommittee Final Report25•The ability to work hierarchically with the components of the architecture: an expert inextended MHD should be supported by the framework in assembling a MHD solver withvariant capabilities or to explore new methods. At the same time, an expert in RF sources
should be able to use a framework-provided ‘default’ MHD component without
becoming an MHD expert.

•

Rapid prototyping capabilities, or the ability to run selected components in a lowerfidelity mode for quick tests and proof of concept simulations: this also refers to theability to quickly compose, compile, and launch ‘what-if’ scenarios using the framework.

•

Collaboration support, such as human interactions via videoconferencing, shared codedevelopment and distribution tools, and a shared testbed of hardware and software usedby everyone on the same FII, or across multiple FIIs: this underlies the concept ofvirtual FII centers of research.

•

Data analysis tools, that can be used across physics regimes, mesh and discretization
techniques (PIC, AMR, finite elements), and disparate codes: this includes visualization
but may also include statistical summaries, consistency checkers, etc.

These additional considerations are more generally characteristic features of problem-solving
environments.
While an overall governance structure will be mandated for all of the FIIs within the FSP, as described
elsewhere in this report, a successful proposal should also be required to address some local
governance issues related to the computational science infrastructure proposed. Software version
control, bug tracking, and code configuration and maintenance methodologies are of little help unless
all the stakeholders use the proposed system. Users will be required to follow some standards, but if
too onerous they will be ignored. Particularly in situations where stakeholders are geographicallydispersed, a priori agreements need to be worked out on timely responses to issues reported, and thelevel of support that individual component suppliers are expected to provide.
f. VERIFICATION AND VALIDATION REQUIREMENTS
Since the goal of the FSP is to build models capable of accurate prediction, it must be in a position
to assess the reliability or accuracy of these models at all phases throughout the project. Assessmentof predictive models has been divided into two distinct activities: verification, which assesses thedegree to which a code correctly implements the chosen physical model and validation, whichassesses the degree to which a code describes the real world. The former is essentially amathematical problem (in a broad sense) while the latter is essentially a physical problem. Overall,
the goal of verification and validation is an assessment of the extent to which a simulation represents true
system behavior sufficiently to be useful.
As documented in IPPA 2002 and the 2002 Snowmass Fusion Summer Study documents, predictive
capability based on scientific understanding is a key goal of the fusion energy sciences program. The
accuracy of our predictions, when mapped to the fusion energy mission, has significant economic
consequences. Reactor scale devices are expensive, and uncertainty in the underlying science requires
extra margin in their design. Formal verification and validation regimes have been defined andapplied to ‘high consequence’ applications like national defense, environmental protection andnuclear power and in some cases linked to the regulatory schemes for the systems in question. WhileFESAC ISOFS Subcommittee Final Report26

we can learn much from work of this rigor, we need to introduce a validation and verification
governance regime appropriate to the goals and scope of the FSP. Cost/benefit/risk tradeoffs will need
to be made as the project management allocates manpower and other resources. It is also importantto recognize that verification and validation is an iterative process carried out over the life of a
project, not a one-time test.
1. Verification
The verification process attempts to identify and quantify errors in the computational model and its
solution. As such it must logically precede validation. Sources of error include algorithms,
numerics, spatial or temporal gridding, coding errors, language or compiler bugs, convergencedifficulties and so forth.
The most powerful tool for verification of an individual model is comparison with analytic solutions
to the same conceptual (physical) model. This is not always easy since analytic solutions are often
only possible in very simple regimes. Comparison between codes is also useful, pointing out theimportance of maintaining diversity and breadth in the code library. Codes that use radicallydifferent approaches to their solutions provide the most thorough tests. Internal checks forconsistency and convergence are, of course also essential, by changing gridding, timesteps, andsometimes even solution algorithms.
For an integrated suite of physics models, verification can present more of a challenge, since there istypically no analytic solution available and there may not be other existing coupled computationalmodels. But there are various options, depending on the type of coupling. First, when available, iscomparison with other coupled computational models (either completely independent code efforts,
or multiple approaches to coupling implemented within a given code). Also, in some cases a coupledapproach can be compared with a ‘brute-force’ direct simulation of the same physics. For example, a
coupling of turbulence and transport can be benchmarked against a background-evolving turbulencecode for test problems where the timescales are not too disparate, or a coupling of one-dimensionalcore transport and 2- or 3D edge transport can be compared with an edge transport calculation that
extends all the way into the core. Or, a code that integrates different kinds of physics into a single
set of equations can be operated in limits where one kind of physics is expected to dominate and thencompare with an existing code that calculates the dominant physics.
In all of these approaches it is helpful to operate within a computational problem solving
environment that facilitates side-by-side execution and comparison of multiple computational
approaches.
2. Validation
A successful validation regime must begin with planning. An FII must clearly define the goals of thepredictive code - what is driving the need for the calculation? Since validation is not a mathematicalprocess, it is only really meaningful in the context of a well-defined application, such as an
overarching area of assessment as defined by an FII. The validation regime attempts to assess
quantitatively the ability of a code to predict and to define the boundary between acceptable andunacceptable extrapolations; this defines applications for which the code can be trusted. Next, the
FII participants must identify the critical issues, design real and numerical experiments, specify
metrics and define assessment criteria. Planning is the place where resources are balanced against
other elements of the project. Since validation necessarily involves experimental groups who are
FESAC ISOFS Subcommittee Final Report27

outside the simulation project and funded independently, contacts should begin at the onset of FII
planning. Critical diagnostics may need to be developed and deployed as part of the overall program,and synthetic diagnostics developed and employed in the simulations. New diagnostic techniques canalso be discovered by this process.
The principal validation activity is the design, execution, and analysis of dedicated experiments, both
real and numerical. Comparison with historical data from existing archives is useful but almost
certainly not sufficient. The crucial comparisons are those designed to test important features of themodel and evaluate critically and quantitatively. The experiments must be designed to collectessential data for comparison, particularly initial and boundary conditions needed by the code.
Experiments should challenge the codes in fundamental ways and explicitly test the model’s
assumptions. A hierarchy of measurements and comparisons of increasing difficulty should be established, for example progressing from global to local variables and from steady state to transientconditions.
Experimental design needs to be collaborative between the code project and experimental team.
Typically it will require use of simulation as part of that design. In this manner the experiments canbe optimally useful and can stress critical parameters and measurements. Use of the codes at this stage
help build the collaborative environment, tools and working methods that will be necessary during
the measurement and analysis phases. The groups must form a team for the purposes of validation.
The team should not have the goal of proving the code is correct, but dispassionately evaluating its
status. At the same time, the experimental team must be frank and forthcoming about limitations
and errors in the experimental data. The availability and quality of data is a critical need for thevalidation program and raises the very large issues of error analysis and experimental data validation.
Typically evaluation of random errors is straightforward while estimation of systematic errors is not.
Often the latter is no more than a ‘seat of pants’ estimate. Although experiments must be developed
collaboratively, independence should be maintained in data collection and analysis.
Each FII, and ultimately the FSP as a whole, must define metrics by which the comparison between
the code and experiments are to be evaluated. The metrics need to take account of all sources of error:
experimental, both random and systematic; and, assumptions and approximations in the model aswell as convergence or numerical errors. Using these metrics, the FII team then must make anassessment of its status. The assessment is essentially a statement of confidence in the code in aparticular area and confidence in the ability of the code to extrapolate or predict. (The importanceof the latter suggests greater weight be given to predictive tests rather than postdictive comparisons.)
Standard statistical analyses for hypothesis testing may be appropriate for this task.
Finally, the process and results need to be well documented. This should include a description of theexperiment, the full set of experimental data and metadata, the assumptions, parameters, inputs andoutputs from the code, a description of the analysis procedures and error estimates, along with themetrics and assessment. This should be kept as part of the documentation of the FII. Specificvalidation requirements in topical areas are given in the Appendix.

FESAC ISOFS Subcommittee Final Report28

g. INFRASTRUCTURE REQUIREMENTS
We have discussed the FII concepts, the architecture issues, and verification and validation of the FII
developing capabilities. All of this hinges on resources available to produce results, i.e., thecomputational capabilities available to the FSP. This section highlights some of the salient features ofthis fundamental project need that is external to the resources of the FSP.
1. Computational Resources
To realize the benefits of the developing simulation capability will require three rather distinct typesof computing resources:1.

interactive or rapid turnaround resources for short to moderate times, at all ranges of relevant
memory (e.g., including largest numbers of processors for short times) for purposes of codedevelopment and debugging, testing of physics formulations, code components, andnumerical methods;

2.

substantial computer resources for very long periods of time for production runs andparameter surveys. In this case a figure of merit is the number of usable flops available overthe course of a year, not the maximum achievable flops, and need not necessarily be on thelargest, fastest machines; and,3.

the largest memory, fastest processor machines to allow exploration of extremely challenging
physics regimes having high resolution requirements, large Reynolds number, highdimensionality and the like, to push a verified, validated computational capability into a
regime that is wholly new.

Each of the fundamental areas of fusion theory are now pushing the limits of computation of thetypes listed as 2) and 3) above. For example, the key challenge in performing Extended MHD
computations relevant to the hot plasmas of modern fusion experiments is to increase thedimensionless parameter characterizing inverse plasma collisionality, the Lundquist number, S.
Present Extended MHD calculations have achieved 18 Gflop/sec (GF) on 384 processors of an IBM
SP3. This performance limits both the accessible Lundquist number (~107) and the problem time(~1 msec). These values are several orders of magnitude less than are required to accurately simulatepresent fusion experiments. It is estimated that a 1000-fold increase to 20 Tflop/sec (TF) sustained
performance could allow values of S approaching 109 and the problem time to approach a tenth of asecond or more, enabling validation of the mathematical models and comparison with present experiments. Further extensions into the 100s of TF regime would likely be needed to treat some key problems for burning plasma devices.
-

At the present time roughly 10 3 s of a turbulent discharge can be modeled at minimal spatial grid resolution with 120 hours on 128 processors on the NERSC IBM SP (115 MF sustained
performance per processor
1.5 TF hours). This time needs to be increased by a factor of 10-100
to address transport time scales. Furthermore additional physics associated with kinetic electrons(which necessitates an increase in computing resources ~50 to 100) and electromagnetic couplingmust be included in the models in order to allow a quantitative understanding. Codes in the SciDAC
Plasma Microturbulence Project have recently added this physics capability, but presently there areinsufficient computer resources to carry out the scientific studies. It is estimated that 103 to 10 4 TF
hours are required to include kinetic electron dynamics for transport time scales.
FESAC ISOFS Subcommittee Final Report29

A gyrokinetic edge code simulation would require a capability in the 20 TF (sustained performance)
range to simulate up to nominal background relaxation timescales. Full-shot simulation, or
simulations requiring coupling to the largest spatial scales, e.g. for edge-localized modes, would
require at least an additional order of magnitude.
Codes which solve the full, hot plasma wave equation in 2D and 3D, to all orders in Larmor radius
divided by scale length and including all cyclotron harmonics have been developed under the SciDAC
program to study wave heating, current drive and plasma flow drive. These codes scale well and haveachieved efficiencies ~40% relative to theoretical maximum using 1600 processors on the NERSC
Seaborg machine. High resolution solutions in 2D to study fast wave mode conversion typicallyrequire a few TF hours per toroidal mode calculated. A full antenna spectrum of ~50 toroidal modes
then would require ~100 TF hours. A low-resolution solution in 3D for fast wave propagation
required ~ 30 TF hours.
As we seek to integrate the disparate plasma models, it is realistic to expect that the level offundamental physics detail that can be incorporated will be dictated by the available computerresources. Two to three orders of magnitude increase in effective computing may be required toachieve the program goals. This increase can come from several sources: more, and more problem
efficient, computer hardware; improved physics analysis resulting in more accurate reduced models;and, improved algorithmic and mathematical methods.
2. Network and Storage
Requirements for network connectivity and mass storage are driven by the vast quantities of data thatwill be produced by the proposed simulations and by the geographical distribution of participants andcomputational resources. Precise quantitative predictions are difficult since the frequency with which
simulations will be performed, the amount of data generated, the amount of that data that will need
to be stored or transferred, how quickly after a simulation data will be needed and how manysites/people will use the data are all uncertain at this point. In part, these depend on the FIIs chosen,
and in part on other issues such as researcher proximity. Still, even under the most prosaic imagined
situations, rough estimates yield numbers which are large enough to warrant serious attention. Overthe next five years, three dimensional, non-linear MHD and turbulence codes may be generating on
the order 1 PByte per simulation. We may further hypothesize that, integrated over the entire
project, it would be desirable for major simulations, each representing a full, integrated experimental“shot” to be completed on the order of once a week. (This also is practically possible. Consider:
1000x 1000x100 spa tial zones, 10 vari ables per zone, 106 time steps, an dassuming 100 floating-point operations per variable per space-time point, such a calculation would
take no more than a few percent of 50 TFlops machine to complete in a week.) By this estimate,aggregate rates in the 10’s of PByte per year should be planned for. It is not sufficient to simply
archive this data; scientific progress will be linked to our ability make effective use of it.
Meeting these challenges will require a highly capable network, massive storage infrastructures, alongwith advanced middleware and network services to “glue” them together. All of these elements are
required resources for access by the FIIs. Simply moving the estimated simulation data once would
require dedicated links at several Gbps. Distributing the data to multiple sites over a wide area
network, while technically feasible in the project’s time frame, is probably not an economicallyreasonable approach. Instead, the researchers should plan on moving only that part of the datanecessary for visualization or post-run analysis. Assuming a data set for local analysis or visualization
is 10-20 GBytes (a reasonable guess for the RAM capacity of a workstation in the next five years), andthat 10 or 20 seconds is a reasonable waiting time, one calculates a requirement for burst transfer rates
FESAC ISOFS Subcommittee Final Report30

up to 1 Gbps. Alternately, post-processing or visualization engines could be co-located with the datastore and the results streamed to end users. An HDTV stream requires about 50 Mbps, a large
display wall, fed with uncompressed data could use 1 Gbps. Thus the network requirement may be
estimated conservatively as one that supports a large number of users (perhaps 50-60) located at mostof U.S. fusion sites each transferring bursts of 1 Gbps at duty cycles of a few percent.
Data storage in a reliable and robust repository (or repositories) is another formidable challenge. In
analogy with experimental data, results generated for major simulations will be of archival quality.
The estimates shown above, suggest that data will accumulate at 10’s of PBytes per year. As with the
network requirements, these rates are not technically insurmountable but do require significant
thought as every decision concerning archives of this magnitude will have a serious impact on theability of the team to carry out the scientific program and on the economics of the project.
Architectural decisions include where to store the data, whether to centralize or distribute the archive,
whether to support data replication at remote sites, how to integrate post-simulation analysis, how tointegrate with experimental archives, and how to guarantee data integrity and consistency. With sucha large quantity of data, automated mechanisms for constructing data digests, databases or summaries
and new and improved ways to efficiently mine the repositories to extract knowledge will need to be
developed.
With distributed resources for software development, computing, storage, analysis and visualization,
the project will require advanced network middleware that supports distributed computing andcollaboration. At the same time the apparently conflicting requirements for transparency andsecurity in a widely distributed environment point up the need for efficient and effective network
services. Central management of Public Key Infrastructure (PKI) or equivalent technologies using‘best practices’ and providing around the clock support is essential. It is equally essential that the user
authentication framework adopted is such that common policy can be negotiated among thecollaborating sites. Mutually agreed upon tools and protocols for resource authorization is also
important. For such a large user base and with the need for close collaboration with experimentalgroups on validation tasks, global directory and naming services may be a key technology and may
help to anchor the wealth of distributed metadata. A hierarchical infrastructure with well-managed‘roots’ can provide the necessary glue for many collaborative activities. A global name service could
also solve the longstanding problem for our field of variable name translation between codes orexperiments. Since users, including partners at universities, private companies and international sites
are interested only in end-to-end performance, real-time network performance monitoring andproblem resolution tools which work across administrative domains will be essential. Finally, a hostof collaboration services including teleconferencing, distributed applications and remote visualization
will be required.
3. Summary of infrastructure requirements
High-end infrastructure requirements are summarized in the table below. In addition, the project willneed local infrastructure consisting of medium scale computing clusters for development, testing, andpost-processing; powerful desktop and visualization systems; medium scale storage systems and wellprovisioned local area networks.

FESAC ISOFS Subcommittee Final Report31

NOW
Computers~4Tflop/s~2TByte memory

SOON(1-3 years)
10’s Tflop/s
TBytes memory

Storage
Networks~50TByte/year
0.1 Gbps to desktop
0.62 Gbps backbone~1 PByte/year
10’s PByte/year
1 Gbps to the desktop >1 Gbps to the desktop
10 Gbps backbone
100 Gbps backbone

Table III.1. High-end fusion simulation infrastructure requirements.

FESAC ISOFS Subcommittee Final Report
32

LATER
(> 3 years)
100’s Tflop/s
10’s TByte memory

IV. THE FSP PATH
a. PROJECT ROADMAPThe goals of the FSP both near, five-years, and longer term, ten-years and fifteen-years, areambitious. To meet them careful but flexible planning coupled with a well-designed program
governance and program management structure is required. To put governance and management
requirements in context, Fig.IV.1 below provides a summary roadmap of the FSP goals.

We expect a 15 year timeline is required to produce the FPS
FII A

FII A Phase 2

FII B

FII B/C Phase 2

FII C

FII D Phase 2

FII D

FII E

Software
Structure

Exploration ofseveral computationalframeworks selectedby the FIIs.

FIIs become moreinteroperable, someconsolidated. Perhaps
new FIIs started.

Single integrated
System drawing fromtotal FII experience.

New
Capability

Better understandingof isolated integrated
phenomena.

Wider and more indepth integration. New
physical processes added.

Full simulation of all
relevant processes infusion systems.

Normalized
Integration
Actitvity

0

5 yr

FusionPlasma Simulator
(FPS)

10 yr

15 yr

Funding Years
Figure IV.1: Fifteen-year roadmap for the Fusion Simulation Project

The plan shows a fifteen year timeline, with significant value and specific milestones at the end of fiveand ten years. As noted in the 2002 Snowmass Fusion Summer Study, the full extent of the Fusion
Plasma Simulator project is expected to require funding on the order of $0.4B throughout the fifteen
year period.
The three major phases of the project are described in the following paragraphs.
First five years:
During the first five years, we will initiate several Focused Integration Initiatives (FIIs) as describedin Section III. These will be concentrated on specific high-profile physics integration issues that areconsidered to be the most critical, and are also prototypical of the integration issues faced by thewhole initiative. Each FII will be quasi-independent of the others, but there will be efforts made at

FESAC ISOFS Subcommittee Final Report — 11/8/02
33

coordination between initiatives, looking towards future integrations. We expect at the end of the
five-year period to have substantial new capability in each of the FIIs and to gain new scientificinsights during this time frame.
Each of the FIIs will be expected to develop a computational framework best fitted to its task. Since
there is not universal agreement on the best way to solve many of these integration problems, thecomputational framework also needs to allow rapid prototyping of different solution techniques oversome range of hardware architectures.
New computational algorithms will be developed to treat the unique mathematical problems present
in fusion science. For example, many of these arise from the presence of the strong magnetic field,
which adds an extreme anisotropy to the plasma and results in temporal and spatial anisotropy ofparticle motion. Also, the mathematical equations describing plasma waves are higher order parallelto the field than across it. These physical effects lead to sparse matrices with peculiar properties, tothe need for very specialized gridding techniques, and to the need to deal with “stiff” equations on
disparate time scales.
5-10 years:
During the second five-year period, we expect several things will occur. One is that we will begin tocombine select FIIs into larger and more comprehensive integration activities. Another is that we will
introduce new FIIs as required. A third is that we will take select FIIs to the next level ofdevelopment.
During this period, there will also be a comparative reassessment of the issue of computationalframeworks. We expect the frameworks to grow in maturity, integrating such things as advanced
graphics and user interfaces, and also that some down-selection and solidification will occur. A
unified system for effectively managing the increasing complexity of the project will become apriority. We also envision that improved algorithms will enter the project as the nature of thecouplings of the new integration phenomena becomes clearer. For example, some new algorithms
may achieve greater efficiency by combining individual components rather than by treating eachcomponent as a block in a high-level algorithmic diagram.
10-15 years:
During the final five-year period, the focus will be on comprehensive integration. There will be alink between all the physics components of the project. The mathematical framework will largely bein place for the integration.
Part of the challenge of this final integration will be to include multiple levels of description of thesame phenomena. For example, there would be an option for plasma equilibrium to be computed
either in the 2D axisymmetric approximation or fully in 3D, including the effects of small magnetic
islands. Plasma rotation could be included in either of these calculations or neglected. Each level
represents a tradeoff between computational efficiency and physical fidelity.
As the integration proceeds into this final phase, we expect the simulation capabilities to be exercised
more, and to have new and more comprehensive comparisons between the simulation andexperiment. In many cases this will lead to a validation of the model, but we expect that in somecases it will serve to identify shortcomings or inadequacies of the model that will be subsequently
addressed. This final integration will succeed only if all the fundamental components have beenadequately addressed and if the component integration is carried out correctly.
FESAC ISOFS Subcommittee Final Report34

We expect that the Fusion Plasma Simulator produced by this fifteen year project will be a living
software system that will continue to grow and be modified many years into the future. It will serve
an invaluable role as an intellectual integrator of many experimental results and approaches, and will
be heavily relied upon to reach decisions regarding the development path of fusion energy.
b. THE FSP AND OTHER OFFICE OF SCIENCE ACTIVITIESWe have stressed throughout this report that the FSP will reach across disciplinary boundaries inorder to bring together all relevant expertise required to develop an integrated simulation capability
for magnetic fusion systems. This expertise is resident in several units within the DOE Office of
Science. Primary among these are the activities within the Office of Fusion Energy Sciences and the
Office of Advanced Scientific Computing. Other activities are and will continue to be directly
relevant to the success of the FSP. Within the DOE, these include the recently developed SciDAC
initiative and materials sciences research within the Office of Basic Energy Sciences.
The SciDAC (Scientific Discovery through Advanced Computation) is one of the new, and criticalstrategic programs within the Office of Science. One of the SciDAC program’s principal goals has
been to assemble interdisciplinary teams and collaboratories to develop the necessary state-of-the-art
mathematical algorithms and software, supported by appropriate hardware and middleware
infrastructure, to use terascale computers effectively to advance fundamental research in science thatis central to the DOE mission. Substantial success has been achieved towards this goal. The SciDAC
success provides an argument for the timeliness of the FSP. Perhaps no area of science is more
central to the SciDAC mission than fusion, and five projects were launched under SciDAC auspices
in FY 2001 to develop and improve the physics models needed for integrated simulations of plasma
systems to advance fusion energy sciences.
One of these projects, the Center for Extended Magnetohydrodynamic Modeling (CEMM), has beenable to speed up an Extended MHD modeling code through synergistic interactions with appliedmathematicians on another SciDAC team (Terascale Optimal PDE Simulations). The resulting
algorithmic improvements have already decreased running times by a factor of two, and further
exploitation of certain matrix structures will yield even more improvement.
Another SciDAC fusion project coordinates a multi-institution program on ‘Numerical Computationof Wave-Plasma Interactions in Multi-Dimensional Systems.’ An applied mathematician on thisteam was able to recognize and exploit Kronecker product structure in some of the equations
underlying simulations within this project. The introduction of this and other such insights into thefusion sciences context has led to codes that are now running two to ten times as fast as previous
versions. More detailed physics can now be introduced earlier in the modeling and simulation
process, thereby greatly accelerating the pace and scope of the science that can be explored.
A third SciDAC has given rise to the U.S. Fusion Grid (http://www.fusiongrid.org), which is now
being tested on DIII-D and C-Mod scientific data analysis. Developed under the auspices of the
National Fusion Collaboratory SciDAC project, the Fusion Grid presently combines experimental
data that is stored on servers at MIT, GA, and PPPL with a computational code located at PPPL toprovide greatly improved data analysis throughput rates combined with instant access to the latest
versions of a PPPL numerical code. The Fusion Grid is thus beginning to provide a collaborative
fusion research environment that is transcending geography.
FESAC ISOFS Subcommittee Final Report
35

These and other SciDAC projects, even though currently funded at only a modest level, can bethought of as part of a pilot program. As such, SciDAC will ultimately have succeeded best if itspawns major application-specific initiatives precisely like the one being proposed in this document.
Early SciDAC success stories provide compelling proof-of-concept evidence to strongly suggest thatappropriately funded interdisciplinary teams, focused on a full-scale integrated program, will
successfully deliver a greatly enhanced simulation capability to fusion energy sciences. Such acapability is absolutely essential for realizing our nation’s goal of commercially viable fusion power ina realistic timeframe.
Further, there are significant needs in materials modeling of fusion device hardware which must be
met to provide a complete FSP predictive capability. This is work that would be carried out incollaboration with the Office of Basic Energy Sciences (OBES). It would range from basic theory offundamental material processes on the atomic, mesoscopic and continuum levels to the simulation ofcomplex surface and bulk phenomena. Surface processes of interest include sputtering and othererosion mechanisms, implantation, re-deposition and co-deposition of tritium and surface
restructuring, and roughening among many surface problems of interest. Bulk processes including
crystal lattice displacement damage, the creation of atomic and cluster defects, microstructure
evolution, dimensional instabilities, and a variety of embrittlement processes, will also needattention. It is significant that the basic approach to multi-scale modeling of materials, e.g. thepassage from atomistic simulations to mesoscopic simulations to continuum simulations, isconsistent with and complementary to the multi-scale, multi-year paradigms for the FSP. It would
be beneficial for a mutual working relationship to develop between the FSP and OBES tocomplement those between OFES and OASCR.
We anticipate that the relationship between the FSP and other Office of Science activities willcontinue to evolve and mature during the life of the FSP. Experience has shown that many advances
in basic science have been achieved in the pursuit of goal driven activities such as the FSP. We
strongly believe that this pattern will emerge again in the context of this project, thus benefiting both
the goals of enhanced energy production and the advancement of fundamental science.
c. ROADMAP IMPLEMENTATION: P ROJECT GOVERNANCE & MANAGEMENTThe FSP initiative will be focused, highly interdisciplinary, and will involve a significant number ofpeople. For these reasons it is extremely important that careful attention be given to governance ofthe project. The governance structure needs to effectively balance the professional requirements ofthe creative and individualistic people who will carry out the work with the programmatic needs forfocus and timely delivery of results. In addition the structure has to work effectively with the two
DOE programs offices, OASCR and OFES, that will support and manage the initiative.
There will be two elements of guidance and oversight needed to reach the technically complex goals
indicated in the roadmap, Fig. IV-1, on any of the indicated time scales. The first element, project
governance, is the process of coming to the best possible technical judgments when evaluating options
to reach project goals. This would include agreements about software architectures, selections ofemphasis for physics fundamentals, the down selection of FIIs, and a multitude of other issues of thissort. In addition, there are issues of project management: actual implementation of the broad
technical decisions across the FSP, e.g. software standards, tracking of progress, issues ofaccountability, organization of project reviews, assisting in the representation of the project, and etc.

FESAC ISOFS Subcommittee Final Report36

In the case where several institutions or groups have co-equal technical shareholding status in the FSP,
e.g. if the 3-5 multi-institutional FIIs are enfranchised as recommended in this report, the abovedrawn distinction has important functional implications. For this circumstance, a sketch of a
proposed governance structure is provided below. An analogous set of issues has been addressed by
the Community Climate Systems Model (CSSM) activity; see http://www.ccsm.ucar.edu . While
there are significant differences between the nature of the science involved in the CCSM and theinitiative discussed here, there nonetheless are sufficient similarities that the CCSM activity can help
suggest an optimal structure. The organizational chart suggested is:

SCIENTIFIC STEERING COMMITTEEADVISORY BOARDFII A

SOFTWARE STANDARDSCOMMITTEE

FII B

FII C

FII D

Figure IV.2. Organizational Chart for the Fusion Simulation Project.

The functions of these organizational groups are:
Scientific Steering Group (SSG): This group provides the overall scientific direction and vision for the
project. It provides oversight and coordination of scientific activities. It is the key group forassuring that integration is effected. A primary function of this group is outreach at the technical level.
The SSG ensures the verification and validation function of the FSP. The SSG will also need to work
closely with the program management on the topic of resource allocation issues.
Advisory Board: This group is made up of people with scientific breadth that are not directly engaged
in the FSP. The group will provide scientific and management advice to both the SSG and program
management. A fundamental role of the Advisory Board is to address resource adequacy and FSP
collaboration throughout the Office of Science.
Software Standards Committee (SSC): This committee is comprised of representatives from each of
the FIIs. It is critical that some level of standards and common practice be made across the FSP withrespect to software and collaborative tools. The SSC will work to assure the maximum realistic
uniformity in software choices throughout the project. The SSC ensures that each FII has a plan for
tracking and using software utilities and components developed by other FIIs.

FESAC ISOFS Subcommittee Final Report37Focused Integration Initiatives (FIIs): Each FII will have the responsibility of carrying out theoverarching research plan to which it is committed. Each FII group oversees the scientific direction
for its integrated simulation, including determination of required fundamental research, and couplingwith experiment. This is where the real work gets done, in fusion science, computational science, andin the cross-disciplinary activities that involve verification and validation.

FESAC ISOFS Subcommittee Final Report38

V. SUMMARY
The goal of the Fusion Simulation Project (FSP) is to develop the computational capacity toperform integrated simulations of toroidal magnetic confinement devices and provide a validated
predictive capability. The panel envisions a program proceeding through three five-year phases, thefirst of which is detailed in this report and would be comprised of focused integration initiatives(FIIs). This development will be made feasible by close coupling of the integration initiative researchwith ongoing core program activities in theory, experiment, computer science and applied mathcarried out under the auspices of DOE OFES and OASCR. Our vision for the initiative is detailedin Section III of this report, with the path discussed in Section IV, and a fifteen-year overview
roadmap delineated in Fig. IV.1.
Numerical modeling has played a vital role in magnetic fusion for most of its history, with increases
in the scope and reliability of simulation enabled by advances in hardware and numerics and throughimprovements in basic theory. Knowledge gained by this approach has covered the entire range ofproblems in the fusion energy sciences. A summary of the current state of fusion plasma simulation
can be found in the Appendix, where theoretical issues are also surveyed. Some progress in
integrated modeling has been made as well, leading to important insights on topics as diverse as major
disruptions, turbulence regulation by flows, and the design of compact stellarators.
Achieving the goals of the FSP will require significant collaborative advances in physics, appliedmathematics and computer science. The wide range of temporal and spatial scales, extreme
anisotropies and complex geometry, make this problem among the most challenging incomputational physics. The numerical challenges for fusion simulation are outlined in Sections IId
and IIIe. Methods for simulating phenomena coupled over disparate space and time scales, and overdifferent dimensionality will require qualitative improvements, innovations and strong collaborations
across all of the constituent disciplines. This disciplinary integration will be an essential element of
the project. The project must develop software methodologies and frameworks for designing, building, maintaining, and validating the simulation software. Computer science issues raised by thisinitiative include: the choice of an architecture for interconnecting code modules; data models;performance monitoring and optimization; provision for flexibility and extensibility; and, tools forenabling human collaborations over long distances. These and related topics are also discussed in
section IIIe.
Assessment of predictive models has been divided into two distinct activities: verification, whichassesses the degree to which a code correctly implements the chosen physical model, and validation,
which assesses the degree to which a code describes the real world. Overall, the goal of verificationand validation is an assessment of the extent to which a simulation represents true system behavior
sufficiently to be useful. Verification is particularly difficult for integrated models where analytic
solutions may not be available in any regime. The validation process puts a premium on close
collaboration between computational and experimental groups. To succeed, a central feature of thisinitiative must be an intensive and continual close coupling between the simulations efforts andexperiments. The requirements for verification and validation are summarized in section IIIf.
The Fusion Simulation Project will require significant improvements in computational infrastructure.
These include advances at major computational facilities which are shared across the Office of Science
community as well as deployment and enhancements to local or topical computing centers. The
simulations envisioned here will also produce truly prodigious quantities of data and will require
FESAC ISOFS Subcommittee Final Report39

investments in advanced storage systems at all levels. With geographically dispersed resources andresearchers, the wide-area network becomes a crucial element in the computing environment, withassociated collaborative tools and protocols. Timely upgrades to the communication network andlocal infrastructure will be required. Of particular concern is connectivity to university orinternational partners. Infrastructure requirements are detailed in Section IIIg.
New funding necessary for the success of the FSP is presently estimated at approximately $20M peryear for each of five years. To achieve the greatest productivity, this new research should be split
between OFES and OASCR, with fusion scientists funded by OFES, and applied mathematicians,computer scientists, and the computational toolkits provided under the auspices of OASCR. This
joint undertaking represents a significant opportunity for the DOE Office of Science to create acapability that will advance the understanding of fusion energy to a level unparalleled worldwide.

FESAC ISOFS Subcommittee Final Report40VI. ACKNOWLEDGEMENTS
W

e acknowledge helpful contributions to this report from members of the fusion and appliedmathematics communities, with particular thanks to speakers at the May 23 ISOFS workshop, andto the speakers and participants of the September 17-18 FSP workshop. We thank the support staff
without whom this activity would not have been possible, including Lucille Kilmer, Deanne Eggers,
Ron Winther, Beulah Koz, and Marcia Freels. We also thank the Theory Coordinating Committee(TCC) for their letter, and the members of the PSACI PAC for the concepts and suggestions
provided in their 18 June 2002 letter to this Subcommittee, and most particularly appreciate their
enthusiasm for the Fusion Simulation Project. We are grateful for the excellent and substantive technical contributions: in the computational science sections of this report, by David Brown and David
Keyes; and, in the fusion sections, by Jeff Candy, Ron Cohen, Nasr Ghoniem, Greg Hammett, Wayne
Houlberg, David Humphreys, William Nevins, Ron Stambaugh, and Ron Waltz.

FESAC ISOFS Subcommittee Final Report41

Attachment: ISOFS Charge Letter
February 22, 2002

Professor Richard D. Hazeltine, Chair
Fusion Energy Sciences Advisory Committee
Institute for Fusion Studies
University of Texas at Austin
Austin, TX 78712
Dear Professor Hazeltine:
This letter provides a charge to the Fusion Energy Sciences Advisory Committee (FESAC) to assist
the Office of Fusion Energy Sciences (OFES) in preparing a roadmap for a joint initiative with the
Office of Advanced Scientific Computing Research (OASCR). Recent reports, such as the FESAC
report “Priorities and Balance within the Fusion Energy Sciences Program,” the “Report of theIntegrated Program Planning Activity” (IPPA), and the NRC report “An Assessment of the
Department of Energy’s Fusion Energy Sciences Program,” have identified a predictive
understanding as a measure of the quality of the science and the maturity of the knowledge base of afield. The IPPA report lists several challenging10-year objectives for the fusion program, including“develop fully integrated capability for predicting the performance of externally-controlled systems
including turbulent transport, macroscopic stability, wave-particle physics, and multi-phaseinterfaces.” This objective, as well as several other IPPA objectives related to innovative confinement
configurations, will require significantly enhanced simulation and modeling capability. Therefore,
the goal of this initiative should be to develop an improved capacity for Integrated Simulation and
Optimization of Fusion Systems.
The initiative should be planned as a 5-6 year program, which would build on the improved
computational models of fundamental processes in plasmas that are being developed in the basetheory program and in the SciDAC program. Rough estimates are that an integrated simulation
initiative would require a total funding level of about $20 million per year, with funding for theplasma scientists provided by OFES and funding for the applied mathematicians, computer scientists,and computational resources provided by OASCR. Thus, the roadmap should include not only
human resources but also computer and network resources.
Please carry out the preparation of the roadmap using experts outside of FESAC membership, as
necessary, including experts recommended by the Advanced Scientific Computing Advisory
Committee. The sub-panel of experts should obtain community input through a series of workshops
covering at least the following questions:••

What is the current status of integrated computational modeling and simulation?
What should be the vision for integrated simulation of toroidal confinement fusion systems?

FESAC ISOFS Subcommittee Final Report
42

•
•
•
•

What new theory and applied mathematics are required for simulation and optimization offusion systems?
What computer science is required for simulation and optimization of fusion systems?
What are the computational infrastructure needs for integrated simulation of fusion systems?
How should integrated simulation codes be validated, and how can they best be used toenable new scientific insights?

The ultimate product should be a roadmap document similar to the one developed for the Genomes
to Life Initiative (http://www.doegenomestolife.org/roadmap/index.html). Please conduct aworkshop on the first two questions above and provide a summary document with overall program
goals and objectives, major program deliverables, and a brief description of the OFES and OASCR
funded elements of the program by July 15, 2002, so that OFES would be able to include a
description of the program in the FY 2004 OMB budget request. Please complete work on the final
roadmap by December 1, 2002, in order to provide the detailed information needed by OFES andOASCR to develop detailed program plans, program announcements and grant solicitations.
I appreciate the time and energy that members of FESAC and FESAC sub-panels have provided tothe continuing efforts to develop program plans and roadmaps for the OFES program. I am confident that the Committee’s recommendations on a roadmap for Integrated Simulation and
Optimization of Fusion Systems will form a sound basis for beginning a joint OFES/OASCR program.

Sincerely,

James F. Decker
Acting Director
Office of Science

FRONT COVER ATTRIBUTION LIST(APPEARS ON BACK COVER)

B. Carreras, W. Dorland, L.Garcia
G. Hammett, G. Kerbel, C. Kim
J-N. Leboeuf, V. Lynch, W. ParkS. Parker, B. Rogers, C. Sovinec
D. Spong, R. Sydora, NCSX Team
W7-X Team