[SciDAC II: The TeraScale Supernova Initiative Perspective]{.underline}

Anthony Mezzacappa (ORNL)

Coauthors: John Blondin (NCSU),

Eric Myra (SUNYSB), and Doug Swesty (SUNYSB)

**[Recommendations]{.underline}**

1.  A follow-on program (SciDAC II) is imperative in order to bring the
    successes documented herein to fruition and to address our Grand
    Challenge scientific problem. A concerted 5-10 year effort (from
    now) is required to develop fully general relativistic
    three-dimensional supernova models that will pin down the explosion
    mechanism and make accurate predictions of critical
    supernova-associated phenomena such as their gravitational radiation
    emission. No doubt, similar horizons exist for other
    SciDAC-sponsored applications.

2.  The development of computational simulations that can fully utilize
    the capabilities of cutting-edge parallel architectures is a
    multi-year activity. The funding model for both application and ISIC
    teams must reflect this reality. The kind of computational science
    that has been done under the current SciDAC program could not have
    been done under a three-year funding model. Therefore, we strongly
    recommend a five-year SciDAC follow-on program.

3.  We envision that our future primary mode of multidisciplinary
    collaboration will be through SAPP. SAPP has funded TSI-embedded
    mathematicians and computer scientists, as well as some of the
    groups we have most closely collaborated with (e.g., Beck's
    Logistical Networking group), as documented below. The TeraScale
    Supernova Initiative plans a major expansion in the areas of applied
    math and computer science to become, through SAPP and embedding, a
    single-application micromodel of SciDAC.

4.  Based on what we will document here, we also envision significant
    continued interaction with the ISICs. We have benefited from the
    ISICs in ways that are complementary to the benefits we have enjoyed
    through SAPP. The successes we will document below were often the
    result of the combined efforts of SAPP and ISIC teams. We imagine
    SAPP and the ISICs as equal partners in the next SciDAC round.

5.  The next generation SciDAC program should focus on applications that
    can benefit from computational capabilities at the 100+ TFlops
    level. Computational science at this scale is opening new scientific
    doors and is not being funded elsewhere. The program should also
    ensure that the appropriate computational resources are in place at
    the DOE labs to handle such computing needs.

6.  The Office of Science should demand rigorous verification and
    validation (V&V) from its application teams. This is a cornerstone
    in the foundation of a mature field. Some journals (such as AIAA
    journals) have established policies of not accepting computational
    work where the authors have not *published* rigorous verification
    and validation benchmarks for their codes on problems relevant to
    those they are simulating. The Office of Science should *demand*
    that work that has the imprimatur of SciDAC funding should manifest
    and maintain rigorous and continuing V&V efforts, including
    publication of results of V&V test problems. Furthermore, the Office
    of Science should insist that application teams publish numerical
    details sufficient to allow *replication* of the results by other
    researchers. The simple publication of plots or pictures with no
    algorithmic details should not suffice.

**[Background]{.underline}** Explosions of massive stars (core collapse
supernovae) are arguably the most important link in our chain of origin
beginning with the Big Bang and ending with the formation and evolution
of life on Earth. They are the dominant source of most elements in the
Periodic Table between oxygen and iron, and are believed to be
responsible for producing half of all elements heavier than iron. These
explosions are the most energetic events in the Universe, give birth to
neutron stars and stellar mass black holes, and serve as cosmic
laboratories for particle and nuclear physics at extremes that are
inaccessible in terrestrial experiment. For these and other reasons,
core collapse supernova science has been an area of research
traditionally supported by the Nuclear Physics and High Energy Physics
Programs within the Office of Science, and now, under SciDAC, by the
Office of Advanced Scientific Computing Research.

Core collapse supernovae are radiatively-, and perhaps magnetically-,
driven turbulent events. The intense radiation of neutrinos from the
proto-neutron star at the center of the explosion is believed to power
them, and neutrino production, transport, and interaction in the core
defines the dynamics of stellar core collapse. In fact,
three-dimensional neutrino transport is the single most important
component of a supernova model and is what defines this application as
ultimately a petascale application.

![](media/image1.pct){width="2.7777777777777777in"
height="2.7777777777777777in"}

*Visualization of a stellar explosion by J.M. Blondin (NCSU), A.
Mezzacappa (ORNL), and K.-L. Ma (UCD).*

An understanding of core collapse supernovae will be obtained through a
series of simulations in which the physics and the dimensionality are
layered in a systematic way. That is, simulations in one, two, and three
spatial dimensions with, in each case, increasingly sophisticated and
more inclusive physics must be performed. Such a systematic approach
will be in sync with the development of computer architectures as we
move from the terascale to the petascale. Ultimately, as we now discuss,
petascale platforms will be required.

The neutrino transport equations are multidimensional integro-partial
differential equations. The rapid time scales associated with radiation
transport and especially radiation interaction with matter necessitate
the use of implicit differencing in time. Underlying the solution of the
nonlinear algebraic equations that result from such differencing is the
solution of large, sparse linear systems of equations. These are
structured equations, with dense blocks on the diagonal resulting from
local neutrino interactions. Bands above and below the dense blocks
round out the sparsity pattern and are associated with the nonlocal
spatial transport of neutrinos. There are N bands above and below the
diagonal for N spatial dimensions.

These sparse linear systems are tera- to peta-scale systems depending on
the resolution in "phase space" (three space dimensions, two neutrino
angles, and neutrino energy). They are solved efficiently through the
use of matrix-free implementations of Krylov-subspace algorithms. As for
the CPU requirements, it is first important to note that the solution of
our sparse systems is dominated by the dense blocks. Assuming an
$f´N^{2}$ algorithm, where $N$ is the dimension of each dense block

![](media/image2.jpeg){width="2.625in" height="2.625in"}

*The large, sparse linear systems arising in 1-, 2-, and 3-D supernova
simulations are block structured.*

And where$f\text{<<}N$, and assuming 200 TF sustained speeds (20% of
peak on a PF machine), a well-resolved three-dimensional simulation
would require \~ 1 month to complete−i.e., 200 TF-months. In addition,
we would need to run \~10 simulations to cover different supernova
progenitor masses (10-40 Solar masses), different equations of state,
etc. Thus, in reality, we would need \~ 2000 TF-months to cover the
science. This example illustrates two points: (1) A PF machine is
required to complete such a simulation in a reasonable amount of time.
(2) The development of algorithms for the efficient solution of our
integro-partial differential equations is paramount.

In addition to the need to develop algorithms to solve the equations
governing a supernova, there is also a critical need to develop ways to
manage and analyze the massive amounts of data that will be produced in
supernova simulations in the near future. Ongoing simulations are
generating data at \~500 Mbps over the course of days, for a total of
10s of TB. Simulations in the next few years will generate 100s of TB of
data over the course of months. Storing, moving, analyzing, and
ultimately visualizing such data for scientific discovery will present a
significant challenge. Developments in supercomputer capabilities must
be met by developments in the computational science infrastructure that
will enable effective use of these capabilities and the workflows that
will ultimately lead to scientific discovery.

![](media/image3.jpeg){width="3.125in" height="2.0125in"}

*Network of "depots" set up throughout the U.S. to enable efficient data
storage and transfers for the DOE TeraScale Supernova Initiative. (S.
Atchley, M. Beck)*

**[SAPP/ISIC Collaboration Models]{.underline}**

In our view two primary factors motivate continued funding for both
models in a SciDAC follow-on program: (1) The daunting computational
science challenges faced by the application teams and their significant
application-specific needs warrant first and foremost a fully
integrated, multidisciplinary team of application scientists, applied
mathematicians, and computer scientists working should-to-shoulder to
take on these challenges for scientific discovery. (2) The commonality
of needs across application domains (e.g., in data management,
networking, and visualization) and the need to take advantage of and to
adapt existing technologies to each application domain, and to further
develop such technologies, warrant continued funding for ISIC teams as
well.

Thus, first and foremost in our view, funding for application teams in a
SciDAC follow-on program should allow for significant funding for both
discipline specific (e.g. NP, HEP, Fusion, BER, BES) personnel and
embedded math and computer science (OASCR) personnel. In particular, the
previous limit of 25% SAPP funding should be lifted. The presence of
Math/CS personnel within our collaborative team has been and will
continue to be critical to our success. Many of the pre-SciDAC
relationships between TSI physicists and Math/CS personnel were born out
of the critical need to collaborate on problems unique to our
application (e.g., the development of preconditioners *tuned to* the
non-linear systems arising out of multigroup flux-limited diffusion and
Boltzmann neutrino transport). Similarly, our visualization needs are in
many ways specific to our application and, generally speaking, are very
demanding. MICS funding for TSI-embedded applied math and visualization
researchers has allowed us to make significant advances in both areas as
they pertain to our application, which in turn has enabled our science.

Moreover, in order for the application teams to fully leverage the
efforts of the ISIC teams, the applications must have sufficient
embedded personnel dedicated to this task. At an early phase of the
current SciDAC program we were faced with a daunting many-to-one
interface problem, with one application scientist working with numerous
ISIC researchers. Obviously, interfacing with an ISIC team is a task
that requires significant time and effort on the part of the application
team. It is necessary to explain the issues we face to the members of
the ISIC team, provide application software (and guidance on its usage)
and data to ISIC team members, etc. It may also be necessary to refactor
application software info a form that is compatible with ISIC
technologies. These tasks must be carried out by an application team
member that is familiar with the intimate details of the application
project. Also, these tasks cannot be accomplished by an ISIC team member
that is unfamiliar with the details of the application. Finally, the
application developers cannot successfully carry out many such liaison
tasks while simultaneously trying to carry out application-specific
research. Thus, in order to fully leverage the efforts of future ISIC
teams the application teams should be staffed with a sufficient number
of MICS-funded personnel tasked to act as liaisons with these MICS
teams.

ISIC teams should further strive to maintain collaborative research with
application teams as a major goal, along with their existing goal of
product development. ISIC technologies should not be developed in a
vacuum. Our greatest successes in working with the ISIC teams have come
from situations where full-blown research collaborations have developed.
We have had limited success in incorporating products that were
developed and then "thrown over the fence." Computational science
research applications are sufficiently specialized that few black-box
solutions are useful. Instead we require software that can be tuned and
tweaked for our individual needs. Collaboration should also mean that we
are writing papers with ISIC team members about the development of new
technologies and/or the application of technologies to new applications.

ISICs that produce software should be funded long-term to ensure
stability of the software and to harden this software into usable
products. Furthermore, continued funding should be tied to
"productization," maintenance, and development of documentation for the
software. Successful examples of Office-of-Science-funded software that
have been "productized" are MPICH and PETsc. Productization and
long-term support has lead to widespread adoption of these packages.
This is not true for other products that are prototyped and then left at
that stage. Also, ISIC-developed products should be "open-sourced" to
insure continued development. A great measure of success for SciDAC
ISICs would be to see ISIC-developed software fomenting robust
open-source development activity on open-source sites such as
sourceforge.

**[Case Study 1: Multidisciplinary Collaboration Bridging Astrophysics
and Nuclear Physics]{.underline}** The dynamics of a core collapse
supernova depends not only on the "macrophysics" (i.e., neutrino
transport and magnetohydrodynamics) but the "microphysics" (i.e.,
nuclear and particle physics) operative in a dying stellar core. In
particular, the entire stage for the core collapse supernova is set by
the "capture" of electrons by the nuclei in the stellar core during its
collapse. In turn, the amount of electron capture on these nuclei
depends on their "nuclear structure" (i.e., their states and how these
states are occupied by the protons and neutrons in the nucleus).
Determining this nuclear structure is in and of itself a TeraScale
application. Working closely with TSI nuclear physicists at ORNL and
their collaborators, TSI astrophysicists implemented electron capture
rates that were computed with state of the art nuclear structure theory
and demonstrated that a more realistic treatment of the stellar core
nuclear physics had a significant impact on the supernova dynamics, in
turn demonstrating the efficacy of such multidisciplinary collaboration,
the need to further push the nuclear structure frontiers, and the need
to develop experimental facilities to supplant the predictions made by
nuclear structure theory (a program of nuclear measurements at the
Spallation Neutron Source has in fact been proposed, motivated in part
by supernova science and the TSI developments).

**[Case Study 2: Multidisciplinary Collaboration Bridging Physics and
Applied Math (Preconditioners for TeraScale Linear
Systems)]{.underline}** As we discussed earlier, at the heart of any
core collapse supernova simulation is the solution of Tera- to
Peta-Scale (for future three-dimensional models) sparse linear systems
of equations. Together with Dennis Smolarski (Santa Clara University),
Ed D'Azevedo (ORNL), and Victor Eijkhout (UTK)−i.e., a team of
TSI-embedded applied mathematicians−we have developed efficient
preconditioners for both multigroup flux-limited diffusion and Boltzmann
neutrino transport. These preconditioners are being used in conjunction
with Krylov subspace algorithms for the solution of our sparse linear
systems. In the Boltzmann neutrino transport case, the "ADI"
preconditioning scheme has been implemented in spherically symmetric
supernova models in the TSI code AGILE-BOLTZTRAN. The ADI preconditioner
significantly reduced the time required for the sparse system solves,
thereby significantly reducing the run times for each simulation, which
has allowed us to explore readily the impact of new microphysics on
supernova dynamics. The spherically symmetric models are also the
fiducial models against which two- and three-dimensional models are
interpreted. Efforts are now underway to extend the ADI preconditioner
to the two- and three-dimensional Boltzmann neutrino transport cases.
"Sparse Approximate Inverse" preconditioning has been developed for
multigroup flux-limited diffusion neutrino transport and has enabled our
ongoing two-dimensional supernova simulations being performed with the
TSI code V2D. These are the first realistic two-dimensional supernova
simulations \[with realistic (all relevant terms) multigroup neutrino
transport\] to be performed. They are computationally intensive,
requiring months to complete, and are currently running on 1024+ Seaborg
processors at NERSC. Scalable algorithms for nonlinear and linear system
solution were, therefore, essential to our ability to successfully
perform these simulations.

**[Case Study 3: Multidisciplinary Collaboration Bridging Physics and
Computer Science (Data Management, Networking, and
Visualization)]{.underline}** One of the initial "discoveries through
advanced computing" by TSI was the discovery of the "SASI": a global
instability of the nacsent supernova shock wave that may impart large
asymmetries to the supernova and possibly aid in driving the explosion.
The initial discovery was made using 2D hydrodynamics simulations using
\"Eagle\" (an IBM SP3) at ORNL. Because the SASI is a low-wavenumber,
global mode, this research quickly moved to large-scale 3D simulations
as the only way of studying this phenomenon without the strong influence
of an assumed axisymmetry.

The initial 3D SASI simulations were computed on a grid of 300 million
zones using the IBM SP3 at NERSC (Seaborg). These models produced of
order 300 GB of data per run, but they were only capable of following
the early linear evolution of the SASI. The most recent 3D SASI models,
computed on the Cray X1 at the ORNL CCS, use a growing grid to
accommodate the expanding supernova shock as the SASI becomes nonlinear.
The largest simulation to date was computed on a grid with 1.5 billion
zones and produced 4 TB of data.

Analysis and visualization of these TeraScale data sets on the
supercomputing platforms on which they were generated is not practical.
Initially this was the only solution available, so we began by running
visualization software in batch mode on Seaborg. This provided a means
of generating 3D images from our simulation data, but the process was
extremely inefficient. One visualization would identify an interesting
feature of the flow. A new visualization script would then be written to
further investigate that feature. After a 24-hour turn-around, the new
visualization would reveal that the origin of that feature was in a
different part of the flow not visible in the current rendering. The
lack of interactivity provided by this method effectively stifled the
process of scientific discovery.

Moving such large data sets from NERSC or CCS to NC State University for
analysis and visualization on a local linux cluster was clearly the only
viable solution for interactive analysis of these TeraScale simulations,
but data movement quickly became the rate-limiting step. Micah Beck\'s
Logistical Computing and Internetworking group provided a simple,
ready-to-use solution. After a quick and relatively easy installation of
their LoRS tools, we were able to move data to and from the Logistical
Network at 10 to 20 times the rate at which we had been moving data
between CCS and NC State.

Implementing the LoRS tools enabled us to interactively visualize our 3D
data for the first time. We quickly discovered that our first generation
of 3D models suffered from a numerical problem (as is often the case) at
the surface of the accreting proto-neutron star. Without the ability to
probe the full 3D data, we would not have found this problem. With
minimal effort we were quickly on to a next generation of models with
much improved fidelity.

In hindsight the LoRS software was much more valuable than a simple file
transfer tool. Given the large, collaborative nature of TSI, and in
particular the direct involvement of several research groups in
scientific (SASI) visualization \[Toedte (ORNL), Ma (UCD), McCormick
(LANL), and Huang (UTK)\], we found that the Logistical Network was an
ideal way to temporarily store and share our TeraScale datasets with
researchers around the country.

The importance of working with computer scientists to implement these
data management, networking, and visualization tools was exemplified by
the TSI discovery of strong rotational flow generated by the SASI in
three-dimensional simulations, which has ramifications for the spin up
of the central compact object at the center of the explosion, known as
the proto-neutron star. How neutron stars are born spinning, and
equivalently how pulsars are formed, is a question associated with the
question of how core collapse supernovae occur, TSI's primary scientific
question. When these 3D simulations were first moved to the Cray X1 at
the ORNL CCS, this machine was not yet in production mode and we did not
have an efficient method for moving data off the machine. Instead, we
forged ahead with the largest 3D SASI simulations performed to date, but
kept only global diagnostics computed on the fly. From this first
large-scale 3D run we saw that the nonlinear evolution of the SASI led
to a very large deposition of angular momentum on the proto-neutron
star. But without access to the full data, we could not identify the
physical mechanism for this exciting result. We knew there was valuable
science to be learned from these simulations, but without the tools for
discovery, we were left with idle speculation.

Working with Nagi Rao and Steven Carter at ORNL, we implemented an
existing (but not well known) file transfer tool, bbcp, to move data
efficiently from the Cray X1 to the TSI data cluster at NC State
University. With the 3 TB of data on a local, dedicated cluster we were
able to visualize these results and study the transition of the SASI
into a non-axisymmetric mode that generated the strong rotational flow
recorded in earlier runs.

Scientific visualization is an end goal of the significant data
management and networking efforts we have just documented. These efforts
have certainly been matched by efforts in visualization. TSI-embedded
researchers Bachta, Baker, and Toedte have been instrumental to TSI's
scientific discovery and the dissemination of TSI's scientific results
to the science community, the DOE, and the public. This required close
and extensive interaction between them and TSI scientists, over
countless hours. As a result, new representations for TSI's
multidimensional, complex data and, generally, new and illuminating
renderings of the complex fluid flow in the stellar core after collapse
and during explosion and of the multifrequency neutrino radiation field,
in both spatially two- and three-dimensional simulations, were
developed.

One of TSI's visualization researchers (Toedte) has played a central
role in the development of the significant visualization infrastructure
at ORNL and in its use for TSI science. Moreover, TSI science−in
particular, TSI simulation data−has helped drive this infrastructure
development, which is now benefiting a number of application areas. The
ORNL visualization infrastructure developed during the SciDAC program
includes a Powerwall, a visualization cluster driving the Powerwall, and
the availability of a visualization software suite that includes
EnSight.

Ultimately, to enable scientific discovery, we must establish an
end-to-end computational science infrastructure to manage the scientific
workflow, from simulation data generation and storage, to data movement,
to data analysis, and ultimately to scientific visualization. A recent
task undertaken by TSI researcher K. DeNisco (SUNYSB) has been the
connection of the SPA (Scientific Process Automation) workflow designed
by X. Xin and T. Critchlow of the SDM ISIC. Previous work in
collaboration between Swesty, Myra, Xin, and Critchlow had resulted in
the development of a process automation workflow that handled management
of batch jobs. DeNisco has assumed the task of integrating this workflow
with the SUNYSB operating environment. The workflow is currently running
from group workstations at Stony Brook and can manage the execution of
batchjobs on Seaborg at NERSC and the transfer of data to HPSS and to
the SUNYSB RAID systems. DeNisco is currently working on extending the
workflow to automatically post-process the data when it arrives at Stony
Brook. In short, these developments have greatly facilitated the
scientific workflow for the SUNYSB TSI group, and efforts are underway
to now extend these capabilities to other TSI sites. In parallel with
this effort, the Logistical Networking team continued to work actively
with TSI to improve the NCSU-ORNL scientific workflow, from customizing
scripts to move data directly between archival storage and the
Logistical Network, to implementing a netCDF interface \[in
collaboration with Rob Ross (ANL) of the SDM ISIC\] now allowing us to
move files into and out of the Logistical Network *directly* from our
simulation code and visualization software *without* significant new
code development.

**For further information, contact:**

Dr. Anthony Mezzacappa

Phone: 865-574-6113

<mezzacappaa@ornl.gov>