**Scientific Discovery through\
Advanced Computing:\
Progress and Opportunities**

**Input from the Laboratory Community to the\
Office of Advanced Scientific Computing Research\
Office of Science\
U.S. Department of Energy**

**September 2005**

**SciDAC: Progress and Opportunities**

Table of Contents

[Executive Summary 1](#executive-summary)

[1. Accelerating Scientific Discovery through Advanced Computing
3](#accelerating-scientific-discovery-through-advanced-computing)

[1.1 Scientific Discovery through Simulation
3](#scientific-discovery-through-simulation)

[Accelerator Physics 3](#accelerator-physics)

[Astrophysics 4](#astrophysics)

[Climate Modeling 5](#climate-modeling)

[Combustion 6](#combustion)

[Fusion 7](#fusion)

[Quantum Chromodynamics 8](#quantum-chromodynamics)

[1.2 Enabling Science with Data 10](#enabling-science-with-data)

[Particle Physics 10](#particle-physics)

[Climate 10](#climate)

[Fusion 11](#fusion-1)

[1.3 Validating New Approaches 11](#validating-new-approaches)

[Chemistry 11](#chemistry)

[Combustion 11](#combustion-1)

[2. The Landscape of Scientific Discovery through Advanced Computing
13](#the-landscape-of-scientific-discovery-through-advanced-computing)

[2.1 Leadership in Computing Infrastructure
13](#leadership-in-computing-infrastructure)

[Computing Platforms 13](#computing-platforms)

[Networking Infrastructure 15](#networking-infrastructure)

[2.2 Science-Driven Enabling Technologies
15](#science-driven-enabling-technologies)

[2.3 Computational Science Outreach 15](#computational-science-outreach)

[2.4 SciDAC and Scientific Computing within Office of Science
16](#scidac-and-scientific-computing-within-office-of-science)

[3. Future Opportunities 17](#future-opportunities)

[3.1 Accelerator Physics 17](#accelerator-physics-1)

[3.2 Astrophysics 18](#astrophysics-1)

[3.3 Biology 19](#biology)

[3.4 Chemistry and Materials Science
20](#chemistry-and-materials-science)

[3.5 Climate 21](#climate-1)

[3.6 Combustion 22](#combustion-2)

[3.7 High Energy and Nuclear Physics with Colliding Beams
23](#high-energy-and-nuclear-physics-with-colliding-beams)

[3.8 Fusion Energy 24](#fusion-energy)

[3.9 Quantum Chromodynamics 25](#quantum-chromodynamics-1)

[4. Advancing Science through Enabling Computational Technologies
26](#advancing-science-through-enabling-computational-technologies)

[4.1 Cross-Cutting Opportunities to Advance Simulation Sciences
28](#cross-cutting-opportunities-to-advance-simulation-sciences)

[4.2 Cross-Cutting Opportunities to Link to Experimental Sciences
29](#cross-cutting-opportunities-to-link-to-experimental-sciences)

[4.3 Integrated Software Infrastructure Centers (ISICs)
30](#integrated-software-infrastructure-centers-isics)

[Building on the Successful ISIC Interaction Model
31](#building-on-the-successful-isic-interaction-model)

[4.4 Scientific Application Partnerships (SAPs)
32](#scientific-application-partnerships-saps)

[Building on the Successful SAP Interaction Model
33](#building-on-the-successful-sap-interaction-model)

[4.5 Software Integration, Maintenance, and Support (SIMS)
34](#software-integration-maintenance-and-support-sims)

[Components of SIMS 34](#components-of-sims)

[Selecting and Evaluating SIMS 35](#selecting-and-evaluating-sims)

[4.6 Meeting Future Application Needs: Topics of Emphasis
35](#meeting-future-application-needs-topics-of-emphasis)

[5. Engaging the Broader Computational Science Community
38](#engaging-the-broader-computational-science-community)

[5.1 Benefits of SciDAC Institutes 38](#benefits-of-scidac-institutes)

[5.2 The SciDAC Institutes Model 39](#the-scidac-institutes-model)

[6. SciDAC-2 Requirements for Computing and Networking Facilities
41](#scidac-2-requirements-for-computing-and-networking-facilities)

[6.1 Computing Requirements 41](#computing-requirements)

[6.2 Networking Requirements 43](#networking-requirements)

[7. Budgets 44](#budgets)

[8. Relationships with Other Agencies and Programs
45](#relationships-with-other-agencies-and-programs)

[8.1 Interactions with Other Agencies
45](#interactions-with-other-agencies)

[8.2 Relationship to Other DOE Programs
45](#relationship-to-other-doe-programs)

[References 47](#references)

[Participants 48](#participants)

[Appendix: SciDAC-2 Planning Workshop Reports
51](#appendix-scidac-2-planning-workshop-reports)

[SciDAC Software Integration, Maintenance, and Support (SIMS)
52](#scidac-software-integration-maintenance-and-support-sims)

[SciDAC-2 Integrated Software Infrastructure Centers (ISICs)
68](#scidac-2-integrated-software-infrastructure-centers-isics)

[Computational Science Institute 75](#computational-science-institute)

[Accelerating Scientific Discovery in Experimental Science via Advanced
Computing
84](#accelerating-scientific-discovery-in-experimental-science-via-advanced-computing)

[SciDAC Computer Science Institute
104](#scidac-computer-science-institute)

[Role of SciDAC-2 in Predictive DOE Mission Driven Systems Biology
117](#role-of-scidac-2-in-predictive-doe-mission-driven-systems-biology)

# Executive Summary

The nation's scientific enterprise is in the midst of two fundamental
transformations, one enabled by the increasing application of
computational simulation to long-standing challenge problems, the other
driven by the rapid adoption of advanced infrastructure for the capture,
storage, transmission, sharing, and analysis of large-scale experimental
data. Together, these two technologies are dramatically improving our
nation's ability to solve important engineering problems and to make
critical discoveries in many scientific domains that in the past have
been considered intractable because of their extreme complexity,
multidisciplinary nature, or lack of available data analysis capability.

During the past five years, the U.S. Department of Energy's SciDAC
program (SciDAC-1) has demonstrated that important scientific
accomplishments are possible through simulation and modeling with the
focused collaboration and active partnership of domain scientists,
applied mathematicians, and computer scientists. Successes have been
documented in such areas as accelerator design, chemistry, combustion,
climate modeling, and fusion. SciDAC activities have also demonstrated
that large-scale simulation offers some of the most cost-effective
opportunities for answering a number of scientific questions in such
areas as the fundamental structure of matter, the production of heavy
elements in supernovae, and the functions of enzymes.

In spite of tremendous progress, however, much remains to be done, and
even more compelling opportunities for scientific discovery lie ahead.
In FY 2004, DOE's Office of Science launched an aggressive program to
develop and deploy leadership-class computing facilities and announced a
20-year scientific facilities roadmap that will provide a rich
scientific infrastructure for the next two decades. Each of the
Department of Energy's Office of Science (SC) programs---Basic Energy
Sciences, Biological and Environment Research, Fusion Energy Sciences,
High-Energy Physics, and Nuclear Physics---has identified a crucial need
for continued and sustained advances in scientific computing. The
creation of additional SciDAC-like scientific application partnerships
is necessary for the fulfillment of these program's missions.

To address these needs, we recommend that the SC's Office of Advanced
Scientific Computing Research (ASCR) make significant investments for FY
2007 to establish a SciDAC-2 program. SciDAC-2 can build on the success
of the first five years of the program (SciDAC-1) and extend the program
in three important ways:

-   **Enrich the scientific application partnerships.** The major source
    of acceleration in simulation-based science has been the strength
    and depth of partnerships among application domains, computer
    science, and applied mathematics. ***SciDAC-2 should improve and
    strengthen the scientific application partnership activities that
    formed the core of the successful SciDAC-1 program.***

-   **Engage experimental science.** The use of advanced computing
    technologies to accelerate scientific discovery is not limited to
    modeling and simulation; it can also be used to improve experimental
    science. New data management and analysis tools are required for
    both small and large-scale experiments such as those that will be
    carried out in the current and future facilities outlined in the SC
    20-year roadmap. ***SciDAC-2 should address this opportunity of
    improving productivity and capability of experimental science
    through the development and application of advanced data and
    analysis capabilities, computation in support of experiment, and
    technologies for the automation of experiments.***

-   **Empower new scientific communities.** Some areas of science do not
    yet fully benefit from large-scale computing. Some communities are
    not fully aware of successes and methods from other scientific
    areas; in other cases methods, tools, or conceptual approaches are
    relatively immature or inadequately developed. ***SciDAC-2 should
    address this opportunity through improved outreach activities
    including the creation of SciDAC Institutes aimed at broad inclusion
    of new communities and improved workforce development to ensure a
    robust supply of next-generation researchers.***

These SciDAC-2 investments should be supported by computational science
and experimental science application investments from the DOE programs
in BES, BER, FES, HEP, and NP and by additional ASCR infrastructure
investments in leadership-class computing facilities, the National
Energy Research Scientific Computing Center, and the Energy Sciences
Network. The resulting research portfolio will accelerate research
progress in advanced energy systems (e.g., solar, hydrogen, fusion),
biotechnology (e.g., genomics, cellular dynamics), nanotechnology (e.g.,
sensors, storage devices), and environmental modeling (e.g., climate
prediction, pollution remediation).

# 1. Accelerating Scientific Discovery through Advanced Computing

Over the five years of the SciDAC program, researchers within DOE's
Office of Science have achieved key scientific insights in a number of
areas of national importance, including fusion, combustion, climate
modeling, high energy and nuclear physics, and astrophysics. These
advances have been accomplished through the development of
state-of-the-art simulation codes that run on DOE supercomputers. The
results of these simulations, together with associated theory and
experiment, help ensure that the United States maintains a leadership
role in science and technology.

The key to the success of the SciDAC program has been the power of
multidisciplinary teams that bring together experts in the scientific
discipline, computer science, and applied mathematics. Multidisciplinary
teams have achieved progress that could not have been made in any other
way. It is increasingly hard for a small team of experts in a single
area to develop a state-of-the-art simulation code that uses the latest
mathematical algorithms and runs effectively on today's complex computer
architectures. Today, simulation science is a team sport. SciDAC teams
have pushed the forefront of science to the limits of the capability
computing resources available to SC researchers. They have also built
unique new infrastructure in applied mathematics, computer science, and
distributed computing technology that has enabled scientific progress
across a range of disciplines and that will lead to further advances in
the coming years.

We now describe some of the scientific advances for which the SciDAC
program is responsible. They are grouped into three categories. In
*Scientific Discovery through Simulation,* we show examples of
computational models used to solve problems central to the DOE science
mission. In *Enabling Science with Data,* we describe how the data
management and computing tools are used to disseminate and analyze large
data sets obtained from large experimental facilities or high-end
simulation or to provide remote access to experimental facilities. In
*Validating New Approaches,* we discuss examples of new algorithms and
software that developed under SciDAC to the point that we can see their
potential for solving important scientific problems.

## 1.1 Scientific Discovery through Simulation 

### Accelerator Physics

The long-term future of experimental high-energy physics research using
accelerators depends on the successful development of novel ultra
high-gradient acceleration methods. New acceleration techniques using
lasers and plasmas have already been shown to exhibit gradients and
focusing forces more than 1000 times greater than conventional
technology. The challenge is to control these high-gradient systems and
then to string them together. Such technologies would enable the
development of ultra-compact accelerators. The potential impact on
science, industry, and medicine of placing such compact accelerators in
research organizations, high-tech businesses, and hospitals is
staggering. In plasma-wakefield accelerators a drive beam, either an
intense particle beam or laser pulse, is sent through a uniform plasma.
The space charge or radiation pressure creates a space-charge wake on
which a trailing beam of particles can surf.

SciDAC investigators in the Accelerator Science and Technology (AST)
project have developed a suite of particle-in-cell (PIC) codes to model
such devices. These were used to model the E-164x particle beam
accelerator experiments at SLAC and the laser-plasma experiments at the
L'OASIS lab at LBNL. In the E-164x experiment, 3-4 GeV energy gain was
observed in \<10cm while in the L'OASIS experiments, \~100 MeV
monoenergetic electron beams were observed. Electromagnetic PIC
simulations predicted key physics that was later observed in
experimental results published in *Nature* (Figure 1).
![](media/image1.png){width="2.463888888888889in"
height="3.285416666666667in"}

In the area of RF accelerators, the AST project, in collaboration with
the SciDAC-funded mathematicians and computer scientists, have been
greatly extending the electromagnetic simulation capabilities available
to this community, and applying them to a variety of current problems in
accelerator design. For example, an international team comprising DESY,
KEK, FNAL, CEBAF, and SLAC is collaborating on the development of a
low-loss (LL) accelerating cavity for the International Linear Collider,
the highest priority future accelerator project in high energy physics.
SLAC has applied the new electromagnetic modeling tools to optimize the
LL design's performance, improve its efficiency, and ensure the
stability of beam transport in a linac using such cavities.

### Astrophysics

**Core Collapse Supernovae.** Simulations of core-collapse supernovae,
enable under SciDAC, have been systematically exploring the role of
different physics component---neutrino transport, turbulent fluid flow,
rotation, magnetic fields, gravity, weak interactions, and equations of
state on the explosion. A series of 3D hydrodynamics simulations of the
Stationary Accretion Shock Instability (SASI) (Figure 2) has led to the
discovery of a vigorous nonaxisymmetric mode that is capable of
generating substantial angular momentum in the postshock flow. This new
ingredient of core-collapse supernovae, which can be modeled only in
full three-dimensional simulations, may have a dramatic impact on both
the supernova mechanism itself and on the neutron star left behind by
the supernova event. These discoveries were enabled by breakthroughs in
networking developed under SciDAC that made it possible to move
terabytes of data between the NCCS, where the simulations were
performed, and NCSU for analysis and
visualization.![](media/image2.jpeg){width="2.7527777777777778in"
height="1.7104166666666667in"}

**Type Ia Supernovae**. One of the goals of the SciDAC Supernova Science
Center is to understand the dynamics of nuclear burning in stars and
ultimately to derive possible mechanisms for transition to detonation in
type Ia supernovae. The initial focus of this effort has been on
detailed quantification of the microscale physics of nuclear flames.
Some of the significant scientific results to date are the elimination
of classical flame instabilities (such as Landau-Darrieus) as
significant sources of combustion enhancement, and the quantification of
the role of buoyant instabilities and flame-generated turbulence as a
mechanism for enhanced combustion (Figure 3). The starting point of this
work was the adaptive mesh refinement algorithms and software for
low-Mach number flows developed under SciDAC with modifications to parts
of the algorithm were required to accommodate the substantially
different thermodynamics and transport properties of nuclear matter in
stars. The use of the low-Mach number models, AMR, and high-end parallel
computation increased by five orders of magnitude the rate at which
simulation results could be obtained over the use of more traditional
methods.

![](media/image3.jpeg){width="6.023611111111111in"
height="2.457638888888889in"}

### Climate Modeling

Because of the complex feedbacks within the Earth system, precisely
estimating the magnitude and rate of global warming, or understanding
its effects on other aspects of climate, such as precipitation, is a
daunting scientific challenge. Thanks to a wealth of new observational
data and advances in computing technology, current climate models are
able to reproduce the global average temperature trends observed over
the 20th century and provide evidence of the effect of human activity on
today's climate. Our understanding of global climate change and ability
to predict climate over decades to centuries will enable us to develop
science-based solutions to reduce and minimize the impacts of climate
change and help us better plan for our nation's future energy needs.

SciDAC-1 projects have achieved major advances in both simulation and
distributed data access capabilities in climate, as the following
examples show.

**Linking ocean circulation with dynamic ecosystems.** An important
climate feedback mechanism can be investigated with the addition of
carbon cycle, atmospheric chemistry, and ocean biogeochemistry in the
coupled climate system model. In the ocean code POP2, an ecosystem model
simulates phytoplankton growth and sea color from chlorophyll
concentration (Figure 4). The simulated ocean ecosystem also produces
trace gases such as dimethyl sulfide that are important in the formation
of atmospheric sulfate aerosols.
![](media/image4.png){width="2.5229166666666667in"
height="2.4895833333333335in"}

**Dynamic vegetation and carbon cycle dynamics.** The CLM3 land model
has been extended with both the CASA and the CN interactive carbon cycle
with links to the atmosphere through the new coupler. The net primary
production from the CASA model demonstrates that the carbon cycle can be
reasonably simulated within a global climate model. These new
capabilities are being used to examine the effect of nitrogen deposition
on land biogeochemistry.

SciDAC funding has enabled DOE researchers to participate with NSF and
NASA researchers over an extended period in the design and
implementation of the next-generation Community Climate System Model
(CCSM). Two major software challenges were addressed by the CCSM
Consortium Project: performance portability of the model among high-end
computer architectures, and structured code design for improved
extensibility. The product of that effort is documented in a special
dedicated issue of the *International Journal of High Performance
Computing Applications*. The CCSM3 model is a major contributor to the
current international climate change assessment, with over 6,400 years
of simulation performed on three different high-end computer
architectures.

### Combustion

In many practical combustors the fuel and air are not premixed. For
these systems, rapid mixing of the reactants is desirable to maximize
heat release rates, thus enabling smaller combustion chamber volumes,
and minimizing the production of certain pollutants. Above a critical
value, however, enhanced mixing can lead to local extinction or even
destabilization of the entire flame, adversely affecting efficiency,
pollutant generation and safety. Accurate engineering models for
nonpremixed combustion require models for this mixing process. Current
models are based on the simplifying assumption that mixing of all scalar
fields occurs on the same time scale as the turbulence. Experimental
measurement of local mixing in flames is extremely difficult and even
pointwise measurements have only recently become possible. High-fidelity
numerical simulations known as direct numerical simulation (DNS),
enabled by SciDAC, have made it possible to quantify mixing time scales
numerically. SciDAC researchers have performed the world's largest
three-dimensional DNS of turbulent nonpremixed combustion with detailed
CO/H~2~/air chemistry to study the interplay of mixing and reaction,
particularly when strong finite-rate chemistry effects are involved.
Figure 5 presents an instantaneous snapshot from the DNS showing a
comparison between passive and reactive scalar mixing as depicted by the
dissipation rates of the mixture fraction, hydroperoxy radical, and
hydroxyl radical. These simulations have demonstrated a range of mixing
timescales varying by as much as a factor of three and are a providing
the necessary quantitative data needed to construct high-fidelity mixing
models for engineering simulations.

The DNS was performed by using a state-of-the art DNS solver, S3D,
developed at Sandia National Laboratories. SciDAC support has enabled
S3D to be ported and optimized on the NERSC IBM SP and the ORNL CrayX1
and CrayXT3 platforms. More recently, the DOE SciDAC project
"High-Fidelity Terascale Simulations of Turbulent Combustion" has
extended S3D to include multi-physics modules for spray, soot, and
optically thick radiation.
![](media/image5.png){width="3.911111111111111in"
height="3.502083333333333in"}

### Fusion

Sustained fusion power generation requires that we understand the
detailed physics of the fluid of ions and electrons, or plasma, in a
fusion reactor. Computational techniques for simulating the growth and
saturation of turbulent instabilities in this plasma are critical for
developing this understanding of how to control the plasma and hence to
develop a successful fusion reactor. Researchers have used the most
powerful DOE leadership class computing capabilities to perform the
highest fidelity simulations to date of the flow of charged particles in
fusion plasmas. These simulations have led to key insights in the
understanding of fusion reactors by accurately describing what is
happening to particles in the plasma as turbulence occurs.
Accomplishments of this type contribute to the fuller understanding of
fusion plasmas leading to a successful fusion reactor design.

A principal focus of the much of the research on burning plasmas is
driven by the International Thermonuclear Experimental Reactor (ITER).
Pellet injection is currently seen as the most likely refueling
technique for the ITER reactor. In the past, progress in the development
of an efficient refueling strategy for ITER has required time-consuming
and expensive experiments. SciDAC computer codes have now qualitatively
reproduced these experimental results (Figure 6), resulting in
significant progress toward the scientific goal of achieving a
simulation-based predictive capability for pellet injection in
tokomaks.![](media/image6.jpeg){width="6.0159722222222225in"
height="1.5034722222222223in"}![](media/image7.jpeg){width="5.9875in"
height="1.4965277777777777in"}

### Quantum Chromodynamics

The long-term goal of nuclear and high energy physicists is to identify
the fundamental building blocks of matter and to determine the
interactions among them that lead to the physical world we observe.
Remarkable progress has been made through the development of the
Standard Model of nuclear and high energy physics, which provides
fundamental theories of the strong, electromagnetic and weak
interactions. Our knowledge of the Standard Model is incomplete,
however, because it has proven difficult to extract many of the most
interesting predictions of quantum chromodynamics (QCD), the component
of the Standard Model that describes the strong interactions. The
technical solution has been to solve QCD on a space-time lattice. The
enormous scale of resources required has resulted in lattice QCD being
recognized as one of the grand challenges of computational science.

With the startup of SciDAC, a national collaboration was formed
involving almost every lattice QCD scientist in the United States. Two
examples of the collaboration's recent SciDAC-enabled results are shown
in Figures 7 and 8, where clear agreements between lattice QCD
calculations and experimental results are seen, thereby demonstrating
the scientific understanding of the underlying physics involved:

![](media/image8.jpeg){width="3.9347222222222222in"
height="3.0131944444444443in"}

**Figure 7.** The semileptonic form factor for the decay of a D meson
into a K meson, a lepton, and a neutrino, as a function of the momentum
transferred to the leptons. The orange curve is the lattice QCD
prediction by the Fermilab lattice and MILC Collaborations \[Phys. Rev.
Lett. 94, 011601 (2005)\], and the blue points are the experimental
results of the Focus Collaboration \[Phys. Lett. B607, 233 (2005)\],
which confirmed the prediction.

![](media/image9.jpeg){width="4.058333333333334in"
height="3.1319444444444446in"}

**Figure 8.** As the pion mass used in the lattice QCD calculation by
the Lattice Hadron Physics Collaboration approaches its physical value
the result approaches the experimental value as determined by a Chiral
extrapolation \[Thomas et al., PRL86, 5011 (2001)\].

The collaboration used the SciDAC model of involving scientists,
computational scientists and computer scientists successfully meet three
challenges:

-   Created a portable software environment that improved the science
    delivered for a given level of computational resources by more than
    a factor of two, significantly reduced the time required by a
    lattice QCD theorist to set up a given problem, and dramatically
    improved the portability of lattice QCD code from one hardware
    environment to another.

-   Developed commodity based hardware and low-level software
    computational systems that by 2005 achieved the price performance of
    \$1/megaflops sustained by the application.

-   Integrated the portable software environment and hardware systems
    with problem specific lattice QCD codes to achieve significant
    scientific results.

## 1.2 Enabling Science with Data

### Particle Physics

The Particle Physics Data Grid project has transformed the practice of
scientific data management and analysis for several major U.S. physics
experiments, including D0 at Fermilab, BaBar at the Stanford Linear
Accelerator Center (SLAC), and the STAR nuclear physics experiment at
Brookhaven National Laboratory (BNL). The reliable, automated transfer
of terabytes of data allows "next day" analysis and enabled the
publication of the first results of a recent d+Au run only four months
after the run. A factor of 50 improvement has been made in file
discrepancies (now \<0.02% error rate). The Open Science Grid
infrastructure created by this work has also become a general-purpose
source of capacity computing for researchers from outside physics.

The U.S. physics community has been working, with SciDAC support, to
create a distributed computing infrastructure capable of distributing
and processing the petabytes of data to be produced by the Large Hadron
Collider (LHC) experiments. In a significant milestone, two major LHC
experiments, ATLAS and CMS, demonstrated in 2004 their ability to
distribute, reconstruct, and analyze data at 25% of the rate that will
be required at the start of 2007. Key to this success was their ability
to harness computing and storage resources across U.S. laboratory and
university sites.

### Climate

The Earth System Grid (ESG) has addressed the challenges of enabling
management, discovery, access, and analysis of the important datasets
that represent global climate simulations. ESG has two secure portals to
make the simulation output from the models available to hundreds of
researchers worldwide. The National Center for Atmospheric Research has
one portal dedicated solely to disseminating the nearly 100 TB of
simulations from the latest version of the Community Climate System
Model (CCSM) under various emission scenarios. At Lawrence Livermore
National Laboratory, the Program for Climate Model Diagnosis and
Intercomparison is serving a subset of the CCSM data, as well as output
from 22 other models, to scientists participating in the
Intergovernmental Panel for Climate Change (IPCC) Working Group I, which
focuses on the scientific basis of climate change. This portal has been
serving data to nearly 400 projects in the IPCC community for over 10
months, and the average data movement is about 300 gigabytes per day.
The IPCC archive contains over 30 TB, from which over 200 research
papers were written in the first three months of the archive
availability.

### Fusion

The National Fusion Collaboratory's remote collaboration services allow
U.S. scientists to conduct experiments at the JET European fusion
experiment, the largest operating tokamak in the world, without
traveling. Access Grid technology provides multiple video images and a
unified audio stream between the DIII-D National Fusion Facility in San
Diego and the JET control room, and data can be analyzed as soon as it
is produced. This successful prototype validates the concept of a
collaborative control room for the next-generation burning plasma
experiments, including ITER.

## 1.3 Validating New Approaches

### Chemistry

One of the principal challenges in computational chemistry is the
reduction of the power law scaling of current quantum chemistry
algorithms for systems with large numbers of atoms and electrons, that
is, alternative approaches to handling the electron correlation problem
for many electron systems. A collaboration of SciDAC-funded chemists and
mathematicians has developed multiresolution and novel separated
representations that have yielded fast, accurate, and reliable
prediction of molecular electronic structure free of basis set error.
Most exciting are prototypes for computing in six and higher dimensions
from which highly accurate many-body wavefunctions can be constructed
(Figure 9).![](media/image10.jpeg){width="3.1034722222222224in"
height="2.3201388888888888in"}

### Combustion

SciDAC researchers have been developing a new class of simulation
capabilities for reacting flows and applying them to obtain new
understanding of problems in combustion of both fundamental and
practical importance. Based on low-Mach number asymptotics and adaptive
mesh refinement, these methods represent a significant departure from
the traditional approaches used for detailed simulation of reacting
fluid flows. To validate this approach, the researchers have used these
codes to simulate combustion for a variety of problems for which there
is good experimental data but for which there are open questions that
computation could conceivably answer. An example is the combustion of an
ammoniated hydrocarbon fuel to understand how the process by which
fuel-bound nitrogen in biomass fuels is converted into pollutants such
as NO~x~. The simulations illuminated the principal chemical kinetic
mechanisms for the formation of NO~x~, improved the standard kinetics
models for representing that process, and identified the hydrodynamic
mechanism for the observed large sensitivity of the integrated NO~x~
production to the inputs. ![](media/image11.jpeg){width="2.5in"
height="3.2958333333333334in"}

In the area of turbulent flame simulations, the new code capabilities
have made it possible to simulate laboratory-scale premixed turbulent
flames without engineering models, thus enabling direct comparison with
and augmentation of laboratory experiments (Figure 10). Detailed
comparison of the calculations with laboratory experiments show
excellent agreement with available data, demonstrating that the new
approach is able to accurately capture turbulent flame dynamics using
high-fidelity models for chemical kinetics and transport. This leads to
a new approach to studying laboratory-scale combustion by using
first-principles fluids models directly to simulate such flows. In
contrast, the more traditional approach uses first-principles
simulations to obtain parameterizations for use in heuristic engineering
models, which introduces a layer of indirection and uncertainty.

# 2. The Landscape of Scientific Discovery through Advanced Computing

Clearly, SciDAC-1 has enabled exciting scientific advances by the
application and combination of advanced modeling, theory, experiment,
and computation. In short, SciDAC-1 has arguably changed the landscape
of advanced scientific computing. And the field continues to change at a
rapid pace. New advances in mathematics, algorithms, computer science,
and an ever-changing array of new computer architectures make the field
of computational science one of continuing challenges.

A key lesson learned from SciDAC-1 is that the complex environment in
which computational science is performed requires a multidisciplinary
approach. Computing facilities, in conjunction with enabling
computational technologies, provide the needed resources to drive
innovation and advance science. The resulting software infrastructure
developed within the Office of Science programs and SciDAC-1 also
enabled the various programs to carry out their specific scientific
missions.

In the sections that follow, we highlight the key features of the new
landscape surrounding scientific discovery through advanced computing.
One of the critical components is the infrastructure of computing and
networking resources. Another equally important component is the
enabling computational technologies, both in applied mathematics and in
computer science. Moreover, there is an increasing need for outreach
activities that provide a forum for addressing multidisciplinary
scientific problems. By responding to these challenges and
opportunities, a SciDAC-2 program can move scientific discovery and
advanced computing to the next level.

## 2.1 Leadership in Computing Infrastructure

Sustained, year-after-year exponential growth in computing, storage, and
networking capabilities has resulted in an ongoing revolution in
information technology. This trend, along with the nation's critical
needs for scientific discovery and engineering innovation, drives the
advance of ultra-scale computing and information processing
capabilities.

### Computing Platforms

For the past 40 years, the DOE has been a world leader in using
supercomputers to advance scientific research. Today, the computing
landscape in the DOE Office of Science includes a comprehensive range of
different classes of facilities for meeting the diverse computing needs
of the research community:

-   A facility providing computing resources tailored for specific
    scientific applications and serving as a focal point for an
    application community to ensure the leadership of the United States
    in critical areas of science. In 2004, DOE established a Leadership
    Computing Facility (LCF) as a partnership between ORNL, ANL, and
    PNNL for open science. The LCF is the realization of the UltraScale
    Scientific Computing Capability, which was identified as the second
    priority in DOE's Office of Science's "Facilities for the Future of
    Science" report. In this plan, this facility will increase by a
    factor of 100 the computing capability available for open,
    unclassified scientific research. The National Center for
    Computational Sciences is the primary facility in this role. The
    focus of this facility is on supplying capability computing to
    unclassified science research. This shift in landscape toward a
    leadership-class computer architecture will have a profound impact
    on the kind of research that will be possible in SciDAC-2.

-   A facility providing high-end computing resources for all of the
    Office of Science with a mixture of capacity and capability
    computing. The National Energy Research Scientific Computing Center
    (NERSC) is the primary facility in this role, with other DOE labs
    also supplying some capacity computing to the DOE science mission.
    In addition, a large number of Linux clusters in the DOE labs also
    handle capacity computing requirements. By serving the needs of the
    many users with a requirement for capacity computing, these
    facilities help to maintain the strong science programs within the
    Office of Science.

-   Specialized application computing facilities offering specialized
    hardware to be exploited by unique features of the application. An
    example of such a system is the QCDOC machine (BNL). A key metric in
    the deployment of these systems is cost-effectiveness.

-   A facility (or facilities) for assessing the promise of experimental
    and new computing technologies for scientific applications. It is
    expected that the function of experimental computing facilities
    currently supported within the Advanced Computing Research Testbed
    (ACRT) at ORNL will be realized within ASCR's Research and
    Evaluation (R&E) program in FY06.

The size and type of systems that will become available over the next
five years will differ dramatically from what the scientists utilized
during SciDAC-1. With the Office of Science's push for leadership
computing, systems at the facilities will grow beyond 100 Tflop/s during
SciDAC-2 and will be approaching petascale performance by 2010, with
systems consisting of 20,000 to 100,000 processors. Thus, codes
currently running on thousands of processors need to scale to tens of
thousands or hundreds of thousands of processors. Many additional
science codes will need to make the jump from hundreds of processors to
thousands of processors.

During the same period, specialized computing platforms---both custom
machines and application-specific commodity configurations---are
expected to remain important for lattice QCD calculations, as well as
other applications. For such specialized computing, SciDAC-2 must
continue to advance specialized hardware solutions and to invest in
robust and portable software environments able to take advantage of
these systems.

In order to ensure that computing platforms are well suited for the
complex applications anticipated during the next decade, SciDAC-2 should
consider developing partnerships with computer vendors. Such
partnerships would go beyond mere evaluation of planned platforms and
would involve the following:

-   Teams of scientists and computational mathematicians who would
    modify and optimize their applications for future systems through
    the use of performance modeling, simulators, and prototypes of new
    hardware

-   Teams of computer architects from major U.S. computer vendors who
    would interact directly with the scientific applications teams

-   Teams of computer scientists who would work with both applications
    scientists and computer architects to analyze and abstract the
    requirements of scientific applications so that they can be
    addressed in hardware design and to develop the software
    environments that will allow scientists to extract the maximum
    performance and capability from that hardware

### Networking Infrastructure

As part of its mission, DOE has also taken a leadership role in
providing advanced networking. For 20 years, ESnet has been widely used
by application scientists, and the establishment of the specialized and
ultra-high-scale computing platforms discussed above is expected to
place increasing demands on wide-area networks. SciDAC-2 will need to
address significant technical issues that include packet switching,
routing, and jitter; high-speed transport protocols; and quality of
service.

## 2.2 Science-Driven Enabling Technologies

Enabling computational technologies aid the application developers in
realizing the highest possible performance on all computer systems. The
following are examples of successful enabling technologies developed
under SciDAC-1:

-   Key tools that allowed the computational astrophysics project to
    visualize phenomena never seen before

-   Software integration libraries that allowed the computational
    chemistry project to solve problems that had previously been
    considered impossible to do because of their complexity

-   New standards for system software integration that are being used in
    production at several DOE labs and many universities

-   New math libraries that not only have significantly improved the
    time to solution of several application areas but also have enabled
    new science not previously possible

In SciDAC-2, the role of the enabling computational technologies will
become even more important to the science applications as computing
systems grow by two orders of magnitude and their architectures become
more diverse. In addition, new tools need to be developed that will
focus on the information and visualization analytics challenges
resulting from large-scale simulations and the growing datasets from the
experimental sciences.

## 2.3 Computational Science Outreach

As the complexity of the emerging science increases, it will be
imperative to engage the best and brightest intellects from all areas of
computing science---applications scientists, computer scientists, and
mathematicians. To this end it is recommended that SciDAC-2 support the
formation of SciDAC Institutes to encourage discussion of fundamental
scientific issues, provide a venue for focused collaborations, train the
next generation of multidisciplinary scientists, and disseminate results
across disciplines.

## 2.4 SciDAC and Scientific Computing within Office of Science

Scientific computing facilities and the development of new approaches to
computational sciences are integral to the mission of each program
within the Office of Science. SciDAC-1 has already proven the value of
involving scientist in diverse areas working together with
mathematicians and computer scientists to create novel approaches to
common problems. While the most revolutionary effects have necessarily
been on small science teams without the resources to undertake major
research and development in scientific computing, the larger,
traditionally self-reliant teams have also contributed to the benefits
of creating widely applicable technologies. Distributed data and
workflow management ("Grids") is an excellent illustration of where the
alignment of DOE computer science and DOE program resources have begun
to create a broadly applicable environment benefiting U.S. science.

SciDAC-2 should be a component in a wider strategy for Office of Science
scientific computing. This strategy should align the powerful
program-level motivation of mission responsibility with the higher-level
motivation of ensuring that the Office of Science achieves its overall
mission by advancing scientific computing on an appropriately broad
front.

# 3. Future Opportunities

Over the past five years, numerous reports presented at SciDAC and other
scientific meetings have delineated the need for yet greater simulation
fidelity and new theoretical models. Indeed, recent and expected growth
in computational capabilities (along with the methodological advances
achieved by SciDAC-1) means that the scientific opportunities associated
with advanced computing today are far greater than five years ago, when
SciDAC-1 began. These endeavors will be further complicated by
increasingly complex computer architectures and the need to scale
scientific computing codes to the petascale regime.

Looking to the future, we also see a timely and significant opportunity
to increase both the utility of DOE experimental facilities and the
effectiveness of DOE experimental science, through judicious and
targeted application of advanced computing and information technology.
The driving factors are the profound importance of DOE experimental
facilities and science to the nation; the urgent challenges currently
faced by experimental sciences as a result of rapidly growing data
volumes and new experimental techniques; and the significant progress in
advanced computation, distributed computing, and interdisciplinary and
interlaboratory collaboration achieved within the SciDAC-1 program. A
broader SciDAC-2 program can empower experimentalists by automating
important steps in the experimental process, from the operation of
experimental facilities, to the informed design of experiments through
simulation techniques, to the collection, comparison, and analysis of
experimental data and model results.

We now describe some of the many challenges and opportunities that
remain in these scientific domains that advanced computing technologies
can help address. This list is not exhaustive but is meant to be
representative of the opportunities that can be exploited within a
SciDAC-2 program.

## 3.1 Accelerator Physics

As the Office of Science paper "Accelerator Technology for the Nation"
points out, accelerators play a crucial role, not just in DOE research,
but also in the entire scientific enterprise:

Essentially all we know today and will learn in the future about the
fundamental nature of matter is derived from probing it with directed
beams of particles such as electrons, protons, neutrons, heavy ions, and
photons. The resulting ability to "see" the building blocks of matter
has had an immense impact on society and our standard of living. Over
the last century, particle accelerators have changed the way we look at
nature and the Universe we live in and have become an integral part of
the Nation's technical infrastructure. Today, particle accelerators are
essential tools of modern science and technology \[1\].

There are a number of opportunities for high-performance computing to
benefit accelerator physics, some of which are discussed below.

**Development of compact accelerators.** The long-term future of
experimental high-energy physics research using accelerators depends on
the successful development of novel ultra high-gradient acceleration
methods. New acceleration techniques using lasers and plasmas have
already been shown to exhibit gradients and focusing forces more than
1000 times greater than conventional technology. The results to date
demonstrate a proof of concept for compact wakefield accelerators; the
challenge is to control these high-gradient systems and then to string
them together. Continued development and application of PIC codes are
required to plan and interpret experiments, to model parameters not
available to current experiments, and to study the key physics that
needs to be understood before a 100+ GeV collider based on plasma
techniques can be designed and tested.

**Create new data management and analysis opportunities for
experiments.** The current Data Grids, built under SciDAC-1, have proved
quite successful. For the second generation under SciDAC-2, integrated
data management and analysis capabilities are needed to handle the
statistically challenging, and often real-time, data from accelerators
including RHIC, LHC, and the Spallation Neutron Source (SNS).

**Deploy effective remote operations for LHC**. The Large Hadron
Collider at CERN is the world's highest energy particle accelerator and
will produce petabytes of data beginning in 2007. One of the critical
limiting steps in utilizing this facility will be managing the high data
volume that is generated. Requirements for an effective collaborative
working environment go far beyond present collaboration technologies by
requiring the coupling of a large number of simultaneous heterogeneous
participant sites and the seamless sharing of data and applications in a
secure, yet easily accessible, integrated environment.

## 3.2 Astrophysics

Supernovae are among the most dramatic events in the cosmos. The nuclear
synthesis that occurs in supernovae is the source of most of the heavy
elements in the universe. They provide a natural laboratory for the
study of matter and energy at extreme conditions, bringing together
nuclear physics, particle physics, fluid mechanics, radiation transport
and other physical processes. Supernovae have been a topic of intense
study in astrophysics, from both an observational and a theoretical
perspective. Because of the inherent complexity, however, a detailed
understanding of the mechanics of supernovae remains elusive.

**Accurately predict core-collapse supernovae*.*** Accurate prediction
of core-collapse supernovae will lead to the understanding of element
synthesis; neutrino, gravitational wave, and gamma ray emissions;
neutron star spin-up and kicks; and photon polarization. To achieve such
predictions will require complete three-dimensional, multiphysics
core-collapse supernova simulations with Boltzmann (exact) neutrino
transport.

**Accurately predict dynamics of Type Ia supernovae.** Type Ia
supernovae are the brightest thermonuclear explosions in the universe.
They are thought to arise in binary star systems in which a white dwarf
gradually accretes matter from its companion red giant star. As the mass
of the white dwarf increases to approximately 1.4 solar masses, the
carbon/oxygen core of the star begins to undergo rapid thermonuclear
burning resulting, ultimately, in the explosion of the star. Type Ia
supernovae are significant scientifically because of their use as
standard candles in determining the expansion rate of the universe.
Despite extensive interest in and observations of Type Ia supernovae,
the mechanisms of the transition from white dwarf to supernova are not
well understood. Accurate modeling of the entire process from ignition
to the explosion of the star requires codes that can operate across a
broad range of space and time scales from slow convective motion to
early deflagrations propagating a small fraction of the sound speed to
the final unbinding of the star.

**Next-generation experimental cosmology.** In order to handle \~1
Gbyte/sec data streams from the Large Synoptic Survey Telescope (LSST),
near-real-time analysis and associated information management
capabilities are essential. Science with LSST will require responsive
queries on an object catalog that will itself reach hundreds of
terabytes in size; retrieval from the tens of petabytes of image data
will be required for many studies.

## 3.3 Biology

The 21st century has been called the "biological century"---an era when
advances in biology, spurred by achievements in genomic research
(including the sequencing of the human genome) will bring revolutionary
and unconventional solutions to some of our most pressing and expensive
challenges in energy, the environment, and medicine. We will understand
how living organisms interact with and respond to their environments so
that we will be able to use biological processes to produce clean
energy, remove excess carbon dioxide from the atmosphere, and help clean
up the environment. SciDAC-2 opportunities in biological research
include the following.

**Systems-level simulation of cells and microbial communities.**
Predictive modeling of microbial behavior is one of the critical
capabilities required for harnessing microbial communities for energy
production, carbon sequestration, and environmental remediation. Such
predictive models provide a powerful tool for scientific discovery
through the design and control of pathways or even entire microbial
organisms with novel capabilities in hydrogen production, metal
reduction, or cleanup of aromatic hydrocarbons in a radiation-rich
environment. Achieving predictive capabilities will require integrated
heterogeneous models and simulations of intra- and intercellular
metabolic, regulatory, and signaling pathways as well as dynamic
functions of microbes and microbial communities.

**Biological data analysis and model building.** Analysis of
experimental data and model building are key to reaping the full
benefits of high-throughput experimental methods in biology. The amount
of microarray, proteomics, and structural data being generated is
rapidly outpacing the ability of biological scientists to analyze it.
Methods for large-scale data analysis, along with reliable methods for
automated construction of biological models, are sorely needed. Data
from experiments and even the biological literature can all be mined in
search of underlying patterns. Data mining requires a computing
infrastructure able to support high-throughput data processing. Core
technological advances must reduce the effective complexity of
databases, in order to make data analysis sufficiently scalable to
tackle an increasing body of experimental knowledge.

**Structural modeling and simulation for biological systems.** Molecular
modeling is a unique tool for obtaining new insights into the relation
between structure, dynamics, and function, at a level of detail that is
difficult or expensive to obtain from experiment. Novel methods and
algorithms are needed to harness the computational power of
next-generation supercomputers to successfully simulate biologically
relevant time and length scales for protein complexes. Accurate
large-scale simulation of biological systems can be accomplished only if
the emphasis in this field shifts toward the use of more accurate
physical models and use of methods that have been developed to
investigate chemical systems. Greater complexity must be achieved by
including more realistic, multicomponent environments, and greater
breadth by using comparative analysis methods that allow the
simultaneous analysis of multiple molecular trajectories.

**Data analytics environment for genomic data.** The avalanche of data
to be produced at DOE's four GTL facilities represent a tremendous
challenge for information technology---and an opportunity for
biologists. To exploit this opportunity, we must develop an integrated
data analytics environment that incorporates the biophysics and
bioinformatics with the structural and cellular modeling tools, provides
access to distributed and multidisciplinary databases and federated
queries across the GTL facilities, and thus supports new discoveries by
facilitating experiment-to-experiment and experiment-to-simulation
comparisons.

## 3.4 Chemistry and Materials Science

Chemistry and materials science represent a large and diverse set of
research goals including fundamental chemical reaction dynamics,
nanoscale science and technology, catalysis, magnetism, and
superconductivity. The computational issues revolve around the ability
to predict the atomic geometry, electron charge density or magnetization
of small atomic cluster or crystals as well as more exotic properties
such as superconductivity. Understanding chemical reaction dynamics is
key to reducing emissions and improving combustion, to petroleum
refining, and to fertilizer production. There is an increasing interest
in biomimetic materials and the quantitative study of biochemical
processes, for example, in protein structures and enzyme kinetics.

The understanding of these issues will provide the insight needed for
improved photovoltaics for solar energy, solid state lighting, hydrogen
production and storage, combustion, and the conversion of biomass into
usable fuels. Atomic scale processes are critical for climate modeling
and environmental remediation including heavy element (actinide)
transport in soils. The computational challenges that arise within both
theoretical science and DOE user facilities are enormous, but so are the
opportunities to overcome those challenges in ways that enable
fundamental advances in science and/or more efficient operation of
unique facilities. The following are just a few examples.

**Nanoscience.** It is widely recognized that one of the key enabling
technologies of the 21^st^ century will be the ability to accurately
control the placement of individual atoms in clusters and on solid
surfaces. Such unprecedented control over atomic structures will require
an unprecedented level of theory and computation to understand and
ultimately design for desired properties. A detailed understanding of
the electrical conductivity in carbon nanotubes is still not in
hand---an essential ingredient to utilize these materials in devices.

**Data management for the nanosciences*.*** Providing integrated
information and workflow management for DOE's five Nanoscience Research
Centers will spur new discoveries by facilitating
experiment-to-experiment and experiment-to-simulation comparisons.

**Enabling innovative science at the LCLS*.*** To realize the full
scientific promise of the Linac Coherent Light Source (LCLS) requires
the timely development and application of advanced computing tools for
the design of experiments and for data handling and analysis---tools
that LCLS does not have the expertise to develop alone.

**Catalysis.** Through the combination of leadership-scale computing and
continued improvements in theory and algorithm, computational simulation
can enable the design of catalysts from first principles.

-   **Electronic structure of molecules, clusters, thin films, and
    crystals.** There is a need to expand the current algorithms and
    theoretical models to at least one order of magnitude: from hundreds
    of atoms to thousands for high accuracy methods, and from thousands
    to tens of thousands of atoms for more approximate methods.

## 3.5 Climate

Climate models must continue to improve to meet the increasing demands
of the climate science, and to an even greater extent, the climate
policy communities. Better accuracy and precision are required of
climate models at the regional scales so that the impacts of potential
climate change can be ascertained with sufficient confidence that
response strategies can be devised. Further, although there is a fairly
complete understanding about the effects of different atmospheric
concentrations of greenhouse gases on the climate, relatively little is
known about the feedbacks between a changing climate and the processes
that ultimately determine those concentrations, particularly the most
important greenhouse gas, carbon dioxide. Consequently, future climate
models will require much higher spatial resolution than do current
models as well as the inclusion of representations of many more
physical, chemical and biological processes. Increased resolution not
only provides better regional precision but often results in more
accurate simulation of the smaller scale processes, such as storms and
clouds, that are critical to the correct simulation of climate. A whole
new class of process parameterizations is in the pipeline to be included
in the next generations of models, including dynamic vegetation, ocean
biogeochemistry, and atmospheric aerosol chemistry.

**Carbon cycle modeling effort.** The inclusion of carbon cycle
processes into climate models, particularly those relating to ocean
ecology and dynamic vegetation on land, is one of the most active areas
of research today. Currently, time-dependent atmospheric CO~2~
concentration is a boundary condition for the models that implicitly
incorporates the uptake of the gas by the land and oceans. In recent
years, a significant scientific effort has gone into understanding how
the sources and sinks of CO~2~ function and how they might change in a
changed climate. The purpose of adding the carbon cycle is to progress
beyond the specification of atmospheric carbon dioxide concentrations to
the specification of actual anthropogenic emissions and examine the
feedbacks between the carbon cycle and the climate. Similar to early
efforts to couple the ocean and atmosphere, the additional degrees of
freedom have produced a wide range of results in early simulations of
these Earth-system models that simulate carbon-climate interactions.
Incorporating the carbon cycle involves more than the addition of
process parameterizations to climate models. The nature of the coupling
and our lack of a full understanding of the feedback mechanisms lead us
to believe that there needs to be a more thorough evaluation of the
carbon process models before they are incorporated into the most
comprehensive climate codes. Model confidence can then be gained through
improved quantitative methods for validating terrestrial and ocean
ecosystem models against observational datasets and the extraction of
more model-relevant quantities from the observed data.

**Coupled model integration and evaluation capability.** Given the
complexity and highly nonlinear dynamics present in climate models,
changes in any part of the model will have an impact on the entire
system. As a result, evaluating whether any proposed improvement in
process representations, resolution, or numerical algorithms achieves
the desired result is a difficult and time-consuming process that
imposes high barriers to change. What is needed, therefore, is a model
integration, testing and evaluation facility to rapidly incorporate new
ideas. This integration capability will significantly lower the barriers
to the adoption of new ideas into full models.

**Next-generation data requirements.** High-speed network bandwidth
linking distributed archives of experimental, observational, and
simulation data are required for scientific discoveries by the large
national and international climate research communities. The size of the
data holdings has grown substantially and will continue to grow as new
observational data are taken from environmental monitoring and
simulations play an ever-larger role in filling the theoretical gaps in
our understanding of complex environmental systems. Researchers studying
global and regional impacts of climate change now need access to both
future petabyte-scale simulation datasets *and* large and diverse
experimental datasets from the DOE Atmospheric Radiation Measurement
(ARM) program, NASA satellites, and other sources. This urgent need can
be met by the creation of an integrated environmental data portal that
allows scientists, and ultimately other stakeholders, to pose complex
questions against what will ultimately be petabytes of simulation and
observational data.

## 3.6 Combustion

Combustion currently provides 85% of the energy needs of the United
States. Whatever mix of energy sources we utilize in the future we will
almost certainly continue to rely heavily on combustion. At the same
time fundamental scientific issues need to be answered that can be
adequately understood only with advanced computing. These include the
factors that control the production of particulate matter (soot) as well
as polluting gases and optimally efficient designs.

**Lean premixed combustion.** New combustion systems based on ultra-lean
premixed gaseous burners have the potential for dramatically reducing
pollutant emissions in transportation systems, heat and stationary power
generation, and a broad range of science and technology applications.
Ultra-lean flames, however, are highly susceptible to fluid-dynamical
combustion instabilities, making robust and reliable systems difficult
to design. An additional complexity in the development of ultra-lean
burners is the range of possible fuels and fuel mixtures. Recent
research suggests that certain low-concentration additives, such as
hydrogen, in the fuel stream can significantly enhance the stability of
turbulent hydrocarbon flames. The combustion of hydrogen itself is also
the focus of increased interest because of its importance to alternative
fuels involving hydrogen-rich refinery gas and syn-gas (a mixture of
predominantly carbon monoxide, hydrogen, and water formed in a coal
gasification process). New computational tools developed under SciDAC-1
can answer key questions about flame stability and combustion dynamics
for ultra-lean flames, providing the key scientific underpinnings to the
design of new combustion systems.

**Molecular mixing in nonpremixed turbulent combustion.** Many
combustion systems such as aircraft engines and diesels do not mix fuel
and oxidizer prior to burning. For these types of combustors, mixing
controls the combustion process. Standard assumptions about mixing
behavior are inadequate for the development of high fidelity model. A
more detailed understanding of mixing and reaction dynamics is required
that can be determined only from numerical simulation; the level of
detail required is not accessible experimentally with current
techniques. Multiphase issues also play an important role in many
nonpremixed systems. For example, in diesel engines, the fuel is
injected as a liquid spray whose breakup and atomization plays a
critical role in the combustion process. Understanding the formation of
soot particulates and their subsequent oxidation is a key factor for
improving emissions from diesel engines.

**High-throughput laser imaging for combustion science*.*** U.S.
leadership in combustion science must be sustained through support for
integrated workflow, information, and data management solutions able to
handle next-generation laser imaging data. One important component of
this work is to facilitate the comparison of experimental measurements
with simulation data. As high-fidelity simulations of laboratory-scale
experiments become more routine, there will be increasing emphasis on
improving the accuracy of analysis algorithms and providing tools to
facilitate the fusion of simulation and experimental data.

## 3.7 High Energy and Nuclear Physics with Colliding Beams

Experiments at colliding beams are the key to advancing our
understanding of the physics of our universe on the smallest length and
time scales, and at the level at which the fundamental particles
transition into the matter that we know today. The inevitable physical
scale of experiments requires international, even global collaboration.
Huge volumes of data are needed to achieve precision in a world where
the intrinsic unpredictability of quantum processes must be offset by
massive statistical weight. The greatest computing challenge for such
physics is how to use the intellectual energy of intercontinental
collaborations to extract Nobel Prize-class understanding from the truly
daunting volumes of data.

High energy physics and nuclear physics experiments are entering a
period of revolutionary challenges and opportunities. The LHC will make
amazing discoveries (the most amazing of all would be to observe nothing
new). The nuclear physics colliders are just at the beginning of an
experimental understanding of the nature of matter under the most
extreme cosmological and astrophysical conditions. Instruments with an
analog data rate of a petabyte a second will yield petabytes per year
after brutal data selection and compression.

Opportunities to empower or revolutionize the science appear in two
related areas, both of which should bring more general benefits to
science.

**Open Science Grid at the core of distributed data-intensive science.**
The Open Science Grid is already a vital production environment for
physics and presents a tantalizing taste of what the environment could
become. Extremely challenging problems of resource management,
monitoring, and troubleshooting become critical in production
environments. SciDAC-2 will be able to advance science by addressing
such issues in the context of OSG.

**Amazing discoveries from an ocean of data.** The analysis of petabytes
of selected and compressed data is also extremely difficult. Current
techniques have aggregate response times measured in months to years;
scientific intuition is best served by weeding out the crazy ideas in
minutes. SciDAC-2 and other initiatives such as the planned Research and
Evaluation program could provide the innovative architectures and the
innovative computer science that can be married to the data intensive
challenges of high energy physics and nuclear physics to bring
revolutionary benefits.

## 3.8 Fusion Energy

Fusion as a power source is potentially inexhaustible, produces no
troublesome emissions, is safe, and has few, if any, proliferation
concerns. It creates no long-lived waste and runs on fuel readily
available to all nations. Computational simulation techniques and
distributed computing technologies are fundamental to the goal of
developing environmentally and economically sustainable fusion reactors.
For example:

-   Sustained fusion power generation requires that we understand the
    detailed physics of the fluid of ions and electrons, or plasma, in a
    fusion reactor. Computational techniques for simulating the growth
    and saturation of turbulent instabilities in this plasma are
    critical for developing this understanding of how to control the
    plasma and hence to develop a successful fusion reactor.

-   A tight engagement of U.S. fusion scientists in the International
    Thermonuclear Experimental Reactor (ITER), to be located in France,
    will be essential both for it to succeed and for the United States
    to profit from its operation. Advanced distributed computing
    technologies are key to this tight engagement in a distributed
    international project.

The U.S. decision to participate in ITER and the advances achieved in
SciDAC-1 create both challenges and opportunities that a SciDAC-2
program will be well positioned to address. The following are three
representative examples.

**Coupling computation to experiment at ITER.** Building on the
successes of the SciDAC-1 Fusion Collaboratory, SciDAC-2 researchers
will be able to perform quasi-real-time analysis of experimental data
from long-running experiments such as those planned for ITER.

**Building ultra-high-fidelity predictive models.** A SciDAC-2 thrust
focused on improving simulation fidelity and realism can produce
predictive models able to simulate plasma behavior under realistic
conditions. Advanced mathematical algorithms, leadership-class computing
capabilities, and support for data-intensive applications will lead to a
new ability to understand plasma evolution from a statistical
perspective.

**Remote participation in ITER.** ITER's location in France will require
the development of "remote control room" capabilities that allow
physicists in the United States and elsewhere to stand shift without
incurring prohibitively expensive travel costs. The creation of such
capabilities raises difficult networking, security, and distributed
computing challenges.

## 3.9 Quantum Chromodynamics

The long-term goal of nuclear and high energy physicists is to identify
the fundamental building blocks of matter and to determine the
interactions among them that lead to the physical world we observe.
Remarkable progress has been made through the development of the
Standard Model, which provides fundamental theories of the strong,
electromagnetic and weak interactions. Our knowledge of the Standard
Model is incomplete, however, because it has proven very difficult to
extract many of the most interesting predictions of quantum
chromodynamics (QCD), the component of the Standard Model that describes
the "strong force," by which gluons bind quarks together to form
hadrons. QCD theorists use computer simulations to determine the most
probable arrangements of gluons and quarks inside a particle and then
use these configurations to determine the particle's properties.
Calculating all the possible positions of quarks and gluons would be
impossible, so theorists simplify the problem by imagining space and
time not as a continuum, but as a lattice---a four-dimensional grid of
discrete points at which quarks and gluons can reside. This approach,
called lattice QCD, transforms an impossible problem into one that is
extremely difficult but, in principle, solvable. Among the major goals
of lattice QCD are to determine the range of validity of the Standard
Model; understand the internal structure of nucleons and other strongly
interacting particles, and determine the properties of strongly
interacting matter under extreme conditions, such as those that existed
immediately after the "Big Bang" and are produced today in relativistic
heavy-ion experiments.

**Physics challenges.** To meet nuclear and high energy physics
challenges of the future, the teraflops (sustained to the application)
systems developed during SciDAC 1 will need to be enhanced two orders of
magnitude, to 100 teraflops systems. For example, to reduce the pion
mass to the 200 MeV level will require of scale 100 teraflops-years of
resources sustained to the application. The nuclear physics scope is
also expanding beyond the hadronic QCD mostly associated with the
experimental program at CEBAF to include significant efforts in finite
temperature QCD that will (one hopes) explain the quark-gluon liquid new
being seen at RHIC.

**Enhanced computing environment.** Enhancing the present lattice QCD
environments to be able to effectively use 100 teraflops and even
petaflops systems planned to be made available from other program
resources is a significant task: some of the hardware environments may
scale to hundred of thousands of processors, and other hardware
environments will include the complexities of multicore processors just
coming into the market. Making effective use of these new systems will
require significant enhancement of the portable software environment and
the low-level software with its associated hardware development systems.

# 4. Advancing Science through Enabling Computational Technologies

The successes of SciDAC are many and broadly shared. In addition to the
scientific accomplishments highlighted earlier in this document, another
SciDAC success is cultural: demonstrating the power of multidisciplinary
and multi-institutional teams working toward a common goal. Such teams
are used to help meet many of the goals of SciDAC:

-   **Improve science through the effective delivery of advanced
    enabling computational technologies (ECTs).** The SciDAC program
    structure works to meet this goal in two ways. First, it delivers
    robust software tools that are generally applicable and widely
    available to the scientific computing community. Linear and
    nonlinear solvers, mesh generation and refinement software, scalable
    data management tools, distributed computing software, and
    visualization tools are all examples of this type of infrastructure.
    Second, the SciDAC program structure allows for smaller, targeted
    efforts that integrate advanced applied mathematics and computer
    science technologies into selected DOE applications projects. This
    type of relationship results in strong interactions and maximizes
    the likelihood of success for the insertion of the specific
    technology into that application. In both cases, technology
    development efforts are driven by application needs.

-   **Improve development of enabling computational technologies.** DOE
    has a long history of developing enabling computational
    technologies. Prior to SciDAC, these efforts were performed largely
    by small teams focusing on single-niche technology areas. In
    SciDAC-1, ECT development efforts became more collaborative in two
    ways, both of which benefit application scientists. First,
    multidisciplinary teams were formed that combined domain experts,
    mathematicians, and computer scientists. Such teams are more
    effective in addressing the large-scale computational science
    challenges present today. Second, SciDAC-1 created horizontally
    integrated teams from across the DOE complex that focused the
    efforts of experts in a single technology area (for example,
    solvers, meshing tools, data management). These collaborative
    endeavors advance the development of ECTs in many ways: they
    encourage the development of best-in-class software, improve the
    usability of software for application scientists, and eliminate
    duplication of effort.

-   **Manage the software life cycle more effectively.** In addition to
    the goals mentioned above, an effort focused on supporting the
    software lifecycle clearly is needed. Software is an increasingly
    critical component in the scientific endeavor, and it is necessary
    to ensure that the enabling computational technologies developed as
    part of SciDAC can continue to meet the needs of computational
    scientists once initial development is complete.

Two SciDAC-1 program elements have been critical in meeting the first
two goals described above: Integrated Software Infrastructure Centers
(ISICs) and Scientific Application Partnerships (SAPs). These
synergistic program elements have worked closely with DOE application
communities to enable the scientific accomplishments of SciDAC-1 (see
Figure 11). ISICs and SAPs have proven valuable as a means of
simultaneously attacking cross-cutting technical problems and using
application science drivers to motivate a focused multidisciplinary
attack on critical problems. Both program elements have been extremely
successful and should be preserved and enhanced, rather than replaced or
overhauled. Recognizing the importance of the third goal above, SciDAC-2
introduces a new program element---Software Integration, Maintenance,
and Support (SIMS)---which will bring economies of scale to the support
of SciDAC software.

![](media/image12.jpeg){width="6.0in" height="3.861111111111111in"}

**Figure 11**. Interrelationship between the Office of Science programs,
the computing facilities and the SciDAC program elements.

These program elements also play an important integrating role beyond
the applications they support: they enable *vertical* integration within
ASCR and *horizontal* integration across the Office of Science. Within
ASCR, ISICs allow new ideas generated in the base research program to be
deployed on the Office's high-end computing facilities and networks and
to be promulgated to the larger national and international computational
science community. Conversely, the close coupling of ISICs and SAPs with
their applications teams helps them identify future directions for ASCR
research programs.

Across the Office of Science, ISICs and SAPs serve as the means by which
ASCR computational technologies are deployed to application teams to
enable new science. These programs have increased significantly the
number and quality of interactions between scientists funded by ASCR and
the other program offices. In many cases, these interactions not only
have enhanced simulation capabilities but also have driven the
development and prioritization of ISIC technologies---and thus have
indirectly influenced future directions in the ASCR base research
program. The reason ISICs have been successful in both horizontal and
vertical integration is that they collectively span a broad range of
mathematics and computer science topics that are relevant to many, if
not all, Office of Science applications.

We now describe these program elements---ISICs, SAPs, and SIMS---in some
detail. In each case, we describe the role of the program element and
discuss the characteristics that define its success. We include a brief
description of some topical areas that are important to meet the needs
of application scientists in SciDAC-2.

To set the stage, we begin with a discussion of the cross-cutting
opportunities for SciDAC-2 in both simulation and experimental science.

## 4.1 Cross-Cutting Opportunities to Advance Simulation Sciences

Each of the program offices within the Office of Science supports
computational science research that contributes to their specific
research missions and agendas. Each program has computational science
research communities, which could comprise hundreds of members, all
interested in the application of advanced computing within their
discipline and seeking to pose and answer a few significant scientific
grand challenge questions of national importance. SciDAC-1 has
encouraged these communities by helping to identify the needed
application software and by collecting the objectives, requirements, and
implementations deemed necessary for the high-end computers within the
DOE computing facilities. Following is a small but important list of
some of these opportunities:

-   **Managing model complexity.** Increasingly complex phenomena will
    be simulated in SciDAC-2 over a wide range of temporal and spatial
    scales, using potentially different physics models at each scale.
    Examples include deducing macroscopic behavior directly from the
    constituent atoms or molecules in materials modeling and addressing
    large-scale magnetic confinement in plasma physics using micro-scale
    turbulence simulations. Advances in combining simulations at
    different scales are required, using different physics and
    understanding the error introduced in such simulations.

-   **Discretization of spatial models.** The advent of increased
    computational resources allows an increase in the fidelity of
    numerical simulations. For example, complete simulations of complex
    scientific devices, the details of microbial cell communities, and
    capturing of landscape features in regional climate models will be
    possible with SciDAC-2 computational resources. All of these
    examples rely on accurately capturing complex geometry and utilizing
    advanced discretization techniques that faithfully represent the
    mathematical equations.

-   **Fluid flow modeling.** Many scientific applications---plasma
    physics, combustion, astrophysics, and climate---use similar fluid
    flow algorithms that have an arbitrary choice for the tradeoffs
    between accuracy and computational cost. There is an opportunity for
    a SciDAC Science Institute to address the fluid flow models to allow
    each domain to make more suitable choices in this tradeoff.

-   **Managing computational complexity.** Often, as the model becomes
    more complex or increased resolution is used to obtain more accurate
    answers, the cost of solving the problem also increases. Ideally,
    the increased cost will be offset by the increase in computational
    resources, but this is true only if the cost grows linearly with the
    size of the problem. This ideal scenario is available for some
    application areas, but more must be done to achieve scalability as
    the problem size grows, to obtain computational efficiency on
    hierarchical multiprocessor systems, and to increase robustness for
    more application areas.

-   **Managing software complexity.** Just as the models and mathematics
    are increasing in complexity, so too are the software engineering
    challenges. Tools developed by different communities must be used
    together in a seamless fashion to achieve the scientific goals
    outlined above. For example, advanced meshing techniques, solvers,
    visualization, and data management are all likely to be incorporated
    into application codes. Technology that encourages modular software
    development, handles language interoperability issues, and eases
    experimentation with different tools will strengthen the development
    of state-of-the-art scientific simulation codes.

-   **Understanding petascale computational resources.** The effective
    utilization of next-generation supercomputers will require a
    rethinking of the current algorithms. The development of tools that
    address systems software issues, algorithm scalability, and
    application performance on such architectures is critical to
    realizing the full potential of DOE's unparalleled computational
    resources.

-   **Understanding simulation results.** Simulations on petascale
    architectures will generate petabytes of data. Understanding that
    data and extracting useful information from it continue to be
    challenges for applications scientists. Advances in visualization
    techniques, data mining, and feature detection will enable the key
    scientific insights that can be expected from the investment in
    SciDAC-2.

## 4.2 Cross-Cutting Opportunities to Link to Experimental Sciences

In addition to the previously listed opportunities, each of the Office
of Science programs has extensive efforts supporting the experimental
sciences and the establishment of new facilities as outlined in the DOE
Office of Science's "Facilities for the Future of Science" report.
Several of these new experimental facilities, either under construction
or in the planning stages, have significant investments in both
experiments and simulation. In addition, the proposed Genomics/GTL
facilities are expected to work together as a distributed enterprise,
cross-cutting both simulation and experiment in a highly coordinated
fashion. This situation leads to new opportunities for the integration
of experiment into theory, modeling and simulation to advance science.
In many cases the result is new informatics-driven science, requiring
new data management and analytic approaches to drive the discovery
process. In SciDAC-2, new program areas will be added to address this
important new area of computing.

The following are examples of common areas that can be exploited, either
through collaborative application projects or through cross-cutting
technology projects.

-   **Remote control rooms for international experiments.** The
    worldwide experimental programs in fusion and high energy physics
    will be centered on facilities located in Europe: the International
    Thermonuclear Experimental Reactor (ITER) and Large Hadron Collider
    (LHC). Enabling remote participation by U.S. scientists will
    contribute to the success of ITER and LHC and also maximize the
    value of these unique facilities to the United States.

-   **Real-time computing at beamlines.** A new generation of more
    powerful x-ray and neutron sources and far more sensitive detector
    technology are combining to create a data explosion at facilities
    across the DOE system. A combination of on-demand access to
    computing, enhanced algorithms capable of quasi-real-time analysis,
    and telecollaboration technologies can enable far more efficient use
    of beamlines.

-   **Distributed data management and computing.** Climate, physics,
    biology, and nanosciences (among others) face similar needs for the
    distributed management and analysis of large volumes of data
    produced at different facilities, analyzed by means of often complex
    workflows, and consumed by a large and distributed user community.

-   **Information management.** In addition to large data volumes, DOE
    experimental science disciplines face challenges associated with
    increasingly complex data due to different experimental modalities
    and analysis techniques. Again, there are opportunities to achieve
    technological and methodological advances that benefit multiple
    disciplines.

-   **Cybersecurity for open facilities.** DOE user facilities face the
    increasingly difficult task of operating open facilities in an
    increasingly hostile security environment while maintaining utility
    of the facilities and preserving the free flow of information.

## 4.3 Integrated Software Infrastructure Centers (ISICs)

ISICs are large, multi-institutional teams of mathematicians and
computer scientists chartered to accelerate the development of and
protect the long-term investments in scientific codes, to achieve
maximum efficiency on high-end computers, and to enable a broad range of
scientists to use simulation in their research . These centers have been
individually and collectively praised for developing and delivering
leading-edge computational technologies to a diverse set of scientific
applications. ISICs also have emerged as national repositories of
high-performance scientific computing software and expertise.

In SciDAC-1, seven ISICs were funded to address application needs in the
areas of solvers and optimization, mesh generation and discretization,
adaptive mesh refinement, software interoperability, scientific data
management, performance evaluation, and systems software. ISICs have
succeeded by

-   inserting advanced mathematics and computer science technologies
    into SciDAC applications by encapsulating research results in robust
    software tools;

-   developing and delivering new enabling computational technologies in
    response to articulated and anticipated application needs;

-   providing expertise and consulting advice on a range of topics in
    mathematics, computer science, and software engineering---as well as
    providing entrée to the broader scientific computing community; and

-   nurturing the next generation of SciDAC computational scientists
    through active participation in conferences and workshops, and via
    the training of graduate students and postdocs.

ISICs develop and deliver robust software tools that are either directly
usable by computational science application teams or are necessary for
the effective use of high-end computing facilities. As one SciDAC
scientist said, "The \[ISIC\] isn't merely making the hard things
easier, it is making science possible that we thought was too complex to
try." ISICs are well positioned to work with SAPs and have effectively
done so in SciDAC-1. ISICs will also support the SIMS program objectives
of advancing software interoperability and reusability and promoting
sound scientific software engineering practices. ISICs should also be
frequent participants in the workshops sponsored by the SciDAC
Institutes, for example, by offering intensive immersion opportunities
in key enabling technology areas.

Clearly, ISIC teams serve many functions, including the creation of
advanced technologies, code development, application interactions, and
general outreach to the scientific computing community. Funding levels
should be commensurate with this level of activity.

### Building on the Successful ISIC Interaction Model

ISICs have been widely praised and are considered essential to the
success of SciDAC-2. They are working extremely well: they are
developing world-class computational capabilities and working closely
with application teams to insert this technology into their simulation
codes. In discussing the future of ISICs, we have heard consistently the
message, "Don't change the current model just for the sake of change."
The key characteristics of the successful ISIC model that should be
preserved are as follows:

-   **Emphasize collaboration with, and impact on, multiple SciDAC
    application teams.** ISICs establish strong collaborations with
    multiple applications teams---without losing their identity as
    ISICs. This strategy ensures that ISIC technology is leveraged
    across multiple applications and the general relevance of ISIC
    topics to the broad computational science community. Working with
    several application teams also helps ISICs focus on the most
    important technology areas, determine priorities for development,
    and produce robust simulation technologies.

-   **Create horizontally integrated, multi-institutional teams.** ISICs
    are formed around a single technology theme with expertise drawn
    from multiple institutions (national laboratories and academia). The
    current set of ISICs have yielded a strong collaborative environment
    across DOE and academia that promotes community efforts to develop
    best-in-class software and integrated tools that no single
    institution could have produced alone. The existing ISICs have shown
    how such multi-institutional teams can accomplish more than the sum
    of their parts.

-   **Provide stable, long-term funding for core technology
    development.** ISICs provide a long-term, stable means for
    "hardening" research software and delivering this software to
    applications teams throughout the Office of Science. Current SciDAC
    projects have demonstrated that it can take five years or more to
    develop a new generation of hardened algorithms and software, and
    the insight on how they can be used effectively in applications.
    This observation is consistent with the NNSA ASC Alliance Centers
    and the NSF Science and Technology Centers***.***

-   **Ensure that the collection of ISICs span a breadth of
    interdependent, relevant topics.** ISICs align with current and
    anticipated needs of Office of Science applications, while
    collectively spanning the basic mathematics and computer science
    research programs to a great extent. This strategy is critical if
    ISICs are to maintain their role as the vertical integration link
    between ASCR's basic research program and its computing facilities,
    as well as the horizontal integration link between ASCR and the
    application program offices.\
    \
    Because ISIC topics are interdependent, they work closely together
    to discuss application code needs (thereby increasing the likelihood
    that they will be met), as well as to leverage each other's
    technology. ISICs are encouraged to support one another as much as
    they are encouraged to support the application teams. For example,
    two or more ISICs might partner with a single application code team.

-   **Keep enough flexibility to respond to near-term opportunities.**
    ISICs need the ability to respond to near-term opportunities to
    interact with application teams and with each other. Such
    interactions can occur on a 12- to 18-month time frame and can
    change over the ISIC's lifetime. Increased flexibility leads to
    increased impact because resources can be placed where they are most
    needed. Acceptable activities for ISIC interactions with an
    application team can take many different forms, including
    demonstrating feasibility of a new algorithmic or modeling approach
    to a scientific area, using new algorithmic and software technology
    to solve specific scientific problems, or embedding new software
    technology in production applications codes.

## 4.4 Scientific Application Partnerships (SAPs)

SAPs are targeted efforts to integrate advanced applied mathematics and
computer science technologies into selected SciDAC applications
projects. Although the stated goal of SAPs is similar to that of ISICs,
the two program elements are complementary.

First, the mathematicians and computer scientists funded by SAPs are
typically in residence with and primarily managed by the application
team. This arrangement results in strong interactions and maximizes the
likelihood of success for the insertion of the specific technology into
that application. ISIC efforts, on the other hand, are larger and focus
on interaction with multiple application teams. They produce, in turn,
robust, usable software that is applicable to the general SciDAC
community in a way that SAP efforts cannot.

Second, SAP provides the mechanism to supply application teams the
computer science and mathematics expertise not covered by ISIC teams.

Examples of how SAP funding has been used effectively are numerous and
include the development of application-specific visualization tools,
data partitioning methods, and adaptive mesh refinement-based simulation
codes.

### Building on the Successful SAP Interaction Model

First and foremost, the SAP program is widely touted as an important
element of the SciDAC-1 program. To enhance the success of SAPs, we
recommend that SciDAC-2 continue the strategies that have proven
successful in the past---and, furthermore, augment the program to
address better the needs of application scientists. In view of its
demonstrated effectiveness and the additional program needs that SAPs
must fill, the size of this program element should increase
significantly, and broader participation should be encouraged.
Specifically, the SAP program element for SciDAC-2 should have the
following characteristics:

-   **Meet specific, targeted application needs.** The strong
    collaboration between the science domain experts, mathematicians,
    and computer scientists involved in SAPs results in successful
    insertion of new technologies directly into application codes.
    Targeted projects that meet needs identified by application
    scientists maximize application impact. Furthermore, collocating
    mathematicians and computer scientists with applications teams has
    proven to be particularly effective, enabling them to provide
    technology directly to the application team and also to act as
    liaisons to foster interactions with ISICs or other external
    technology providers. SAPs should also be used to encourage the
    participation of resident application scientists in ISICs to
    increase the dissemination and insertion of ISIC technology in an
    application area.

-   **Fill topical holes ISICs do not cover.** Although ISICs span a
    broad range of technologies, they cannot be comprehensive. The SAP
    program therefore fills a critical SciDAC program need because it
    can fund the development and insertion of technologies that are
    needed by the application but not part of an ISIC. In the case where
    the application need and ISIC technology overlap, SAP funding should
    be used to fund a co-managed project.

-   **Be flexible enough to address a variety of application needs.**
    Projects appropriate for SAP funding vary significantly in duration,
    scope, and goal. For example, shorter-term efforts are appropriate
    for exploratory projects (e.g., examining a new discretization
    technology) or short-term application needs (e.g., performance
    analysis and tuning, refactoring to support component technologies).
    Longer-term projects that are more involved are also appropriate for
    SAP funding (e.g., the exploration of a new framework for a given
    application area). Thus, SAP funding should be competed separately
    from the initial application and ISIC proposals. This will allow
    projects to have a duration that is suitable to the scope of the
    work, allow applications to fulfill needs that are identified later
    in the project life cycle, and eliminate interactions that are not
    bearing fruit.

-   **Aggressively prepare the next-generation of SciDAC applications.**
    A small percentage of the SAP program should be used to fund
    collaborations between application scientists not currently
    supported by SciDAC and ASCR mathematicians and computer scientists.
    This will seed new SciDAC-2 projects and will provide ISICs the
    opportunity to influence an application code early in the
    development process.

## 4.5 Software Integration, Maintenance, and Support (SIMS)

The DOE Office of Science has supported the development of many software
packages that used extensively by the scientific community. For example,
the National Academy of Sciences report "Getting Up to Speed: The Future
of Supercomputing" specifically mentions NWChem and PETSc as important
tools with wide use; and the NSF Blue Ribbon Panel report
"Revolutionizing Science and Engineering through Cyberinfrastructure"
points to the DOE-supported work on the Globus Toolkit as important for
cyberinfrastructure. No specific effort, however, is focused on
supporting the use of these tools by the DOE community. The SIMS program
element addresses the need to guarantee to computational science
consumers of SciDAC tools that the codes are robust and available on all
major computing platforms and that there are adequate resources to
address any issues.

### Components of SIMS

The SciDAC-2 SIMS program is envisioned as a distributed virtual center
that takes advantage of economies of scale to facilitate three main
functions across the SciDAC software portfolio: software integration,
maintenance, and support.

-   **Integration.** The integration activity addresses common issues
    faced by software developers and users. This activity will support,
    build, and test environments and software repositories. Another
    important activity is the support of software that is no longer
    being actively developed but is critical for computational science
    applications. This software must be usable and integrated with other
    SciDAC tools.

-   **Maintenance.** SIMS maintenance activities are intended to reduce
    the effort spent by tool developers in activities other than those
    in their software development agenda. These activities include bug
    tracking and mitigation, coordination of maintenance with continuous
    development, coordination of releases, revision control and
    recording of main issues, bookkeeping of a question-and-answer
    process for the deployment of scientific software, and the
    implementation of mechanisms to facilitate porting and tuning.

-   **Support.** The goal of a software support infrastructure is to
    provide a place where application and tool developers, along with
    their users, can interact and find prompt and adequate solutions to
    and information on common software development problems. The
    "one-stop shop" software support infrastructure will provide several
    different levels of service, produce written specifications to
    facilitate adoption by application and tool development projects,
    build expertise in successful and common practices, and support
    software in as many platforms as possible (although support of all
    platforms and configurations may be impossible). To an extent, this
    software support infrastructure will also be a conduit to
    application domain expertise.

### Selecting and Evaluating SIMS

A critical component of the SIMS center will be the software that is
chosen for support. The SciDAC program has already developed and will
continue to develop far more software than can be actively supported by
the SIMS center. Thus, criteria for ranking the many candidate projects
are critical. Key criteria include the following:

-   **Demonstrated value to SciDAC.** The software should be of
    demonstrated value to more than one group. The software should
    already run on at least one major leadership computing or
    experimental science platform, it must be correct, and it must be
    widely and freely available.

-   **Commitment to improving software quality.** The developers of the
    software must have a comprehensive plan to improve software
    qualities such as portability, performance, productivity,
    documentation, and maintainability.

The selection and evaluation of software included in the SIMS center and
the activities of the SIMS center itself will be carried out by a
multidisciplinary panel that includes computational scientists
(consumers of SciDAC software, primarily computational scientists and
software developers) and computer scientists and mathematicians
(software developers).

## 4.6 Meeting Future Application Needs: Topics of Emphasis

The next five years will see significant change in computational
science: the first petascale systems will become available, petabyte
datasets will be generated by experimental facilities, and each will
enable new scientific breakthroughs. ISICs and SAPs must build on their
current successes in both application partnering and technology
development. In particular, the next set of ISICs and SAPs must continue
to meet the current needs of the scientific applications teams, as well
as address the technology gaps that will emerge with petascale computing
systems and data-intensive experimental facilities. Some potential gaps
are the scalability and fault tolerance of applications and system
software on 100,000-processor computers, the management of data volumes
a thousand times greater than today, the validation of results as
applications incorporate new physics, and the ability to rapidly develop
ever more complex applications that port efficiently across a variety of
hardware architectures. Technology gaps of experimental facilities
include integrated approaches to managing large and structurally diverse
experimental data, managing complex experimental workflows, linking
experiments and computation, enabling remote access to experimental
facilities, supporting collaborative analysis of large quantities of
data, and enabling secure access to distributed facilities.

ISICs, individually and collectively, must provide a comprehensive,
integrated, scalable, and robust high-performance computing software
infrastructure that will enable effective use of leadership class
computing resources by current and future SciDAC application teams. To
meet this goal, ISICs must address needs for new algorithms and
mathematical software that scale to thousands of processors; accurate
discretization and parallel meshing approaches; software development
environments and methodologies for realizing portable performance; and
scalable scientific data management and analysis tools. In addition,
ISICs are expected to provide consulting expertise and to serve as a
conduit to software resources available elsewhere, for example, the NSF
centers and related NSF "cyberinfrastructure" facilities.

To meet the needs of current and future SciDAC applications, ISICs can
address many topics. These include the following:

-   **Numerical algorithms mathematical software.** Linear, nonlinear,
    and eigen solvers; multigrid and multilevel methods; adaptive
    sampling algorithms; optimization and inverse methods; sensitivity
    analysis and uncertainty quantification. These algorithms and
    mathematical libraries must be scalable and efficient on machines
    with thousands of processors and complex memory hierarchies.

-   **Mesh generation and discretization technologies.** Automatic mesh
    generation; parallel mesh partitioning techniques; adaptive mesh
    refinement algorithms and software frameworks; mesh quality metrics
    and improvement; numerical discretization technology; rapid problem
    definition. These technologies must be tailored to current and
    future applications employing both structured and unstructured
    meshes.

-   **Software development environments and programming paradigms.**
    Software development and code profiling tools; performance
    benchmarking and evaluation; code correctness and validation;
    high-performance software component technologies, code reuse, and
    language interoperability. These tools and capabilities must be
    portable across all SciDAC hardware platforms and provide a common
    programming environment for SciDAC application scientists.

-   **Scientific data management, analysis, and visualization.**
    Large-scale data integration and management; management of diverse
    metadata concerning, for example, provenance and data quality; data
    access and querying; scalable data storage and transfer; feature
    extraction and change analysis; vector/tensor field visualization;
    statistical data analysis. These data analysis algorithms and
    software tools must be scalable to large s, as well as tailorable to
    meet the needs of individual application codes.

-   **System software and tools.** Systems software that scales to tens
    of thousands of processors, supports high performance I/O and
    application-level communication, and provides the highest levels of
    fault tolerance, reliability, manageability, and ease of use for
    system administrators, tool developers and end users.

-   **Distributed computing technologies for experimental facilities and
    science.** Movement and management of large distributed data;
    distributed computing and information sharing technology; workflow
    technologies; methodologies for comparing simulation results and
    experimental data; Open Science Grid development and operations;
    cybersecurity infrastructure for distributed teams and experimental
    science; collaboration technologies for remote access to facilities.
    This software and related infrastructure must scale to support the
    collaborative planning of complex experiments and analysis of large
    quantities of data by distributed teams.

We recommend that one or more projects be funded in each of these broad
areas.

# 5. Engaging the Broader Computational Science Community

SciDAC's most significant accomplishment has been to enable breakthrough
scientific discoveries by engaging applications scientists, computer
scientists and mathematicians within the DOE Office of Science community
in new ways. Involving the broader community of U.S. and international
scientists in the activities of scientific discovery through advanced
computation will increase the impact of the SciDAC model for
collaborative science, with resulting benefits to all of U.S. science.
In addition, many areas of science are at a critical juncture where the
infusion of novel ideas will enable future scientific discoveries whose
impact can only be imagined today. As computational technology and
computing power continue to advance, computation clearly will play an
even more important role in these new discoveries. Programs that train
the next generations of computational scientists and that provide forums
for bringing together the top scientists, computer scientists, and
applied mathematicians to tackle critical scientific problems, will be
essential in enabling these types of advances. We propose the
establishment of SciDAC Institutes to accomplish these objectives.

Institutes have performed extremely useful roles in a number of science
communities. Examples include the Aspen Institute for Physics and the
Mathematical Sciences Research Institute. The SciDAC program will be
enhanced by including one or more Institutes that will provide for a
long-term infusion of new ideas. The key to the success of an Institute
lies in its ability to bring together the top scientists, computer
scientists, and applied mathematicians in a community-based,
interdisciplinary approach to critical problems in computational
science, computer science, and engineering. Like institutes focused on
other scientific disciplines, the SciDAC Institutes will provide a
unique forum where fundamental issues affecting scientific discovery
through advanced computing can be discussed. The Institutes will also be
a focal point for training and educating the next generation of
high-performance computing experts.

## 5.1 Benefits of SciDAC Institutes

SciDAC Institutes will benefit DOE and the scientific community by
building and fostering a broader community of researchers who understand
the challenges of providing and using high-performance modeling and
simulation capabilities and are willing to address these problems
collaboratively. Specifically, these institutes will provide for the
following:

-   **Exploration of new scientific challenges.** An Institute will
    identify challenges in computational and computer science and
    provide strategic direction for researchers in the high-performance
    computing community.

-   **A venue for focused collaborations.** An Institute will enable new
    and focused collaborations between a broad community of researchers,
    including SciDAC researchers, researchers from other agencies (e.g.,
    NSF) and from the broader applications community to develop new
    insights and approaches to solving challenging scientific problems.

-   **Broad dissemination of results.** An Institute will disseminate
    specific technical results through its activities. These include
    technical publications (articles, reports, books, and proceedings),
    advances in software infrastructure (algorithms, software,
    interfaces, and standards), and solutions to specific problems in
    applications of high-performance computing.

-   **A sense of community among computational scientists.** An
    Institute will foster a sense of community among researchers engaged
    in computational and computer science research and engage this
    community to help solve its most challenging problems.

-   **Education and training of the next generation of computational
    scientist.** An Institute will provide a key aspect in the training
    of the next generation of computer and computational scientists by
    engaging students as well as faculty and senior researchers in its
    programs.

To reap the benefits described above, a SciDAC Institute must gain a
reputation for instigating seminal research through ideas and specific
software development projects. The Institute must develop an identity
and reputation for leadership in computational science. Ultimately, the
strength of an Institute will be its ability to foster a community
around high-performance computing and to help solve some of this
community's most challenging problems.

## 5.2 The SciDAC Institutes Model

SciDAC Institutes will be loosely patterned on successful institute
models in other communities, such as the Aspen Institute for Physics and
the Mathematical Sciences Research Institute. It will not be necessary
for SciDAC Institutes to have a single location; however, facilities
used must be conducive to research and collaboration.

The Institutes will sponsor a variety of activities, which will vary in
format to satisfy different purposes and needs. Activities that might be
included in an Institute are the following:

-   **Critical topic programs** to address a particular problem of
    critical interest to the HPC community. These programs would last
    for an extended period of time. Graduate students and postdocs will
    attend the entire program, and faculty and laboratory research staff
    will spend possibly shorter periods of time advising, reviewing, and
    interacting with the students and postdocs.

-   **Short courses and summer schools** to give researchers at all
    levels training in new HPC concepts, technology, techniques, and
    software. These will provide graduate students, postdocs, and
    junior-level researchers access to skills not provided by
    traditional curriculums and will allow more senior researchers and
    industry the opportunity to acquire new skills.

-   **Collaborative workshops** to facilitate the interaction of
    researchers working on related problems. The primary focus of the
    collaborative workshops will be to present new ideas for future work
    and for discussions and interactions to develop collaborations and
    new directions. The collaborative workshops will also provide a
    forum for presenting recent results to a lesser extent.

-   **"Coding camps"** to facilitate rapid progress in the collaborative
    development of software and facilitate technology transfer between
    projects. Camps would typically be from one to two weeks and
    attended by the developers of software projects.

Staff will include a director, who would have overall responsibility for
the Institute, overseeing both scientific and administrative aspects.
Support staff will include a small number of full-time and part-time
administrative staff to handle activity proposals, correspondence,
travel and housing arrangements, and communications, such as web site,
newsletters, and reports. The organization of the Institutes will also
include a steering committee or advisory panel representing the DOE
computational science community to assist the director in setting the
scientific directions for the institute. Careful consideration must be
given to the process by which activities for each SciDAC Institute are
proposed, evaluated, selected, and executed. For example, the SciDAC
Institutes will *not* include conferences, research projects tied to a
specific program or call, and activities that result in intellectual
property.

# 6. SciDAC-2 Requirements for Computing and Networking Facilities

The goals of SciDAC-2 and the computational and networking goals of the
DOE programs are tightly linked. This plan relies on the availability of
computing and networking facilities to be provided by other program
components of the Office of Science in order to enable computational
science under SciDAC-2. As a consequence, while our vision for SciDAC-2
does not include any large capability or capacity computing systems or
any broadly deployed increase in networking capability, fulfilling the
goals of SciDAC-2 requires significant deployment of computational and
networking resources on the part of the DOE programs. The additional
requirements for SciDAC-2-level computational science have been
developed elsewhere.

## 6.1 Computing Requirements

Future computational requirements for many of the scientific
applications under SciDAC-2 have been investigated in detail in the
"Federal Plan for High-End Computing," a report of the High-End
Computing Revitalization Task Force (HECRTF; see www.cra.org) and in the
SCaLeS report \[2\].

The HECRTF solicited input from leading applications scientists in a
variety of disciplines relevant to the SciDAC program who use high-end
computing to advance their research. They were asked to identify
important scientific challenges addressable by high-end computing and to
estimate the additional computational capability (as a multiple of
present high-end capability) needed to achieve the goal. A summary of
the breadth of opportunity for such scientific and technological
advances from this HECRTF report can be found in Table 1. The estimates
of additional capability needed to achieve the goals ranged from 100 to
1,000 times the current capability of today's high-end computing
resources. Examples of the detailed analysis undertaken by these
applications researchers can be found within the full HECRTF report.

Table 1 Computing needs for scientific discovery.

+----+----------+------------------------+-----------------------------+
| ** | **Appli  | **SciDAC-2 Science     | **Potential Outcome with    |
| Ar | cation** | Challenge**            | 100 to 1,000 times Current  |
| ea |          |                        | Capability**                |
| ** |          |                        |                             |
+----+----------+------------------------+-----------------------------+
| >  | Astr     | Simulation of          | Yield understanding of the  |
|  * | ophysics | astrophysical          | conditions leading to the   |
| *P |          | environments such as   | origin of the heavy         |
| hy |          | stellar interiors and  | elements in the universe    |
| si |          | supernovae             |                             |
| cs |          |                        |                             |
| ** |          |                        |                             |
+----+----------+------------------------+-----------------------------+
|    | Hig      | Achieve detailed       | Guide experiments to        |
|    | h-Energy | understanding of the   | identify transition from    |
|    | Physics  | effects of strong      | quantum chromodynamics to   |
|    |          | nuclear interactions   | quark-gluon plasma          |
|    |          | so that the validity   |                             |
|    |          | of the Standard Model  |                             |
|    |          | can be tested to       |                             |
|    |          | determine whether      |                             |
|    |          | physics beyond the     |                             |
|    |          | Standard Model occurs  |                             |
|    |          | at extreme sub-nuclear |                             |
|    |          | distances              |                             |
+----+----------+------------------------+-----------------------------+
|    | Acc      | Accurate simulations   | Optimize the design,        |
|    | elerator | of the performance of  | technology, and cost of     |
|    | Physics  | particle accelerators  | future accelerators, and    |
|    |          |                        | use existing accelerators   |
|    |          |                        | more effectively and        |
|    |          |                        | efficiently                 |
+----+----------+------------------------+-----------------------------+
|    | Nuclear  | Realistic Simulations  | By developing a             |
|    | Physics  | of the characteristics | quantitative understanding  |
|    |          | of the quark-gluon     | if the behavior of this new |
|    |          | plasma                 | phase of nuclear matter,    |
|    |          |                        | facilitate its experimental |
|    |          |                        | discovery in heavy ion      |
|    |          |                        | collisions                  |
+----+----------+------------------------+-----------------------------+
| >  | Catalyst | Calculations of        | Reduce energy costs and     |
| ** | Science  | homogeneous and        | emissions associated with   |
| Na | /N       | heterogeneous catalyst | chemicals manufacturing and |
| no | anoscale | models in solution     | processing.                 |
| -s | Science  |                        |                             |
| ci | and      |                        | Meet federally mandated     |
| en | Te       |                        | NO~x~ levels in automotive  |
| ce | chnology |                        | emissions                   |
| ** |          |                        |                             |
+----+----------+------------------------+-----------------------------+
|    | N        | Simulate the operation | Take miniaturization of     |
|    | anoscale | of nanoscale           | electronic devices to a     |
|    | Science  | electronic devices of  | qualitatively new level     |
|    | and      | modest complexity      | enabling faster computers,  |
|    | Te       |                        | drug delivery systems, and  |
|    | chnology |                        | consumer and military       |
|    |          |                        | electronics                 |
+----+----------+------------------------+-----------------------------+
|    | N        | Simulate and predict   | Enable the discovery and    |
|    | anoscale | mechanical and         | design of new advanced      |
|    | Science  | magnetic properties of | materials for a wide        |
|    | and      | simple nanostructured  | variety of applications     |
|    | Te       | materials              | potentially impacting a     |
|    | chnology |                        | wide range of industries    |
+----+----------+------------------------+-----------------------------+
| >  | St       | Simulations of enzyme  | Provide ability to          |
| ** | ructural | catalysis, protein     | discover, design and test   |
| Li | and      | folding, and transport | pharmaceuticals for         |
| fe | Systems  | of ions through cell   | specific targets and to     |
| >  | Biology  | membranes              | design and produce hydrogen |
| Sc |          |                        | and other energy feedstock  |
| ie |          |                        | more efficiently            |
| nc |          |                        |                             |
| es |          |                        |                             |
| ** |          |                        |                             |
+----+----------+------------------------+-----------------------------+
|    | Signal   | Develop atomic-level   | Yield understanding of      |
|    | Tran     | computational models   | initiation of cancer and    |
|    | sduction | and simulations of     | other diseases and their    |
|    | Pathways | complex biomolecules   | treatments on a molecular   |
|    |          | to explain and predict | level; and the prediction   |
|    |          | cell signal pathways   | of changes in the ability   |
|    |          | and their disrupters   | of microorganisms to        |
|    |          |                        | influence natural           |
|    |          |                        | biogeochemical cycles such  |
|    |          |                        | as carbon cycling and       |
|    |          |                        | global change               |
+----+----------+------------------------+-----------------------------+
| >  | Climate  | Resolve additional     | Provide U.S. policymakers   |
|  * | Science  | physical processes     | with leading-edge           |
| *E |          | such as ocean eddies,  | scientific data to support  |
| ar |          | land use patterns, and | policy decisions            |
| th |          | clouds in climate and  |                             |
| >  |          | weather prediction     | Improve understanding of    |
|  a |          | models                 | climate change mechanisms   |
| nd |          |                        | and reduce uncertainty in   |
| >  |          |                        | predictions of climate      |
|  A |          |                        | change                      |
| tm |          |                        |                             |
| os |          |                        |                             |
| ph |          |                        |                             |
| er |          |                        |                             |
| ic |          |                        |                             |
| >  |          |                        |                             |
| Sc |          |                        |                             |
| ie |          |                        |                             |
| nc |          |                        |                             |
| es |          |                        |                             |
| ** |          |                        |                             |
+----+----------+------------------------+-----------------------------+
|    | Weather  | Enable dynamical       | Provide critical support to |
|    | and      | prediction of          | deployed naval, air, and    |
|    | Sh       | frequency and          | land forces in local,       |
|    | ort-term | intensity of           | regional, and global combat |
|    | Climate  | occurrence of          | environments: Lives saved   |
|    | Pr       | hurricanes/typhoons,   | and economic losses avoided |
|    | ediction | and severe winter      | due to better severe        |
|    |          | storms 90 days in      | weather prediction          |
|    |          | advance                |                             |
+----+----------+------------------------+-----------------------------+
|    | Solid    | Improved statistical   | Provide prioritized         |
|    | Earth    | forecasting of         | retrofit strategies         |
|    | Science  | earthquake hazards     |                             |
|    |          | (fault-rupture         | Reduced loss of life and    |
|    |          | probabilities and      | property                    |
|    |          | ground motion)         |                             |
|    |          |                        | Damage mitigation           |
+----+----------+------------------------+-----------------------------+
|    | Space    | Realistically simulate | Provide decision makers     |
|    | Science  | explosive events on    | (both civilian and          |
|    |          | the sun, the           | military) with status and   |
|    |          | propagation of the     | accurate predictions of     |
|    |          | energy and particles   | space weather events on     |
|    |          | released in the event  | time scales of hours to     |
|    |          | through the            | days                        |
|    |          | interplanetary medium, |                             |
|    |          | and their coupling to  |                             |
|    |          | Earth's magnetosphere, |                             |
|    |          | ionosphere, and        |                             |
|    |          | thermosphere           |                             |
+----+----------+------------------------+-----------------------------+
| >  | Su       | Simulate the fate and  | Predict contaminant         |
| ** | bsurface | transport of           | movement in soils and       |
| En | Conta    | radionuclides and      | groundwater and provide a   |
| er | mination | organic contaminants   | basis for developing        |
| gy | Science  | in the subsurface      | innovative technologies to  |
| >  |          |                        | remediate contaminated      |
|  a |          |                        | soils and groundwater       |
| nd |          |                        |                             |
| >  |          |                        |                             |
|  E |          |                        |                             |
| nv |          |                        |                             |
| ir |          |                        |                             |
| on |          |                        |                             |
| me |          |                        |                             |
| nt |          |                        |                             |
| ** |          |                        |                             |
+----+----------+------------------------+-----------------------------+
|    | Magnetic | Optimize balance       | Support U.S. decisions      |
|    | Fusion   | between self-healing   | about future international  |
|    | Energy   | of plasma and heat     | fusion collaborations       |
|    |          | leakage caused by      |                             |
|    |          | electromagnetic        | Integrated simulations of   |
|    |          | turbulence             | burning plasma crucial for  |
|    |          |                        | quantifying prospects for   |
|    |          |                        | commercial fusion           |
+----+----------+------------------------+-----------------------------+
|    | Co       | Understand             | Understand detonation       |
|    | mbustion | interactions between   | dynamics (e.g., engine      |
|    | Science  | combustion and         | knock) in combustion        |
|    |          | turbulent fluctuations | systems                     |
|    |          | in burning fluid       |                             |
|    |          |                        | Solve the "soot" problem in |
|    |          |                        | diesel engines              |
+----+----------+------------------------+-----------------------------+

## 6.2 Networking Requirements

Future networking requirements are discussed in the report "DOE Science
Networking Challenges: Roadmap to 2008" \[3\]. Some of these networking
requirements are summarized in Table 2.

As evidenced by ESnet traffic, networking bandwidth usage has increased
by a factor of 10 every 46 months since 1990, and all indications are
that this exponential will continue through the period of SciDAC-2.
However, making wide-area network "pipes" bigger will solve only part of
the problem. To achieve S&T goals, the end-to-end problem must also to
be solved; that is, the data must make it all the way from the end
source to the end sink at the desired speeds and with the desired levels
of quality of service. The source drivers are diverse: all of the
variety of high-end computational systems, experimental raw data
sources, plus massive data stores. Examples of data sinks include
high-end computational systems, massive data stores, visualization
systems, and a nearly infinite sea of low-end computational systems.

The challenge is to integrate the science requirements, end sources and
sinks, transport protocols, jitter control, packet switching, circuit
switching, routing, WAN components, etc. into a system the meets the
end-to-end requirements and thereby the overall science goals. The
SciDAC-2 approach is ideal for meeting this challenge by optimally
enhancing scientific discovery potential by vertically integrating
networking capabilities from science applications at the top down
through fundamental network technologies underneath.

Table 2. Networking Requirements

  -------------- -------------- -------------- ---------------- -----------------
  **Science      **Today        **5 years      **5-10 Years     **Remarks**
  Areas**        End2End        End2End        End2End          
                 Throughput**   Throughput**   Throughput**     

  High Energy    0.5 Gb/s       100 Gb/s       **1000 Gb/s**    High-bulk
  Physics                                                       throughput

  Climate (Data  0.5 Gb/s       160-200 Gb/s   **N x 1000       High-bulk
  & Computation)                               Gb/s**           throughput

  SNS            Not yet        1 Gb/s         **1000 Gb/s +    remote control
  NanoScience    started                       QoS for control  and time-critical
                                               channel**        throughput

  Fusion Energy  0.066 Gb/s\    0.198 Gb/s\    **N x 1000       Time-critical
                 (500 MB/s      (500MB/\       Gb/s**           throughput
                 burst)         20 sec. burst)                  

  Astrophysics   0.013 Gb/s\    N\*N multicast **1000 Gb/s**    computational
                 (1 TBy/week)                                   steering and
                                                                collaborations

  Genomics Data  0.091 Gb/s\    100s of users  **1000 Gb/s +    High-throughput
  & Computation  (1 TBy/day)                   QoS for control  and steering
                                               channel**        
  -------------- -------------- -------------- ---------------- -----------------

# 7. Budgets

ASCR currently supports basic research in applied mathematics and
computer science, and operates advanced computing and networking
facilities. In FY04 the SciDAC component of the ASCR budget was about
\$39M. This was complemented by about \$20M SciDAC funding in the other
program offices of the Office of Science supporting computational
science projects. We recommend a SciDAC-2 budget in ASCR that increases
from \$39M/year to \$80M/year starting in FY07, with the expectation
that a commensurate increase in funding request will augment the
computational science project budgets in the other SC offices in order
to take advantage of the new opportunities provided by SciDAC-2.

This FY07 funding request will enable ASCR to increase significantly the
number of scientific results that can be obtained through advanced
computing. By connecting to experimental science, the use of enabling
computing technology developed and deployed by ASCR will increase the
scientific productivity of all SC facilities.

In the category of "Integrated Software Infrastructure Centers" (ISICs),
an increase of funding from \$18M (FY04) to \$31M (FY07) is recommended.
Funding at the current level supports three mathematics and four
computer science ISICs. The increased level would support about 10 ISICs
that will be selected in a competitive process, with the expectation
that the new ISICs will focus on activities supporting experimental
science. Recompetition of existing ISICs may allow funding for new
opportunities in mathematics and computer science (such as
visualization) that were insufficiently covered in SciDAC-1.

In the category of "Scientific Applications Partnerships" (SAPs), an
increase of funding from \$14M to \$40M is recommended. This budget
level will support both the expansion of the current SAP program to
encompass additional DOE science areas (\~\$6M+current \$14M=\$20M) and
the development of new SAP activities relating to experimental science,
at a comparable level (\~\$20M).

In the category of "Software Infrastructure Management and Support
(SIMS)," new funding at the level of \$5M is recommended. Funding for
this new component of the SciDAC program will be used to build and
support test environments, coordinate software releases, implement
mechanisms to facilitate porting and tuning, and generally ensure that
SciDAC tools are robust and widely available on diverse platforms.

Finally, in the category of "SciDAC Institutes," new funding at the
level of \$4M is recommended. The Institutes are presented as a new
component of the SciDAC program, providing a forum for researchers from
multiple disciplines to address key scientific issues. An initial annual
funding of \$4M will support administrative and technical costs,
facilities, and travel, as well as conference expenses associated with
the events the Institutes sponsor. It is envisioned that, initially,
approximately eight events will be held per year, focusing on frontier
science.

# 8. Relationships with Other Agencies and Programs

DOE has a specific and unique mission. At the same time, DOE activities
often interact with and complement activities in other federal agencies
or national programs. This section briefly discusses such interactions.

## 8.1 Interactions with Other Agencies

While many agencies (including NSF, DOD, NASA, NOAA, and NIH) have
computational science and engineering programs, DOE brings complementary
strengths to these efforts, motivated by its strong mission needs and
supported by deep expertise at its laboratories.

-   DOE, like DOD, NSAS, NOAA, and NIH, is focused on scientific
    problems critical to its mission; NSF, on the other hand, has a
    broad scientific portfolio. For example, in computer science and
    applied mathematics, DOE funds only activities critical to
    scientific computing, while NSF supports a wide range of activities
    important to the advancement of information technology as a whole.

-   In fulfillment of its mission, DOE often establishes large,
    nationwide, integrated programs aimed at solving its most
    challenging scientific and engineering problems, problems not unlike
    those outlined here. NSF, on the other hand, seeks to stimulate the
    creativity of single investigators or small, local research groups.

-   To achieve its goals, DOE, like the other mission agencies, must
    support the full life cycle (research, development, deployment, and
    support) of the major software systems created in its research
    programs. This is a responsibility beyond the traditional scope of
    research but is essential to realizing the full potential of
    advances in computing technology for scientific discovery.

-   In each of these three areas, DOE has distinct and often unique
    mission needs that require focused action. Close coordination with
    other agencies is important to ensure effective technology transfer
    both to DOE from other agencies (e.g., basic research results from
    NSF) and from DOE to other agencies (e.g., new computational methods
    and software to NIH and DOD).

## 8.2 Relationship to Other DOE Programs

The investments in modeling and simulation outlined here complement the
investments in the experimental programs supported by DOE. The modeling
and simulation effort will draw heavily on DOE's experimental programs
to provide basic insights into and data on complex systems as well as
the fundamental processes that govern their behavior. The experimental
programs also provide the means of validating the computational models
and simulations, and validation is essential to advancing the stare of t
the art in scientific simulation. In turn, the computational modeling
and simulation will do the following:

-   Provide insights into fundamental physical, chemical, and biological
    processes than would otherwise be unattainable

-   Maximize the return on investments in experimental facilities by
    more quickly and efficiently harvesting scientific results

-   Provide the technical capability to design and construct innovative,
    new experimental facilities.

Computational modeling and simulation provide a strong link between the
Office of Science and the other DOE offices. By demonstrating how
real-world devices and systems can be simulated from a knowledge of the
fundamental physical, chemical, and biological processes involved, the
investment in scientific computing connects the basic research programs
in Office of Science with the energy and environmental research and
development programs in the offices of Fossil Energy, Energy Efficiency,
and Environmental Management.

# References

1\. "Accelerator Technology for the Nation," U.S. Department of Energy,
Office of Science,
http://www.er.doe.gov/Sub/Occasional_Papers/8-Occ-Accelerator-Technology-for-the-Nation.pdf.

2\. Phillip Colella, Thom H. Dunning, Jr., William D. Gropp, and David
E. Keyes, eds., "A Science-Based Case for Large-Scale Simulation"
(Washington, D.C.: DOE Office of Science, July 30, 2003).

3\. Roy Whitney and Larry Price, eds., "DOE Science Networking
Challenges: Roadmap to 2008" (Washington, D.C.: DOE Office of Science,
2003).

# Participants

Following is a draft list of participants in the SciDAC-2 planning.

+-----------------------------------+----------------------------------+
| > Abdulla, Ghaleb                 | LLNL                             |
+-----------------------------------+----------------------------------+
| > Agarwal, Deborah                | LBNL                             |
+-----------------------------------+----------------------------------+
| > Ahern, Sean                     | ORNL                             |
+-----------------------------------+----------------------------------+
| > Allcock, William                | ANL                              |
+-----------------------------------+----------------------------------+
| > Anderson, Gordon                | PNNL                             |
+-----------------------------------+----------------------------------+
| > Apra, Edoardo                   | PNNL                             |
+-----------------------------------+----------------------------------+
| > Armstrong, Robert               | SNL                              |
+-----------------------------------+----------------------------------+
| > Ashby, Steven                   | LLNL                             |
+-----------------------------------+----------------------------------+
| > Babnigg, Gyorgy                 | ANL                              |
+-----------------------------------+----------------------------------+
| > Bailey, David                   | LBNL                             |
+-----------------------------------+----------------------------------+
| > Bair, Ray                       | ANL/U of Chicago                 |
+-----------------------------------+----------------------------------+
| > Basney, James Alan              | University of Illinois           |
+-----------------------------------+----------------------------------+
| > Batchelor, Don                  | ORNL                             |
+-----------------------------------+----------------------------------+
| > Beckman, Pete                   | ANL                              |
+-----------------------------------+----------------------------------+
| > Bernholdt, David                | ORNL                             |
+-----------------------------------+----------------------------------+
| > Bethel, Wes                     | LBNL                             |
+-----------------------------------+----------------------------------+
| > Bland, Buddy                    | ORNL                             |
+-----------------------------------+----------------------------------+
| Bowers, Kevin                     | D. E. Shaw Research and          |
|                                   | Development                      |
+-----------------------------------+----------------------------------+
| > Bownas, Jennifer                | ORNL                             |
+-----------------------------------+----------------------------------+
| > Brightwell, Ron                 | SNL                              |
+-----------------------------------+----------------------------------+
| > Brown, David L.                 | LLNL                             |
+-----------------------------------+----------------------------------+
| > Camp, William                   | SNL                              |
+-----------------------------------+----------------------------------+
| Carter, Jonathan                  | NERSC                            |
+-----------------------------------+----------------------------------+
| > Cary, John                      | UC Boulder                       |
+-----------------------------------+----------------------------------+
| > Cattaneo, Fausto                | U of Chicago                     |
+-----------------------------------+----------------------------------+
| > Childers, Lisa                  | ANL                              |
+-----------------------------------+----------------------------------+
| > Colella, Phil                   | LBNL                             |
+-----------------------------------+----------------------------------+
| Collart, Frank                    | ANL                              |
+-----------------------------------+----------------------------------+
| > Collis, Scott                   | SNL                              |
+-----------------------------------+----------------------------------+
| > Cook, Kem                       | LLNL                             |
+-----------------------------------+----------------------------------+
| Critchlow, Terence                | LLNL                             |
+-----------------------------------+----------------------------------+
| > Davenport, James                | BNL                              |
+-----------------------------------+----------------------------------+
| > DeLuca, Cecelia                 | NCAR                             |
+-----------------------------------+----------------------------------+
| > Diachin, Lori                   | LLNL                             |
+-----------------------------------+----------------------------------+
| > Dongarra, Jack                  | U of Tennessee                   |
+-----------------------------------+----------------------------------+
| Donohue, Timothy                  | U of Wisconsin-Madison           |
+-----------------------------------+----------------------------------+
| > Drake, John                     | ORNL                             |
+-----------------------------------+----------------------------------+
| > Drummond, Leroy (Tony)          | LBNL                             |
+-----------------------------------+----------------------------------+
| > Feng, Wu                        | LANL                             |
+-----------------------------------+----------------------------------+
| Foster, Ian                       | ANL                              |
+-----------------------------------+----------------------------------+
| Frank, Edward                     | ANL                              |
+-----------------------------------+----------------------------------+
| Fredrickson, Jim                  | PNNL                             |
+-----------------------------------+----------------------------------+
| > Gannon, Dennis                  | Indiana U                        |
+-----------------------------------+----------------------------------+
| > Garimella, Rao                  | LANL                             |
+-----------------------------------+----------------------------------+
| Geist, Al                         | ORNL                             |
+-----------------------------------+----------------------------------+
| > Giometti, Carol                 | ANL                              |
+-----------------------------------+----------------------------------+
| Givan, Scott                      | Oregon State University          |
+-----------------------------------+----------------------------------+
| Glass, Elizabeth                  | ANL                              |
+-----------------------------------+----------------------------------+
| > Glimm, James                    | BNL/Stony Brook U                |
+-----------------------------------+----------------------------------+
| > Gorin, Andrey                   | ORNL                             |
+-----------------------------------+----------------------------------+
| > Gottschalk, Erik                | FNL                              |
+-----------------------------------+----------------------------------+
| > Gracio, Deborah                 | PNNL                             |
+-----------------------------------+----------------------------------+
| > Gropp, William                  | ANL/U of Chicago                 |
+-----------------------------------+----------------------------------+
| > Grossman, Jeff                  | Berkeley - UC                    |
+-----------------------------------+----------------------------------+
| > Hargrove, Paul                  | LBNL                             |
+-----------------------------------+----------------------------------+
| > Harrison, Robert                | ORNL                             |
+-----------------------------------+----------------------------------+
| > Hendrickson, Bruce              | SNL                              |
+-----------------------------------+----------------------------------+
| > Henson, Van                     | LLNL                             |
+-----------------------------------+----------------------------------+
| > Hovland, Paul                   | ANL                              |
+-----------------------------------+----------------------------------+
| Hules, John A.                    | LBNL                             |
+-----------------------------------+----------------------------------+
| > Istrail, Sorin                  | Brown U                          |
+-----------------------------------+----------------------------------+
| > Jackson, Keith                  | LBNL                             |
+-----------------------------------+----------------------------------+
| > Jakobsson, Eric                 | NCSA                             |
+-----------------------------------+----------------------------------+
| > Jefferson, David                | LLNL                             |
+-----------------------------------+----------------------------------+
| > Jones, Phil                     | LANL                             |
+-----------------------------------+----------------------------------+
| > Kazimierz, Gofron               | ANL                              |
+-----------------------------------+----------------------------------+
| > Keahey, Kate                    | ANL                              |
+-----------------------------------+----------------------------------+
| > Kendall, Ricky                  | ORNL                             |
+-----------------------------------+----------------------------------+
| > Kesselman, Carl                 | USC - ISI                        |
+-----------------------------------+----------------------------------+
| > Keyes, David E.                 | Columbia U                       |
+-----------------------------------+----------------------------------+
| > Khaleel, Mohammad               | PNNL                             |
+-----------------------------------+----------------------------------+
| > Kimmerly, Bill                  | PNNL                             |
+-----------------------------------+----------------------------------+
| > Knotek, Michael                 | Knotek Consulting                |
+-----------------------------------+----------------------------------+
| > Kolda, Tammy                    | SNL                              |
+-----------------------------------+----------------------------------+
| > Kolker, Eugene                  | BIATECH Nonprofit Research       |
|                                   | Center                           |
+-----------------------------------+----------------------------------+
| > Koontz, Annette                 | PNNL                             |
+-----------------------------------+----------------------------------+
| > Kruger, Scott                   | Tech X Corporation               |
+-----------------------------------+----------------------------------+
| > Kumfert, Gary                   | LLNL                             |
+-----------------------------------+----------------------------------+
| Langston, Mike                    | U of Tennessee                   |
+-----------------------------------+----------------------------------+
| > Larson, Jay                     | ANL                              |
+-----------------------------------+----------------------------------+
| > Lauret, Jerome                  | BNL                              |
+-----------------------------------+----------------------------------+
| Liebman, Michael                  | Windber Research Institute       |
+-----------------------------------+----------------------------------+
| Lightstone, Felice                | LLNL                             |
+-----------------------------------+----------------------------------+
| > Livny, Miron                    | U of Wisconsin                   |
+-----------------------------------+----------------------------------+
| Locascio, Phil                    | ORNL                             |
+-----------------------------------+----------------------------------+
| > London, Rich                    | LLNL                             |
+-----------------------------------+----------------------------------+
| Maltsev, Natalia                  | ANL                              |
+-----------------------------------+----------------------------------+
| > Mancini, Derrick                | ANL                              |
+-----------------------------------+----------------------------------+
| > Mann, Reinhold                  | ORNL                             |
+-----------------------------------+----------------------------------+
| > Marques, Osni                   | LBNL                             |
+-----------------------------------+----------------------------------+
| > Matz, Karen                     | PNNL                             |
+-----------------------------------+----------------------------------+
| > Mauth, Jeffrey                  | PNNL                             |
+-----------------------------------+----------------------------------+
| > McCune, Charles                 | PPPL                             |
+-----------------------------------+----------------------------------+
| > Mellor-Crummy, John             | Rice U                           |
+-----------------------------------+----------------------------------+
| > Meza, Juan                      | LBNL                             |
+-----------------------------------+----------------------------------+
| > Michaels, George                | PNNL                             |
+-----------------------------------+----------------------------------+
| > Middleton, Don                  | NCAR                             |
+-----------------------------------+----------------------------------+
| > Miller, Bart                    | Wisconsin University             |
+-----------------------------------+----------------------------------+
| > Minnich, Ron                    | LANL                             |
+-----------------------------------+----------------------------------+
| Mitchell, Julie                   | U of Wisconsin - Madison         |
+-----------------------------------+----------------------------------+
| > Mount, Richard                  | SLAC/Stanford                    |
+-----------------------------------+----------------------------------+
| > Mural, Richard                  | Celera Genomics                  |
+-----------------------------------+----------------------------------+
| > Myers, James                    | NCSA                             |
+-----------------------------------+----------------------------------+
| > Naughton, Thomas                | ORNL                             |
+-----------------------------------+----------------------------------+
| > Nichols, Jeff                   | ORNL                             |
+-----------------------------------+----------------------------------+
| > Oehmen, Christopher             | PNNL                             |
+-----------------------------------+----------------------------------+
| > Oldfield, Ron                   | SNL                              |
+-----------------------------------+----------------------------------+
| > Osni, Marques                   | LBNL                             |
+-----------------------------------+----------------------------------+
| > Papka, Michael                  | ANL                              |
+-----------------------------------+----------------------------------+
| > Parker, Steven                  | U of Utah                        |
+-----------------------------------+----------------------------------+
| > Peters, Carolyn                 | ANL                              |
+-----------------------------------+----------------------------------+
| > Peterson, Elena                 | PNNL                             |
+-----------------------------------+----------------------------------+
| > Petravick, Donald               | Fermilab                         |
+-----------------------------------+----------------------------------+
| > Pieper, Gail                    | ANL                              |
+-----------------------------------+----------------------------------+
| > Plimpton, Steve                 | SNL                              |
+-----------------------------------+----------------------------------+
| > Popescu, Razvan                 | BNL                              |
+-----------------------------------+----------------------------------+
| > Pordes, Ruth                    | FNL                              |
+-----------------------------------+----------------------------------+
| > Pouchard, Line Catherine        | ORNL                             |
+-----------------------------------+----------------------------------+
| > Quarrie, David                  | LBNL                             |
+-----------------------------------+----------------------------------+
| > Rahn, Larry                     | SNL                              |
+-----------------------------------+----------------------------------+
| > Rintoul, Mark                   | SNL                              |
+-----------------------------------+----------------------------------+
| > Rotenberg, Eli                  | LBNL                             |
+-----------------------------------+----------------------------------+
| > Roux, Benoit                    | Cornell U                        |
+-----------------------------------+----------------------------------+
| > Ryne, Rob                       | LBNL                             |
+-----------------------------------+----------------------------------+
| > Sadayappan, P.                  | Ohio State U                     |
+-----------------------------------+----------------------------------+
| > Samatova, Nagiza F.             | ORNL                             |
+-----------------------------------+----------------------------------+
| > Saphir, William C.              | LBNL                             |
+-----------------------------------+----------------------------------+
| > Schilling, Christophe           | Genomatica, Inc.                 |
+-----------------------------------+----------------------------------+
| > Schissel, David                 | General Atomics                  |
+-----------------------------------+----------------------------------+
| > Schmidt, Mike                   | Iowa State U                     |
+-----------------------------------+----------------------------------+
| > Schulthess, Thomas              | ORNL                             |
+-----------------------------------+----------------------------------+
| > Schwan, Karsten                 | Georgia Tech                     |
+-----------------------------------+----------------------------------+
| > Shinn, Neal                     | SNL                              |
+-----------------------------------+----------------------------------+
| > Shoshani, Arie                  | LBNL                             |
+-----------------------------------+----------------------------------+
| > Simon, Horst D.                 | LBNL                             |
+-----------------------------------+----------------------------------+
| > Smith, Barry                    | ANL                              |
+-----------------------------------+----------------------------------+
| > Sofia, Heidi                    | PNNL                             |
+-----------------------------------+----------------------------------+
| > Steffen, Craig                  | U of Illinois/NCSA               |
+-----------------------------------+----------------------------------+
| > Stevens, Rick                   | ANL/U of Chicago                 |
+-----------------------------------+----------------------------------+
| > Straatsma, TP                   | PNNL                             |
+-----------------------------------+----------------------------------+
| > Strauss, Charlie                | LANL                             |
+-----------------------------------+----------------------------------+
| > Strelatz, Richard               | LANL                             |
+-----------------------------------+----------------------------------+
| > Studham, Scott                  | ORNL                             |
+-----------------------------------+----------------------------------+
| > Summers, Michael                | ORNL                             |
+-----------------------------------+----------------------------------+
| > Swain, William (Tom)            | U of Tennessee                   |
+-----------------------------------+----------------------------------+
| > Tang, William                   | PPPL/Princeton                   |
+-----------------------------------+----------------------------------+
| > Tipparaju, Vinod                | PNNL                             |
+-----------------------------------+----------------------------------+
| > Turner, David                   | LBNL                             |
+-----------------------------------+----------------------------------+
| > Uberbacher, Ed                  | ORNL                             |
+-----------------------------------+----------------------------------+
| > von Laszewski, Gregor           | ANL                              |
+-----------------------------------+----------------------------------+
| > Wells, Jack                     | ORNL                             |
+-----------------------------------+----------------------------------+
| > Wenaus, Torre                   | BNL                              |
+-----------------------------------+----------------------------------+
| > White, Julia                    | ORNL                             |
+-----------------------------------+----------------------------------+
| > White, Vicky                    | FNL                              |
+-----------------------------------+----------------------------------+
| > White, James (Trey) III         | ORNL                             |
+-----------------------------------+----------------------------------+
| > Whitney, Roy                    | JLab                             |
+-----------------------------------+----------------------------------+
| > Williams, Dean N.               | LLNL                             |
+-----------------------------------+----------------------------------+
| > Womble, David E.                | SNL                              |
+-----------------------------------+----------------------------------+
| > Yates, Kim                      | LLNL                             |
+-----------------------------------+----------------------------------+
| > Zacharia, Thomas                | ORNL                             |
+-----------------------------------+----------------------------------+
| > Zemla, Adam                     | LLNL                             |
+-----------------------------------+----------------------------------+
| > Zhang, Duan                     | LANL                             |
+-----------------------------------+----------------------------------+
| > Ziebarth, John P.               | LANL                             |
+-----------------------------------+----------------------------------+

# Appendix: SciDAC-2 Planning Workshop Reports

This appendix comprises summaries of the workshops as well as white
papers that contributed to the formulation of the plan.

## SciDAC Software Integration, Maintenance, and Support (SIMS)

**A report based on a workshop held at**

**Argonne National Laboratory,**

**July 18-19 2005**

**Introduction and Scope of the** **Program**

The US Department of Energy (DOE) has made a substantial investment in
the development and utilization of high end software for computational
resources across the DOE complex. This covers the full gamut of
application software in the various scientific domains, Mathematics
tools, and Computer Science infrastructure. The progress and development
of computational hardware is still following some form of Moore's law
with faster processors, larger memory footprints and more complex and
robust interconnection networks. To facilitate the utilization of
current and new hardware resources and the complex software
infrastructure being developed by the Scientific Discovery through
Advanced Computing (SciDAC) program, a center to deal with the software
integration, maintenance and support (SIMS) is essential to assist the
developer communities to interact with their respective user communities
and make use of the computing resources.

The SIMS program is envisioned as a distributed virtual center that
takes advantage of the economies of scale to facilitate software
integration, management, and support issues across the SciDAC software
portfolio of the Office of Science. There are essentially three classes
of users: 1) the Integrated Software Infrastructure Centers (ISICs)
currently mathematics and computer science based; 2) the scientific
application code developers (SAPD), this includes the SciDAC Scientific
Application Pilot projects (SAP); and 3) end scientific application
users (ESAPU). There are complex interactions among these user
communities, for example in some domains the ESAPU and SAPD communities
will be the same and for other domains the communities will be more
diverse. The center will necessarily function in concert with these user
communities toward common goals that serve to meet the needs of all
involved.

Each software development effort currently mitigates SIMS issues with
their own tools, procedures, and practices that have evolved over the
time of their project; all too often, this is an unfunded mandate or an
afterthought of the successful research project that has developed the
software. The main SIMS center functions will start with the
identification and assimilation of the best of current techniques for
mitigating these issues, and build upon them based on economies of
scale. These functions will include promulgating best practices and
procedures, developing, refining, and evolving those practices and
procedures, fostering developer and user communities via direct
interactions, training and mentoring, enabling services such as
automated build and regression procedures, acting as a focal point and
facilitator for information flow among stakeholder communities,
including interagency, interorganizational, and international moieties
and finally incorporating proven open-source practices where
appropriate. These functions will necessarily expand to meet
unanticipated requirements as the software portfolio expands to meet the
missions of the Office of Science.

Which software should be a part of the SIMS supported software? This is
one component of the center that must have some flexibility and obvious
ties to the funding profile for the program. Since the software adopted
by the SIMS center will have applicability to one or more of the three
user groups outlined above, their input on the viability and usefulness
of the software is important. A process will be required to select and
prioritize the most appropriate software to be embraced with the SIMS
resources. The selection process will have to be able to deal with the
full gamut of software from applications to the specific mesh topology
generation library and meet the needs of the majority of SIMS users. We
have developed "a draft selection criteria" that could be used as a
template for respondents to the program call.

**Draft Selection Criteria for SIMS Software**

A critical component of the SIMS center will be the software that is
chosen for support by the center. There is more software that has been
and will be developed by the SciDAC program than can be actively
supported by the SIMS center, so a set of criteria for prioritizing the
many candidate projects is critical.

Criteria for software selection:

-   *Widely applicable:* the software should be of demonstrated value to
    more than one group -- both application and 'downstream' tool
    providers are acceptable,

-   *Available:* the software should be freely downloadable, either from
    a SIMS software repository or from an institutional web page,

-   *Mature:* the software should have significant functionality that is
    stable. This does not imply that the software cannot evolve over
    time, but rather that it should not be in an early, prototype stage,

-   *Correct:* the software should accurately implement a functionality
    or algorithm. This should be demonstrated by the developers before
    the software is accepted for SIMS maintenance,

-   *Relevant:* the software should be applicable to current computing
    platforms and application needs,

-   *Documented:* because the primary mechanism by which users learn to
    use the software is through the documentation, significant and
    fairly complete documentation should be provided by the developers.

Additional criteria could include robustness, portability, and
interoperability, but it is assumed that the SIMS center is designed to
help address exactly these issues for candidate software projects.

An additional criterion is required of the software development team.
*They must be willing to interact extensively with the SIMS center
personnel and, as appropriate, adopt 'best practices' defined by the
center.* This is critical to ensuring the relevance of the SIMS center
by allowing the software developers to provide feedback to center
personnel. In addition, this will help promote the interoperability goal
of the center if "maintained" software has a common look and feel.

It is also important to ensure that the overall portfolio is balanced
among

-   *different classes of tools*: both mathematical and computer science
    tools should be supported; supporting only linear algebra tools or
    only discretization tools will not serve the broader SciDAC
    community, and

-   *mathematical or computer science tools and community application
    codes*: SciDAC has funded extensive software development of both
    general math and computer science tools and specific application
    codes that are intended for use by a broader community. Both types
    of software should be supported by SIMS. Software developed and used
    by only one group should not be supported by SIMS.

**Program Funding Profile and Model**

The funding profile for this program is straightforward in the sense
that the baseline funding required to field a distributed center to
cover just the Office of Advanced Scientific Computational Research
component of this effort should include enough effort to support a
significant fraction of the current ISICs and SAPs that are ready for
support efforts. As the selection criteria of the program is applied to
this class of software moieties, the projects that have the most need
will get funded in order of the prioritization. The model presented here
applies to any software entity regardless of the primary funding source
within the Office of Science and the SciDAC program.

The funding should follow a two-fold mechanism where funding goes from
DOE to the SIMS center and from DOE to the supported software effort
directly. The SIMS center should not be in the business of directly
funding efforts. The software supported by the SIMS center should be
reviewed on an annual basis by a group of users and software experts to

-   ensure that ongoing, supported projects are active in their
    maintenance efforts in conjunction with SIMS personnel and still
    relevant to SciDAC needs,

-   evaluate the level of support required by ongoing projects as
    projects may need less funding once they have incorporated SIMS
    technologies and move into a long-term maintenance mode,

-   ensure that new projects are included as they mature and become
    ready for SIMS support, and

-   ensure that the overall portfolio remains balanced.

**SIMS Center Functions**

The three main functions---integration, maintenance, and support---are
somewhat arbitrary delineations of the actual functions that go into the
program. At the workshop we divided the efforts of the participants
along these lines, knowing full well the multiple levels of synergism
around these three functions. The following subsections are from these
three breakout groups.

**[Integration]{.underline}**

**Issues to consider**

**Who are the customers - users or developers?**

The primary customers of the center are the developers of SciDAC
software. The users are customers as well, for they will take advantage
of many of the same services or productivity improvements motivated by
the developers. Potential metrics might be customer counts, locations,
and backgrounds.

**Software support versus software engineering**

Many of the critical interaction points between developers and users
similarly bridge support and software engineering. The needs of
developers span the areas of support and engineering, for example, in
bug reporting, tracking, and resolution. Additional examples include
build systems and testing environments.

**Centralized tools versus software repository**

An area of discussion was whether the center should provide centralized
services more like SourceForge (tools) or NetLib (repository). Because
of the emphasis on developers as customers, the center should provide
services more like the former, but such services are a superset that
could easily include software repositories and value added beyond
repositories.

**Standards versus best practices**

The center should not attempt to "enforce" standards on the customers,
but should strive to identify and propagate best practices. The center
should take advantage of *de facto* standards to provide re-usable and
interoperable services and knowledge. Beyond that, the center could be a
portal for the interface standards developed by domain experts for and
within SciDAC. Promulgating best practices and procedures should be the
function of the center. This document deliberately avoids the word
\"standards\" in its discussion, preferring instead words such as
"exemplar," "best-in-class," or "community standards." The rationale for
this choice is significant and should not be lost upon the reader. The
scientific computing community has seen many standards come and go, some
of them never achieving usefulness or even relevance. Moreover,
contention is inevitable when one group imposes standards on another.
Therefore we intentionally choose our wording to emphasize carrot over
stick. The ability for parties to pick-and-choose, enhance, or even
reject SIMS "exemplar" solutions in their specific context. Competitive
proposals should follow suit, emphasizing how they will advance
best-practices and help others over defining new hurdles for others to
achieve compliance.

**Developing tools versus providing services**

The emphasis of the center should be in providing services. Tool
development should be motivated by the needs of the customers and should
heavily leverage existing tools.

**Analogies with existing ISICs**

The center should be similar to an ISIC, in the sense that it is a
virtual team, roughly the size of an ISIC, with a primary focus intended
to serve the other SciDAC projects. A possible difference is the absence
of or reduced emphasis on a research agenda for the center itself. There
is a possible tension between the benefits of leveraging the center for
research in effective software engineering for scientific computing and
the potential distraction from the primary goal of practical support of
SciDAC developers and users.

**Training and marketing**

With the development of experts and knowledge bases associated with the
center, training should be an integrated and common service provided by
the center. Such training could be provided through conference
tutorials, workshops, hands-on workshops, Access Grid events, *etc.*
Promotion of the center tools, services, and expertise should be coupled
to the training to ensure that the center has the greatest positive
influence.

**What to Centralize**

**Sourceforge-like tools**

Sourceforge is a centralized collaborative tool to assist with the
development of software. This includes services like source revision
control systems, issue trackers, mailing lists, task lists, document and
web-page management, release tools, and discussion forums. Additional
needs for SciDAC might be specialized security features, flexible
support for licensing strategies, and additional support tools specific
to scientific computing or the SciDAC mission.

**Development testbeds**

The center could arrange for allocations at the major computing centers
and provide that time to developers for the purposes of porting,
software testing, and environment regression testing. The center could
also provide pools of various relevant systems for compilation and
testing. Beyond the hardware, the center could arrange for automated
testing on various systems, perhaps linked to source revision tools
hosted by the center.

**Binaries and source repository**

The center could augment the collaborative-development tools with
services to build software stacks relevant to SciDAC applications and
verify those stacks on relevant platforms. The components of the
verified software stacks could then be made available at a central
location. User would be able to retrieve the components with the
knowledge that they work together.

To enable the users to take full advantage of this integration work, the
center could provide tools to install and remove components on relevant
systems. These tools could automatically retrieve software components,
dependent components, and updates. Such tools should leverage existing
best practices.

**Stewardship for orphaned software**

A major issue that motivated SIMS was orphaned software, where a
research project creates useful software but is not funded to maintain
it (and instead is funded to do other research). It is unrealistic for
the center to take full responsibility for providing support for an
arbitrary collection of orphaned software, but the center would be in a
unique position to make a positive impact.

By hosting tools for collaborative development and maintenance of
software, and by attracting SciDAC projects and user communities, the
center would be storing the main artifacts and knowledge bases for
SciDAC software. This situation offers great flexibility and protection
for software when the original developers cannot continue to maintain
it. The same tools that helped the original developers could allow other
interested parties to continue or renew support and maintenance with
much of the historical investment intact. These parties could be
volunteers, stakeholders, other funded projects, or center staff. The
central availability of the software artifacts and knowledge bases thus
provides an "insurance policy" for orphaned software and a strong
incentive for development projects to take advantage of the centralized
tools.

Software would thus have a good base level of support "built in" through
previous use of the tool infrastructure, plus a flexible environment for
increasing levels of support. The center could choose to fund more
services, as described elsewhere, for critical orphaned software.

**Common build machinery**

The center could provide expertise in build environments, and could
further enable interoperability of build environments for different
software components, particularly for components that depend on one
another. One mechanism for greater interoperability is the definition of
common macros for configuration. Software that is dependent on
particular libraries or tools could then rely on these macros to
simplify the build process. Dealing with mixed-language build
environments across relevant platforms is another area where expertise
and tools could be centralized and reused.

**Portal to SciDAC interface definitions**

A primary goal of various SciDAC projects is the definition of standards
to promote interoperability of software components. The center could
provide a portal for these standards, so that developers have a one-stop
shop for useful interfaces for their own software projects.

**Associated experts**

Experts in various areas of software engineering and support could be
associated with the center. The association could be as a funded member
of the center, a volunteer stakeholder, or an ISIC member. The center
should leverage such experts in multiple ways and incorporate positive
feedbacks. For example, an expert might first help a particular SciDAC
project meet a particular goal. The expertise developed by this work
should then be documented and used by the center to develop similar
solutions for other SciDAC projects. Results could include case studies
and tool products. The center could further provide training based on
these results.

**What to Not Centralize**

The integration working group thought that it was inappropriate for the
center itself to be the distribution point for funding of SciDAC
software support. The "right" answer is for maintenance and support to
be an expected, reviewed, and funded component of each SciDAC proposal.
We had some discussion of using only a portion of the funding set aside
for SIMS to be used for a center, with the remaining funding distributed
to SciDAC projects through a separate proposal process.

**[Maintenance]{.underline}**

**Goals**

The maintenance activities in SIMS are intended to attenuate the effort
spent by tool developers in activities other than the ones in their
software development agenda. These activities include the definition of
mechanisms for bug tracking and mitigation, coordination of maintenance
with continuous development, coordination of releases, revision control
and recording of main issues, bookkeeping of a question and answer
process for the deployment of scientific software, and the
implementation of mechanisms to facilitate porting and tuning.

The above activities can become very specific or unfold in a variety of
ways depending on the software tool. In fact, tools that are intended
for the solution of common problems in a scientific and engineering
applications (e.g. DOE ACTS Tools), domain specific tools (e.g. NWChem,
Chombo, ESMF) or more general tools (e.g. CCA software, Babel, PERC
tools, visualization tools) can have a variety of different requirements
for maintenance. The following is a nonexhaustive list of possible
requirements that we have identified: build mechanisms, configuration
management, bug tracking, user list, standard testing, porting, version
control, launch monitor, performance analysis and optimization,
compliance to standards, dependency management, standard certification
(documentation, peer reviewing), language interoperability,
collaboration environment, auto documentation (e.g., doxygen), standard
requirements tracking (e.g., Telelogic's DOORS), debuggers and
development tools, and benchmarking.

For SSMIS purposes, and SciDAC tools in particular, the above list can
be further specialized. In the sequence, we discuss maintenance
requirements that are important in the SciDAC/SSMIS context.

**Requirements**

[Mechanisms to facilitate portability:]{.underline}

Scientific collaboration in computational sciences is predicated on
interoperable software. Interoperable software, in turn, is predicated
on individual codes being portable. There is a specific activity of
"porting" which is generally understood to be the process of adding
support for the software to work on a new architecture (e.g., porting
from Linux to 64bit Linux, RedStorm, or BG/L). But it is also well known
among practitioners that porting codes between similar architectures in
different institutions is a non-trivial task. Machines may have similar
hardware but may differ in switching fabric, system configuration, or
even simply versions of installed software. Porting is an experiential
skill that cannot be captured and encoded in a text. Its exercise is all
about resolving clashes when the reality of the system deviates from the
programming model that the application developer had presupposed in
their implementation. A great deal of porting work is devoted not to the
scientific software itself but its auxiliary software (makefiles and
scripts) charged with converting its scientific software from its
general source code form to its machine-specific executable form. We
refer to this type of auxiliary software as "the build." Because of its
function, the build must be equipped to handle all requisite arcane for
porting. Proposals are tasked, therefore, to describe their approach for
fomenting and promulgating best-in-class build and portability tools and
techniques to the SciDAC community. We encourage them to provide
services to SciDACs wishing to increase the portability of their codes,
insofar as it can be achieved through software engineering mechanisms,
such as build tools. There are another classes of portability issues
excluded from SIMS purview that involve algorithmic changes (i.e.
porting between vector, shared, or distributed memory) of performance
tuning on new architectures (assumed to be purview of other ISIC). We
also allow that SIMS center may tie compliance to its build/portability
best practices to the level of support it provides to external
customers. Put another way, if a customer has an obscure/custom
mechanism to handle portability, there is no economy of scale to
motivate the SIMS center to provide support for it.

[Evaluation and improvement of maintenance tools:]{.underline}

There are hundreds of tools available to assist in software
maintenance. For example, there are 400 build tool projects and 335
version control projects listed at http://www.freshmeat.net. No single
development team can evaluate all, or even a significant fraction, of
the available tools. Therefore, one role for the SIMS Center is to
evaluate and provide a forum for SciDAC developers to discuss the
relative merits of various maintenance tools. Another activity for the
SIMS Center is to document several recommended tools for software
maintenance, including tools for automatic documentation, build tools,
version control tools, tools for testing, static analysis tools, and
tools for defect tracking. Some of the needed tools, for example tools
for code instrumentation or performance analysis, may be available
through or under development by the CS ISICs. In such cases, the SIMS
center can assist SciDAC developers in the use of these tools and
provide feedback to the CS ISICs on the requirements of the SciDAC
developers. Finally, the SIMS Center may be called upon to extend or
augment existing maintenance tools to meet the needs of SciDAC software
development processes.

[Application collaboration projects:]{.underline}

As the complexity of physics models, numerical algorithms, and
computational hardware increases, it becomes more difficult for a single
computationalist to have all of the knowledge to meet the challenges.
This provides incentives to not only have large computational projects,
but to have those projects develop a user-base. The issues of working in
large groups, and having a user-base are generally less familiar in the
applications area, and the experiences of the CS and math ISICs should
be communicated to the other offices. The main difference between the
application areas and the ISICs, is that the ISICs typically have both
\"downstream\" and \"upstream\" dependencies (e.g., many applications
depend on PETSc and PETSc depends on MPI); whereas the application areas
typically only have downstream dependencies. Despite this singular
difference, the lessons learned from SIMS have the potential to
dramatically increase the productivity of computationalists across the
Office of Science.

[Black box testing:]{.underline}

Ongoing maintenance of a software product necessitates a formal release
process. The objective of such a process is to ensure that each
successive release provides the specified functionality with an
acceptable confidence level. The details of the release criteria are
established in a certification protocol prescribing essential
verification and validation (V&V) activities. These activities generally
include peer review of design documentation, code inspection, and
software testing. The proposal should elaborate all-important components
of certification. For example key elements of design review, such as
requirements traceability, etc., should be identified. The range of code
review techniques should include both manual and automated techniques.
Certification testing should be addressed as a quantitative process for
establishing software fitness for use.

The proposed SIMS center will develop (possibly domain-specific)
templates for certification of releases. These templates will include
specific guidelines for establishing and executing software
certification for a software package. SIMS proposals should describe a
strategy for template development, validation, and application. The
template development strategy should outline the categories of V&V
practices to be included and potential sources for detailed definition
of those practices. The certification template validation strategy
should involve case studies and/or metrics collection. Validation should
be structured both to refine proposed practices and establish
credibility with the SciDAC developer community. Finally the proposal
should provide a plan for application of appropriate certification
templates to maintenance of selected current and all future SciDAC
projects. This plan may include direct involvement where SIMS staff
actively performs release certification tasks for selected packages
and/or an educational approach where the SIMS center provides training
and coaching in certification practices.

[Defining and communicating best practices:]{.underline}

The development of simulation software is in many cases driven by the
needs of applications. However, it is often the case that promising
technologies developed by the high end computing community may not yet
be in a form suitable for distribution and use among engineers and
domain scientists. Therefore, a careful review and analysis of
development practices is needed to make cutting edge software technology
suitable for distribution and adoption by a much wider and less expert
user population. The adoption of good practices by developers SciDAC can
also reduce the duplication of efforts. SIMS can achieve these goals by
promoting the sharing of software modules, by reducing the time in
locating relevant software and information through the use of
appropriate indexing and search mechanisms, by facilitating the adoption
of useful standards, and by promoting the use of the best available
software and libraries.

[Frontline for development groups:]{.underline}

The multidisciplinary nature of the SciDAC program often requires an
interchange of software and expertise among the various Applied
Mathematics and Computer Science ISICs. These development groups usually
have distinct timelines, expectations and needs. However, tardiness in
responding to the needs of a particular project may deter the
development process of such a project. This situation can ultimately
lead to difficulties to developers of scientific application codes that
rely upon the tools developed by the SciDAC ISICs as well as non-SciDAC
users of SciDAC ISIC and scientific application codes. Therefore, SIMS
will be a catalyst for the SciDAC ISICs in order to make sure that
inter-groups requirements are timely and properly addressed.

**[Support]{.underline}**

**Goals**

The goal of a software support infrastructure is to provide a place
where application and tool developers, and their users can interact and
find prompt and adequate solutions to and information on common software
development problems. The One-Stop-Shop software support infrastructure
will provide several different levels of services (see Figure 1),
produce written specification to facilitate the adoption by application
and tool development projects, build expertise in successful and common
practices, and support software in as many platforms as possible, but
support all platforms and configurations of these may be impossible. To
an extent, this software support infrastructure will also be a conduit
to application domain expertise.

**Target Communities**

SIMS support will be geared toward two target communities: the Software
Developers and the Users of SciDAC developments. Moreover, we foresee
that support infrastructure will have a greater impact as it will
indirectly reach out and benefit the computational sciences community at
large. In this context, the community of Software Development Projects
is represented by SciDAC tool and application development teams (see
Software Selection and Metrics sections). The Users community includes
SciDAC developers, non-SciDAC developers that are users of SciDAC
produced technology, and user support personnel at the DOE High
Performance centers.

**Support Tools and Services**

The SIMS center is envisioned to provide expertise and services to two
groups of scientists: those who develop software and those who use it.
It is expected that proposals will draw this expertise from both experts
employed by the SIMS center, and from the greater SciDAC community
enabled by an on-line facility or other mechanism maintained by the SIMS
center. In Figure 1, the SIMS function is broken down in terms of
whether the service targets SciDAC users or developers against whether
the service is provided by an SIMS funded expert or is community-based.

SciDAC software will be developed in a collaborative manner and an
important part of any SIMS proposal will be activities to enhance
interaction and community-based codes. The successful proposal will have
sufficient provision for community-based services including but not
restricted to:

-   Mailing lists

-   Wiki and collaborative text processing

-   On-line forums

-   Website maintenance for application centers

In addition direct resources for developer collaboration is encouraged,
such as SourceForge-like web-based developer and user resource, and code
collaboration servers that provide an on-line "meeting place" for
community development, trouble shooting, and the like.

![](media/image13.png){width="4.761805555555555in"
height="3.6881944444444446in"}

Figure 1. Chart of Distributed Support Services for the target
communities

Broad education and outreach activities are important. This may include
hosting workshops, coding camps, and tutorials that promote SciDAC
software adoption and promulgate best practices in software engineering.
Also salient are more passive forms of education including on-line
tutorials, documentation, FAQs, and so forth.

Software Download and Update (download + tracking) services. These
services will provide customers with access to code releases, as well as
providing information on updates. Furthermore, projects will have the
option of providing anonymous, read-only access to version control
systems for their project. The tracking services will provide developers
with quantitative estimates of the number of users of their code.

Assisting the developer community in developing standards for software
installation, downloading, version control, and using interfaces.

Assistance in the form of access and resources for reproducing problems
reported by users. Examples include:

-   Identifying issues such compiler problems, interoperability
    problems, conflicts between packages, etc.

-   Reproducing bugs on similar platforms to those of the users,
    particularly when the platform in question would not otherwise be
    available to the developers.

-   Cataloging of reported problems in order to identify patterns.

Tools to enable community support for software. These tools will
facilitate communication between the developers and end-users as well as
among the end-users themselves.

**Value Added to Developers**

A central bug and feature request interface for users will allow the
detection and identification of common problems and solutions. This
provides pre-emptive handling of problems, allowing developers to share
experiences and solutions rather than having each tool group
independently discover, diagnose, and solve problems related to problems
with compilers, operating systems, runtime middleware, or hardware
problems (such as problems related to power supplies). An additional
advantage of common bug reporting is the ability to serve as a central
contact point to work with software and hardware vendors to resolve
problems that are identified by the SIMS center.

Another service that the SIMS center will provide for developers is
timely updates on software issues relating to common tools, such as
updates to build tools, compilers, and runtime libraries. This is of
particular value to developers as updates to these tools may introduce
incompatibilities with previous releases (for examples, changes in the
semantics of the weak pragma in a C compiler or in the command line for
autoconf), and the identification of these issues, along with common,
tested workarounds, can significantly benefit developers.

The SIMS center should also coordinate feedback from DOE HPC facilities
to the developers of tools, both for bug reporting and feature requests.
In practice, many tool developers receive little if any feedback from
users at these centers. This feedback, particularly feature requests,
provides valuable information for tools developers that can guide future
development relevant to SciDAC users.

**Value Added to User Community**

SciDAC software development represents a large DOE investment, and broad
acceptance of the tools and applications produced by the SciDAC program
will leverage this investment across diverse scientific domains. More
important, the widespread use of SciDAC-funded software will greatly
benefit the scientific end-user. The SSMIS user support function will
provide an ideal means for providing this value to the end-users. A
proactive outreach effort will in effect "market" the software
components to SciDAC and non-SciDAC researchers. This can occur via
direct contact with interested scientists, collaboration with the
various DOE computer centers, and workshops. Education efforts will
focus on lowering the barriers to acceptance of the most widely
applicable software products.

Closely tied to the marketing function is that of software selection.
There can often be a bewildering amount of overlap between scientific
software tools. SIMS will provide users a source of qualifying
information to facilitate their choice of appropriate software.

End-users will also see great benefits in receiving comprehensive and
timely responses to their queries regarding SciDAC software. However, it
must be recognized that user support queries span a wide range of
priorities. Therefore the user support organization will define distinct
levels of response, with different expectations (response time, depth of
analysis, etc.) associated with each level. In this way users will
receive the most appropriate level of support for their individual
queries.

**Support Meets Maintenance**

Porting user software to all the different programming environments
supported by the center would be one of the support services provided by
this center. In addition to porting user software, the center would also
provide services to solve any problems the user may face in porting the
center's supported tools. Also, support services pertinent to
installation of software would also be provided by the center. The
center would also provide technical expertise about reliable software
development tools and platforms (e.g. compiler versions, OS versions,

Tool patches for a given environment) that are supported by the center
as development environments. Issues such as requests for build support
or failure of QA tests for supported software will also be addressed by
the center as support services. Services provided by the center for
support and maintenance have a clear synergy; above mentioned support
issues fall under such category as they have a clear overlap with
maintenance issues.

**Support Meets Integration**

The support tools and services that the central software center provides
will mesh with the various integration efforts of SIMS. Specifically,
the two efforts are encouraged to share a common defect tracking system
so that knowledge gained through integration activities may be queriable
by support staff. In addition, we recommend that documents created by
the two efforts share a common version control system and collaboration
web site facilitate communication. These technical communication systems
will enable the support center to identify cross-cutting solutions that
apply to new areas of support. It will also enable integration efforts
to identify high-priority software needs.

**Deliverables and Funding**

Setting the expectations of what the software support infrastructure can
deliver will be determine by level of funding and the way the
infrastructure is setup. Basically, the levels of funding will
determine:

• x-number of users to be supported by the infrastructure

• y-number of development teams that will be supported

Thus, given a budget amount, the software support infrastructure will
not be able to support whole collection of SciDAC software deliverables.
And under this assumption, there will be a support prioritization
process (see selection section above) by which the levels of support to
tools and application development projects will be determined. In
addition, the software support infrastructure should look at ways of
supporting legacy code that is still in used and valuable to the SciDAC
community.

**[Metrics for SIMS Success]{.underline}**

Proposals for the development and operation of the Center shall include
metrics to be used as part of a regular evaluation of the effectiveness
of the Center. Metrics should include both measures of the impact and
value of the Center to the target audience in addition to metrics for
evaluating internal operations. Illustrations of external target
variables include the following:

-   Number of users of the resources

-   Number of development programs that use the Center

-   Tracking bugs, user requests, responses

-   Time to response

-   Web site traffic and/or downloads

-   Procedures for soliciting and implementing community/user feedback

Proposals should include a sufficient level of detail to enable external
reviewers to evaluate proposed metrics, tools to capture the
information, procedures for evaluation and response to these measures,
and milestones.

The Center is responsible for self-promotion, outreach, and education
efforts to increase community and sponsor awareness and usage of the
Center resources. Metrics shall be identified that illustrate the level
of activity carried out in these areas and corresponding response of the
target audience to these efforts.

The operational effectiveness of the Center will be evaluated on a
regular basis. Proposals will include metrics for the following:

-   Document performance of service providers including, but not limited
    to, distributed components, for example, contracted build gurus

-   Illustrate evaluation of return on investment and processes to
    reallocate resources

-   Identify areas of growing/decreasing needs of users; Identify single
    points of failure in the Center

The sponsor will provide individual SciDAC projects with funding for
maintenance and support of users. The Center will maintain a database of
common metrics for each SciDAC project that receives maintenance and
support funds, documenting the projects' activities in these areas.
Proposals for development and operation of the Center shall include
metrics to be gathered from these groups and integration/analysis of
these metrics with Center statistics.

[Other Issues with Respect to Metrics]{.underline}

All proposals shall include a description of relationships, existing
resources or staffing, and capabilities that will enable the proposal
author(s) to successfully initiate and operate the Center.

**Workshop Participants**

Apra, Edoardo Armstrong, Robert

Pacific Northwest National Laboratory Sandia National Laboratory

edoardo.apra@pnl.gov rob@sandia.gov

Beckman, Pete Colella, Phillip

Argonne National Laboratory Lawrence Berkeley Laboratory

beckman@mcs.anl.gov PColella@lbl.gov

Diachin, Lori Drummond, Leroy (Tony)

Lawrence Livermore Nat'l Laboratory Lawrence Berkeley Nat'l Laboratory

diachin2@llnl.gov LADrummond@lbl.gov

Glimm, James Gropp, William

Stony Brook University Argonne National Laboratory

glimm@ams.sunysb.edu gropp@mcs.anl.gov

Hovland, Paul Kendall, Ricky

Argonne National Laboratory Ames Laboratory

hovland@mcs.anl.gov rickyk@scl.ameslab.gov

Keyes, David Kruger, Scott

Columbia University Tech X Corporation

david.keyes@columbia.edu kruger@txcorp.com

Kumfert, Gary Larson, Jay

Lawrence Livermore Nat'l Laboratory Argonne National Laboratory

kumfert@llnl.gov larson@mcs.anl.gov

Marques, Osni Minnich, Ron

Lawrence Berkeley Nat'l Laboratory Los Alamos National Laboratory

oamarques@lbl.gov rminnich@lanl.gov

Naughton, Thomas Schulthess, Thomas

Oak Ridge National Laboratory Oak Ridge National Laboratory

naughtont@ornl.gov schulthesstc@ornl.gov

Showerman, Mike Smith, Barry

National Center for Supercomputing Applications Argonne National
Laboratory

mshow@ncsa.uiuc.edu bsmith@mcs.anl.gov

Swain, William (Tom) Tipparaju, Vinod

University of Tennessee Pacific Northwest Nat'l Laboratory

swain@cs.utk.edu vinod@pnl.gov

Turner, David White, Julia

Lawrence Berkeley National Laboratory Oak Ridge National Laboratory

dpturner@lbl.gov whitejc@ornl.gov

White, III, James (Trey) Whitney, Roy

Oak Ridge National Laboratory Jefferson Lab

trey@ornl.gov whitney@jlab.org

-   

# 

## SciDAC-2 Integrated Software Infrastructure Centers (ISICs)

A Report from the SciDAC-2 Planning Committee

**Introduction:** Integrated Software Infrastructure Centers (ISICs) are
large, multi-institutional teams of mathematicians and computer
scientists chartered in SciDAC-1 to accelerate the development of
scientific codes, to achieve maximum efficiency on high-end computers,
and to enable a broad range of scientists to use simulation in their
research. These centers have been individually and collectively praised
for developing and delivering leading-edge computational technologies to
a diverse set of scientific applications. ISICs also have emerged as
national repositories of high performance scientific computing software
and expertise.

To plan for SciDAC-2, a significant amount of effort was expended to
gather input from the community on this important program element. In
particular, in October 2004, a workshop was held in Bodega Bay, CA that
focused on the future of the applied mathematics ISICs, and in February
of 2005, a smaller workshop was held in Washington, DC that focused on
the future of the computer science ISICs. Combined attendance at these
workshops was over 50 people and included mathematicians, computer
scientists, and application researchers. In addition, at the SciDAC 2005
conference in San Francisco an open panel discussion on the future of
SciDAC was held to gather input from the nearly 250 conference
attendees. The ISICs were highlighted in this discussion. Moreover, in
past 8 months, the ISIC PIs were asked to prepare white papers
discussing the future of the ISIC program. All of these discussions and
white papers provided valuable input for the creation of this report.

In addition, the SciDAC-2 planning committee held a virtual workshop on
this topic in July, 2005. Teleconference discussions were led by Steven
F. Ashby and existing SciDAC ISIC and application researchers, as well
as leading scientists in the broader community, were invited to
participate. Discussion of the aspects of the ISIC model that worked
well, along with the aspects that could be improved, was encouraged.
Input on topics of emphasis that would meet the needs of SciDAC-2
application researchers was solicited.

**Executive Summary of the Findings.** Through a holistic evaluation of
the ISICs' integrative role within OASCR and across the Office of
Science, we concluded that the current ISIC model is successful and that
the basic structure should be preserved and refined rather than
significantly overhauled. There are aspects that can be improved to
strengthen the ISICs, and we articulate this list as well as highlight
the relationship of the ISICs with the new program elements proposed for
SciDAC-2. We conclude with a list of topical areas of emphasis that
should be considered to meet application needs in the SciDAC-2 program.

**The OASCR Program**: OASCR's primary role in the Office of Science is
to *enable large-scale computational science*. To accomplish its
mission, OASCR has invested heavily in three different areas:

-   *High-end computing facilities (HEC)*: the development and
    deployment of leadership class computing and networking hardware
    infrastructure

-   *Enabling computational technologies (ECT):* the encapsulation of
    mathematics and computer science research results into robust
    software tools that are directly useable by computational science
    application teams (e.g., fusion energy science, climate modeling,
    and combustion) or necessary for the deployment of high end
    computing facilities (e.g., scalable systems software).

-   *Basic research (Research)*: advances in mathematics (e.g.,
    numerical algorithms, discretization methods, multiscale modeling
    techniques) and computer science (e.g., scalable operating systems,
    components technologies, performance metrics) that support longer
    term computational science goals.

Figure 1 shows the hierarchical relationship among these three program
elements. Basic research provides the foundation for all other
activities. Successful endeavors in basic research lead to the
development of robust enabling computational technologies that directly
support HEC facilities. In turn, new HEC architectures can drive basic
research in computer science and algorithm development. To a lesser
extent, HEC architectures drive ECT development, and ECT development can
lead to new areas of basic research. The SciDAC program, through the
mathematics and computer science ISICs, supports ECT activities to an
extent that was not previously possible. *The ISICs have become the
primary mechanism for encapsulating and delivering basic research
results to the HEC facilities. The ISICs thereby enable vertical
integration within the OASCR program.* This has been successful, in
part, because the ISICs collectively span a broad range of mathematics
and computer science topics. ***We conclude that the current ISIC model
works well to vertically integrate programs within OASCR and should be
preserved. Furthermore, it is important that the topics chosen in
SciDAC-2 be mutually interdependent and aligned with the mathematics and
computer science activities in the base research program.***

**OASCR's relationship to other SC programs**: The science directorates
within the Office of Science have program hierarchies analogous to those
that support the computational science goals in OASCR. The left pyramid
in Figure 2 shows how a typical science mission is accomplished through
leveraged investments in large-scale experimental facilities,
simulation, and theory. Consider, for example, the Office of Fusion
Energy Sciences (OFES). At the top of the FES investment triangle are
facilities such as the DIII-D tokomak device at General Atomics.
Large-scale experiments are conducted on this device and are supported
by sophisticated simulations that predict various physical phenomena
such as plasma confinement and transport. The simulations are supported
by theoretical research in the plasma sciences. Of course, the
experiments generate data that drive advances in simulation and theory.

There are linkages between all levels of the OASCR hierarchy and the
simulation components of the other offices. The SciDAC program,
primarily via the ISICs and the Scientific Application Partnerships
(SAP) program, has significantly increased the number and quality of
interactions between OASCR and the other SC offices. In particular, SC
application scientists rely on OASCR HEC and ECT to perform their
large-scale simulations. In many cases, these interactions have enhanced
simulation capabilities while simultaneously driving the development and
prioritization of OASCR enabling technologies. These interactions also
have engendered new areas of OASCR basic research, particularly in
applied mathematics and data management. *The ISICs have become one of
the primary means by which OASCR researchers interact with SC
application scientists; they also are the primary mechanism for
integrating OASCR capabilities into SC scientific applications. The
ISICs thereby enable horizontal integration across the Office of
Science.* This has been successful, in part, because the ISICs
collectively span a range of relevant topics, and each ISIC supports
more than one application team. This last point is important because it
avoids the tunnel vision that might result if the ISICs were vertically
integrated and aligned per application. This mode of interaction
requires active management, but it also results in more broadly
applicable and robust simulation technologies. ***The strong
horizontally integrative role the ISICs play within the Office of
Science supports the conclusion that the ISIC model be preserved.
However, the set of topics chosen in SciDAC-2 should be re-examined with
an eye toward meeting the needs of the full range of Office of Science
applications, especially the next-generation SciDAC applications.***

**Planning for the SciDAC-2:** The SciDAC program is extremely
successful: it has not only delivered on its scientific promises, but
also has helped to integrate OASCR vertically and the Office of Science
horizontally. The ISICs in the SciDAC-2 program should be focused on
developing key enabling computing technologies and be driven by strong
collaborations with application centers for research, development, and
deployment. They should preserve the best of the ISIC program from
SciDAC-1, but be enhanced to encourage stronger integration and
collaboration with application teams, with each other, and with the new
SciDAC program elements.

There are many aspects of the current ISIC model that are working well
and should be continued in SciDAC-2. In particular, creating
multi-institutional teams and providing long term funding stability for
core technology development has resulted in a strong collaborative
environment across the DOE complex and a set of integrated tools that no
single lab could have produced alone. To strengthen the ISICs, OASCR
should re-examine their focus areas to ensure that they collectively
span the basic mathematics and computer science research programs as
much as possible *and* are well-aligned with the future needs of SC
application scientists. Moreover, we believe that this is best done by
choosing a set of mutually interdependent topics as opposed to creating
vertically integrated ISICs.

We believe that the ISICs could be improved in the following ways.

First, *we advocate that the ISICs be given increased funding agility to
respond to opportunities to interact with application teams.*
Interactions with applications often occur on an 18-36 month time frame
and can change over the lifetime of the ISIC. Increased flexibility will
lead to increased impact because resources can be placed where they are
most needed and will also lead to a tighter coupling with application
teams because it places the burden of success on those developing the
technology. We note that acceptable activities for ISIC interactions
with an application team can take many different forms including
demonstrating feasibility of a new algorithmic or modeling approach to a
scientific area, using new algorithmic and software technology to solve
specific scientific problems, or embedding new software technology in
production applications codes.

Second, *ISIC should participate in the proposed SciDAC Institutes
program to expand their outreach activities*. The ISICs have been
successful in working with funded SciDAC application teams, but, in
general, have not broadly disseminated their technology. Now that ISIC
technology is maturing, an increased emphasis on outreach to new
simulation development efforts and to applications that are important to
the Office of Science but not (yet) funded by SciDAC can significantly
increase ISIC impact. To facilitate this outreach, the ISICs should
participate in the proposed SciDAC Institutes program element.
Activities of these institutes may include workshops to define new
application areas and summer school activities to provide an intensive
immersion experience in an enabling technology or application area. The
ISICs should participate extensively in these activities by providing
expertise, lectures, tutorials, and notes as needed.

Third, *the ISICs must interact with the proposed Software Integration,
Maintenance, and Support program element to ensure robust, useable
software is delivered to the SciDAC community.* The likely role of the
SIMS program is to sustain the ISIC software investment made in
SciDAC-1, to promote software interoperability and reusability, and to
promote sound scientific software engineering practices. The ISICs
should provide new technology developments and software to this program
and leverage it extensively to harden and maintain the software.

Fourth, *the ISICs need to improve coordination of their activities and
interactions with the SC application teams.* In addition to facilitating
the use of ISIC technologies by the application teams, the ISICs could
leverage each other's efforts more.

Fifth, *the ISICs must maintain their critical role as the vertical
integration link between basic research and HEC facilities within
OASCR*. As was done in the first round of SciDAC, this can be
accomplished through the development of explicit criteria for successful
proposals in which ISIC teams articulate their connection to HEC
facilities and/or basic research.

The relationship between the ISICs and the other SciDAC program elements
is summarized in the figure below. Each ISIC should directly support
multiple simulation or data-intensive SciDAC application areas.
Moreover, they should coordinate with each other and work together to
support the needs of applications. They will be active participants in
the proposed SciDAC Institutes program element and use this as one
mechanism for reaching out to the broader community. The ISICs will
leverage the SIMS program element to harden and maintain their software
and will be active participants in promoting sound scientific software
engineering practices.

![](media/image14.png){width="4.934722222222222in" height="3.70625in"}

**The relationship between the ISIC and SAP program elements.**
Scientific Application Partnerships (SAPs) are targeted efforts to
integrate advanced applied mathematics and computer science technologies
into selected SciDAC applications projects. Although the stated goal of
SAPs is similar to that of the ISICs, the two program elements are
complementary. First, the mathematicians and computer scientists funded
by SAPs are typically in residence with and primarily managed by the
application team. This arrangement results in strong interactions and
maximizes the likelihood of success for the insertion of the specific
technology into that application. ISIC efforts, on the other hand, are
larger and focus on interaction with multiple application teams. They
produce, in turn, robust, usable software that is applicable to the
general SciDAC community in a way that SAP efforts cannot. Second, SAP
provides the mechanism to supply application teams the computer science
and mathematics expertise not covered by ISIC teams. The ISIC program
element could leverage the SAP program element to encourage the
participation of resident application scientists in ISICs to increase
the dissemination and insertion of ISIC technology in an application
area.

**The relationship between the ISICs and OASCR's base research
program.** The ISICs have a separate mission - applications-driven
software development -- from that of the OASCR base program, which is
long-range research in applied mathematics and computer science.
However, there are significant linkages in both directions between the
two programs. The first is that the base program is a source of new
ideas, which, as they reach a sufficient level of maturity, can be
turned into robust software by the ISIC program. In addition, the tight
coupling between the ISICs and applications has led to the
identification of new areas of research and development, some of which
feed back into the base program as areas for fundamental research.
Finally, the base program is a potential user of software tools coming
out of the ISICs. ISIC software serves as a base on which to build new
algorithms, models and tools, without having to write everything from
scratch. This is an increasingly important role, as algorithms become
more complicated, and as we start to use computation as an experimental
tool to investigate new models, such as in the new multiscale
mathematics program.

-   **ISIC topics of emphasis.** ISICs, individually and collectively,
    must provide a comprehensive, integrated, scalable, and robust
    high-performance computing software infrastructure that will enable
    effective use of leadership class computing resources by current and
    future SciDAC application teams. To meet this goal, ISICs must
    address needs for new algorithms and mathematical software that
    scale to thousands of processors; accurate discretization and
    parallel meshing approaches; software development environments and
    methodologies for realizing portable performance; and scalable
    scientific data management and analysis tools. To meet the needs of
    current and future SciDAC applications, ISICs can address many
    topics. These include the following:

-   **Numerical algorithms mathematical software.** Linear, nonlinear,
    and eigen solvers; multigrid and multilevel methods; adaptive
    sampling algorithms; optimization and inverse methods; sensitivity
    analysis and uncertainty quantification. These algorithms and
    mathematical libraries must be scalable and efficient on machines
    with thousands of processors and complex memory hierarchies.

-   **Mesh generation and discretization technologies. Automatic** mesh
    generation; parallel mesh partitioning techniques; adaptive mesh
    refinement algorithms and software frameworks; mesh quality metrics
    and improvement; numerical discretization technology; spatial
    coupling techniques; multi-physics, multi-scale modeling; error
    quantification; rapid problem definition. These technologies must be
    tailored to current and future applications employing both
    structured and unstructured meshes.

-   **Software development environments and systems software.** Software
    development and code profiling tools; performance benchmarking and
    evaluation; code correctness and validation; high performance
    software component technologies, code reuse, and language
    interoperability, systems software that scales to tens of thousands
    of processors, supports high performance I/O and application-level
    communication, and provides the highest levels of fault tolerance,
    reliability, manageability. These tools and capabilities must be
    portable across all SciDAC hardware platforms and provide a common
    programming environment for SciDAC application scientists and
    provide ease of use for system administrators, tool developers and
    end users.

-   **Scientific data management, analysis, and visualization.**
    Large-scale data integration and management; management of diverse
    metadata concerning, e.g., provenance and data quality; data access
    and querying; scalable data storage and transfer; feature extraction
    and change analysis; vector/tensor field visualization; statistical
    data analysis. These data analysis algorithms and software tools
    must be scalable to large data sets, as well as tailorable to meet
    the needs of individual application codes.

> **Distributed computing technologies for experimental facilities and
> science.** Movement and management of large distributed data;
> distributed computing and information sharing technology; workflow
> technologies; methodologies for comparing simulation results and
> experimental data; Open Science Grid development and operations;
> cybersecurity infrastructure for distributed teams and experimental
> science; collaboration technologies for remote access to facilities.
> This software and related infrastructure must scale to support the
> collaborative planning of complex experiments and analysis of large
> quantities of data by distributed teams.

## Computational Science Institute

We present here a plan for the establishment of a Computational Science
Institute comprising one- to two-week workshops, with the objective of
broadening the use of high-end computing and computational science
within the scientific community. Such an institute will fill an
important gap in computational science today. Specifically, although
several other scientific and mathematical institutes exist, none has as
its primary objective the development of an interdisciplinary approach
to computational science and engineering enabled by high-end computing.
The focus of the Institute's activities will be on large-scale computing
problems requiring large computational resources, including massively
parallel computers and large-scale data.

**Strategies**

The main activity of the Institute will be intense workshops that will
bring together the leading experts in a particular area to work on a
specific topic of interest. The organizers will submit proposals that
will be reviewed by an advisory board and approved by the board. The
goal of each workshop will be to address one or two problems of general
interest and to foster new collaborations among the participants. We
envision that for the duration of the workshop, the Institute will
become the "center of the universe" for that topic. The workshops will,
in general, last 2--3 weeks, with participants coming from various areas
including computer science, mathematics, and a specific scientific
discipline.

Other activities that the Institute may support include the following;
these are further discussed in the section "Community Participation and
Outreach."

-   ***Visiting faculty program*** to encourage faculty on sabbaticals
    to come to the Institute, either as participants in one of the
    workshops or for an extended period of time.

-   ***Postdoctoral fellowships*** to foster the careers of entry-level
    scientists in interdisciplinary topics of high-end computational
    science.

-   ***Graduate student research internships*** ("predocs") to support
    predoctoral associates who may naturally accompany their university
    professor or research advisor to participate in the Institute's
    programs.

-   ***Outreach program*** to disseminate the scientific results of the
    Institute to a broader community, by maintaining an electronic
    archive of the major success stories from the SciDAC program as well
    as those from the Institute activities.

####### **Organizational Structure**

In order to serve the scientific community and reach its aspirations,
the Institute should have a Director, whose main goal will be to set the
overall research direction and to oversee the management of the
Institute. In addition, the Institute may have a Deputy Director, an
Associate Director, or other staff as is necessary to implement its
programs. Leveraging the administrative staff from an existing
institution, lab, or university may be an attractive approach to
organizing the administration and implementation of the Institute.

The Institute will have a Board of Governors to assist the Director and
to provide advice and oversight for the overall management of the
Institute. A Scientific Advisory Committee will be appointed by the
Board of Governors, with the agreement of the Director, to help set the
research direction for the Institute and to review the workshop
proposals.

####### **Scientific Opportunities**

Many areas of science are at a critical juncture and require the kind of
new ideas that the Institute would provide. These come from across the
science spectrum, and advances in any one are likely to provide the
basis for advances in others. In some cases it is not clear how to go
beyond current paradigms, for example how to increase the time and
length scales in molecular dynamics simulations. In others, the
researchers have not adequately explored how to take advantage of
terascale computing resources. The Computational Science Institute will
establish programs for scientific luminaries from a variety of
organizations to focus on specific problems in collaborative research.
Examples of such problems are described in the following. A fuller list
is given in the Appendix.

**Biology, Chemistry, Materials Science, and Nanoscience.** In biology,
some of the outstanding issues are concerned with the development of
statistical learning methods, how to deal with extremely large s, how to
make the transition from flux balance models to full kinetics in whole
cell simulation, and how to construct models of early evolution starting
from the RNA world. In chemistry, important challenges are finding
electronic structure algorithms that scale linearly with system size,
especially in ab initio quantum mechanical simulations of large systems,
the calculation of accurate potential energy surfaces for understanding
catalysis and reaction dynamics, and the predicting the behavior of
molecules at ultrafast (femto-second) time scales. At the frontier of
biology and chemistry lies the development of completely novel
strategies to simulate large-scale biomolecular systems composed of
several million atoms with atomic detail. In materials science, a
profoundly difficult problem remains the prediction of the various
phases and properties of matter with quantitative accuracy including
exotic magnetic and superconducting phases and granular materials. How
to deal with large plastic deformations in real materials is a problem
ripe for attack. In nanoscience, a challenge is to determine the
electrical and thermal transport in nanostructures and charge separation
(one may note the obvious connection to biological ion channels).
Moreover, the ability to synthesize new nanostructures will increasingly
rely on computation, yet the same problems of multiple minima, and force
fields that plague protein structure determinations need to be addressed
in this context.

**Fusion and Plasma Science.** The computational challenges in fusion
energy and plasma science are both scientifically stimulating and
temporally urgent. One challenge is to assemble the best physics models
into a realistic predictive, integrated modeling capability; this will
involve stronger coupling to the experimental community in meaningful
verification and validation tests and will enable the international
plasma physics community to harvest the key scientific knowledge from
major projects. An associated challenge, which requires strong alliance
with the computational materials science community, is to investigate
the best materials (beyond ferrites) for fusion systems; of particular
interest are materials such as vanadium and liquid metal compounds. At a
more fundamental physics level, the grand challenge of direct numerical
simulation of multiscale kinetic dynamics is a perfect fit to the SciDAC
approach of strong alliances between computer science, applied
mathematics, and applications, to enable the best possible utilization
of terascale/petascale computational resources. Progress in this area
will also clearly be of interest in the computational astrophysics area.

**Astronomy, High Energy Physics, and Nuclear Physics.** There remains
the need for novel hardware and algorithmic approaches to the study of
quantum chromodynamics. These would have important implications for the
understanding of the origin of the universe, the quark-gluon plasma, and
particle masses. In astrophysics, cosmology, the structure of galaxies,
and the origin of dark matter are outstanding problems.

**Applied Mathematics and Computer Science.** Multiscale modeling is a
common theme in many scientific areas and will be a likely early target
of the Institute. Computer architecture and software issues, such as
computing at the exascale, reconfigurable computing, quantum computing,
and interactive supercomputing, need attention. New methodologies for
robust optimization would have a broad impact in computational science.
A number of other topics in high-performance computing cross-cut various
application areas. Verification and validation, for example, demands a
new level of scientific fidelity tests as the computational capabilities
dramatically improve. With the tremendous increase in observational data
from new experiments as well as from advanced simulations, data
management and associated extraction of information is a key challenge.
The use of statistical and machine learning approaches needs to be
explored. A third area is to substantively demonstrate the acceleration
of scientific progress through effective utilization of most advanced
platforms; this will involve the implementation and exercising of
current codes on the most advanced platforms to produce new scientific
insights that would otherwise have been impossible.

####### **Relationship to Other Institutes** 

A number of existing programs currently operate similar to Computational
Science Institute. These institutes all have significant visiting
researcher components and have responsibilities for community-building
and are thus distinguished from institutes that have a specific research
mission performed by in-house researchers. None of the existing
institutes has a central focus on computational science. Arguably, some
of them do occasionally host workshops on computational topics; but
because they are disjoint, these institutes do not have the potential to
develop the cross-disciplinary tools and techniques that are considered
under the Computational Science Institute. Nevertheless, they do have
operational models that can serve as role models for the Institute. We
review here two of these.

The ***Santa Fe Institute*** was founded in 1984 and focuses on "complex
systems research." The institute has approximately 35 researchers in
residence year-round, about double that number during the summer; the
administrative staff consists of 25 people. The institute holds around
two dozen workshops each year, as well as annual summer schools. The
group publishes a working paper series, a book series, and a public
lecture series. Research topics of recent focus have included cognitive
neuroscience, computation in physical and biological systems, economic
and social interactions, evolutionary dynamics, network dynamics, and
robustness. In focusing on a broad range of technical and political
subjects, the Santa Fe Institute has extensive experience with
cross-disciplinary research. Compared to the Santa Fe Institute,
however, the Computational Science Institute would be more focused on
computational science problems.

***The Kavli Institute for Theoretical Physics (ITP)*** was founded in
1979 and is located at the University of California Santa Barbara. It is
funded by the National Science Foundation and the University of
California. The ITP seeks to foster collaborations between subfields in
theoretical physics, but typically not more broadly. It has resident
scientists and a regular visitor program, plus an artist-in-residence
program and a journalist fellow program. Current research topics include
complexity in strongly correlated electron systems, mathematical
structures in string theory, friction, the supernova gamma-ray burst
connection, topological phases and quantum computation, spintronics, and
new physical approaches to molecular and cellular machines. The
Computational Science Institute would be similar in structure to ITP.
However, it would focus on computational science applied to a number of
disciplines, and would attempt to foster collaborations broader than the
typical physics community.

In addition, there exist an Institute for Pure and Applied Mathematics
at UCLA and several institutes for theoretical physics in other
countries, including The Perimeter Institute for Theoretical Physics
(Canada), The Nordic Institute for Theoretical Physics (Denmark,
Finland, Iceland, Norway, Sweden), Landau Institute for Theoretical
Physics (Russia), Institute for Theoretical Physics (Austria), Yukawa
Institute of Theoretical Physics (Japan), National Institute of
Theoretical Physics (Australia), Institute for Studies in Theoretical
Physics and Mathematics (Iran), Institute of Theoretical Physics
(Switzerland), and Institute for Theoretical Physics (The Netherlands).
), the Centre Europeen de Calcul Atomique et Moleculaire CECAM (Lyon),
and the Centre de Recherche en Mathematique (Montreal). Most of these
programs have regular visiting scholar programs, semi-permanent research
staff, and intensive workshop series.

####### **A Critical National Role**

The Computational Science Institute will play a critical role in the
national scientific computing efforts.

SciDAC. SciDAC 1 has focused on building software tools and algorithms
that provide infrastructure for software systems that enable science at
scale on large-capability computers. In some communities, however, it
has not led to actually doing the science or even posing those problems
that should be tackled. The Computational Science Institute will bring
together world leaders to address such problems, providing the
intellectual leadership that bridges the gaps between the
science-at-scale problems and the capability providers.

***DOE.*** DOE has core research programs in all of the general
scientific domains (BES has chemistry and materials, BER has biology and
climate, ASCR has computer science and mathematics, etc.). Interaction
between these programs is sometimes difficult, although the actual
problems and techniques might best be met through an interdisciplinary
approach. Moreover, the mission-driven nature of the agency can make it
difficult to accommodate semester-long sabbaticals for such exchanges.
The Computational Science Institute provides a unique opportunity to
bring together the right interagency mindshare to tackle
multidisciplinary problems in a timeframe amenable to the DOE laboratory
environment.

***NSF***. The National Science Foundation supports a number of
institute programs nationwide. Most of these institutes are focused on
specific problem domains (such as theoretical physics, biology) or on
mathematical techniques. Institutes that are focused on collaborative
interdisciplinary research are relatively rare. Recent NSF initiatives,
such as a focus on leadership-class computing, cyberinfrastructure, and
other large-scale computational science activities, would benefit from
the activities outlined for the Institute. We envision that a number of
NSF-funded researchers will participate in the programs delivered by the
Institute.

**Venue and Infrastructure Support**

One of the most important qualities of a good venue is that it be
relatively self-contained so that participants can remain on-site and do
not need their own transportation. The venue must provide adequate
conference facilities, including rooms for plenary presentations,
breakouts for smaller groups, and office space.

Especially for institute workshops lasting more than two weeks, lodging
and meals must be available at reasonable cost. One must expect that
participants in these longer workshops will wish to bring their
families. Dormitory accommodations are not considered conducive to the
mission of the Institute in attracting and encouraging participation of
leaders in the computational and computing sciences. In addition, the
venue should be in an attractive location and provide recreational
activities for the participants and their families.

Examples of desirable venues include established institutes such as the
Aspen Center for Physics and self-contained conference centers and some
university campuses. Less desirable are facilities that are easy to step
away from or that do not provide the freedom from local interruptions
needed for focusing on the subject of the workshop.

#### Infrastructure

The required infrastructure can be divided into three groups:
administration, space, and computing and laboratory resources. The
minimum administrative support includes a director as well as permanent
logistics and outreach support. Advisory and review panels will also be
needed to evaluate the workshops supported by the institute. During the
workshops, additional logistics support will be required to ensure the
smooth operation of the workshops, as well as support for the computing
infrastructure.

A typical workshop will require rooms that support plenary gathers of
approximately 50 people, four breakout rooms for at most 20 people each,
and smaller interaction spaces. In addition, lab space with computing
resources will be required. Shared office space for all participants
will be needed to support the longer-term workshops and is highly
desirable for shorter workshops. Space also will be required for
administration staff and for a resource center that contains reference
materials.

The venue must provide access to computing facilities. At a minimum,
such access includes adequate network access, collaboration tools (e.g.,
one or more Access Grid-equipped rooms), visualization workstations, and
a permanent Web presence. While the venue is not expected to have its
own high-performance computing resources, the Institute should arrange
for access to external HPC resources through cooperative agreements with
supercomputing centers.

***Virtual vs. Concrete Organization***

A natural question to ask is whether the Institute should be "virtual."
A virtual institute takes advantage of infrastructure at established
institutes and provides overall direction and coordinates access to
unique resources, such as high-performance computing, visualization, and
computational science reference material (e.g., an online-library).
However, a virtual institute lacks its own identity and does not control
what sessions are accepted by the established institutes. If adequate
resources not available to establish a nonvirtual organization, a
virtual institute could be used to build a presence in the community for
the first year or two, evolving to a concrete institute as the program
becomes established.

***Community Participation and Outreach***

Computational science has often been defined as the union of computer
science, applied mathematics, engineering; however, as much a part of
computational science as that union is the applications scientist (the
physicist, biologist, chemist, etc.), who must apply advanced computing
and simulation to a problem in order to advance the state of knowledge
in the fields. Accordingly, we consider the community to be defined by
its scientific purpose, independent of its relationship to the funding
agencies.

How can we obtain community participation? The topic of each program be
designed to advance the state of the art in some specific field or
application; it must tantalize and engage scientists at the top of their
fields to garner the highest-level participants. In order to accomplish
this, the programs will be selected in a competitive process; the
participants themselves will submit proposals for the design of
individual programs.

The Institute must offer something that existing institutes do not.
Specifically, the format, atmosphere, and programming of the Institute
must force the participants to break from traditional, "stovepiped"
discipline-based approaches and adopt a focused, integrated approach to
solving the important problems of the day. The scientists must feel that
they are "gathered in a special place and time for a very special
purpose." Thus, the Institute must provide ample opportunity for small
groups to form and pursue research productively; and it must foster the
formation of permanent collaborative alliances that continue long after
the end of the program.

How can the Institute work to enlarge the community and ensure its
future? Enlarging the computational science community is not the primary
focus of the Institute; nonetheless, the Institute will devote a portion
of its resources to this endeavor.

As noted in the introduction, in addition to its main focus on
workshops, the Institute may support the following:

-   ***Visiting faculty program***, in which the Institute sponsors
    several computational scientists to be in residence at the
    Institute, creating a "critical mass" of expertise in a topic or
    field. The Institute may also sponsor such a program at a university
    or laboratory, temporarily augmenting the staff and creating a core
    of expertise of that institution.

-   ***Postdoctoral program***, in which the best new Ph.D. scientists
    are attracted to the Institute and provided a high degree of freedom
    to explore the topics of their interest. Initially, these postdocs
    may be established at a university or laboratory that emphasizes
    computational science. In the steady state, the program will exist
    at the Institute itself. Such fellowships will help to "seed" the
    community with young scientists steeped in the philosophy of
    computational science. At least one new fellow should be selected
    each year for a two- or three-year appointment.

-   ***Graduate student research internships***, in which emerging
    scientists are exposed to the interdisciplinary SciDAC
    communityThese young scientists will be among the vanguard of
    leaders in computational science and engineering in the future. It
    will be useful to coordinate these internships with the CSGF
    program. For example, one could envision that the CSGF fellows be
    given an option to attend one of the workshops as part of their
    practicum.

-   ***Outreach program***, through which the Institute publishes
    reports, lecture notes, and proceedings, making them available to
    the public.

An attractive and important activity, that does not fit the Computation
Science Institute's focus on forefront scientific research but
nonetheless has complementary value, is the organization of short (1--2
week) *training courses* on a specific area in computational science,
with the intention of brining participants quickly up to speed in a new
scientific area. The general outline would follow a short-course format,
with some in-depth labs. This component, organized as a SciDAC summer
school activity, would likely benefit from coordinated activities with
the SciDAC Computational Institute. **We strongly encourage the creation
of such a school as a separate line-item in the SciDAC-2 plan.**

**Funding Model**

The Institute will require have an initial annual funding of at least
\$2M for both capital and operating expenses (see the table below). This
is viewed as a minimum level of viability based on administrative costs
and participant support. Additional funding will be needed as the
Institute expands from summer workshops mode to operation throughout the
year.

We strongly encourage the hosting institution to provide leveraging
funding. An example might be a university "contributing" its indirect
back to the institute. The Institute will also seek sponsorship by other
organizations, for example, private foundations, industry, or other
government agencies (e.g., NSF, DoD, NASA).

**Cost Estimate**

+---------------------------------------------------------+------------+
|                                                         | \$k        |
+---------------------------------------------------------+------------+
| Annual Costs                                            |            |
+---------------------------------------------------------+------------+
| Director                                                | 350        |
+---------------------------------------------------------+------------+
| Administrative and technical support (e.g., Web)        | 200        |
+---------------------------------------------------------+------------+
| Rental of facilities (approx ¼ year, total 10,000       | 125        |
| square feet, furniture etc.)                            |            |
+---------------------------------------------------------+------------+
| M&S                                                     | 50         |
+---------------------------------------------------------+------------+
| Director's travel                                       | 40         |
+---------------------------------------------------------+------------+
| Advisory Board travel etc                               | 15         |
+---------------------------------------------------------+------------+
| Subtotal                                                | 780        |
+---------------------------------------------------------+------------+
|                                                         |            |
+---------------------------------------------------------+------------+
| Per Workshop Costs (assumed two weeks each, 30          |            |
| participants)                                           |            |
+---------------------------------------------------------+------------+
| 10 Participants "level 1"                               | 120        |
|                                                         |            |
| Salary Support @ \$9k each                              |            |
|                                                         |            |
| Travel @ \$1k each                                      |            |
|                                                         |            |
| Per diem @ 2k each                                      |            |
+---------------------------------------------------------+------------+
| 10 Participants "level 2"                               | 80         |
|                                                         |            |
| Salary Support @ \$5k each                              |            |
|                                                         |            |
| Travel @ \$1k each                                      |            |
|                                                         |            |
| Per diem @ 2k each                                      |            |
+---------------------------------------------------------+------------+
| 10 Participants "level 3"                               | 30         |
|                                                         |            |
| Salary Support @ \$0 each                               |            |
|                                                         |            |
| Travel @ \$1k each                                      |            |
|                                                         |            |
| Per diem @ 2k each                                      |            |
+---------------------------------------------------------+------------+
| Director and Admin staff participation (5)              | 15         |
|                                                         |            |
| Travel @ \$1k each                                      |            |
|                                                         |            |
| Per diem @ 2k each                                      |            |
+---------------------------------------------------------+------------+
| Subtotal per workshop                                   | 245        |
+---------------------------------------------------------+------------+
|                                                         |            |
+---------------------------------------------------------+------------+
| Annual Costs + Costs for 4 Workshops                    | 1760       |
+---------------------------------------------------------+------------+
| Indirect Costs (upper acceptable limit) 15%             | 264        |
+---------------------------------------------------------+------------+
|                                                         |            |
+---------------------------------------------------------+------------+
| TOTAL                                                   | 2024       |
+---------------------------------------------------------+------------+

**Appendix: Examples of Compelling Scientific Problems**

1.  modeling and simulation of RNA

2.  whole cell simulation from flux-based to full kinetics

3.  interactive supercomputing -- what would the architecture look like?

4.  computer architecture problems at the exascale -- after Moore's law

5.  quantum computers -- what science (if any) could we do (besides
    crypto)?

6.  atomistic biomolecular systems - not simplifying but reformulating
    the problem for the large scale (10M molecules)

7.  linear scaling methods for large scale (beyond 7-9^th^ order of
    today), with accuracy

8.  statistical or machine learning methods for petabyte s

9.  beyond double precision (quad-quad) -- where is it needed? Where is
    LESS needed? Where is multiprecision needed?

10. statistical learning and data analysis

11. long-time evolution of large time scales and length scales (perhaps
    spatial vs. statistical techniques, or hybrid) -- modeling
    atomistically

12. computational models for astrophysical systems (today, no quality
    control -- what defines an excellent computation?)

13. subgrid-scale modeling with focus on, say transport, angular
    momentum

14. numerical methods for granular flows

15. multiphysics models -- calibration, integration?

16. structural modeling for large deformations

17. programming languages for scientific computing

18. bridging time scales -- spatial and statistical

19. synthesis of carbon monofibers -- long-time evolution

20. chemical sputter, erosion of plasma (predictive capabilities
    lacking)

21. ultrafast, ultraintense science

22. integrated modeling -- MM for fusion

23. V&V in multiscale simulation -- can we extrapolate beyond what
    experiment predicts?

24. more realistic benchmarks for most advanced platforms

25. approximation theory -- angular momentum not conserved with
    grid-based methods

26. application-inspired challenge problems

27. coupled models subscale to macroscale -- how connected? design
    optimization

28. uncertainty -- building robustness in opt. Roblems; V&V in
    ultrascale modeling

29. superscale modeling --high-fidelity models receiving input from
    large scale; downscale

30. MEMS and simulation of large-scale problems; control problems in
    coupled systems

31. role of databases -- designing reference databases for application
    areas

32. agent-based modeling in social and economic policy

33. data mining for nongeometric problems

34. high-performance data-intensive computing -- how to get integrate
    sensors, networks

35. reconfigurable computing -- effect on productivity

36. optimization of multiscale structure -- esp. for material/system
    behavior over long time

37. density functional theory --exchange correlation potentials that are
    accurate and reliable

38. fast methods in quantum chemistry

39. energy initiative -- e.g., catalysis to improve combustion systems

40. microelectronics -- devices for next generation, low-temp
    superconductivity

41. petascale data analysis data into models -- similar to NSF
    data-driven application science

42. new models to exploit advanced machines -- e.g., physics

43. computational cosmology

44. ultrafast science

45. potential energy surface --find global minima in problems with, say,
    10^5^ local minima

## Accelerating Scientific Discovery in Experimental Science via Advanced Computing

Editors: Ian Foster, <foster@mcs.anl.gov>, Vicky White, <white@fnal.gov>

> "\[DOE\] user facilities provide resources that speed up experiments
> by orders of magnitude and open up otherwise inaccessible facets of
> nature to scientific inquiry. Many of the important discoveries made
> in the physical sciences in the second half of the twentieth century
> were made at---or were made possible by---user facilities. Moreover,
> most of these user facilities, which were justified and built to serve
> one scientific field in the physical sciences, have made significant
> contributions to knowledge and technology in many other fields,
> including biology and medicine." -- Dr. Hermann Grunder, in testimony
> to Congress, 2003.

**Summary**

We present a plan for SciDAC-2 activities aimed at engaging with
experimental facilities and experimental science. This plan aims to
achieve significant advances in the experimental sciences via the
development and application of advanced computing and information
technology. The plan is motivated by three factors: the profound
importance of DOE experimental facilities and science to the nation; the
urgent challenges currently faced by experimental sciences due to
rapidly growing data volumes and new experiment techniques; and the
significant advances in advanced computation and interdisciplinary and
interlaboratory collaboration achieved within the SciDAC-1 program.
These factors lead the authors of this report to conclude that there
exists a timely and significant opportunity to increase both the utility
of DOE experimental facilities and the effectiveness of DOE experimental
science, via precisely the sort of focused and directed
interdisciplinary research program that has been applied so successfully
within the simulation sciences in SciDAC-1. Much as SciDAC-1 is
empowering theorists by automating (via numerical simulation) important
steps in the process of exploring the implications of theory, so a
comparable program can empower experimentalists by automating important
steps in the experimental process, from the operation of experimental
facilities to the collection and analysis of experimental data.

The rest of this report is structured as follows. In Section 2, we
explain the specific contributions that we expect the program to make to
DOE and the nation. In Section 3, we propose an organizational structure
for a program, based around the concepts of the Science Application
Partnerships (SAPs) and Integrated Software Infrastructure Centers
(ISIC) that proved so successful in SciDAC-1. In Section 4, we list
examples of cross-cutting technology areas that could be suitable for
ISICs, and in Section 5 we discuss potential risks for a program in this
area.

The material in this report has been developed via an intensive process
of consultation during June-July 2005, culminating in a workshop held at
Argonne National Laboratory on August 2-3. This material also builds on
and relates to earlier workshops such as those on Data Management and
the National Collaboratory program. Appendix A lists participants in
that workshop, and Appendix B provides additional information on
application opportunities developed during that meeting.

**Benefits to DOE and the Nation**

Experiment driven research in science and engineering, and the design,
construction, and operation of experiment facilities, are at the heart
of DOE's mission. Forty percent of DOE's FY06 Basic Energy Sciences
budget request is devoted to experiment facilities, which support more
than 18,000 users. A large fraction of DOE's scientists and engineers
are engaged in experimental research. The aggressive future facilities
program and associated scientific rationale laid out in the report
"Facilities for the Future of Science" emphasize that experimental
science will continue to be crucially important to the nation's
prosperity, health, and security.

DOE user facilities consistently receive the highest possible reviews
for their utility and usability. However, new facilities are delivering
revolutionary advances in data rates, data volumes, and experimental
accuracy, and facility users are struggling to keep up. It is also
becoming increasingly clear that making curated data in complex fields
broadly available stimulates new methods, more science, and accelerates
progress.

It is in this context that we propose here a program aimed at
*accelerating scientific discovery in experimental science*. The goal of
the program is simple: to bring certain key advanced information
technologies to bear on challenging problems arising in modern
experimental facilities and research within DOE. In so doing, the
program will deliver benefit to the nation in several important
respects.

Firstly, projects funded within this program will contribute directly to
existing DOE experimental facilities and research programs, enhancing
their usability and accessibility, enabling new classes of experiment,
broadening access to larger and more diverse communities, and enabling
cross-cutting comparisons of experiments with other experiments and with
simulation results. The following are examples of projects that might be
funded under such a program, and the benefits that can be expected to
accrue from their work.

-   *Data management for the nanosciences*. Nanoscience and
    nanotechnology require investigators to combine data from large and
    complex groups of experimental and computational sources---even when
    investigating just a single nanostructure. The five new Nanoscience
    Research Centers (NSRCs) are gearing up to produce a flood of data
    in different modalities. However, the nanoscience community lacks
    the scientific data management systems to support the resulting
    complex workflows and challenging data and information handling. A
    SciDAC-2 program focused on addressing these issues can spur new
    discoveries by facilitating the comparison of experiment with
    experiment and experiment with simulation---and also produce tools
    of value to other scientific disciplines.

-   *Timely analysis of LHC data.* From 2007 on US scientists will
    analyze petabytes of data yearly from the Large Hadron
    Collider(LHC). The high energy physics community plans to create a
    10^18^-byte archive by 2012 for data from the four major LHC
    experiments. A project that builds on results obtained within the
    SciDAC-1 Particle Physics Data Grid (PPDG) project to enable rapid
    dissemination and analysis of this data within the U.S. physics
    community and interoperability with the worldwide LHC Computing Grid
    will greatly accelerate the pace of scientific discovery and
    maintain U.S. leadership.

-   *Enabling innovative science at the LCLS*. The extremely bright and
    short x-ray pulses from the Linac Coherent Light Source (LCLS) will
    enable unique scientific research. Yet to realize the full
    scientific promise of this facility will require the timely
    development and application of advanced computing tools for the
    design of experiments and for data handling and analysis---tools
    that LCLS does not have the expertise to develop alone. A focused
    SciDAC-2 program can create the partnerships required to achieve
    these advances.

-   *Enabling climate research and policy studies in a petabyte age*.
    Climate scientists can now access almost 100 terabytes of climate
    simulation data, thanks to the SciDAC-1 Earth System Grid. (In
    March-June 2005 alone, scientists downloaded 20 TB of
    Intergovernment Panel on Climate Change (IPCC) data and produced 250
    scientific articles!) But researchers studying global and regional
    impacts of climate change now need access to both future
    petabyte-scale simulation datasets *and* large and diverse
    experimental datasets from the DOE Atmospheric Radiation Measurement
    (ARM) program, NASA satellites, and other sources. This urgent need
    can be met by the creation of an integrated environmental data
    portal that allows scientists, and ultimately other stakeholders, to
    pose complex questions against what will ultimately be petabytes of
    simulation and observational data.

-   *Creating new analysis opportunities for RHIC experiment data*. The
    Relativistic Heavy Ion Facility (RHIC)has a rich experimental
    scientific program that has attracted a community composed of
    hundreds of physicists from around the world to study what the
    universe may have looked like in the first few moments after its
    creation. RHIC has been a leading force in studying the Quark Gluon
    Plasma and has generated petabytes of data a year with anticipated
    future data rates of GB/sec in the next few years. A program that
    builds on the data management and data-grid work of the SciDAC-I
    program can develop second generation integrated analysis
    capabilities for the statistically challenging, and often real-time,
    data from experiments at RHIC thus opening up analysis paths that
    scientists would not otherwise consider.

-   *Next-generation experimental cosmology*. The Large Synoptic Survey
    Telescope (LSST) aims to achieve a precision measurement of the
    nature of dark energy. LSST will do this by surveying more than
    20,000 square degrees to an unprecedented depth in 6 filters over a
    period of 10 years, creating a 100+ Pbyte database. LSST will take
    about 2000 exposures per night with its 3.4 gigapixel camera,
    generating about 14 Tbytes of raw images and 28 Tbytes of calibrated
    images per night. Each image will be analyzed in near real-time for
    data quality control and the ability to alert on transient
    astrophysical events such as supernovae and gamma-ray bursts. LSST's
    data torrent presents significant challenges for near real time
    analysis of data streams of \~1 Gbyte/sec, provenance, end-to-end
    simulations, and information management. LSST also needs to build on
    the Earth Systems Grid, and other SciDAC-funded efforts to enable
    asynchronous analysis and grid based data access to its enormous
    data sets. Solving these problems in a general, reuseable and
    extensible way will benefit all data-intensive experimental science.

-   *Community informatics for combustion science*. The SciDAC-1
    "Collaboratory for Multi-Scale Chemical Science" produced integrated
    collaborative data sharing and information management solutions that
    are enabling community-based management of chemical reference data
    and rich experiment/model comparisons. New experimental techniques,
    such as high-throughput laser-imaging of the combustion process,
    will provide a vast new source of community data for comparison with
    simulations. A SciDAC-2 project extending CMCS techniques to
    automate the community curation and analysis of high-throughput
    terabyte-scale data would be a tremendous step towards an integrated
    experiment/theory/simulation combustion knowledge base supporting
    U.S. leadership in combustion science.

-   *Coupling computation to experiment at ITER.* The International
    Thermonuclear Experimental Reactor (ITER) is a burning plasma
    magnetic confinement experiment, the next major step in an
    international program aimed at proving the scientific viability of
    controlled fusion as an energy source. Support for quasi real-time
    analysis of experimental data will bring unprecedented improvements
    by greatly shortening the time-scale for the comparison of
    experiment to simulation resulting in more informed experimental
    decisions.

Second, this program will encourage identification of common
requirements across different DOE experimental programs. A coordinated
program can exploit these commonalities and achieve cross-fertilization
of ideas among different facilities, experimental programs, and computer
scientists throughout the DOE laboratory system and its university
partners. The following are examples of areas in which commonalities can
be exploited, via either collaborative application projects or
cross-cutting technology projects.

-   *Remote control rooms for international experiments*. The worldwide
    experimental programs in fusion and high-energy physics will be
    centered, for the next decade at least, on facilities located in
    Europe: the International Thermonuclear Experimental Reactor (ITER)
    and Large Hadron Collider (LHC), respectively. Yet the physical
    relocation of U.S. scientists to Europe is impractical. Thus we see
    a cross-cutting need for *remote control rooms*: working
    environments for off-site personnel engaged in experimental
    operations that are every bit as productive as those onsite.
    Building on the initial success of Fusion remote control rooms
    (FusionGrid) under SciDAC-1, the enabling of remote participation by
    U.S. scientists will contribute to the success of ITER and LHC and
    also maximize the value of these unique facilities to the U.S.

-   *Real-time computing at beamlines*. A new generation of more
    powerful X-ray and neutron sources and far more sensitive detector
    technology are combining to create a data explosion at facilities
    across the DOE system. When a single experiment can produce one
    thousand megapixel images, analysis becomes a substantial
    bottleneck. A combination of on-demand access to computing, enhanced
    algorithms capable of quasi-real-time analysis, and
    tele-collaboration technologies can allow for far more efficient use
    of beamlines.

-   *Distributed data management and computing*. Climate, physics,
    biology, and nanosciences (among others) face similar needs for the
    distributed management and analysis of large volumes of data
    produced at different facilities, analyzed via often complex
    workflows, and consumed by a large and distributed user community.

-   *Information management*. In addition to the problems due to large
    data volumes, DOE experimental science disciplines face challenges
    associated with increasingly complex data: expanded experimental
    modalities, different analysis techniques. Again, there are
    opportunities to achieve technological and methodological advances
    that benefit multiple disciplines.

-   *Cybersecurity for open facilities*. DOE user facilities face the
    increasingly difficult task of operating open facilities in an
    increasingly hostile security environment. Policies designed to
    protect facilities against cyber attack can easily reduce the
    utility of those facilities to their users, and hinder the free flow
    of information needed for productive experimental science.

Third, the program will develop linkages and coherent approaches that up
to now have been neglected but which have strong potential to empower
future revolutionary advances in experimental facilities and science.
For example:

-   It will produce the technological and methodological advances needed
    to link computation tightly with experiment, manage large quantities
    of complex experimental data, and enable remote participation in
    experiments.

-   It will develop expertise in both discipline-specific information
    technology and underlying computer science, and establish the
    cross-disciplinary connections needed for further progress.

-   It will engage talented faculty at U.S. universities in addressing
    critical problems of importance to DOE, and train the students that
    DOE needs to provide the next generation of technologically literate
    experimental scientists.

#### Program Organization and Scope

In defining a program of this sort, we must address the question of what
structure and activities are most likely to result in broad and deep
impacts on DOE facilities and science. A new program of say \$30M a year
will represent a tiny fraction of the total Office of Science
budget---indeed, it will correspond to only a small fraction of the
total information technology budget across Office of Science
experimental programs.

We believe that the strategies pioneered and refined within the SciDAC-1
program represent a good starting point for an effective program.
Specifically, we point to the following ideas:

-   The *Scientific Application Pilot Program* (SAP) supports projects
    that use application science drivers to motivate a focused and
    directed multidisciplinary attack on some critical problem(s).
    Successful SAPs produce concrete advances in their specific
    scientific disciplines, provide exemplars that can guide work in
    other related disciplines, and advance the state of the art in
    methods and tools.

-   *Integrated Science Infrastructure Centers* (ISICs) attack
    cross-cutting technical problems relating to algorithms and
    software. Focused on development rather than fundamental research,
    ISICs increase the effectiveness of the entire program by
    concentrating computer science expertise on the most challenging
    technical problems faced by application groups.

-   Both SAPs and ISICs are funded at a level that allows substantial
    progress to be made within a short amount of time, and integrated
    program management works to achieve cross-fertilization across
    different projects.

We believe that the same organizational structure can be applied to good
effect within the program described here. We have already noted several
scientific application problems that can clearly benefit from a SAP-like
approach; in Appendix A, we both expand upon these descriptions and
introduce others. Thus, we advocate that a SciDAC-2 program support
**Integrated Science Infrastructure Centers** focused on experimental
sciences. The following characteristics define an effective project of
this sort.

The project is focused on achieving substantial and quantifiable
advances in scientific practice within one or more fields of
experimental science.

-   Results are available from preliminary studies that suggest with
    high likelihood that the proposed advances are achievable within the
    project timeframe.

-   The project has strong support from the relevant program
    office(s)---as evidenced, for example, by significant matching
    funding.

-   The results of the work are to be useful outside the individual
    discipline(s) within which the work is performed.

We are also convinced that there are big opportunities for cross-cutting
technology development projects that can simultaneously exploit
commonalities across different application projects and work to overcome
stovepipes within different application communities. (See Section 4 for
an analysis.) Thus, we advocate that the ASDES program support
**Integrated Science Infrastructure Centers** focused on experimental
sciences (EISICs). The following characteristics define an effective
EISIC project.

-   The project is focused on producing, in a timely fashion,
    well-defined software tools and/or algorithms for which a clear need
    can be demonstrated across several DOE experimental science
    communities.

-   Several DOE experimental science communities have expressed a
    commitment to adopting the results of the work.

-   Results are available from preliminary studies that suggest, with
    high likelihood, that the proposed tools can be developed within the
    project timeframe.

**Cross-Cutting Technology Opportunities**

While each DOE experimental facility and research program has its own
unique concerns, analysis of technical requirements shows that there are
also significant overlaps in technology requirements. We present here
five examples of cross-cutting technology areas identified during the
August 2-3 workshop as compelling EISIC candidates.

**Information management**. A first quasi-ubiquitous requirement is for
tools for creating, searching, and managing information, whether raw
data, metadata associated with data, or new data and metadata generated
via analysis and simulation---in a manner as automated as possible.
Applications in such domains as the nanosciences, environmental
sciences, biology and combustion need rich semantic representations of
the semantic content of data objects, their processing history, and
their relationships to other objects. They also need more structured and
automated ways of creating new data and to generalize traditional
process description methods (i.e., workflows) to include experiment
management, incremental experiment planning, and interactive steering.
They need ways to present data and information to a broad community of
users, building on technologies such as those found in the Earth System
Grid to provide deeper semantics, better federation across data storage
containing heterogeneous data types, and integration of real-time data
from a variety of observational sources. Another set of requirements
that arose repeatedly in discussions was to generalize current
information management techniques to address the unique characteristics
of experimental data, addressing issues of data quality and confidence,
and the realities of missing, corrupted, and erroneous data. Such
concerns may require rethinking data discovery, data representation, and
management techniques. An EISIC in this area would find customers from
such diverse domains as combustion, environmental sciences, biology,
physics and nanosciences.

**Distributed computing technology**. SciDAC-1 supported pioneering work
on basic services for securely and reliably locating, moving, and
accessing large scale data sets---work that has underpinned the
successes of SciDAC-1 projects such as Earth System Grid, Fusion
Collaboratory, and Open Science Grid. A need that was raised repeatedly
while preparing this report was for tools that could build on these
services to enable dependable and predictable data management and
support for complex workflow and access policies within and across
large, distributed, collaborative projects. One overarching requirement
is to deliver end-to-end performance guarantees to individuals and
communities, within the constraints of user, facility, and community
policies concerning resource availability. A 2004 DOE Office of Science
report on Scientific Data Management
(<http://www.sc.doe.gov/ascr/Final-report-v26.pdf>) identified common
needs across diverse disciplines. An EISIC in this area would find eager
customers from such diverse domains as combustion, environmental
sciences, biology, nanosciences, and physics.

**Comparing experiment and simulation**. While SciDAC-I projects are
exploring aspects of this problem related to community analysis
environments, feature detection, feature-based indexing, and data
translation, the cross-cutting requirement to handle comparisons of
high-throughput, terascale data has not been address in previous SciDAC
research activity. Science applications will require timely integration
of simulation and observation data in several ways, for example in
climate research an integrated environmental data portal that allows
scientists and policy makers to pose complex questions against what will
ultimately be petabytes of simulation and observational data is needed.
New discoveries in nano sciences can be made possible by enabling data
comparison of experiment with experiment and experiment with simulation.
In order to maximize the LSST science return, results from massive
surveys need to be compared to the detailed predictions of the massive
cosmological simulations of the formation of galaxies and large scale
structure. This will enable validating the simulation models and
algorithms and the functionality of the data analysis pipelines. One
accepted mechanism for choosing between different cosmologies is to
simulate the universe using different assumptions and compare with
observation. A key in this process is the accurate modeling of the
uncertainties, both statistical and systematic in the observational
errors. This area of work may involve developing new interpolation
techniques, new approaches to comparative scientific visualization, and
on-line statistical or quantitative analysis approaches as well as
efficient data management inclusive of efficient network transfer.

**Open Science Grid**. SciDAC-1 enabled the creation of a
national-scale, multi-agency distributed computing and storage
infrastructure, the Open Science Grid (OSG). This infrastructure is
already delivering millions of CPU hours and many terabytes of storage
capacity to physicists, biologists, and chemists who need substantial
*capacity* for loosely coupled data analysis tasks but do not require
the high-end *capability* offered by tightly coupled supercomputers. An
EISIC focused on updating and expanding in order to provide a stable,
production level distributed environment for DOE science will find many
customers. Application projects already using OSG capabilities include
numerous projects in high energy and nuclear physics, and genome and
proteome analysis projects in biology. The infrastructure can also
support the many DOE facilities such as ALS and the NSRCs that do not
have the ability or mandate to run large dedicated computational or
storage facilities, yet whose efficiency and effectiveness can be
improved via increased use of computation. Ultimately, OSG should be
able to deliver large-scale computing and storage capacity to any
application project that needs such capabilities.

**Security for experimental science**. Scientific experiments have
become global, collaborative, and distributed endeavors in which a broad
range of resources must be accessed, whether to complete an experiment,
run a simulation, or search databases. While numerous projects are
building the applications and middleware to implement the infrastructure
needed by distributed science, including such issues as authentication
and authorization, little attention has been paid to the cybersecurity
of these environments. Often in fact, distributed science traffic
bypasses existing site cybersecurity mechanisms. However, just as these
new distributed collaborative science models are developing, so too are
new, more malicious cybersecurity attacks. An EISIC to work with science
projects and sites to develop cybersecurity solutions to support user
facilities and distributed science would address a critical need. This
project would work to bridge the ever-widening gap between the tools
needed to protect distributed science and the cybersecurity tools and
techniques in use today.

**Risks**

We analyze briefly factors that have the potential to diminish the
effectiveness of a SciDAC-2 program focused on experimental sciences.

A first risk is that a program in this area does not receive strong
proposals. However, we view that result as extremely unlikely. Several
SciDAC-1 projects that bear on this area have already achieved
significant success and have demonstrated the potential to continue to
make a significant impact. The set of "application opportunities"
provided in Appendix B, and the enthusiastic response to the August 2-3
planning workshop despite little notice (see Appendix A), are two
additional indications of the breadth and depth of interest in this area
across the DOE system.

A second risk is that the DOE system and its university collaborators do
not possess the expertise needed to conduct an effective program of work
in this area. Again, the sense of the authors of this report is that
this is far from the case. Several successful projects have explored
(and in some cases pioneered) relevant techniques. Interactions at the
August 2-3 workshop suggested that within the DOE community lurks a
tremendous number of unexploited ideas

A third risk is that projects in this area will not achieve a lasting
impact on experimental science within DOE. As in the case of simulation
science, the transition from "pilot" to broadly adopted "practice" can
be fraught with difficulties, particularly when funding is tight. The
ultimate success of a program focused on experimental sciences will
depend as much on appropriate outreach to relevant DOE programs as it
will on the technical accomplishments of the projects. However, we see
no particular reason to believe that experimental projects face greater
challenges than projects focused on simulation.

Appendix A: Workshop Participants

# 

  -------------------------------- --------------------------------------
  Abdulla, Ghaleb                  LLNL

  Agarwal, Deborah                 LBNL

  Allcock, William                 Argonne

  Anderson, Gordon                 PNNL

  Babnigg, Gyorgy                  Argonne

  Basney, James Alan               NCSA

  Cook, Kem                        LLNL

  Foster, Ian                      Argonne

  Gofron, Kazimierz                Argonne

  Gottschalk, Erik                 Fermi

  Grossman, Jeff                   LBNL

  Hendrickson, Bruce               SNL

  Jackson, Keith                   LBNL

  Keahey, Kate                     Argonne

  Kesselman, Carl                  USC Information Sciences Institute

  Kolda, Tammy                     SNL

  Koontz, Annette                  PNNL

  Lauret, Jerome                   BNL

  Livny, Miron                     U. Wisconsin Madison

  London, Rich                     LLNL

  Mancini, Derrick                 Argonne

  Mauth, Jeffrey                   PNNL

  McCune, Charles                  Princeton Plasma Phy Lab

  Middleton, Don                   NCAR

  Mount, Richard                   SLAC

  Myers, James                     NCSA

  Oldfield, Ron                    SNL

  Papka, Mike                      Argonne

  Peterson, Elena                  PNNL

  Petravick, Don                   Fermilab

  Popescu, Razvan                  BNL

  Pordes, Ruth                     Fermilab

  Pouchard, Catherine              ORNL

  Rahn, Larry                      SNL

  Rotenberg, Eli                   LBNL

  Samatova, Nagiza                 ORNL

  Schissel, David                  General Atomics

  Schulthess, Thomas               ORNL

  Shinn, Neal                      SNL

  Shoshani, Arie                   LBNL

  Strelatz, Richard                LANL

  Summers, Michael                 ORNL

  von Laszewski, Gregor            Argonne

  Walter, Howard                   LBNL

  Wells, Jack                      ORNL

  Wenaus, Torre                    BNL

  White, Vicky                     Fermilab

  Williams. Dean N.                LLNL
  -------------------------------- --------------------------------------

**Appendix B: Example Application Opportunities**

We present here brief descriptions of challenges identified within
different experimental science facilities and domains, and ideas as to
how SciDAC-2 projects could help address those challenges. We emphasize
that this set is chosen to be representative, but is far from
comprehensive.

**B.1 Experimental Combustion Science**

Contact: Larry A. Rahn (<rahn@sandia.gov>), Sandia National Laboratories

Experimental combustion science is entering a new era of advanced
laser-based measurements that are designed to pursue the discovery and
validation of phenomena, models, parameters, and approaches that, with
concurrent modeling and simulation research, will enable predictive
design of innovative combustion systems. Currently planned SciDAC
ultra-scale simulations in combustion are targeting the specially
designed flames that are the subject of many of these advanced
measurements, providing unprecedented opportunities to explore the
fundamental multi-dimensional turbulence-chemistry interactions that
underlie our understanding of many ignition, stability, emission, and
dynamics issues in turbulent combustion. Others of these advanced
experiments support the development of chemical and thermodynamic models
that span multiple spatial scales and facilitate a systems approach to
combustion science.

The advanced experimental facilities that have come online around the
world include several that are part of the Phase II expansion of the
DOE/BES Combustion Research user facility (CRF) at Sandia National
Laboratories. The latter include high-throughput laser-imaging
experiments in laboratory turbulent flames and optically accessible
internal combustion engines, combined optical (including laser and
accelerator light sources) and mass spectrometry kinetics measurements,
and laser spectroscopic measurements in support of quantitative
diagnostics. Especially notable are laser-imaging measurements of
turbulent flames at the CRF, a few US Universities, and several EU
laboratories that are associated with a decade-long US/DOE-led
international collaboration (<http://www.ca.sandia.gov/TNF/>) that
shares experimental data and modeling results. This community is
currently facing the challenge of transitioning from sharing
point-measurement data and modeling parameters using traditional
hand-built web pages containing Zip files of data, to sharing
experimental data that exceed 1 TB per flame condition, and comparing
with multi-TB direct simulation results. Over the next 5 years,
experimental data from the expanding collection of standard flames
(about 12 currently, with more complex flames planned) documented in the
international turbulent combustion community is expected to exceed 20
TB. Other experiments, such as those in kinetics (e.g., see *Science*
**2005,** 308, 1887-1889), are just now beginning to publish data to the
combustion community with the help of new technologies and leadership
developed under SciDAC-1 such as the Collaboratory for Multi-scale
Chemical Science. Such collaborative approaches are enabling new
approaches to analysis and model development that cut across scales and
disciplines, as well as across institutions and agencies (including DOE,
NSF, NIST, and NASA) to broadly facilitate chemical science.

The international experimental combustion science community requires
significant new infrastructure and tools to fully exploit the
unprecedented opportunities they face. These new capabilities require
interdisciplinary approach and enabling collaborative tools that have
been initiated in the SciDAC-1 program. But the work has only begun, and
significant new challenges are emerging. The collaborative mining,
analysis, and modeling of phenomena revealed by multi-dimensional
combustion data is a new challenge for both experiments and simulations.
Scalar gradients, for example, and their correlation with chemical
reaction and turbulent fluid features must be quantitatively
characterized, modeled, and compared between experiment and simulation.
Collaborative research results must be tracked, their pedigree
documented, and published in new web-based forums from which additional
science pursued and applications can be developed. New approaches must
be developed (including technical, administrative, and funding) that
support continuous innovation, development, and operation of the
infrastructure and tools that enable and collaborative scientific
discovery from ultra-scale data sets. Furthermore, these approaches must
also support needed multi-scale integration data products and models
needed for emerging combustion systems science.

**B.2 Advanced Computing for the Linac Coherent Light Source**

Contact: R. London ( <rlondon@llnl.gov>) LLNL

When it begins operation in 2009, the Linac Coherent Light Source (LCLS)
will produce ultra-short x-ray pulses ten billion times brighter than
currently available. This revolutionary capability will enable a new
field of research that is becoming known as "Ultrafast Science." This is
untrodden territory that will certainly pose unique challenges in
modeling and data analysis, especially with the anticipated break-though
beyond current detector technology. The timely development and
application of advanced computing tools for the design of experiments
and for data handling and analysis will ensure the realization of the
full scientific promise of LCLS. The two examples below will probably
prove to be only the tip of the LCLS iceberg.

In contrast to existing x-ray sources, the LCLS pulses will interact
strongly with the materials to be studied. Predicting these interactions
via advanced simulations would greatly aid in the design of experiments.
This is especially true for biomolecular imaging and high energy density
physics experiments. The aim of first is the determination of the 3-D
structure of single, uncrystallized, biological molecules. By removing
the need to crystallize molecules, large new classes of structures can
be determined. This experiment would benefit from several developments.
A complete computational simulation is needed to design the experiment.
This will require the development of more efficient and adaptable
massively parallel techniques to extend molecular dynamics simulations
to long times and the extension of molecular dynamics models to include
new physics, such as electron transport. The 200 MB/second data rates
will create several data challenges, for example recording, managing and
moving the data and analysis to enable data classification, angular
orientation and reconstruction to produce 3-D models from the 2-D image
collection. Simulation algorithms need to be developed to produce
synthetic data sets that can be used to test newly developed data
analysis algorithms. Another LCLS experiment aims to study the structure
and ultrafast evolution of high-density plasmas using the LCLS x-ray
pulses to create and/or probe the plasmas. Major computational
developments are required to design and analyze these experiments
through simulations of plasma generation, atomic kinetics in dense
plasmas, spectroscopy, and equation of state.

A multidisciplinary effort involving computer scientists, physicists and
biophysicists during the next few years prior to the turn-on of the LCLS
could have a large impact on making the biomolecule and high energy
density experiments a success by helping to design the experiment and by
developing the necessary tools to handle and analyze the data. Similar
computational focus would also greatly enhance other experiments.

**B.3 An Earth Science Knowledge System**

Contact:

Global society demands that climate scientists develop a quantitative
knowledge of the Earth system at sub-continental to global scales to
enable the future prediction of its evolution. They will gain this
knowledge through advanced data analysis, successive generations of
Earth system models and the ability to merge the two through real-time
data assimilation capabilities.

SciDAC-1 brought major advances in climate modeling and the management
and sharing of the distributed terascale data via Grid technology. Even
so, the climate research community has a daunting task of locating,
acquiring, and integrating collections of *both* simulation and
experimental data that underpin the evolving multidisciplinary study of
the Earth systems. Data sources can span models (global and regional)
and observational data including ground measurements, radars, satellite,
ocean buoys, and many others. It is diverse, distributed, real-time in
some cases, and petascale in volume with multiple agencies as providers.
Also, current models are not suited to take advantage of the huge
progress made in the collection of data by instruments.

An earth science knowledge system that would bring together simulation
and experimental data from a variety of sources and a variety of sensors
would accelerate global studies and enable these to be leveraged and
bear insight into Earth science processes, trends, and interactions. For
example,

-   What processes determine the evolution of the Earth system at the
    regional scale, and how is climate evolving region by region?

-   How do human activities (of all kinds -- energy production, land
    use, deforestation, reforestation, etc) influence the evolution of
    the Earth system?

-   Can life become accustomed to near and long-term changes in the
    Earth system?

Additional projected impacts would include greatly enhanced scientific
productivity along with new resources for developing decision-making
tools for societal and environmental impacts including, water resources,
air quality, and optimal energy production and usage strategies.

**B.4 Remote Operation of the LHC Accelerator, CMS Experiment, and
ITER**

Contact: Erik Gottschalk (erik@fnal.gov), Fermi National Accelerator
Laboratory

Contact: David P. Schissel (schissel@fusion.gat.com), General Atomics,
DIII-D National Fusion Facility

Over the next decade the worldwide experimental programs in fusion and
high-energy physics will be centered on facilities located outside the
US, in Europe. These facilities will be the largest and most expensive
scientific instruments ever built. For fusion, the International
Thermonuclear Experimental Reactor (ITER) is a burning plasma magnetic
confinement experiment located in France. For high-energy physics, the
Large Hadron Collider (LHC) is the world's highest energy particle
accelerator located in CERN, straddling the border between France and
Switzerland. Carrying out large-scale experimental research activities
in interna­tional collaborations is a technically demanding problem,
requiring the provisioning of a working environment for off-site
personnel engaged in experimental operations that is every bit as
productive as what is onsite. With these facilities being located
outside the US, an effective collaborative infrastructure is required
both for the success of the projects and to maximize the value of ITER
and LHC to the US programs.

An effective integrated collaborative working environment requires
off-site scientists to be actively and effectively engaged in all
aspects of the experiment including experimental design sessions,
experimental operations, data analysis working sessions, hardware and
software training, seminars, and ad-hoc discussions. These requirements
go far beyond present day collaboration technology by requiring the
coupling of a large number of simultaneous heterogeneous participant
sites and the seamless sharing of data and applications in an integrated
environment. The solution to these challenging problems requires the
partnering of domain experts and computer scientists to design, evaluate
and deploy effective remote operations capabilities. Solutions can be
developed and tested on present day fusion and high energy physics
experiments for immediate scientific benefit and also in preparation for
future distributed science facilities such as LHC and ITER. Building on
the successful initial deployment of the collaborative control room for
fusion science (FusionGrid) developed under SciDAC-1 will greatly
accelerate the success of this effort across these two scientific
disciplines.

**B.5 Enabling Discovery in Experimental Biological Science**

Contact:

*Advanced analytical technologies*: Building predictive models at the
systems level faces a significant challenge of understanding extremely
high-dimensional, multi-model, distributed and dynamically changing
experimental data. Not only the size but the extreme richness and
complexity of the data for which the analytical and mining technologies
are quite limited. Most of the data interprestation/prediction tools do
not provide any confidence intervals.

*Data surety and uncertainty quantification*: Experimental biological
data is extremely noisy, incomplete, and unreliable due to propagated
errors from devices, experimental protocols, limitations of experimental
technologies leading to high false positive or false negative rates.
Lack of methods for statistical and robust quantification of inherent
errors and errors propagation prevents predictive biology and proper
data interpretation. Replications across samples and multiple runs have
to be enforced along with the tools quantifying the reproducibility of
the experiments.

*Technologies for dynamic updates of data in databases*. A single
project can go over 10s of iterations of experimental protocols over a 3
month time-frame capturing various features and characteristics.
Updating the database and making it possible to compare across these
protocols and the ones produced by other research groups is a challenge.
Databases are not well designed to keep track of schema evolution.

*Transparent data movement, data access, query, retrieval, and
representation.* The facility-centric mode of data production generates
data centralized in few locations that should be shared, accessed, and
queried by very large and diverse community of users. Data movement
(terabyte scale is routine) is typically done by emailing to data
producers and waiting for their ability to find this data, create its
copy on some media (hard-drive)and fed-exing the media. For long-term
collaboration, setting up a specialized infrastructure that overcomes
the issues of cybersecurity, firewalls, diverse institutional policies,
etc is tedious and time consuming process resolved on a case by case
basis. Querying of these huge data files generated over incompatible
experimental protocols with different analysis tools used for
postprocessing of the raw data is impossible.

*Facility for data processing and analysis tools.* Various tools are
being developed that require advanced sophisticated users to use, access
to proper software and hardware infrastructure to run them. Having a
facility that provides expertise on how to use these tools, transparent
access to run them and send the results back (like Blast at NCBI),
compare the results across different tools (e.g., SEQUEST vs. MASCOTT)
is desirable but currently impossible. Each small research group builds
their own infrastructure, has huge learning curve on how to use tools,
and covers quite a limited set of them.

*Distributed workflows on top of distributed LIMS*. Data processing is
an iterative complex process that requires advanced analysis tools that
have access to data generated at experimental facilities. The process is
often repetitive subject to re-run the analysis once the DB is updated
or new tools are developed. Plus, the existing LIMS systems that capture
the experimental data are incompatible, do not run across distributed
sites; have limited abilities to capture experimental protocols.
Entering the metadata into LIMS is a pain. The analysis tools and
problem solving environments do not operate on top of LIMS, which makes
analysis of raw data impractical. PSEs have quite limited analysis
capabilities -- basic tools are included.

*Enabling community annotation and data curation efforts*. Annotation
and curation of data is largely community-centric (e.g., Shewanella
federation). Capturing the process by multiple research groups is
impossible. Physical presence in the same room and email exchnages is
almost the only way. The growing gap between the exponentially growing
number of genomes sequenced and omics data produced and the amount of
curated/annotated data with the human in the loop involved is
frightening. String need for facilitating infrastructure with the proper
access controls, data provenance, etc.

**B.6 Enhancing Material Science Beamlines**

Contact:

DOE materials science beam lines are experimental facilities at X-ray
and neutron sources dedicated to analysis of structural, electronic and
magnetic properties of materials. These beam lines cost typically \$3-6M
dollars and are heavily oversubscribed. In many cases, the data rates
are now outpacing the IT infrastructure co-located at the beam lines.
Since these facilities are typically oversubscribed it makes sense to
maximize their value by maximizing efficiency. Optimizing user beam time
can help significantly improve user throughput on instruments thus
accelerating scientific research overall.

There is an element of discovery at these facilities. An initial
overview of the data's full phase space covers many degrees in freedom
leading to high dimensional data arrays. These data need to be processed
into forms that humans can understand, in order to direct the experiment
quickly to the next phase. It is therefore essential to introduce high
performance computing for fast online analysis. This would allow more
aggressive experiments since the feedback loop for experiment steering
will be closed in pace with the data acquisition. It would also allow
more users to come through the facility, because the confidence level in
the quality of the data is increased.

For example, one can collect about 1000 mega pixel images in about an
hour. Presently it takes more than an hour to load and process this data
set, so that the idle time of the experiment runs about 50%. We need
real time processing of the data so that they are properly reconstructed
at the end of a scan in order to direct the next measurements.

New paradigms of efficient beam time allocation would be enabled by
telepresence and teleoperation because the total time for a useful
experiment can now be one day instead of one week. This has not been a
traditional area for materials science.

**B.7 Large-Scale Computational Services in Support of ITER Operations**

Contact: David P. Schissel (schissel@fusion.gat.com), General Atomics,
DIII-D National Fusion Facility

The International Thermonuclear Experimental Reactor (ITER) is a burning
plasma magnetic confinement experiment, the next major step in an
international program aimed at proving the scientific viability of
controlled fusion as an energy source. Located in France, it will be the
largest and most expensive scientific instrument ever built for fusion
research. As is already known from present day experiments, experimental
operations are critically guided by near real-time data analysis.
SciDAC-1 developed large-scale non-linear fusion codes to perform plasma
simulations that could be deployed in support of off-line experimental
data analysis. Yet, the ability to have these massive supercomputer
applications support quasi real-time analysis of experimental data will
bring unprecedented improvements by greatly shortening the time-scale
for the comparison of experiment to simulation resulting in more
informed experimental decisions. This will require the dynamic coupling
of experiment and high-powered computation with coordinated deployment
of world wide distributed resources such that computations can be
scheduled as required to deliver analyzed data back to the experiment
within a guaranteed time frame. The ability of the US to deploy this
capability for ITER will require the solutions to a variety of problems
including network quality of service, computational scheduling,
large-scale data management, and visualization. This work will leverage
the results of the initial SciDAC-1 investment in these codes that has
placed the US in a leadership position in fusion research.

**B.8 Empowering RHIC Users with 2nd-Generation Analysis Tools**

Contact: Jerome Lauuret (<jlauret@bnl.gov>), Brookhaven National Lab

The US-based RHIC facility and scientific program have attracted a
community composed of hundreds of physicists from around the world to
study what the universe may have looked like in the first few moments
after its creation. The world-class scientific research facility located
at the Brookhaven National Laboratories has been a leading force in the
study of the Quark Gluon Plasma and generates PByte scale of data a year
going toward GB/sec data rates in the out years. Our collaboration, as a
Nuclear Physics community with computer scientists leading the field of
data management in the SciDAC-1 program, has allowed us to create a
production level data-grid for data movement and to increase the
scientific output. Leveraging on our past success, our active community
is looking to further develop tools and integrated analysis capabilities
allowing reaching the far corners of the statistically challenging and
real-time analysis through analytics. The scientific benefit of making
possible to efficiently reach the full coverage of the phase space of
science through cataloguing and global metadata management of
distributed data will open opportunities scientists would not otherwise
consider.

-   Consensus in common interfaces and architectures;

-   Interoperation and collaboration with peer grid infrastructures;

**B.9 ARM/ACRF Program Needs for Service Development**

Contact: Annette Koontz (<Annette.Koontz@pnl.gov>) Pacific Northwest Lab

The Atmospheric Radiation Measurement (ARM) Program is the largest
global change research program supported by the [U.S. Department of
Energy (DOE)](http://www.doe.gov/engine/content.do). It was created to
help resolve scientific uncertainties related to global climate change,
with a specific focus on the crucial role of clouds and their influence
on radiative feedback processes in the atmosphere. The primary goal of
the ARM Program is to improve the treatment of cloud and radiation
physics in global climate models in order to improve the climate
simulation capabilities of these models. ARM scientists research a broad
range of issues that span remote sensing, physical process investigation
and modeling on all scales. ARM\'s site operators focus on obtaining
continuous field measurements and providing data products to promote the
advancement of climate models.

ARM\'s climate research sites serve as a national scientific user
facility for collaborative research primarily with university,
government agency, and national laboratory researchers. These sites
provide significant research capability for the global scientific
community. The resulting new [ARM Climate Research Facility
(ACRF)](http://www.arm.gov/acrf/index.stm) has been designated a
national user facility for the purpose of providing this unique asset
for the study of global change to a broader national and international
research community. Proposed projects at the ACRF are reviewed by the
[ACRF Science Board](http://www.arm.gov/acrf/science_board.stm), a
highly respected group of scientists who assist with reviewing proposals
for use of the facility.

ACRF provides users access to the data through the ARM Archive at the
Oak Ridge National Laboratory (ORNL). It also supports multi-national
field campaigns at its measurement facilities. Given the user population
of both U.S. citizens and foreign nationals, systems security and user
authentication are critical issues for the success of the program. We
need to deal with the conflicting requirements of security and easy
access by users.

The ACRF needs tools that can be applied to globally distributed systems
in each of the following areas:

-   Secure user authentication with single logon per session for both
    shell and web access - our users are not anonymous

-   Intrusion detection technologies - both host and network based

-   Automated system management technologies for detecting and
    correcting vulnerabilities on globally distributed systems

The ACRF manages many dozens of instrument data streams that transmit
their data across a variety of network paths. A variety of home-grown
tools are used to manage the data flow and movement of the data on its
way to the ARM Archive. There is no small amount of data movement
software available publicly and commercially. The ideal solution would
provide integrity, timeliness, security, and efficient use of bandwidth
and scale. The current solutions tend to only address a subset of these.
The ACRF would benefit from an effort to combine/enhance/rewrite the
variety of tools already available. Examples include sftp, gridftp,
bbftp, ldm, and bittorrent. While ACRF could continue to manage the data
flow with available tools, one can imagine several tasks with hundreds
or thousands of sensors that need real-time, secure, and reliable data
flow across limited bandwidth.

As stated above, the goal of the ARM program includes "acquire . . .
high quality measurements." One of the keys of this goal is the ability
to describe and communicate the quality of the data to scientists. The
ARM Program has expended a lot of effort in this regard, but has yet
only tackled the "tip of the iceberg." The ARM Program would benefit
from work in establishing file format standards, software tools, and
conventions specific to tracking, managing, and communicating data
quality.

The ACRF provides infrastructure support to the ARM Program and is not
funded to perform research in these areas.

**B.10 U.S. LHC Experiments**

Contact: Torre Wenaus (wenaus@bnl.gov) Brookhaven National Lab

The Particle Physics Data Grid and other SciDAC pilot partnerships
between computer scientists and particle physicists have driven and
signficantly increased the capabilities of distributed data movement,
and data and job management technologies. The performance and robustness
of data movement has been improved by an order of

magnitude, a typical year improved the efficiency of distributed job
scheduling and execution by a factor of 2, data from many mass storage
systems are now accessible to the grid, and joint projects across the
DOE and NSF have built and operated a common infrastructure (now the
Open Science Grid) which has supported simple applications from more
than 10 different research groups for more than 18 months.

As part of a global collaboration of more than 2000 researchers, more
than 500 US scientists at over 100 US universities and laboratories will
play leading roles in analyzing the 100s of petabytes of data yearly
from the Large Hadron Collider. Thus over the next 2-3 years we must
increase the capability, scale and robustness of our data processing
systems by at least an order of magnitude through the development and
validation of higher level services that support automated management of
massive data processing.

The US LHC experiment Computing Technical Design Reports require that
the data processing system manage and prioritize worldwide access to
hundreds of datasets of up to several hundred Terabytes each and
simultaneous use of the computing ensemble across more than twenty
independent data analysis groups and hundreds of individual users. The
requirements include such metrics as "submission of collection of
O(1000) jobs should happen within few seconds; a typical data analysis
task will translate into submission of O(1000) jobs; a failure rate of
less than the third root of the number of jobs in large tasks of up to
O(10000) jobs in 2008; end-to-end job failure rates at less than 5%".

SciDAC-2 will allow us to grasp the opportunity to continue the
successful physics and computer science collaborations in developing,
deploying, and refining increasingly capable data management middleware
and services. The ability to process the datasets in hours rather than
weeks gives the promise of increasing the rate of discovery and
innovation as well as to enrich the data sample through early feedback
of calibration and tuning parameters. The operation and evolution of the
unique OSG multidisciplinary common shared cyberinfrastructure will
service not only the U.S. LHC program but also bring significant benefit
to other data-intensive sciences.

**B.11 Nanoscience and Nanotechnology**

Contact: Derrick Mancini (mancini@anl.gov) Argonne National Laboratory

Nanoscience and nanotechnology is a new frontier with a
multidisciplinary and multitechnique approach that requires
investigators to combine data from a large complex group of experimental
and computational sources for investigating even a single nanostructure.
The nanoscience community lacks the scientific data management systems
to support the complex workflow and burgeoning datasets generated.
Computational tools for nanoscience must advance multiscale simulation
in support of experimental planning and analysis; real-time
reconstruction and processing of high-throughput data; and data-mining
of large and complex s. Simulation tools must be further developed to
produce results expressed in observables in experiments. The
multidisciplinary character requires team approaches with real-time
interaction during experiments and calculations where telepresence and
telecollaboratory is critical.

In nanosciences, various complementary measurements and techniques are
required to fully determine the relevant information. For example, the
complete structural and dynamical analysis of a nanoparticle may require
data from x-ray and neutron scattering; optical, vibrational and NMR
spectroscopies; scanning probe techniques; and electron microscopies.
Database comparison with known structures and properties would be a
crucial factor in defining the modeling. Computational tools including
electronic structure and molecular dynamics calculations would then be
employed to compute appropriate response and structural behaviors, to
test the consistency of the structure/function scheme.

For example, we will use the following typical CNM workflow, or
something similar, to guide the design and validate the implementation
of the system. In this workflow, a user designs, fabricates, and
analyzes samples by repeatedly applying the following process: model a
composite nanostructured material to determine correct parameters for
design and synthesis; deposit materials on selected substrates,
including a possible library of deposited materials; characterize the
resulting samples by analytical and structural methods; lithographically
pattern samples, including a possible library of patterns; characterize
the resulting samples by microscopy and metrology; deposit or
functionalize with additional materials, to create composite or hybrid
structures; characterize the resulting samples by analytical and
structural methods, including 2D and 3D techniques (microscopy and
tomography) requiring significant data processing for reconstruction;
and analyze characterization results including comparison back
to modeling. Note that at each stage in this process, both data products
and associated documentation (structured "metadata" and/or textual
annotations) are created and must be managed. At different stages, we
encounter a need to access both various databases and analysis codes.
Different steps involve physical experimentation, numerical simulation,
and computational data reconstruction and analysis. Our goal is that all
of these activities can be handled within a common data management
environment.

**B.12 The Spallation Neutron Source**

Ian Anderson, ORNL

When it comes into operation in 2006 the Spallation Neutron Source (SNS)
will be the world's premier user facility for materials research using
neutron scattering techniques. After an initial commissioning period and
ramp-up to a full suite of 24 instruments on the first target station,
SNS will produce approximately 1.5 TB of data per day, including raw
data, meta-data, analyzed data, and eventually, published data. These
rates will more than double if a second target station is constructed in
the future. The neutron beams at SNS will be in high demand. Efficient
use of the available beam time will require users to make intelligent,
i.e. informed decisions on the experimental protocols. They must have
the capability to treat, analyze and model their data, within the time
frame of a measurement, as is done, for example, in the medical imaging
industry.

In materials sciences, typically, various complementary measurements and
techniques are required to fully determine the relevant information. For
example, the complete structural and dynamical analysis of a
nano-particle may require data from SANS, EXAFS and RDF methods, as well
as optical, vibrational and NMR spectroscopies, scanning probe and TEM.
Database comparison with known structures and properties would be a
crucial factor in defining the modeling. Computational tools including
electronic structure and molecular dynamics calculations would then be
employed to compute appropriate response and structural behaviors, to
test the consistency of the structure/function scheme.

To turn these promises into reality, the community, and in particular
major user facilities, must empower individual researchers to use rich
resources such as databases, advanced instrumentation and the
availability of grid computing. The SNS is committed to developing a
data management structure including user friendly analysis software
aimed at achieving these goals.

The main features that a "science enabling" data management structure
should possess are:

-   Long term preservation and accessibility of raw data, metadata,
    intermediate data, simulated data and final published data

-   Metadata must be optimized for future retrieval, assimilation and
    re-use.

-   Rapid cataloguing and data mining across distributed data bases.

-   Provenance -- published data must be traceable

-   Data from different facilities/techniques must be shared and
    accessible in real time.

-   Universal authentication processes

-   Rapid data analysis and simulation capability for intelligent
    control

Recognizing the magnitude of this endeavor, the SNS is working with
several other major neutron facilities in the US and abroad to develop a
common Data Management Structure. This collaboration goes under the name
of NeSSI (Neutron Science Software Initiative). Fortunately, a common
data format for storing and transferring raw data files has been
determined in the neutron community, based on HDF structures, however
common Metadata formats have yet to be defined.

There is clearly a significant overlap between the data management
challenges faced by the major neutron and x-ray facilities and other
major science centers. SciDAC-2 program offers a unique opportunity to
provide common solutions to these challenges. Experience with the NeSSI
collaboration has shown that the priorities for collaborative
development lie in the following areas (the first two areas have been
developed from the paper by Ian Foster and Vicky White, the third is new
but equally important):

**B.13 The Solenoid Tracker at RHIC (STAR)**

Contact: Jerome Lauret (<jlauret@bnl.gov>) Brookhaven National Lab

The STAR experiment is one of the two major Relativistic Heavy Ion
Collider (RHIC) experiments at BNL. The STAR experiment records massive
amounts of data. PByte scale data recording overall per year (1012
bytes), the raw data rates envisioned by the RHIC experiment's program
will grow by an order of magnitude by 2008, reaching an online data rate
of 1 GB/sec and making *data management an ever growing challenge*.

By then, the data sets, already spanning over 4.5 Million logical files,
will reach the several 10 Million scales making *cataloguing and
meta-data management* and integrity a focus. Note as we mention
improving the data rates that the *development of libraries of* *known
algorithm for online real time application* (such as suitable for
FPGA/DSP) would greatly benefit the community as a whole: the lack of
CERNlib-like product for online programming has for consequence a
slowdown of R&D and the spread of talented resources trying to solve the
same problem in diverse experiment: improving the data rate with
real-time algorithm implementation (and spending timeless effort
debugging the algorithm developed in VHDL or other "exciting" languages)
. Such example could be the implementation of online cluster finders,
*reducing the data volume* by factors of ten and equally *increasing the
data rate for triggering*.

To date, the STAR computing model has evolved around a data-grid model:
processed data is made immediately available to remote sites where
computing resources may be available. The same model could not be
applied to the production of the raw data due to a severe lack of
*network connectivity* on long distance (average coast to coast less
than 20 MB/sec). Under such conditions, the availability of Tier0
resources or of resources, where connectivity is *from twice to an order
of magnitude higher*, is the only viable path for data processing now
(and this is not speaking to the LHC era). These current needs would
increase proportionally to our data samples. *Without* such
availability, data mining *will be* severely *challenged for the science
to be delivered in a reasonable and* *timely fashion*. However, widely
distributed data processing do raise additional challenges to the well
known areas of *data provenance and meta-data cataloguing*: scalability
and integrity of widely distributed catalogs and its complexity have
caused our community to move toward alternative solutions. In our
experiment, and as an example, we emulate (at API and infrastructure
level) a *federated database* to cope with the need for a site to
maintain (and be the sole maintainer) of its replicas while every site
have immediate knowledge of every replica available. This approach is
questionable as the *federation concept should ideally be implemented at
database level* where best suited. A second concern is the cheer amount
of data transfer needed for calibration purposes: while the data itself
per event is not significantly comparing to the size of the input, a
problem arises whenever several thousand of jobs access the database.
The lack of automatically *load-balancing or self-discovered database
service* does not make large data mining an easy task. Finally, we
believe that the *existence of a meta-data global* *namespace and data
repository* (perhaps shared by discipline) would greatly benefit our
communities in need for simulation: as an example, all RHIC experiments
do run the same Monte-Carlo event generators but not a single database
allows perusing and retrieval of the already produce data sets by the
community, effectively making diverse experiments from within the same
community (often tight to the same run and beam schedule) probably the
same generators using by then useless extraneous CPU cycles. *Associated
to a secured peer-to-peer* like *approach* and a *best use of network
bandwidth*, the collection of such already produced data would be a
tremendous help in consolidating the already existing data sets and
therefore, covering as a common project more of the needed phase space
to reach the scientific conclusions of the program as a whole.

The next level of challenges will come in the form of our distributed
computing program. While our sturdy progress has leaded us to reach a
point where our community has been empower with the ability to perform
their analysis on the Grid via several emerging technologies resting on
past successful storage management components, a few success stories do
not constitute a scalable no a stable solution.

The *lack of long-term support* of the past acquired technologies in the
area of storage for example is *a huge concern* as the *investment in
the second generation of* *storage based tools is not possible* without
solid foundations. We would strongly recommend for *SciDAC-2* to have a
component of *support in targeted areas allowing* for the experiment and
their physics to *move a step "further"*. While we recognize that DOE
programs did not to date make long term support commitments for existing
products, perhaps this could be done in the form of packaging, testing,
interoperability testing and development of enhanced version of tools
based on standard interfaces in a consistent program of work. If so,
interesting problems could be considered: for example, *event* *level
indexing of large data sample* is a known technology using data
management tools to transparently to the user move files across the
nation. The net effect of such technologies is *speeding up user
analysis* by factors proportional to the sub-sampling he makes (tags)
when the data is available locally. Currently file based, it would be
interesting to *stretch* this technology *to its limit* by developing
mechanism which would *gather the event of interest and transfer only
those*.

A few more important areas would be to have middleware components
allowing *relating application level response and cluster* or site level
*events*. The lack of such component renders the *tracking of problems*
a logistic nightmare *for the casual user*. Also, the *harvesting of
remote resource*s via service portals is in our view *requires
well*-*defined, interoperable and common interface*. Not only such
standardized interface (for example, for job submission) would allow
*easier access* to remote resources without compromising the local
scheduling policies or algorithm but also, a young but maturing
*scientist would easily adapt* to another experiment if submitting a job
would be as easy as writing a Web page with *no new-learning overhead*.
We are far from that goal since every system has its own interface. The
availability of *tools designed to create, test and on-thefly* *alter
workflows* and to allow workflows to discover services and adapt
depending on their availability and the user choice (*dynamic workflow*,
*economic model based* *workflows*, time versus reliability) are equally
important to address. With such components, one may better manage highly
prioritize tasks by selecting the most efficient (probably most costly
however cost is defined) services or on the contrary use the most
reliable (yet slower) service for enhanced integrity of the workflow.

Finally, we feel that a better interaction between our detector design
engineers and our simulation team could be achieved by the development
of tools or standards allowing for the exchange of geometry information
between (for example) CAD and "a" Geant framework. Such efforts were
started in Europe but did not reach completion while they would
certainly benefit the detector R&D area.

## SciDAC Computer Science Institute

1\. Summary

We present here a plan for the establishment of a SciDAC Computer
Science (CS) Institute. An institute is designed to attack major
problems in a field of science through a range of collaborative research
interactions, including for example, workshops and focused summer
sessions. The field of computer science is relative new in comparison to
other fields to which the institute concept has been applied; however,
there are significant challenges that prevent the penetration of
high-performance computing into the daily lives of scientists and
engineers, both within the Department of Energy and in the broader
community. It is the opinion of the authors of this whitepaper that an
institute as described can make significant progress towards resolving
these challenges.

Section 2 of this whitepaper presents the case for applying the
institute concept in the field of computer science. Section 3 discusses
in detail some of the activities that might be included in an institute
and some of the technical areas and specific challenges that are
considered to be particularly amenable to a collaborative approach.
Section 4 examines the governance model of an institute, and Section 5
addresses some of the inherent risks.

**2. Benefits of the Computer Science Institute**

Today, science and engineering are critically dependent on the existence
of robust and reliable high-performance computing applications codes.
These application codes are in turn critically dependent on the
algorithms and software developed by the high-performance computer
science community. The SciDAC Computer Science Institute will focus on
systems software and tools, algorithms, and applications software, and
they will complement the efforts of SciDAC centers and other DOE Centers
to advance high-performance computing.

The CS Institute will be primarily focused on enabling this construction
of building these essential communities by providing a forum for
collaborative research and dialogue between leaders in software
application areas, industry, and the high performance computing
community. SciDAC Institute will benefit DOE and the scientific
community by building and fostering a broader community of researchers
who understand the challenges of providing and using high-performance
modeling and simulation capabilities and are willing to address these
problems collaboratively. Specifically, these institute will provide for
the following:

-   Provide for exploration of new scientific challenges. An Institute
    will identify challenges in computational and computer science and
    provide strategic direction for researchers in the high-performance
    computing community.

-   Provide a venue for focused collaborations. An Institute will enable
    new and focused collaborations between a broad community of
    researchers, including SciDAC researchers, researchers from other
    agencies (e.g., NSF) and from the broader applications community to
    develop new insights and approaches to solving challenging
    scientific problems.

-   Broadly disseminate results. An institute will disseminate specific
    technical results through its activities. These include technical
    publications (articles, reports, books and proceedings), advances in
    software infrastructure (algorithms, software, interfaces and
    standards), and solutions to specific problems in applications of
    high-performance computing.

-   Build a sense of community among computational scientists. An
    institute will foster a sense of community among researchers engaged
    in computational and computer science research and engage this
    community to help solve its most challenging problems.

-   Foster education and training of the next generation of
    computational scientist. An institute will provide a key aspect in
    the training of the next generation of computer and computational
    scientists by engaging students, as well as faculty and senior
    researchers in its programs.

To reap the benefits described above, a SciDAC Institute must gain a
reputation for instigating seminal research through ideas and specific
software development projects. The institute must develop an identity
and reputation for leadership in computational science. Ultimately, the
strength of an institute will be its ability to foster a community
around high-performance computing, and to help solve some of this
community's most challenging problems.

Some more specific ways in which the SciDAC program and partners would
benefit from the Computer Science Institute activities include the
following:

-   An understanding of and focus on a common infrastructure for system
    software tools. For example, the optimizing compiler community would
    benefit from the availability of a common shared infrastructure that
    can serve as the base on which different efforts can build
    additional capability, avoiding the problem of each group
    essentially building from scratch, often reinventing functionality
    already available in other efforts.

-   Improved interaction between systems software/tools developers and
    applications developers. As an example, the focus of compiler
    efforts in academia and industry and the form of language features
    in HPF may have taken a very different turn if many applications
    developers were engaged early on in the development process.
    Similarly, DOE Science projects would benefit from improved
    interactions regarding libraries and software products.

-   Better and faster dissemination of developed software/tools. Both
    SciDAC and non-SciDAC projects would benefit from faster
    dissemination of recently developed enabling technologies from other
    DOE (and non-DOE) efforts like PAPI, CCA etc.

The common thread to these benefits is improved collaboration and
communication.

**3. Scope of the Computer Science Institute**

The CS Institute is unique among the other institutes around the world.
It focuses on the enabling technologies that are so critical to
productive use of terascale computers by scientists in all fields. In
SciDAC we are seeing a significant shift to a diverse set of terascale
architectures and software environments. Now, more than ever in the
past, there is a need for a *software ecology* that provides an
efficient, productive, portable environment for scientists to develop
and optimize scientific applications. Many areas of science are at a
critical juncture and require the kind of new software ideas that the
Institute would provide in areas such as scalability, high efficiency,
portability, and fault tolerance. In addition there is an exponential
jump in data volumes and complexity produced by the latest scientific
codes and experimental facilities. In some cases it is not clear how to
go beyond current paradigms. In others, researchers have not adequately
explored how to take advantage of terascale computing resources. The CS
Institute will establish programs for scientific luminaries to gather in
a critical mass to focus on key problems in enabling technologies.
Illustrative examples of such problems are described in the following.

**3.1. Programming Environments and Software Ecology.** Computational
scientists need to minimize the time they spend on the mechanics of
getting their algorithms to run correctly and efficiently, rather than
on the details and shortcomings of particular languages, tools, and
systems. The challenge is to provide programming tools that scale to
very high performance systems. For example, new tools or features that
would enable programmers to find performance bottlenecks more quickly
and perhaps even point the user to possible solutions. Software ecology
refers to the development of software tools that interoperate and fit in
their respective niche in the program development environment. As
another example, conventional task-oriented interactive debugging
techniques don\'t scale beyond a thousand or so processes, which is
inadequate for high-performance systems. Improvements in software reuse
are also to be encouraged, via tools that enable sharing components and
that aid in porting codes and libraries to new systems.

**3.2. HPC Operating Systems**. Processor vendors are adding hardware
support for virtualization technologies that enable new capabilities for
operating and runtime systems for high-performance systems. There is
currently little understanding of how best to utilize these new
capabilities to address the important problems in deploying system
software for large-scale systems. The CS Institute could help bring
together researchers who are developing the virtualization technology
with system software researchers in the HPC community to help advance
this understanding and to help develop standard environments for
configuring and deploying different virtualization environments.

**3.3. Improving Application Performance.** It is widely recognized that
the small fraction of peak performance that is achieved by many
applications is a major problem. In many cases, one significant
contribution to this low performance is not due to parallelism but to
the difficulties in achieving good single node performance. Projects
such as ATLAS and OSKI have shown that significant performance gains can
be made applying source-to-source transformations to dense and sparse
matrix operations. For example, ATLAS has shown that an order of
magnitude improvement in performance is possible over the code produced
by a production-quality optimizing compiler. Other projects have
developed loop transformation tools that can rewrite user code to
achieve better performance. However, these techniques are not routinely
applied to applications, and even the simplest problems, such as dense
matrix-matrix multiplication, are rarely handled well by current
optimizing compilers. A greater understanding of how to apply such code
optimization transformations will help applications increase single node
performance, thus making better use of the available computing
resources.

**3.4. Software Library Portability and Fault Tolerance.** The ability
for applications to be mapped onto hardware effectively is critically
dependent on the software of the overall system. Application programmers
and users will need software that exploits the features of any given
machine without heroic efforts on the programmer's part. Library
software ideally should promote effective parallel processor usage and
efficient memory use while hiding many of the details. It should allow
portability of well-designed application programs between different
machine architectures, handle dynamic load balancing, and also have
fault tolerance. Specific important areas include software for
future-generation computing platforms, support for retargetable
applications, self-tuning software, high-level programming systems, and
computational mathematics software for heterogeneous systems

**3.5. End-to-End Workflows.** As compile times decrease, and debugging
times comparatively increase, the time to write and debug code becomes a
limiting factor. The process to build the software, the \"end-to-end
workflow\" of building software becomes the limiting factor. Solutions
to this problem don\'t tend to come from the teams that build the
components, since the individual tools (compiler, debugger, support
libraries and OS modules) are developed in isolation. The computer
scientists and application developers are the proving ground where the
\"rubber meets the road\", and a whole build system is only used
together the first time in the care of the application developer. A
collaboration where hardware developers, OS developers, compiler
developers, and application programmers work together to compile proof
of concept codes would be an excellent way to find and begin to smooth
the rough edges that inevitably accompany new software build systems,
and would considerably accelerate the shakedown process of new
development systems.

**3.6. File Systems, I/O, and Data Management.** Scientific projects
create geometrically increasing data over time. Increasingly fast
computational resources, increased processing capability, and cheaper
storage media contribute to the flood of data that collaborations want
to store. File systems, IO, and data management are vital for
implementing large advanced file systems. Identifying early problems
with new file systems and techniques for data storage, and understanding
the implications of problems and guiding solutions help avoid the
problem of vendors implementing something that is incompatible with the
way the systems are actually used. An example topic is the development
of HPC extensions to the POSIX I/O API that address the scalability and
performance, including the development of a proposal to the POSIX
standards committee, along with documentation that supports the
performance claims for the extensions.

**3.7. Distributed Computing.** Many of the problems presented so far
have focused on a co-located, large-scale computing capability aimed at
solving large problems. A Computer Science Institute must also address
the question of how to use the computing capacity provided by networked
computing systems. Many of the challenges involve how to store and
manage the large amount of data that is being collected and generated.
Other challenges in distributed computing that could be addressed by a
CS Institute include workflow management and data mining.

**3.8. Networking.** There are significant challenges in delivering the
full capability of a high-performance network to the applications and
services that it supports. Network resource allocation and management
strategies become particularly important as the number of network nodes
in a system reaches several thousand, and these strategies can have a
direct impact on the scalability and performance of applications.
Various levels of software for networking, including application-level
network interfaces, operating system services to support networking, as
well as embedded software inside network devices, present significant
challenges for next-generation systems.

**3.9. Architectures.** Advancements in computer architecture for
next-generation systems have a fundamental impact on the design and
implementation of software for these systems. In many cases, new
architectural features can only be leveraged by changes in the
associated programming model, toolset (compilers), and operating system.
For example, all of the DARPA HPCS vendors are pursuing partitioned
global address space programming models (PGAS) to exploit various
features of their new architectures, Some of these vendors are pursuing
language-based implementations of PGAS that require completely new
toolsets and application development environments.

**4. Activities of the Computer Science Institute**

The Institute sponsors a variety of activities to fulfill its mission.
Activities will vary in format to satisfy different purposes and needs.
We describe four possible formats as *examples* of different types of
activities we anticipate being useful to the DOE computational science
community. We encourage proposers to be flexible with respect to the
format of activities they will host. These example formats include: an
extended summer program for new researchers, intensive training schools,
coding camps for focused problem solving, and workshops intended to
bring together a technical area.

**4.1. Topic-Specific Programs.** This format facilitates focused
multi-disciplinary work to address a particular problem of interest to
the HPC community. It brings together researchers to work on the problem
for an extended period of time. Graduate students and postdocs would
attend the entire program, and faculty and laboratory research staff
would spend possibly shorter periods of time advising, reviewing, and
interacting with the students and post docs. The program would typically
be followed by a workshop with broader attendance at which the results
of the summer work would be presented.

**4.2. Short Courses or Summer Schools.** Intensive schools are intended
to give researchers at all levels training in new concepts, technology,
techniques and software. A collection of experts in the targeted area
will present material in a class or tutorial format, typically around
one week long. The benefit of these schools is to provide graduate
students, postdocs and junior-level researchers access to skills not
provided by traditional curriculums, and allow more senior researchers
the opportunity to acquire new skills.

**4.3. Coding Camps.** "Coding camps" have been used successfully by
many software projects to facilitate rapid progress in the collaborative
development of software, and facilitate technology transfer between
projects. Camps would typically be from one to two weeks and attended by
the developers of software projects. Coding camps would typically be
focused on developing a small but critical piece of software that
requires collaboration from multiple stakeholders. This format has been
employed successfully by the Common Component Architecture Forum to
develop key infrastructure and to collaborate with users of the
technology.

**4.4. Workshops.** Workshops facilitate the interaction of researchers
working on related problems. They provide an environment for presenting
recent ideas and results in a problem area, and providing space for
subsequent discussions and interactions to develop collaborations and
new directions in that area. The effect of a successful workshop is to
draw out common themes and result in a better understanding of the area.

Workshops could be one-time meetings or multiple periodic meetings. The
one-time meetings allow for the exploration of commonality in developing
areas. The periodic meetings allow a group to form long term goals,
exchange ideas and results, and converge on a better understanding of
the area. Such periodic workshops could be targeted at solving long term
problems or develop new standards. Workshops would vary in duration. For
example, meetings of one to three days recurring on a quarterly basis
are a common pattern for standardization efforts, with weeklong events
at the other end of the spectrum.

**4.5. Activities Outside the Scope of the CS Institute*.*** To make
clear the concept of the Institute, we list a number of activities that
are not generally within its scope:

-   It is not a research funding agency, and will not directly fund
    research (except insofar as research takes place during workshops
    and other activities).

    -   But it *promotes* research, by funding workshops that shape
        research agendas, short-course educational programs, and
        graduate student participation in all activities

-   It does not run *conferences* (specifically for showcasing and
    publication of completed research)

    -   But it runs workshops with a collective product that is
        future-oriented, articulating major research issues, setting
        research agendas, fostering collaborations, and making progress
        on grand challenge problems within the fields of its charter

-   It does not engage in work that is constrained by NDA, licensing or
    other intellectual property restrictions

####### 5**. Governance and Structure of the CS Institute**

CS Institute proposals are expected to include a description of the
governance model and structure of the Institute. The administrative
organization of the Institute is up to the proposer, but is expected to
include only a small number of FTEs. Typically, one or more part-time
(co-)director(s) would have overall responsibility for the Institute,
overseeing both scientific and administrative aspects. A small number of
full- and part-time administrative staff to handle activity proposals,
correspondence, travel and housing arrangements, and communications (web
site, newsletters, reports, etc.). Technical support staff for network,
basic computing, and remote collaboration tools would also be a logical
part of an Institute's staff. We expect proposals to include a steering
committee or advisory panel, representing the DOE computational science
community, which would assist the Director in setting the scientific
directions for the Institute.

The governance model should insure that the scientific mission of the
institute is carried out, including the process by which activities are
proposed, evaluated, selected, executed. Activity proposals are expected
to come from the leaders of the activity, but may be solicited or
encouraged by the Director. Activity proposals should be relatively
lightweight (a few pages) in order to promote applications and be able
to turn them around quickly. They should clearly state the level of
financial support requested from the Institute for participants and the
style of the meeting that would best serve the proposed cause. We expect
that some activity proposals will come from groups able to provide
financial support for their projects, but who wish to use the
Institute\'s infrastructure and imprimatur to facilitate and enhance
their activity. We believe such proposals should be encouraged as a
means to stretch the Institute\'s budget. Evaluation could be handled in
a variety of ways, including, for example, by the steering committee or
by a separate peer review process. The criteria for the evaluation and
selection of activity proposals should be clearly articulated. Once
activity proposals are accepted, it is the responsibility of the
proposer to carry out the activity, with the Institute providing the
agreed upon support and infrastructure.

**5.1. Structure of the Institute** There are at least three possible
organizational models for the CS Institute. In the first model, the
Institute is a part of a university or laboratory. An example is the
Kavli Institute for Theoretical Physics. In the second model, the
institute is a separate organization, probably a non-profit corporation.
Examples of this model include the Aspen Center for Physics and the
Mathematical Sciences Research Institute (MSRI). Proposals for the
Institute should describe the organizational model carefully. The third
model is similar to industry consortia where the institute is
co-directed by multiple institutions in a split or rotating fashion. The
model chosen should ensure that maximum impact and name recognition
would be achieved.

**5.2. Activity Location.** Since a primary role of the institute is to
bring researchers together, the barrier to attendance at Institute
activities should be as low as possible. The location must be appealing
to visitors, and must be in an environment where people will feel
comfortable bringing their families. The availability of inexpensive
(dormitory-style) lodging for students and relatively easy accessibility
would also increase participation. The location should of course provide
meeting facilities, space for collaborative and individual work, network
connectivity, and remote presence technology.

**5.3. Evolution of the Institute.** We expect initial funding for the
institute to be modest -- perhaps \$2M a year for five years. However,
we expect that the Institute would become a long-term fixture in the
high-performance computing community. Proposals for the Institute should
include a compelling long-term vision, beyond what would be possible
with the initial funding. We anticipate that the Institute would seek
funding from foundations, private donations, and other federal funding
sources to augment the DOE-funded foundation. An Institute proposal must
also include a plan for succession of Institute leadership, perhaps
through University/Lab leadership, an external advisory board, or other
mechanisms as appropriate for structure of the Institute.

A key means to cultivate and ensure viability of the Institute is to
seek other stakeholders and partners beyond DOE that provide leverage
opportunities. The ability to obtain programmatic support from other
agencies, while at the same time fulfilling DOE mission needs and
achieving performance metrics, would further validate the Institute's
role in meeting the nation's computational challenges.

**5.3. Anticipated outcomes of the Institute.** It is anticipated that
over time the institute will gain a reputation for instigating seminal
research through ideas and specific software development projects. The
Institute must develop an identity and reputation for leadership in
computational science. Ultimately, the strength of the institute will be
its ability to foster a community around high-performance software, and
to help solve some of this community's most vexing problems. In addition
to this high-level objective goal, a number of measurable outcomes may
be produced:

-   Acknowledgments in peer-reviewed publications by researchers that
    formed collaborations or participated in focused workshops that
    inspired research activities

-   Publication of proceedings of institute activities

-   A newsletter regarding institute research

-   Publication of books or course notes from institute training
    activities

-   Graduate students and postdocs trained in high-performance computing
    topics

-   APIs, software standards, or software libraries published

-   Insertion of CS/applied math software technologies into
    computational science applications

-   Development of common software infrastructure which allows multiple
    individual research efforts to work synergistically

**6. Maximizing the value of the Computer Science Institute**

By its very nature a CS Institute is a high risk -- high payoff
proposition, and there are several specific items that must be addressed
successfully to maximize the benefit of the Computer Science Institute.
These fall into three general categories: focus and mission,
organization, and value to the DOE and the computer science community.

**6.1. Focus and Mission.** It is very important for the CS Institute to
have a well-understood and recognized set of specialties. It cannot try
to cover all topics of computer science that are important to the DOE
software mission. To attempt to do this would not provide the type of
brand-name recognition that will be critical to success. On the other
hand being too narrow in focus may result in being viewed as a marginal
contributor. It may be the case that a successful institute will need to
cover three or four core topics, with twelve to eighteen months devoted
to each. In addition the institute must duplicate the work of an
existing SciDAC center. It should focus on topics that are long range,
yet not so removed from current practice that it loses touch with the
needs of DOE.

**6.2. Organization.** Finding effective, highly respected leadership
for the institute is critical. However, a high profile leader with no
time to devote to making the institute a success will lead to failure. A
key role of the leader will be to form a talented and visionary advisory
team that can help engage dynamic, active individuals to run the
workshops. In addition, if an institute is distributed, i.e. it does not
have a central location, it is essential that it is well recognized
identity.

Because a central focus of the institute's activities are workshops and
related activities, it is critically important that initial activities
be perceived as clear successes; this will be essential for the
institute to gain rapid recognition and community respect. People will
not organize or attend meetings hosted by an institute that is not
perceived as being "high profile".

Workshops and summer schools must take active steps to get researchers
to commit sufficient blocks of time to institute activities. For
example, three week events can be difficult for most senior researchers
to attend. One week events might be preferable. Student and postdoc
participation is essential for the success of the institute and the
future of DOE. But it is increasingly difficult to interest students in
computer science.

Finally, the institute will need to work hard to manage intellectual
property issues. The question of who owns the ideas created at the
workshops must be resolved satisfactorily and in advance of CS Institute
activities.

**6.3. Value.** The institute will need to have very clearly defined
metrics for success for each activity it undertakes. It will be
essential to be able to show tangible results and clearly identifiable
outcomes. It also will be essential that these outcomes be clearly
recognized as important to the DOE. There should be a clear path leading
from DOE and SciDAC needs to the results and activities of the
institute.

The deliverables of the institute may be software or software standards,
but they most certainly will include reports and roadmaps. The reports
that come out of the institute workshops should be widely distributed
and respected by the SciDAC and the Computer Science Academe and
Industry as serious contributions to community building.

**7. Cost.**

It is anticipated that \$2 million would be sufficient to start up a CS
Institute and to implement a range of activities focused on
collaborative research. At this funding level, it is reasonable to
expect that approximately half of the funding would be used for
management, administrative assistance, space, and other fixed costs. The
remainder would be used to fund participation in CS Institute
activities.

It is also anticipated that a CS Institute would need to seek leveraging
funding from other government agencies, industry, academia or other
sources to grow and evolve into a long-term presence in the computer
science community.

**Appendix. Model Computer Science Institute**

In this appendix, we give one example of what a CS Institute might be
like. Nothing in this description necessarily reflects the actual CS
Institute as it might be proposed. Nevertheless, this appendix does
demonstrate that a viable CS Institute is possible within the guidelines
and discussion of this whitepaper.

**Charter:** The SciDAC CS Institute (CSI) will promote *research*,
*development*, *standardization* and *education* in scalable software
infrastructure intended to support DOE computational science
applications in nuclear fusion, climate, combustion, etc. The subject
matter includes, but is not limited to, operating systems, debugging
tools, performance measurement, analysis and modeling tools, fault
detection, avoidance, and tolerance, programming models and languages,
runtime systems, communication libraries, algorithms, and load balancing
strategies. The activities of the organization will be directed toward
setting future research agendas

**Governance:** The CS Institute will be directed by a Advisory Board
consisting of 5 to 7 distinguished experts in the relevant computer
science and computational science disciplines, who are affiliated with
different institutions, and who serve three-year terms. One of the
Advisory Board members will be the Director, who will manage all of the
funds, staff, and activities of the organization, but under the
oversight of the Advisory Board. The Advisory Board will have a Chair
who is different from the Director. The Director will work at least 50%
time directing the CS Institute and will recruit and manage a paid
staff, at least some of whom will be collocated with him/her and paid
from CS Institute funds.

Evolution of the CS Institute: Over several years the organization can
be expected to change and grow. One goal of the Director and Advisory
Board should be to attract additional sponsorship, perhaps from other
government agencies or from private corporations (while keeping the CS
Institute and its events noncommercial in nature).

**Expenses:** We would expect that most ordinary infrastructure needs of
the CS Institute (telecom, networking, copying, etc.) would be
contributed by the host institution. But some direct expenses for space
and travel would be charged to the CS Institute, and capital such as
computers and office equipment would be purchased and owned by the CS
Institute.

**Activities:** The CS Institute's primary activity will be to sponsor,
organize and run *workshops, special courses, and outreach* activities
promoting research, development, and education and community building in
the DOE fields of interest.

-   **Broad-based, community-building annual event:** The CS Institute
    will sponsor one annual workshop on software infrastructure issues
    of broad interest to the DOE SciDAC community intended for 200
    people or more and for 1 week's duration. It will be held in an
    attractive location, the same every year is possible, to encourage
    that attendance of as many participants and their families as
    possible, including

    -   up to 20 distinguished invited attendees

    -   self-selected attendees from government, university, private
        industry

    -   at least U.S. 50 graduate students

    -   international scientists and students as space permits

All expenses except travel will be paid for invited attendees and
students; self-selected attendees will be expected to pay fees
comparable to discounted conference rates; and international will be
expected to pay their full share of the cost of the event.

-   **Frequent smaller, more selective, narrower, events in response to
    proposals:** The CS Institute will also sponsor and organize various
    other workshop and educational events several times per year in
    response to specific proposals from the SciDAC community. Typical
    events will be from a week to a month in duration and will involve
    20 to 100 people. The proposed workshops or short courses will be
    forward-looking and relatively narrow in scope. They will *not* be
    research conferences whose primary purpose is to report and publish
    completed research. Instead, they will concentrate on:

    -   producing *white papers* or *calls for funding* recommending
        future research priorities to funding agencies and the SciDAC
        research community

    -   creating time and opportunity for scientists to get to know one
        another face to face, catalyzing cross institutional
        collaborations

    -   Examples of topics that might make suitable workshops topics of
        accompanied by a suitable proposal

        -   load balancing algorithms and strategies

        -   scalable checkpoint/restart techniques for fault tolerance

        -   component standards (CCA)

        -   scalable debugging tools

        -   platform virtualization techniques

        -   nonlinear solvers

        -   short course on Titanium

**Citations:** Participants in CS Institute-sponsored workshops and
events will be expected to cite the workshop in future proposals and
publications that have benefited from attendance at the workshops.

**Metrics of success:** Among the measures of success of the CS
Institute will be:

-   calls for proposals influenced by CS Institute workshops and
    activities

```{=html}
<!-- -->
```
-   citations to CS Institute-sponsored workshops in subsequent
    proposals and research publications

-   some measure of the amount of future research funding influenced by
    CS Institute activities (preferably acknowledged by funding
    agencies)

-   attendees at short course events

-   competition to attend the annual CS Institute Workshop (i.e. number
    of applicants turned down for attendance because of lack of space)

-   conference attendees may be asked to contribute written "nuggets" to
    acknowledge the value of CS Institute activities to them

-   attraction of additional sponsorship

**Budget:**

The CS Institute will need some permanent paid staff and expenses

Director salary 0.5 FTE \$ 140K

Admin/editor 1.0 FTE \$ 75K

Web admin 0.5 FTE \$ 100K

Event coordinators \* 2.0 FTE \$ 300K

Admin to event coord's 1.0 FTE \$ 75K

Office/travel expenses \$ 50K

Space in host institution \$ 100K

\-\-\-\-\-\-\-\-\--

**Total annual fixed costs \$ 840K**

In addition, the CS Institute will have expenses associated with the
events it sponsors. Based on conference expense profiles, all venue and
organizational expenses (housing, meals, meeting rooms, publications,
networking, etc., (but not travel for most attendees)) can be covered by
\$350 per day per person, of which the attendees can be expected to pay
maybe \$50 per day.

Cost of events per person-day \$350

Cost per day paid by most attendees \$ -50

\-\-\-\-\--

Cost to CS Institute per person-day \$300

**3330 person-days per year @\$300** **\$1000K**

**Additional expenses (travel grants, etc) \$ 160K**

**\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\--**

**Total Budget \$2000K**

\* 2 event coordinators needed because, with 4 or more events per year,
planning for events will overlap

## 

## Role of SciDAC-2 in Predictive DOE Mission Driven Systems Biology

Editors:

Deborah Gracio,
[**Debbie.Gracio@pnl.gov**](mailto:Debbie.Gracio@pnl.gov)

Julie Mitchell,
[**mitchell@math.wisc.edu**](mailto:mitchell@math.wisc.edu)

Nagiza Samatova, [**samatovan@ornl.gov**](mailto:samatovan@ornl.gov)

Tjerk Straatsma,
[**Tjerk.Straatsma@pnl.gov**](mailto:Tjerk.Straatsma@pnl.gov)

This report is currently in draft form. Please check with D. Gracio for
an update.

#