**Scientific Discovery Through\
Advanced Computing:\
Progress Report and Future Opportunities**

**Office of Science**

**U.S. Department of Energy**

**August 2005**

**SciDAC: Progress Report and Future Opportunities**

Table of Contents

[Executive Summary 1](#executive-summary)

[1. Accomplishments and Impact of SciDAC-1
2](#accomplishments-and-impact-of-scidac-1)

[Second-Level Heading 3](#second-level-heading)

[Third-Level Heading 4](#third-level-heading)

[2. The Landscape of DOE Leadership in Computing
5](#the-landscape-of-doe-leadership-in-computing)

[3. Opportunities for the Future 7](#opportunities-for-the-future)

[4. Preserving the Best of SciDAC-1 9](#preserving-the-best-of-scidac-1)

[4.1 Integrated Software Infrastructure Centers (ISICs)
9](#integrated-software-infrastructure-centers-isics)

[The Integrating Role of the ISICs
9](#the-integrating-role-of-the-isics)

[Preserving the Best of the ISICs 10](#preserving-the-best-of-the-isics)

[Taking the ISICs to the Next Level
11](#taking-the-isics-to-the-next-level)

[Meeting Future Application Needs: Topics of Emphasis
12](#meeting-future-application-needs-topics-of-emphasis)

[4.2 Scientific Application Pilot Projects (SAPP)
13](#scientific-application-pilot-projects-sapp)

[Preserving the Best of the SAPP Program
13](#preserving-the-best-of-the-sapp-program)

[Building on Success 14](#building-on-success)

[5. Overview of New Initiatives for SciDAC-2
15](#overview-of-new-initiatives-for-scidac-2)

[5.1 SciDAC Institutes 15](#scidac-institutes)

[Benefits of SciDAC Institutes 15](#benefits-of-scidac-institutes)

[The SciDAC Institutes Model 16](#the-scidac-institutes-model)

[Cost of SciDAC Institute 17](#cost-of-scidac-institute)

[5.2 Accelerating Scientific Discovery in Experimental Science via
Advanced Computing
17](#accelerating-scientific-discovery-in-experimental-science-via-advanced-computing)

[Benefits to DOE and the Nation 17](#benefits-to-doe-and-the-nation)

[Program Organization and Scope 19](#program-organization-and-scope)

[Cross-Cutting Technology Opportunities
20](#cross-cutting-technology-opportunities)

[5.3 Software Integration, Maintenance and Support Program
21](#software-integration-maintenance-and-support-program)

[6. SciDAC-2 Requirements of DOE Computing and Networking Facilities
22](#scidac-2-requirements-of-doe-computing-and-networking-facilities)

[6.1 Leadership Systems 22](#leadership-systems)

[6.2 Multi-Program Capability Computing
22](#multi-program-capability-computing)

[6.3 Specialized Application Computing
23](#specialized-application-computing)

[6.4 Terascale Capacity Computing 23](#terascale-capacity-computing)

[6.5 Networking 23](#networking)

[7. FY 2007--FY 2011 SciDAC Program Plan
25](#fy-2007fy-2011-scidac-program-plan)

[8. Organization and Management 27](#organization-and-management)

[9. Relationships with Other Agencies and Programs
28](#relationships-with-other-agencies-and-programs)

[10. Conclusions 29](#conclusions)

[References 30](#references)

[Appendices: SciDAC-2 Planning Workshop Reports
31](#appendices-scidac-2-planning-workshop-reports)

# Executive Summary

1 page: Stevens, Simon

Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Pellentesque
id felis. Vestibulum diam odio, imperdiet vel, placerat non, aliquam
non, nisl. Nullam metus sem, eleifend in, gravida non, consequat nec,
turpis. Nulla tincidunt mauris non velit. Etiam sit amet lacus. Donec
dui wisi, pharetra vel, nonummy at, varius a, ipsum. Etiam quis velit id
elit mattis consectetuer. Pellentesque gravida sodales nisl. Donec vel
erat. In ultricies. Morbi quis arcu a metus venenatis tincidunt.

Aenean pede mi, auctor at, lacinia in, fermentum vitae, orci. In lectus
leo, blandit eu, congue placerat, vulputate et, lectus. Curabitur ac
magna. Quisque tempus dignissim leo. Suspendisse condimentum.
Pellentesque tristique enim sed lorem. Morbi est erat, aliquam in,
sollicitudin quis, aliquet quis, nisl. Nullam ultricies commodo ligula.
Curabitur nec arcu ac metus vestibulum sollicitudin. Nullam sed quam.
Aliquam suscipit facilisis odio. Etiam non augue. Nunc tristique magna
non libero. Nulla quis lacus non enim fermentum tristique. Nullam quam
ligula, porttitor vitae, luctus a, euismod ac, pede. Vestibulum iaculis
varius odio. Curabitur consectetuer aliquam dolor. Vivamus diam. Duis
augue. Aenean iaculis dictum arcu.

Nunc commodo est nec neque. Nullam porttitor, lectus vitae vulputate
bibendum, erat ipsum interdum mi, et gravida felis lectus ut pede. Nunc
eget wisi ac leo sodales vulputate. Nam bibendum ullamcorper mauris.
Morbi lacinia interdum eros. Praesent auctor risus tincidunt mauris. In
elit. Curabitur semper dui a mi. Donec turpis quam, pharetra tincidunt,
consectetuer ac, pellentesque ac, dolor. In sodales mi vel odio tempor
mollis. Donec lectus tortor, bibendum eget, hendrerit id, commodo a,
justo. Maecenas porta enim sed wisi. Cras commodo. Sed non lectus.
Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere
cubilia Curae; Vestibulum feugiat. Maecenas molestie.

Sample bulleted list:

-   Etiam vestibulum cursus quam.

-   Nunc eu turpis ut orci tincidunt vulputate.

-   Class aptent taciti sociosqu ad litora torquent per conubia nostra,
    per inceptos hymenaeos.

Praesent iaculis, leo at nonummy ultricies, diam odio pellentesque
lacus, et tempus sem quam sed nulla. Ut sollicitudin placerat elit.
Pellentesque habitant morbi tristique senectus et netus et malesuada
fames ac turpis egestas. Aenean quis lorem sed magna aliquet volutpat.
Donec mauris nunc, ultrices at, tincidunt quis, rutrum sagittis, ipsum.
Nunc non metus eget justo mollis placerat. Class aptent taciti sociosqu
ad litora torquent per conubia nostra, per inceptos hymenaeos. Nunc
massa erat, tempor vitae, condimentum nec, mollis nec, est. Donec wisi.
Vestibulum ligula. Phasellus nisl mauris, iaculis eu, tempus non,
eleifend eu, ante.

# 1. Accomplishments and Impact of SciDAC-1 

3 pages: Davenport, Tang

Highlights that focus on Return on Investment.

Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Pellentesque
id felis. Vestibulum diam odio, imperdiet vel, placerat non, aliquam
non, nisl. Nullam metus sem, eleifend in, gravida non, consequat nec,
turpis. Nulla tincidunt mauris non velit. Etiam sit amet lacus. Donec
dui wisi, pharetra vel, nonummy at, varius a, ipsum. Etiam quis velit id
elit mattis consectetuer. Pellentesque gravida sodales nisl. Donec vel
erat. In ultricies. Morbi quis arcu a metus venenatis tincidunt.

Aenean pede mi, auctor at, lacinia in, fermentum vitae, orci. In lectus
leo, blandit eu, congue placerat, vulputate et, lectus. Curabitur ac
magna. Quisque tempus dignissim leo. Suspendisse condimentum.
Pellentesque tristique enim sed lorem. Morbi est erat, aliquam in,
sollicitudin quis, aliquet quis, nisl. Nullam ultricies commodo ligula.
Curabitur nec arcu ac metus vestibulum sollicitudin. Nullam sed quam.
Aliquam suscipit facilisis odio. Etiam non augue. Nunc tristique magna
non libero. Nulla quis lacus non enim fermentum tristique. Nullam quam
ligula, porttitor vitae, luctus a, euismod ac, pede. Vestibulum iaculis
varius odio. Curabitur consectetuer aliquam dolor. Vivamus diam. Duis
augue. Aenean iaculis dictum arcu.

Nunc commodo est nec neque. Nullam porttitor, lectus vitae vulputate
bibendum, erat ipsum interdum mi, et gravida felis lectus ut pede. Nunc
eget wisi ac leo sodales vulputate. Nam bibendum ullamcorper mauris.
Morbi lacinia interdum eros. Praesent auctor risus tincidunt mauris. In
elit. Curabitur semper dui a mi. Donec turpis quam, pharetra tincidunt,
consectetuer ac, pellentesque ac, dolor. In sodales mi vel odio tempor
mollis. Donec lectus tortor, bibendum eget, hendrerit id, commodo a,
justo. Maecenas porta enim sed wisi. Cras commodo. Sed non lectus.
Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere
cubilia Curae; Vestibulum feugiat. Maecenas molestie.

Etiam vestibulum cursus quam. Nunc eu turpis ut orci tincidunt
vulputate. Class aptent taciti sociosqu ad litora torquent per conubia
nostra, per inceptos hymenaeos. Nulla augue erat, dapibus sit amet,
volutpat aliquet, porta ac, est. Sed fermentum elementum magna. Sed
luctus. Nam lorem neque, tempus in, tincidunt vitae, varius eget, ante.
Nulla ac nisl. Praesent metus. Vestibulum a augue a dolor faucibus
eleifend. Vestibulum ante ipsum primis in faucibus orci luctus et
ultrices posuere cubilia Curae.

Praesent iaculis, leo at nonummy ultricies, diam odio pellentesque
lacus, et tempus sem quam sed nulla. Ut sollicitudin placerat elit.
Pellentesque habitant morbi tristique senectus et netus et malesuada
fames ac turpis egestas. Aenean quis lorem sed magna aliquet volutpat.
Donec mauris nunc, ultrices at, tincidunt quis, rutrum sagittis, ipsum.
Nunc non metus eget justo mollis placerat. Class aptent taciti sociosqu
ad litora torquent per conubia nostra, per inceptos hymenaeos. Nunc
massa erat, tempor vitae, condimentum nec, mollis nec, est. Donec wisi.
Vestibulum ligula. Phasellus nisl mauris, iaculis eu, tempus non,
eleifend eu, ante. Aliquam sem libero, interdum quis, rutrum sed,
sodales et, elit. In dictum lacinia risus. Cras aliquam. Curabitur ac
ipsum elementum mi malesuada porta. Praesent rutrum.

Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Pellentesque
id felis. Vestibulum diam odio, imperdiet vel, placerat non, aliquam
non, nisl. Nullam metus sem, eleifend in, gravida non, consequat nec,
turpis. Nulla tincidunt mauris non velit. Etiam sit amet lacus. Donec
dui wisi, pharetra vel, nonummy at, varius a, ipsum. Etiam quis velit id
elit mattis consectetuer. Pellentesque gravida sodales nisl. Donec vel
erat. In ultricies. Morbi quis arcu a metus venenatis
tincidunt.![](media/image1.jpeg){width="2.81875in"
height="3.5194444444444444in"}

Aenean pede mi, auctor at, lacinia in, fermentum vitae, orci. In lectus
leo, blandit eu, congue placerat, vulputate et, lectus. Curabitur ac
magna. Quisque tempus dignissim leo. Suspendisse condimentum.
Pellentesque tristique enim sed lorem. Morbi est erat, aliquam in,
sollicitudin quis, aliquet quis, nisl. Nullam ultricies commodo ligula.
Curabitur nec arcu ac metus vestibulum sollicitudin. Nullam sed quam.
Aliquam suscipit facilisis odio. Etiam non augue. Nunc tristique magna
non libero. Nulla quis lacus non enim fermentum tristique. Nullam quam
ligula, porttitor vitae, luctus a, euismod ac, pede. Vestibulum iaculis
varius odio. Curabitur consectetuer aliquam dolor. Vivamus diam. Duis
augue. Aenean iaculis dictum arcu.

## Second-Level Heading

Nunc commodo est nec neque. Nullam porttitor, lectus vitae vulputate
bibendum, erat ipsum interdum mi, et gravida felis lectus ut pede. Nunc
eget wisi ac leo sodales vulputate. Nam bibendum ullamcorper mauris.
Morbi lacinia interdum eros. Praesent auctor risus tincidunt mauris. In
elit. Curabitur semper dui a mi. Donec turpis quam, pharetra tincidunt,
consectetuer ac, pellentesque ac, dolor. In sodales mi vel odio tempor
mollis. Donec lectus tortor, bibendum eget, hendrerit id, commodo a,
justo. Maecenas porta enim sed wisi. Cras commodo. Sed non lectus.
Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere
cubilia Curae; Vestibulum feugiat. Maecenas molestie.

Etiam vestibulum cursus quam. Nunc eu turpis ut orci tincidunt
vulputate. Class aptent taciti sociosqu ad litora torquent per conubia
nostra, per inceptos hymenaeos. Nulla augue erat, dapibus sit amet,
volutpat aliquet, porta ac, est. Sed fermentum elementum magna. Sed
luctus. Nam lorem neque, tempus in, tincidunt vitae, varius eget, ante.
Nulla ac nisl. Praesent metus. Vestibulum a augue a dolor faucibus
eleifend. Vestibulum ante ipsum primis in faucibus orci luctus et
ultrices posuere cubilia Curae.

### Third-Level Heading

Praesent iaculis, leo at nonummy ultricies, diam odio pellentesque
lacus, et tempus sem quam sed nulla. Ut sollicitudin placerat elit.
Pellentesque habitant morbi tristique senectus et netus et malesuada
fames ac turpis egestas. Aenean quis lorem sed magna aliquet volutpat.
Donec mauris nunc, ultrices at, tincidunt quis, rutrum sagittis, ipsum.
Nunc non metus eget justo mollis placerat. Class aptent taciti sociosqu
ad litora torquent per conubia nostra, per inceptos hymenaeos. Nunc
massa erat, tempor vitae, condimentum nec, mollis nec, est. Donec wisi.
Vestibulum ligula. Phasellus nisl mauris, iaculis eu, tempus non,
eleifend eu, ante. Aliquam sem libero, interdum quis, rutrum sed,
sodales et, elit. In dictum lacinia risus. Cras aliquam. Curabitur ac
ipsum elementum mi malesuada porta. Praesent rutrum.

Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Pellentesque
id felis. Vestibulum diam odio, imperdiet vel, placerat non, aliquam
non, nisl. Nullam metus sem, eleifend in, gravida non, consequat nec,
turpis. Nulla tincidunt mauris non velit. Etiam sit amet lacus. Donec
dui wisi, pharetra vel, nonummy at, varius a, ipsum. Etiam quis velit id
elit mattis consectetuer. Pellentesque gravida sodales nisl. Donec vel
erat. In ultricies. Morbi quis arcu a metus venenatis tincidunt.

Curabitur ac magna. Quisque tempus dignissim leo. Suspendisse
condimentum. Pellentesque tristique enim sed lorem. Morbi est erat,
aliquam in, sollicitudin quis, aliquet quis, nisl. Nullam ultricies
commodo ligula. Curabitur nec arcu ac metus vestibulum sollicitudin.
Nullam sed quam. Aliquam suscipit facilisis odio. Etiam non augue. Nunc
tristique magna non libero. Nulla quis lacus non enim fermentum
tristique. Nullam quam ligula, porttitor vitae, luctus a, euismod ac,
pede. Vestibulum iaculis varius odio. Pellentesque tristique enim sed
lorem. Morbi est erat, aliquam in, sollicitudin quis, aliquet quis,
nisl. Nullam ultricies commodo ligula. Curabitur nec arcu ac metus
vestibulum sollicitudin. Nullam sed quam. Aliquam suscipit facilisis
odio. Etiam non augue. Nunc tristique magna non libero. Nulla quis lacus
non enim fermentum tristique. Vestibulum iaculis varius odio.

# 2. The Landscape of DOE Leadership in Computing

2 pages: Nichols

This should tie SciDAC to the Leadership computing program, the next
generation architecture program, the facilities roadmaps and the
computing initiatives in programs like BES, GTL, etc.

More Section 2

# 3. Opportunities for the Future

2 pages: Kendall

Rev up the scientific cases for advanced computing including simulation,
data analysis etc. This should pull from and quote from the major
reports from the last five years.

More Section 3

# 4. Preserving the Best of SciDAC-1

The successes of SciDAC are many and broadly shared. For example, we all
celebrate the scientific accomplishments highlighted earlier in this
document. However, another SciDAC success is a cultural one:
demonstrating the power of multidisciplinary and multi-institutional
teams working toward a common goal. Two program elements deserve credit
for enabling these teams: the Integrated Software Infrastructure Centers
(ISICs) and the Scientific Application Pilot Projects (SAPPs). These
synergistic program elements work closely with the applications to
enable the marvelous scientific accomplishments of SciDAC. As we look
ahead to SciDAC II, we wish to preserve and enhance these program
elements. In this section, we will discuss each in turn, commenting on
their strengths and offering suggestions for improvement. In general, we
believe that both program elements have been extremely successful and
should be refined rather than overhauled.

## 4.1 Integrated Software Infrastructure Centers (ISICs)

### The Integrating Role of the ISICs 

The Integrated Software Infrastructure Centers are large,
multi-institutional teams of mathematicians and computer scientists
chartered \[\\cite SciDAC 1 program plan\]

to accelerate the development of, and protect the long-term investments
in, scientific codes, to achieve maximum efficiency on high end
computers, and to enable a broad range of scientists to use simulation
in their research.

Seven ISICs were funded to address application needs in the areas of
solvers and optimization, mesh generation and discretization, adaptive
mesh refinement, software interoperability, scientific data management,
performance evaluation, and systems software. These ISICs have been
widely praised for their ability to bring leading edge capabilities to
the diverse set of SciDAC applications. In particular, the ISICs have
succeeded by

-   inserting advanced mathematics and computer science technologies
    into SciDAC applications by encapsulating research results in robust
    software tools

-   developing and delivering new enabling computational technologies in
    response to articulated and anticipated application needs

-   providing expertise and consulting advice on a range of topics in
    mathematics, computer science, and software engineering---as well as
    providing entrée to the broader scientific computing community

-   nurturing the next generation of SciDAC computational scientists
    through active participation in conferences and workshops, and via
    the training of graduate students and postdocs.

The ISICs have developed and delivered robust software tools that are
either directly useable by computational science application teams or
are necessary for the effective use of high end computing facilities. As
one SciDAC scientist said, "the \[ISIC\] isn't merely making the hard
things easier, it is making science possible that we thought was too
complex to try."

The ISICs play an important integrating role beyond the applications
they support: they enable vertical integration within OASCR and
horizontal integration across the Office of Science. Within OASCR, the
ISICs serve as the means by which new ideas generated in the base
research program are deployed to the Office's high-end computing
facilities. ISIC software is also a key mechanism for promulgating OASCR
research results to the international computational science community.
Conversely, the ISICs' close coupling to the applications teams they
support enables them to identify future directions for OASCR research
programs.

Within the Office of Science, the ISICs serve as the means by which
OASCR computational technologies are deployed to application teams to
enable new science. The ISICs have significantly increased the number
and quality of interactions between scientists funded by OASCR and the
other program offices. In many cases, these interactions have not only
enhanced simulation capabilities, but also driven the development and
prioritization of ISIC technologies---and thus indirectly influenced
future directions in the OASCR base research program. The reason the
ISICs have been successful in both horizontal and vertical integration
is that they collectively span a broad range of mathematics and computer
science topics that are relevant to several if not all Office of Science
applications.

### Preserving the Best of the ISICs

The ISICs have been widely praised and are considered to be an essential
to the continued success of SciDAC. In discussing the future of the
ISICs, we consistently heard the message, "Don't mess with the current
model for the sake of change." In our view, there are several aspects of
the current ISIC model that should be preserved:

-   *Emphasize collaboration with, and impact on, multiple SciDAC
    application teams.* This leverages the ISIC breakthroughs to
    multiple applications and shows the general relevance of the ISIC
    topics to the broad computational science community. Working with
    several application teams also helps the ISICs to focus and to
    determine priorities that produce more broadly applicable and robust
    simulation technologies.

-   *Create horizontally integrated, multi-institutional teams.* This
    has resulted in a strong collaborative environment across the DOE
    complex that promotes community efforts to develop best-in-class
    software and integrated tools that no single institution could have
    produced alone. The existing ISICs have shown how such
    multi-institutional teams can accomplish more than the sum of their
    parts. The integration also enhances the usability of the ISIC
    software for applications scientists.

-   *Fund a collection of ISICs that span a breadth of relevant topics.*
    The current set of ISICs are well-aligned with the needs of Office
    of Science applications while collectively spanning the basic
    research programs to a great extent. This is critical if the ISICs
    are to maintain their critical role as the vertical integration link
    between OASCR's basic research program and its computing facilities,
    as well as the horizontal integration link between OASCR and the
    application program offices.

-   *Provide long-term funding stability for core technology
    development.* Current SciDAC projects typically take upwards of five
    years to develop a new generation of hardened algorithms and
    software, and the insight on how they can be used effectively in
    applications. This recommendation also is consistent with the
    management of the NNSA ASC Alliance Centers and the NSF Science and
    Technology Centers.

### Taking the ISICs to the Next Level

The current ISICs are working extremely well: they are developing
world-class computational capabilities and working closely with
application teams to insert this technology into their simulation codes.
Of course, there always is room for improvement and we recommend the
following enhancements to the ISIC program:

-   *The ISICs should establish even stronger ties to the application
    teams.* The ISICs should identify ways to strengthen their
    collaborations with the applications teams without losing their
    identity as an ISIC. One way to do this is to assign "code liaisons"
    to key application teams to ensure that ISIC development efforts are
    relevant and ISIC technology is deployed rapidly. SAPP funding has
    been effectively used for this purpose and should be expanded (see
    below).

-   *The ISICs should work together more.* The current ISICs interact on
    an *ad hoc*, as needed basis. The ISICs should work together more
    closely to discuss application code needs (thereby increasing the
    likelihood that they will be met), as well as to leverage each
    other's technology. ISICs should be encouraged to support one
    another as much as they are encouraged to support the application
    teams. For example, SAPP funding could be used to incentive the
    partnering of two or more ISICs with a single code team.

-   *The ISICs should have greater funding flexibility.* The ISICs need
    the ability to respond to near-term opportunities to interact with
    application teams and with each other. Such interactions typically
    occur on a 12-18 month time frame and can change over the lifetime
    of the ISIC. Increased flexibility will lead to increased impact
    because resources can be placed where they are most needed.
    Acceptable activities for ISIC interactions with an application team
    can take many different forms including: demonstrating feasibility
    of a new algorithmic or modeling approach to a scientific area,
    using new algorithmic and software technology to solve specific
    scientific problems, or embedding new software technology in
    production applications codes.

-   *The ISICs should partner with the other SciDAC program elements.*
    The ISICs are well positioned to support the Software Integration,
    Maintenance, and Support (SIMS) program objectives of advancing
    software interoperability and reusability, and promoting sound
    scientific software engineering practices. The ISICs also should be
    frequent participants in the workshops sponsored by the SciDAC
    Institutes, for example, by offering intensive immersion
    opportunities in key enabling technology areas.

### Meeting Future Application Needs: Topics of Emphasis

The next five years will see significant change in computational
science: the first petascale systems will become available and new
scientific breakthroughs will be enabled. The ISICs must build on their
current successes in both application partnering and technology
development. In particular, the next set of ISICs must continue to meet
the current needs of the scientific applications teams, as well as
address the technology gaps that will emerge with petascale computing
systems. Some potential gaps are the scalability and fault tolerance of
applications and system software on 100,000-processor computers, the
management of data volumes 1000 times greater than today, the validation
of results as applications incorporate new physics, and the ability to
rapidly develop ever more complex applications that port efficiently
across a variety of hardware architectures.

The ISICs, individually and collectively, must provide a comprehensive,
integrated, scalable, and robust high performance computing software
infrastructure that will enable effective use of leadership class
computing resources by current and future SciDAC application teams. The
ISICs will address needs for: new algorithms and mathematical software
that scale to thousands of processors; accurate discretization and
parallel meshing approaches; software development environments and
methodologies for realizing portable performance; and scalable
scientific data management and analysis tools. In addition, the ISICs
are expected to provide consulting expertise and to serve as a conduit
to software resources available elsewhere, e.g., the NSF centers.

There are many topics the ISICs can address to meet the needs of current
and future SciDAC applications. These include, but are not limited to:

-   *Numerical algorithms mathematical software*. Linear, nonlinear, and
    eigen solvers; multigrid and multilevel methods; adaptive sampling
    algorithms; optimization and inverse methods; sensitivity analysis
    and uncertainty quantification. These algorithms and mathematical
    libraries must be scalable and efficient on machines with thousands
    of processors and complex memory hierarchies.

-   *Mesh generation and discretization technologies.* Automatic mesh
    generation; parallel mesh partitioning techniques; adaptive mesh
    refinement algorithms and software frameworks; mesh quality metrics
    and improvement; rapid problem definition. These technologies must
    be tailored to current and future applications employing both
    structured and unstructured meshes.

-   *Software development environments and programming paradigms.*
    Software development and code profiling tools; performance
    benchmarking and evaluation; code correctness and validation; high
    performance software component technologies, code reuse, and
    language interoperability. These tools and capabilities must
    portable across all SciDAC hardware platforms and relevant to the
    current or future needs of SciDAC applications.

-   *Scientific data management, analysis, and visualization.*
    Large-scale data integration and management; data access and
    querying; scalable data storage and transfer; feature extraction and
    change analysis; vector/tensor field visualization; statistical data
    analysis. These data analysis algorithms and software tools must be
    scalable to large data sets, as well as tailorable to meet the needs
    of individual application codes.

We recommend funding one or two ISICs in each of these broad areas.

## 4.2 Scientific Application Pilot Projects (SAPP)

The Scientific Application Pilot Program (SAPP) provided support for
targeted efforts to integrate new applied mathematics and computer
science algorithms into the SciDAC applications projects. Although the
stated goal is similar to that of the ISICs, this program element was
used in a complementary fashion and SAPP funding was typically used in
two ways:

-   It encouraged strong liaisons between applications projects and the
    ISICs, by funding personnel co-managed by both teams.

-   It provided a mechanism by which computer science and mathematics
    needs not fulfilled by the ISIC teams could be provided to the
    application teams.

Examples of how SAPP funding was effectively used are numerous and
include the development of application-specific visualization tools,
data partitioning methods, and adaptive mesh refinement-based simulation
codes. The scope of each SAPP project is much smaller than that of an
ISIC. In particular, a SAPP project typically funded a fraction of an
FTE to insert technology into a specific application and the broader
needs of the scientific computing community were not considered. We note
that while each individual project has been small, an application team
could have more than one SAPP project. Total funding for this program
element was about \$2M.

Successes of the SAPP program....

### Preserving the Best of the SAPP Program

There are many aspects of the current SAPP program which are successful
and should be continued in SciDAC-2:

-   The strong collaboration between the application and embedded
    mathematician and computer scientists has encouraged effective
    insertion of new technologies into application codes. In addition,
    the embedded mathematician and computer scientist can serve as a
    liaison between the application team and the ISIC teams.

-   Keeping each individual SAPP project small has encouraged the
    creation of targeted technologies specific for an individual
    application's needs. While this targeted development can maximize
    the impact on a particular application, it does not lend itself to
    the creation of broadly applicable technologies.

-   Allow SAPP funding to fill topical holes the ISICs do not cover.
    Although the ISICs spanned a broad range of technologies, they could
    not be comprehensive and the SAPP program filled a critical SciDAC
    program need. However, SAPP funding should not be used to replace an
    ISIC technology. In the case where there is some overlap between the
    application need and ISIC technology, every effort to fund a
    co-managed project should be made.

### Building on Success

While the SAPP program element in SciDAC-2 should strive to preserve the
best of the program from SciDAC-1, it can be improved in the following
ways.

-   *Increase the size of the program.* The SAPP program was an
    important element of the SciDAC 1 program, but it was small
    percentage of the overall funding. If this program element is
    expanded, it should be clearly described and well-advertised to
    increase participation.

-   *Increase the flexibility of the SAPP program* by competing the
    funding separately from the initial application proposal and
    allowing the duration of SAPP projects to vary. \[LAD: how much does
    this happen now?\] Projects that are appropriate for funding range
    from short term efforts that could be exploratory in nature or meet
    simpler application needs to longer term projects that are more
    involved, for example, the exploration of a new framework for a
    given application area.

-   *Allow ISICs to compete for SAPP funding.* In SciDAC 1, it was
    demonstrated that embedding a computer scientist or mathematician in
    an application project was very effective. It could be equally
    effective to embed an application scientist into an ISIC to increase
    the dissemination of ISIC technology. Moreover, a small percentage
    of SAPP funding could be used to support ISIC collaborations with
    Office of Science application projects that are not funded by
    SciDAC. This could help prepare select application areas for the
    next round of SciDAC funding.

# 5. Overview of New Initiatives for SciDAC-2

\[introductory paragraph needed\]

## 5.1 SciDAC Institutes

Institutes have been found to be very useful in a number of science
communities. Examples include the Aspen Institute in physics and the
Mathematical Sciences Research Institute in math. It is proposed that
SciDAC II include a program element based on the establishment of one or
more institutes. Many areas of science being addressed in SciDAC are at
a critical juncture and require the kind of long-term infusion of new
ideas that institutes could provide. The key to the success of an
institute is the ability to bring together the top scientists, computer
scientists, and applied mathematicians to tackle critical problems. The
SciDAC institutes will provide a forum where many fundamental issues
affecting scientific discovery through advanced computing can be
discussed. The institutes will also be a focal point for training and
educating the next generation of HPC experts.

### Benefits of SciDAC Institutes

SciDAC Institutes will benefit DOE and the scientific community in many
ways. The primary goal of a SciDAC Institute will be to build and foster
a community of researchers who understand the challenges of providing
and using Leadership class capability computing in science and
engineering and are focused on solving these problems. The benefits are
summarized below.

-   A SciDAC Institute will provide a forum for discussion that will
    identify challenges and provide strategic direction to both programs
    and researchers in the high-performance computing community.

-   A SciDAC Institute will provide a forum for collaboration between
    SciDAC projects and bring together the software infrastructure
    centers and the application partnerships. It will do this by
    focusing on the future and the challenges in building a
    high-performance computing capability rather than on past results.

-   A SciDAC Institute is focused on collaborative activities; it will
    leverage DOE's capabilities in high-performance computing as well as
    the research base in the academic community.

-   A SciDAC Institute will reinvigorate the high-performance computing
    research community and will increase the number of graduate students
    and postdocs trained in high-performance computing.

-   A SciDAC Institute will also promote specific technical results
    through its activities. These will include technical publications
    (articles, reports, books and proceedings), advances in software
    infrastructure (algorithms, software, interfaces and standards), and
    coordinated attacks on specific problems in applications of
    high-performance computing.

To reap the benefits described above, a SciDAC Institute must gain a
reputation for instigating seminal research through ideas and specific
software development projects. The institute must develop an identity
and reputation for leadership in computational science. Ultimately, the
strength of the institute will be its ability to foster a community
around high-performance software, and to help solve some of this
community's most challenging problems.

### The SciDAC Institutes Model

SciDAC Institutes will be loosely patterned on successful institute
models in other communities, such as the Aspen Institute in the physics
community and the Mathematical Sciences Research Institute in the
mathematics community.

The institutes will sponsor a variety of activities to fulfill its
mission. Activities will vary in format to satisfy different purposes
and needs and specific activities of an institute will be part of a
successful proposal. Activities that might be included in an institute
include the following:

-   *Critical topic program* to address a particular problem of interest
    to the HPC community for an extended period of time. Graduate
    students and postdocs will attend the entire program, and faculty
    and laboratory research staff will spend possibly shorter periods of
    time advising, reviewing, and interacting with the students and post
    docs.

-   *Short course or summer schools* that will give researchers at all
    levels training in new HPC concepts, technology, techniques and
    software. These will provide graduate students, postdocs and
    junior-level researchers access to skills not provided by
    traditional curriculums, and allow more senior researchers and
    industry the opportunity to acquire new skills.

-   *"Coding camps,"* which have been used successfully by many software
    projects to facilitate rapid progress in the collaborative
    development of software, and facilitate technology transfer between
    projects. Camps would typically be from one to two weeks and
    attended by the developers of software projects.

-   *Workshops* will facilitate the interaction of researchers working
    on related problems. They provide an environment for presenting
    recent ideas and results in a problem area, and providing space for
    subsequent discussions and interactions to develop collaborations
    and new directions in that area.

There are also some activities that are not intended to be a part of the
institute, including conferences, research projects tied to a specific
program or call, and activities that result in intellectual property. It
will not be necessary for SciDAC Institutes to have a single location;
however, facilities used must be conducive to the research and
collaboration.

A SciDAC Institute should be organized to insure that the scientific
mission of the institute is carried out, including the process by which
activities are proposed, evaluated, selected, and executed. There many
possible administrative structures for the institute, but an institute
is expected to include only a small number of regular staff. Staff will
include a director, who would have overall responsibility for the
institute, overseeing both scientific and administrative aspects. Staff
will also include a small number of full-time and part-time
administrative staff to handle activity proposals, correspondence,
travel and housing arrangements, and communications, such as web site,
newsletters, and reports. The organization of the institute will also
include a steering committee or advisory panel representing the DOE
computational science community to assist the director in setting the
scientific directions for the Institute.

### Cost of SciDAC Institute

It is anticipated that each SciDAC Institute would require approximately
\$2M/yr to start up and to implement a range of activities focused on
collaborative research. At this funding level, it is reasonable to
expect that approximately half of the funding would be used for
management, administrative assistance, space, and other fixed costs. The
remainder would be used to fund participation in institute activities.
It is also anticipated that an institute would need to seek leveraging
funding from other government agencies, industry, academia or other
sources to grow and evolve into a long-term presence in the computer
science community.

## 5.2 Accelerating Scientific Discovery in Experimental Science via Advanced Computing

There is a timely and significant opportunity to increase both the
utility of DOE experimental facilities and the effectiveness of DOE
experimental science, through judicious and targeted application of
advanced computing and information technology. The driving factors are
the profound importance of DOE experimental facilities and science to
the nation; the urgent challenges currently faced by experimental
sciences due to rapidly growing data volumes and new experimental
techniques; and the significant advances in advanced computation,
distributed computing, and interdisciplinary and interlaboratory
collaboration achieved within the SciDAC-1 program. These factors create
an opportunity to apply precisely the sort of focused and directed
interdisciplinary research program that has been so successful within
the simulation sciences in SciDAC-1. As SciDAC-1 empowered theorists by
automating (via numerical simulation) important steps in the process of
exploring the implications of theory, so a comparable program can
empower experimentalists by automating important steps in the
experimental process, from the operation of experimental facilities to
the collection and analysis of experimental data.

### Benefits to DOE and the Nation

Experiment facilities and experiment-driven research are at the heart of
DOE's mission. The aggressive future facilities program and associated
scientific rationale laid out in the report "Facilities for the Future
of Science" emphasize that experimental science will continue to be
crucially important to the nation's prosperity, health, and security.
While DOE user facilities receive the highest possible reviews for their
utility and usability, the challenges associated with revolutionary
advances in data rates, data volumes, and experimental accuracy are
mounting.

The goal of a program aimed at *Accelerating Scientific Discovery in
Experimental Science* (ASDES) is simple: to bring key advanced
information technologies to bear on challenging problems arising in DOE
experimental facilities and research.

Projects funded within the ASDES program will contribute directly to
existing DOE experimental facilities and research programs, enhancing
their usability and accessibility, enabling new classes of experiment,
broadening access to larger and more diverse communities, and enabling
cross-cutting comparisons of experiments with other experiments and with
simulation results.

The following are examples of projects that might be funded under such a
program, and the benefits that could be expected to accrue from their
work.

-   *Data management for the nanosciences*. Provide integrated
    information and workflow management for DOE's five Nanoscience
    Research Centers, thus spurring new discoveries by facilitating
    experiment-to-experiment and experiment-to-simulation comparisons.

-   *Timely analysis of LHC data.* Enable U.S. leadership in scientific
    discovery through rapid dissemination and analysis of petabyte
    datasets from the Large Hadron Collider (LHC), building on SciDAC-1
    successes in global Grid Computing to further develop a shared
    national and international Grid infrastructure for physics and other
    sciences.

-   *Enabling innovative science at the LCLS*. Realize the full
    scientific promise of the Linac Coherent Light Source (LCLS) through
    the timely development and application of advanced computing tools
    for the design of experiments and for data handling and
    analysis---tools that LCLS does not have the expertise to develop
    alone.

-   *Enabling climate research and policy studies in a petabyte age*.
    Create an integrated environmental data portal that allows
    scientists and policymakers to pose complex questions against
    petabytes of both simulation and observational data.

-   *Creating new analysis opportunities for RHIC experiment data*. Go
    beyond the current successful data grids to develop second
    generation integrated analysis capabilities for the statistically
    challenging, and often real-time, data from experiments at the
    Relativistic Heavy Ion Collider (RHIC)

-   *Next-generation experimental cosmology*. Provide near-realtime
    analysis and associated information management capabilities capable
    of handling \~1 Gbyte/sec data streams from the Large Synoptic
    Survey Telescope (LSST).

-   *High-throughput laser imaging for combustion science*. Sustain U.S.
    leadership in combustion science through support for integrated
    workflow, information and data management solutions able to handle
    next-generation laser imaging data.

-   *Coupling computation to experiment at ITER*. Inform experimental
    decisions through support for quasi real-time analysis of
    experimental data at the International Thermonuclear Experimental
    Reactor (ITER).

The ASDES program will encourage identification of common requirements
across different DOE experimental programs. The following are examples
of areas in which commonalities can be exploited, via either
collaborative application projects or cross-cutting technology projects.

-   *Remote control rooms for international experiments*. The worldwide
    experimental programs in fusion and high-energy physics will be
    centered on facilities located in Europe: the International
    Thermonuclear Experimental Reactor (ITER) and Large Hadron Collider
    (LHC), respectively. Enabling remote participation by U.S.
    scientists will contribute to the success of ITER and LHC and also
    maximize the value of these unique facilities to the U.S.

-   *Real-time computing at beamlines*. A new generation of more
    powerful X-ray and neutron sources and far more sensitive detector
    technology are combining to create a data explosion at facilities
    across the DOE system. A combination of on-demand access to
    computing, enhanced algorithms capable of quasi-real-time analysis,
    and tele-collaboration technologies can allow for far more efficient
    use of beamlines.

-   *Distributed data management and computing*. Climate, physics,
    biology, and nanosciences (among others) face similar needs for the
    distributed management and analysis of large volumes of data
    produced at different facilities, analyzed via often complex
    workflows, and consumed by a large and distributed user community.

-   *Information management*. In addition to large data volumes, DOE
    experimental science disciplines face challenges associated with
    increasingly complex data due to different experimental modalities
    and analysis techniques. Again, there are opportunities to achieve
    technological and methodological advances that benefit multiple
    disciplines.

-   *Cybersecurity for open facilities*. DOE user facilities face the
    increasingly difficult task of operating open facilities in an
    increasingly hostile security environment while mainting utility of
    the facilities and preserving the free flow of information.

The ASDES program will also establish beneficial linkages between
experiment and computation, foster cross-disciplinary connections,
engage talented faulty at U.S. universities in addressing critical
problems of importance to DOE, and train students for the future
experimental programs.

### Program Organization and Scope

The SciDAC-1 program introduced two strategies---the Integerated
Software Infrastructure Centers (ISICs) and Scientific Application Pilot
Program (SAPP)---that have proved valuable as a means of simultaneously
attacking cross-cutting technical problems and using application science
drivers to motivate a focused multidisciplinary attack on critical
problems.

Thus, we propose the creation of an *Experimental Science Application
Pilot Program* (ESAPP), modeled on the SciDAC-1 SAPP, to support
projects focused on achieving substantial and quantifiable advances in
scientific practice within one or more fields of experimental science.
Strong support from the relevant program offices, as evidenced, for
example, by significant matching funding, as well as timely results and
results that are useful outside the individual disciplines within which
the work is performed, should be used to select ESAPP projects.

We also propose *Integrated Software Infrastructure Centers focused on
experimental sciences* (EISICs) to support the development, in a timely
fashion, of software tools and algorithms for which a clear need can be
demonstrated across several DOE experimental science communities and for
which there is a clear commitment to adopt and adapt the resulting work.

### Cross-Cutting Technology Opportunities

While each DOE experimental facility and research program has its own
unique concerns, analysis of technical requirements shows that there are
also significant overlaps in technology requirements. Some examples
which would find avid customers from diverse domains such as
combustions, environmental sciences, biology, physics and nanosciences
are:

-   *Information management.* Tools for creating, searching, and
    managing information, whether raw data, metadata associated with
    data, or new data and metadata from analysis and simulation.
    Generalize current information management techniques to address the
    unique characteristics of experimental data, addressing issues of
    data quality and confidence, and the realities of missing,
    corrupted, and erroneous data.

-   *Distributed computing technology.* Tools to build on the pioneering
    services of SciDAC-1 (in projects such as Earth System Grid, Fusion
    Collaboratory, and Open Science Grid) for securely and reliably
    locating, moving, and accessing large scale data sets and providing
    associated security and distributed service deployment, monitoring,
    and management functions.

-   *Comparing experiment and simulation.* This important topic, which
    has not been addressed in previous SciDAC research activity, is
    critical if experimental science is to be integrated as a third
    pillar on a par with simulation. It may involve developing new
    interpolation techniques, new approaches to comparative
    visualization, and on-line statistical or quantitative analysis
    approaches.

-   *Open Science Grid.* SciDAC-1 enabled the creation of a
    national-scale, multi-agency distributed computing and storage
    infrastructure, the Open Science Grid (OSG); an infrastructure that
    is already delivering millions of CPU hours and many terabytes of
    storage capacity to physicists, biologists, and chemists.
    Ultimately, OSG will be able to deliver large-scale computing and
    storage capacity to any application project that needs such
    capabilities.

-   *Security for experimental science.* Scientific experiments have
    become global, collaborative, and distributed endeavors in which a
    broad range of resources must be accessed, whether to complete an
    experiment, run a simulation, or search databases. This project
    would work to bridge the ever-widening gap between the tools needed
    to protect distributed science and the cybersecurity tools and
    techniques in use today.

## 5.3 Software Integration, Maintenance and Support Program

# 6. SciDAC-2 Requirements of DOE Computing and Networking Facilities

The goals of SciDAC-2, and the computational and networking goals of the
DOE programs are tightly linked. SciDAC-2 will provide enhanced delivery
of science and technology by integrating science applications,
computational science, computer science, and computer and networking
technology. SciDAC-2 will be focused on all of these layers while the
programs will be largely focused on the science and on the hardware and
networking systems. As a consequence, while SciDAC-2 does not propose
any large capability or capacity computing systems, or any broadly
deployed increase in networking capability, fulfilling the goals of
SciDAC-2 requires significant deployment of computational and networking
resources on the part of the DOE programs.

SciDAC-2 is structured to provide the enhanced delivery of science and
technology for the following DOE environments:

## 6.1 Leadership Systems

The leadership system being developed at CCS (ORNL), \[Should we also
speak to the BlueGene systems (ANL and LLNL) in this section?\] and
anticipated leadership systems in the timeframe of SciDAC-2 all rely on
getting science applications to effectively use very large numbers of
processors. Codes currently running on thousands of processors need to
scale to tens of thousands or even hundreds of thousands of processors.
Many additional science codes will need to make the jump from scale 100
processors into the regime of thousands of processors. Different
leadership systems will likely focus on different suites of applications
as their requirements are better understood. The SciDAC-2 goal for the
leadership systems is to ensure that the programmatic goals can be met
with the order of magnitude or two increase in the number of processors
used by a typical application.

## 6.2 Multi-Program Capability Computing

The capability systems such as Seaborg at NERSC (LBNL), several of the
developmental systems at CCS (ORNL), the larger BlueGene systems (ANL
\[NNSA site systems?\]), and the EMSL system at PNNL are all heavily
used for a mixture of capability and capacity computing (i.e. 10% or
less of the full machine). A number of these systems will be
significantly upgraded during the period of SciDAC-2 in terms of the
number of nodes, speed of the processors, memory, access to networking,
etc. The goal for SciDAC-2 for these systems is very similar to the goal
for the leadership systems, i.e. to ensure that the capability levels of
these systems can be used by the high priority science drivers. This
includes getting applications that today only run efficiently in
capacity modes, (i.e. limited number nodes can be used efficiently), to
be able to scale up to using the full capabilities of these systems.
Many of these codes have unique features that will require the full
deployment of the Sci-DAC-2 approach to integrate the computational,
visualization, networking, etc. elements to effectively deliver the
science objectives.

## 6.3 Specialized Application Computing

Some applications have unique features that that can be exploited by
specialized hardware, where a key metric is cost effectiveness. The most
notable current example is Lattice Quantum ChromoDynamics (Lattice QCD),
where purpose built hardware (QCDOC at BNL) has been assembled and
specially configured commodity PC clusters have been developed under
SciDAC-1 at FNAL and JLab. During the period of SciDAC-2 these
optimizations (both custom machine and application specific commodity
configurations) are expected to remain important for Lattice QCD and
possibly other specific applications. Consequently two of the SciDAC-2
goals for specialized application computing are to continue to advance
the specialized hardware solutions, and to continue the investment in
robust and portable software environments that can take advantage of the
most cost effective solutions whatever they may be. These goals are
especially important when it is realized that to meet the programmatic
science goals in these areas will require systems that push the level of
leadership class computing and beyond.

## 6.4 Terascale Capacity Computing

The bulk of the high-end capacity computing for DOE programs now takes
place on Linux clusters, augmented by a few smaller size BlueGene/L
resources now coming online. Linux clusters touch every programmatic
area in DOE. A SciDAC-2 goal is to use its integrating approach to
significantly increase the efficiency of these clusters, and to expand
the number of processors that can be effectively used. In terms of cost
effectiveness for delivered science, the potential gain from these
efforts is enormous. There is also a significant secondary advantage:
applications tend to move up the capacity / capability / leadership
stack as the scientists and developers supporting them learn the SciDAC
supported skills.

## 6.5 Networking

As evidenced by ESnet traffic, networking bandwidth usage has increase a
factor of 10 every 46 months since 1990, and all indications are that
this exponential will continue through the period of SciDAC-2. However,
making the wide area network "pipes" bigger will only solve part of the
problem. To achieve S&T goals, the end-to-end problem must also to be
solved, i.e. the data must make it all the way from the end source to
the end sink at the desired speeds and with the desired levels of
quality of service. The source drivers are very diverse: all of the
variety of high-end computational systems, experimental raw data
sources, plus massive data stores. Examples of data sinks include
high-end computational systems, massive data stores, visualization
systems, and a nearly infinite sea of low-end computational systems.

The challenge is to integrate the science requirements, end sources and
sinks, transport protocols (TCP, BIC-TCP, HS-TCP, Parallel-TCP, UDT,
Tsunami, SABUL, Hurricane, FRTP, RD-UDP, etc.), jitter control, packet
switching, circuit switching, routing, WAN components, etc. into a
system the meets the end-to-end requirements and thereby the overall
science goals. The SciDAC-2 approach is ideal for meeting this challenge
by optimally enhancing scientific discovery potential by vertically
integrating networking capabilities from science applications at the top
down through fundamental network technologies underneath.

# 7. FY 2007--FY 2011 SciDAC Program Plan

12 pages: Stevens, Simon

This section is the heart of the document and would include a budget
table covering five years and covering the program elements of SciDAC
for each of the budget scenarios.

It would also include a 1-2 page description of each of the proposed
program elements and what scope they would have under each of the budget
scenarios. The descriptions should read similar to the descriptions in
RFPs including the expected outcomes of the programs and the technical
challenges and expected types of participants

More Section 7

# 8. Organization and Management

1 page: Simon

# 9. Relationships with Other Agencies and Programs

1 page: Simon

# 10. Conclusions

(1 page)

# References

(1 page)

# Appendices: SciDAC-2 Planning Workshop Reports

5-10 page summaries of the workshops that contributed to the formulation
of the plan and any white papers that we feel help make the detailed
cases. These would not form part of the budget submission but would be
part of the record for DOE.

-   SIMS: Kendall

-   ISICs: Ashby

-   Science Institute: Stevens

-   Experimental Science: Foster, White

-   Software Institute: Womble

-   System Biology/BER: Gracio