## Fusion Energy Sciences

Developers of the NIMROD code, which is used to simulate fusion reactor
plasmas, collaborated with members of the Terascale Optimal PDE
Simulations (TOPS) Center to implement the SuperLU linear solver
software within NIMROD. As a result, NIMROD runs four to five times
faster for cutting-edge simulations of nonlinear macroscopic
electromagnetic dynamics (Figure 1).

![](media/image1.jpeg){width="2.842361111111111in"
height="2.303472222222222in"}

Figure 1. Full 3D numerical simulation of plasma particle drift orbits
in a tokamak. From Charlson C. Kim, Scott E. Parker, and the NIMROD
team, "Kinetic particles in the NIMROD fluid code," 2003 Sherwood Fusion
Theory Conference, Corpus Christi, Texas.

The Center for Extended Magnetohydrodynamic Modeling (CEMM) collaborated
with the Applied Partial Differential Equations Center (APDEC) to create
the Princeton AMRMHD code, which utilizes adaptive mesh refinement for
magnetohydrodynamic simulations, resulting in 25 to 30 times faster runs
at the highest resolution. A new physical effect discovered by the
AMRMHD code is current bunching and ejection during magnetic
reconnection. (Figure 2)

![](media/image2.jpeg){width="3.0in" height="3.0166666666666666in"}

Figure 2. Current bunching and ejection during magnetic reconnection.
From R. Samtaney, S. Jardin, P. Colella, and T. Ligocki, "Numerical
simulations of magnetic reconnection using AMR," Sherwood Fusion Theory
Conference, Rochester, NY, April 2002.

## High Energy & Nuclear Physics

### Advanced Computing for 21st Century Accelerator Science and Technology (AST)

SciDAC support has enabled the development of four independent,
high-fidelity, particle-in-cell (PIC) codes: OSIRIS (fully explicit
PIC), VORPAL (fully explicit PIC plus ponderomotive guiding center),
QuickPIC (quasi-static PIC plus ponderomotive guiding center), and UPIC
(a framework for rapid construction of new codes such as QuickPIC).
Results from these codes were included in three articles in Nature and
eight in Physical Review Letters. For example, members of the SciDAC AST
project have performed large-scale simulations using the codes OSIRIS
and VORPAL to help interpret laser- and plasma-wakefield accelerator
experiments and to gain insight into the acceleration mechanism (Figure
3).

![](media/image3.jpeg){width="5.047916666666667in" height="2.51875in"}

**Figure 3.** Results from a 3D OSIRIS simulation, performed at NERSC,
of a beam with 2  10^10^ particles, ~r~ = 20 m and ~z~ = 63 m,
propagating through a lithium gas cell of density *n~0~* = 5.143 
10^16^ cm^--3^. The simulation box was 512  128  128 grid cells wide,
and the simulation was run for 3100 time steps. This 3D plot of the
electron density shows the ionization electrons and wakefield as well as
the drive electron bunch. Both isosurfaces and projections of this data
are shown: the drive beam in red-orange-yellow colors, and background
electrons in blue-cyan-green colors. (Image: R. A. Fonseca and L. O.
Silva, Instituto Superior Técnico, Portugal; S. Deng and T. Katsouleas,
USC; and F. S. Tsung and W. B. Mori, UCLA)

Great progress has also been made in the development of reduced
description models for laser/plasma simulation. One such example,
QuickPIC, has been shown for some problems to provide answers as
accurate as OSIRIS but with 2 to 3 orders of magnitude less computation
time. Before the SciDAC effort, a full-scale simulation of a 1 TeV
afterburner would have required 5 million node hours on a supercomputer
(and thus was not done); after SciDAC, using QuickPIC, the simulation
was done in 5,000 node hours on a cluster system.

A cover story and three papers in the September 30, 2004 issue of Nature
announced the first demonstration that a beam of electrons can be
accelerated in a wakefield to a single energy (Figure 4). These results
represent the most significant step so far for laser-based accelerators,
and should stimulate further advances in the near future. Such
accelerators would not only be more compact but would also exceed
conventional sources in peak current, brightness, and shortness of pulse
duration. SciDAC members were collaborators on two of these Nature
papers, and SciDAC codes were used to analyze the experiments; the cover
illustration was a VORPAL simulation.

![](media/image4.png){width="2.3333333333333335in"
height="3.111111111111111in"}

Figure 4. SciDAC codes were used to analyze the experiments in two of
three featured articles on compact particle accelerators in the
September 30, 2004 issue of Nature.

### National Computational Infrastructure for Lattice Gauge Theory

The structure of neutrons, protons, and other strongly interacting
particles in terms of their quark and gluon constituents can be
calculated from first principles by solving QCD on a discrete space time
lattice. With the advent of SciDAC software and prototype clusters and
of DOE-supported dedicated lattice QCD computers, it is now possible for
the first time to calculate physical observables using full QCD in the
regime of large lattice volumes and light quark masses that can be
compared with experiment. Results include nucleon electromagnetic form
factors characterizing the distribution of charge and current in the
nucleon, the longitudinal momentum distribution of quarks in the
nucleon, the transverse structure of the nucleon, the origin of the
nucleon spin, and baryon shapes. Lattice QCD also provides insight into
questions of hadron structure, such as the nature of the pentaquark
states that have been explored in recent experiments.

## Biology & Environmental Research:  Global Climate Research

### Earth System Grid II:  Turning Climate Datasets into Community Resources 

The Earth System Grid II (ESG) project has turned climate data into
community resources by creating a virtual collaborative environment that
links climate centers and users around the world to models and data. In
Fall 2004, ESG started its production service to distribute the
Intergovernmental Panel on Climate Change (IPCC) Working Group on
Coupled Models (WGCM) data to the greater climate community. Over 288
projects have registered to receive IPCC data for analysis. The number
of files downloaded by the climate community totals over 125,000, which
is equivalent to over 22 terabytes of data. The daily download rate is
now averaging over 300 gigabytes. At the IPCC Workshop held March 1--4,
2005, over 130 scientists presented their analysis results of the IPCC
data, which was distribute to them via ESG.

## Basic Energy Sciences:  Chemical Sciences 

### Collaboratory for Multi-Scale Chemical Science (CMCS)

Active Thermochemical Tables (ATcT) represent a significant breakthrough
in chemistry that is directly enabled by SciDAC. ATcT is a new paradigm
of how to obtain accurate, reliable, and internally consistent
thermochemical values and overcome the limitations that are deeply
engrained in the traditional approach to thermochemistry. The
availability of high-quality consistent thermochemical values is central
to chemistry and critical in many areas, such as development of
sophisticated high-fidelity electronic structure computational
treatments or development of realistic and predictive models of complex
chemical environments such as combustion or the atmosphere. As opposed
to the conventional sequential evolution of thermochemical values for
the chemical species of interest, ATcT utilizes the Thermochemical
Network (TN) approach. This approach explicitly exposes the inherent
intricate interdependencies normally ignored by the conventional
treatment and allows a statistical analysis of the individual
measurements that define the TN. The end result is the extraction of the
best available thermochemistry, based on optimal use of all the
knowledge that is available, hence making conventional tabulations of
thermochemical values obsolete.

## Advanced Scientific Computing:  National Collaboratories & Networking

### DOE Science Grid: Enabling and Deploying the SciDAC Collaboratory Software Environment

The Access Grid is an ensemble of resources including multimedia
large-format displays, presentation and interactive environments, and
interfaces to Grid middleware and to visualization environments. These
resources are used to support group-to-group interactions across the
Grid. For example, the Access Grid is used for large-scale distributed
meetings, collaborative work sessions, seminars, lectures, tutorials,
and training. The Access Grid thus differs from desktop-to-desktop tools
that focus on individual communication. The Access Grid concept
complements and extends the concept of the Computational Grid by
exploring and supporting this more complex set of requirements and
functions. The Access Grid has over issued 3,400 certificates to users
across 47 countries. Each institution has one or more AG nodes, or
"designed spaces," that contain the high-end audio and visual technology
needed to provide a high-quality compelling user experience. The nodes
are also used as a research environment for the development of
distributed data and visualization corridors and for the study of issues
relating to collaborative work in distributed environments.

GADU/GNARE, the Genome Analysis and Database Update system developed at
the Mathematics and Computer Science division of Argonne National
Laboratory, has successfully analyzed more than 14 million sequences in
a single year---an amount of data equal to 3,500 average bacterial
genomes. The system is one of the world's first Grid-enabled
high-throughput computational environments designed to provide
researchers with up-to-date sequence information. Globus Toolkit
software and related components are used to harness hundreds of
distributed processors from the GRID2003, NSF TeraGrid, and the DOE
Science Grid, as well as from Argonne and Pacific Northwest national
laboratories, in order to perform high-throughput computations. The
Grid-enabled system allows efficient automation of the major steps of
genome analysis, including data acquisition from diverse genomic
databases and data analysis by various bioinformatics tools and
algorithms. GADU/GNARE also expedites the process of storing the results
of analyses and annotations. The flexible architecture enables users to
tailor the genome analysis process to their individual needs: it can
function in an automated mode as well as interactively through a
Web-based interface.

### Particle Physics Data Grid Collaborative Pilot

The Particle Physics Data Grid provides a coherent grid infrastructure
for 35 sites, 4500 CPUs, and 10 applications, including interoperability
with European Grids. PPDG has given the U.S. Large Hadron Collider
experiments 30--50% more throughput through use of available resources
they do not own.

In 2004 PPDG joined several other Grid projects to form the Open Science
Grid (OSG) Consortium, in order to build and support a
production-quality petascale Grid infrastructure for large-scale
science. The Open Science Grid will ensure that the U.S. plays a leading
role in defining and operating the global Grid infrastructure needed for
large-scale collaborative and international scientific research.

The D0 experiment is routinely using the Data Grid to reprocess its
recorded data---over 500 million events so far---creating datasets for
physics analysis at a dozen computer centers in the U.S., EU, and
Canada. GridFTP has increased the throughput rate of event data by a
factor of 5.

The use of a production grid for managed data movement has consistently
reduced the time to publication for many physics results from the STAR
experiment. The STAR computing groups at BNL and LBNL have operated
production Grid data movement in collaboration with the Scientific Data
Management (SDM) Center. Automated data transfers of up to 5 TB a week
in both directions give physicists "next day" access to data for
analysis. The average throughput is 10 times greater than before the use
of Grid tools, and file discrepancies have been reduced by a factor of
50.

### National Fusion Collaboratory

Scientists at the DIII-D National Fusion Facility in San Diego have been
collaborating for 20 years on the JET European fusion experiment, the
largest operating tokamak in the world. The FusionGrid's remote
collaboration services now allow them to conduct experiments at JET
without leaving San Diego. Access Grid technology provides multiple
video images and a unified audio stream between San Diego and the JET
control room. MDSplus data services allow data to be analyzed in San
Diego as soon as it is available to researchers in the control room.
Web-based displays give real-time tokamak information as well as pulse
cycle and general data analysis status. Similarly, scientists from
France, Germany, and Japan are now participating remotely in DIII-D
experimental operation. The success of FusionGrid's remote collaboration
services validates the National Fusion Collaboratory's design of the
Collaborative Control Room and represents a model for the next
generation burning plasma experiments, including ITER.

### A High-Performance Data Grid Toolkit:  Enabling Technology for Wide Area Data-Intensive Applications 

The ability to manipulate huge data sets has become a key enabler of
advances in science. Climate scientists can run models at resolutions
that were simply unthinkable in the past, and they need even higher
resolutions. Large scientific instruments, such as the Large Hadron
Collider (LHC) at CERN, the Laser Interferometer Gravitational Wave
Observatory (LIGO), and the Sloan Digital Sky Survey (SDSS) are
producing terabytes per day, and in the case of the LHC are projected to
produce several petabytes per year. Manipulation of data sets this size
requires secure, scalable, and robust middleware tools. The DOE SciDAC
DataGrid Middleware Project has produced the tools that are the de facto
standard for Grid projects the world over for these types of problems.
The availability of this middleware layer has significantly reduced the
development effort required in several application projects, eliminating
the need to design and implement complex software modules while enabling
the scientists to focus on building high-level user applications.
DataGrid Middleware has also reduced the support overhead, since the
software has been tried and tested by users all over the world.

### Middleware Technology to Support Science Portals

Recent advances in networking, large-scale simulation capability, and
radar are about to change our ability to predict tornadoes, hurricanes,
and other mesoscale weather events with more precision. The NSF "Linked
Environments for Atmospheric Discovery" (LEAD) project is building a
distributed collaboratory to design and test adaptive workflows that
will link real-time observations directly with large supercomputer
simulations that focus computational power adaptively on very focused
regions where important weather events seem likely to occur. Eventually
these simulations will be able to dynamically retarget the radars to
provide the most accurate data possible back into the simulations. An
essential underpinning of the LEAD project is being provided by SciDAC's
Middleware Technology to Support Science Portals project---a flexible
yet powerful portal framework that allows researchers to analyze and
catalog data, compose experimental and production workflows, launch
simulations, and track the progress of computations.

## Advanced Scientific Computing:  Integrated Software Infrastructure Centers

### Scientific Data Management Enabling Technology Center

Researchers in the SciDAC Scientific Data Management Center received a
patent for their Word-Aligned Hybrid (WAH) method for compressing
database bitmap indexes. This method, implemented in software called
FastBit, can answer a user query 10 times faster than the searching
method used by one of the leading commercial database management
systems. FastBit has already been applied to spatio-temporal data from
two other SciDAC projects, the Terascale High-Fidelity Simulations of
Turbulent Combustion with Detailed Chemistry, and the Terascale
Supernova Initiative.

Researchers in the SciDAC Scientific Data Management Center received a
patent for their Word-Aligned Hybrid (WAH) method for compressing
database bitmap indexes. This method, implemented in software called
FastBit, can answer a user query 10 times faster than the searching
method used by one of the leading commercial database management
systems. FastBit has already been applied to spatio-temporal data from
two other SciDAC projects, the Terascale High-Fidelity Simulations of
Turbulent Combustion with Detailed Chemistry, and the Terascale
Supernova Initiative.

The use of a production grid for managed data movement has consistently
reduced the time to publication for many physics results from the STAR
experiment. The STAR computing groups at BNL and LBNL have operated
production Grid data movement in collaboration with the Scientific Data
Management (SDM) Center. Automated data transfers of up to 5 TB a week
in both directions give physicists "next day" access to data for
analysis. The average throughput is 10 times greater than before the use
of Grid tools, and file discrepancies have been reduced by a factor of
50.

Storage Resource Managers (SRMs) are middleware components developed by
the Scientific Data Management (SDM) Center that provide dynamic space
allocation and file management on shared storage components on the Grid,
enabling diverse storage systems to interoperate. For example, the BNL
to NERSC setup uses an SRM arrangement called a DataMover to move about
10,000 files per month (about 1 GB each) automatically and with a great
reduction in the error rates. The SRM standard has been widely adopted
by various international efforts, including the Earth Systems Grid, the
Particle Physics Data Grid, and Europe's NorduGrid. Although the SRM
specification has become a de facto standard, there is an ongoing effort
to standardize this functional specification through the Global Grid
Forum (GGF).

### Terascale Optimal PDE Simulations (TOPS)

Developers of the NIMROD code, which is used to simulate fusion reactor
plasmas, collaborated with members of the Terascale Optimal PDE
Simulations (TOPS) Center to implement the SuperLU linear solver
software within NIMROD. As a result, NIMROD runs four to five times
faster for cutting-edge simulations of nonlinear macroscopic
electromagnetic dynamics.

### APDEC

The Center for Extended Magnetohydrodynamic Modeling (CEMM) collaborated
with the Applied Partial Differential Equations Center (APDEC) to create
the Princeton AMRMHD code, which utilizes adaptive mesh refinement for
magnetohydrodynamic simulations, resulting in 25 to 30 times faster runs
at the highest resolution. A new physical effect discovered by the
AMRMHD code is current bunching and ejection during magnetic
reconnection. (Figure 2 above)

A cover article in the July 19, 2005 Proceedings of the National Academy
of Sciences reported an unparalleled three-dimensional computer
simulation of a laboratory-scale turbulent rod-stabilized premixed
methane V-flame (Figure 5). The simulation did not resort to models for
turbulence or turbulence-chemistry interactions and was unprecedented in
several aspects---the number of chemical species included (19), the
number of chemical processes modeled (84), and the overall size of the
flame (12 cm high)---and it closely matched the results of a laboratory
experiment. This collaboration between computational and combustion
scientists was supported by the Applied Partial Differential Equations
Center (APDEC).

![](media/image5.jpeg){width="2.55in" height="3.365972222222222in"}

Figure 6. This V-flame simulation, which models a full laboratory-scale
flame by using detailed chemistry and transport, treats a domain more
than three orders of magnitude larger in volume than that of any
previous efforts and represents a major increment in simulation
complexity.