**The DOE Science Grid-Phase 2**

Project Principal Investigators: William T.C. Kramer, National Energy
Research Scientific Computing Center at Lawrence Berkeley National
Laboratory, Ray Bair, Argonne National Laboratory, Al Geist, Oak Ridge
National Laboratory, Keith Jackson, Lawrence Berkeley National
Laboratory, Jennifer Schopf, Argonne National Laboratory, and Scott
Studham, Pacific Northwest National Laboratory; Science Grid Engineering
Working Group: Keith Jackson, Chair, LBNL, Tony Genovese, ESnet, Mike
Helm, ESnet, Von Welch, ANL, Steve Chan, NERSC, Kasidit Chanchio, ORNL,
Scott Studham, PNNL

#  {#section .unnumbered}

#  {#section-1 .unnumbered}

# **Executive Summary** 

Grid technology is evolving to provide the services and infrastructure
needed for building "virtual" systems and organizations. With this Grid
based infrastructure that provides for using and managing widely
distributed computing and data resources in the science environment,
there is now an opportunity to provide a standard, large-scale,
computing, data, instrument, and collaboration environment for science
that spans many different projects, institutions, and countries. The DOE
Science Grid is developing aspects of Grid technology to ensure that it
provides a basis for the creation of the integrated environments that
can combine the resources needed to support DOE's large-scale science
projects that have components located at multiple laboratories and
universities world wide.

The DOE Science Grid Proposal has been working to achieve this vision
for the past 30 months. Its goals were to prototype Grid infrastructure
tools, to research and design appropriate components and to deploy a
multi-lab grid prototype that could be used to demonstrate the
advantages of using grid technology for scientific challenge teams. The
first phase of the project is coming to completion having accomplished
these goals -- yet there is much left to be done in order to make the
use of Grid technology commonplace. Thus, this proposal is for a
two-year extension of the DOE Science Grid effort that will be focused
on deploying a "persistent production grid" and facilitating its use by
applications in strategic science discipline areas of interest to the
DOE. The extension will allow the DOE Science Grid effort to have a
major impact in the SciDAC projects that are due to finish in 2006 and
2007.

#  {#section-2 .unnumbered}

# **Introduction**

# 

+-----------------------------------------------------------------------+
| ![](media/image1.png){width="4.506944444444445in"                     |
| height="3.3805555555555555in"}                                        |
+-----------------------------------------------------------------------+
| 1.  Integrated, Advanced Cyber-Infrastructure -- Very High-speed      |
|     Networks, High Performance Computing, and Grid middleware --      |
|     Enables Advanced Science: A Vision for the U. S. Dept. of Energy, |
|     Office of Science                                                 |
|                                                                       |
| -   Enable the collaborative and interactive use of the next          |
|     generation of massive data producing scientific instruments       |
|                                                                       |
| -   Facilitate large-scale scientific collaborations that integrate   |
|     the Federal Labs and Universities                                 |
+-----------------------------------------------------------------------+

The DOE Science is a project that began in August 2001. The ultimate
goal of the DOE Science Grid project is to define, integrate, deploy,
support, evaluate, refine, and develop (as necessary), the persistent
Grid services needed for a scalable, robust, high-performance Grid
needed to support an advanced cyber infrastructure.  This Grid will
provide persistent services for security, resource discovery, resource
access, system monitoring, and so on, to advanced scientific
applications and problem solving frameworks. By reducing barriers to the
use of remote resources and facilitating large-scale collaboration, we
are making significant contributions to SciDAC wide software standards
and resources, and address the infrastructure for the next generation of
science process. 

The approach is to integrate activities in deployment, research and
development, and application outreach that allow us to refine the tools
and their deployment and support processes, to providing the
capabilities that will enable the DOE Science Grid to be
cost-effectively scaled to support large-scale science
collaborations.  Close collaborations with a variety of application
projects will ensure relevance to SciDAC goals and enable innovative
approaches to scientific computing, via secure remote access to online
facilities, distance collaboration, shared petabyte datasets, and
large-scale distributed computation.

Major accomplishments to date include the design and deployment of a
Grid security infrastructure that is facilitating collaboration between
US and European High Energy Physics Projects and within the US Magnetic
Fusion community; a monitoring and debugging infrastructure that
facilitates managing this widely distributed system and that building of
high performance distributed science applications; deployment of a beta
resource information infrastructure; developing development and
deployment partnerships with several key vendors; and the addition of a
limited number of systems as resources on the Grid. These are important
steps in deploying a realistic scale Grid environment that supports
experimenting with advanced Grid services.

A recent DOE workshop^1^ solicited the opinion of a number of scientific
disciplines as to how the process of doing their science needed to
change in order to make significant scientific advances. Several general
observations came out of this.

The first, and perhaps most significant, observation is that a lot of
science is already, or is rapidly becoming, an inherently distributed
endeavor involving many collaborators that are frequently
multi-institutional. Rapidly increasing data and computing requirements
are routinely addressed with resources that are often more widely
distributed than the collaborators. As scientific instruments become
more automated and complex (and therefore more expensive) they are
frequently used as shared facilities with remote users. For somewhat
different reasons this trend is true of numerical simulation as well^2^.
This leading edge science depends critically, on an infrastructure that
supports the process of distributed science. Future science goals will
generate more even reliance on the distributed infrastructure.

The second important observation of the workshop is that all the science
areas need high-speed networks and advanced grid middleware to couple,
manage, and access the widely distributed, high-performance computing
and storage systems, well beyond what is available today together with
the many medium-scale and desktop systems of the scientific
collaborations, high data-rate instruments, and massive data archives.
Due to the clear trend distributed science activities, all of these
elements operating smoothly together in large-scale collaborations are
required in order to produce an advanced distributed computing, data,
and collaboration infrastructure for the next generation of science.
This is beyond what can be done today in several respects; the number of
participants in a distributed collaboration, the amount of data that can
be managed, the diversity of the use of data, the number of people that
need to discover and use the data, the number of independent
computational simulations that need to be combined in order to represent
more realistic or complex phenomenon or physical system, etc.

The general goal of an integrated, advanced, cyber infrastructure --
commonly known as "The Grid" - is to deliver an overall computing, data,
and collaboration quality of service to scientific projects. That is:

-   Computing capacity adequate for a task is provided at the time the
    task is needed by the science,

-   Data capacity sufficient for the science task is provided
    independent of location, and in a transparently managed, global name
    space,

-   Communication capacity sufficient to support all of the
    aforementioned is provided transparently to both systems and users,
    and

-   Software services supporting a rich environment that lets scientists
    focus on the science simulation and analysis aspects of software and
    problem solving systems, rather than on the details of managing the
    underlying computing, data, and communication resources.

All of these are (or can be) provided by using Grid middleware as the
mechanism for coupling computing, data, instruments, and human
collaborators into an integrated science environment.

-   
-   
-   
-   
-   
-   
-   
-   

### 

### 

1.  
2.  
3.  
4.  
5.  

```{=html}
<!-- -->
```
1.  []{.mark} DOE Science Grid progress to date

The DOE Science Grid project was begun in July 2001 with a kickoff
meeting at the Global Grid Forum in Vienna, Virginia which focused on
the first phase of the project: building a multi-site computational and
data Grid among the participating institutions, The project included
five DOE sites, LBNL (lead), ANL, ORNL, NERSC, and PNNL. Funding was
received in late August 2001.

## Highlights

Highlights of progress are found in the quarterly reports located at
http://doesciencegrid.org/. These include

**Grid Information Services (GIS)** is deployed and running at each
site. The DSG federated these into a larger GIS that allows searches
across the combined DSG resources. Several prototype Virtual
Organization have been constructed based on GISs.

**Certification Authority** **-** DSG designed a scalable approach to
providing and deploying a production CA at ESnet to support the
scientific community. A full production version of the CA is issuing
certificates for DSG users and other DOE science related users. A
Registration Authority (RA) at each site/Virtual Organization now has
the responsibility of verifying the identity and membership of the
applicants. Then the local RA authorizing the CA to issue certificates.
Each RA is an individual who also becomes part of the Policy Management
Authority (PMA), which is the CA's policy oversight body. The PMA meets
periodically, and has approved the current Certificate Policy and
Certificate Practices Statement (CP/CPS). This policy work was vital in
establishing a trust relationship between the DSG CA and the European
Data Grid (EDG) CAs. The result is the first interoperable Grids between
the major US and European High Energy and Nuclear Physics
collaborations. This work has been vital in enabling closer
collaboration and resource sharing between US and European High Energy
Physics communities. Current information about the status of the
certificate policy work can be found at the DSG/ESnet CA website,
<http://www.doegrids.org/>.

**Deploy Globus** on Computing Resources **-** Remote job submission,
data transfer, and information services are deployed and working at all
sites. Interoperability between the sites has been established. This
includes the major production systems at NERSC: the 10 TF Seaborg, the 9
Petabyte HPSS Archive and the Parallel Data Simulation Facility (PDSF).

**Grid Tertiary Storage** - the DSG has facilitated the development and
deployment of GridFTP and/or GSIFTP at NERSC. LBNL, and PNNL. The DSG
project, working with the NERSC Mass Storage Group, contributed to the
design, prototyping, and testing of a version of an HPSS tertiary
storage system, parallel ftp (pftp) server that supports GSI
authentication and tcp buffer tuning. This prototype is being used by
the Earth Systems Grid to move data between ORNL and NERSC, and the STAR
experiment moving data between BNL and NERSC. This version will remain
in use until a redesigned version can be supported by the HPSS
collaboration. PNNL is deploying its new data archive (13 TB disk-based
system); Globus has been included as an authentication method.

**Security Infrastructure** - The communication ports used by Globus are
subject to local firewall policies, so the DSG project examined the
ports used, and how they are used. DSG, in close collaboration with the
production security administrators at the DSG sites, developed a best
practices document for firewall administrators. The document is
available at <http://www.globus.org/Security/v2.0/firewalls.html>.\
\
Risk identification and mitigation is a critical issue for large scale
centers, which are especially concerned about root compromise because of
the potentially very long downtime (weeks) needed to rebuild compromised
a system. Because the Grid exposes supercomputers to the Internet in
many more ways than they are currently exposed, it is very important to
monitor that exposure to make sure that it does not present an
unacceptable vulnerability. NetSaint is a Python-based system for
monitoring remote systems that has been deployed as a prototype. NERSC
and the DOE Science Grid have undertaken a risk and compromise
mitigation study to see what can be done in the Grid environment to help
counteract increased exposure. That study, "Compromise Mitigation for
Grid Applications," may be found at
<http://doescienceGrid.org/Grid/papers>. This work determined which of
these risks can be addressed in the near term, which require
development, and which are inherent risks.\
\
The DOE Science Grid at NERSC also collaborated with the PPDG Site-AAA
project in which the site security staff at BNL, FNAL, JLab and SLAC, in
addition to NERSC, considered the issues and requirements at the site
interface to the grid in the area of authentication and authorization.
The work resulted in two PPDG papers \[PPDG-27, PPDG-28\] as well as
forming a research group in the Global Grid Forum (Site Authentication,
Authorization and Accounting).

**Auditing and Fault Monitoring** **-** ORNL has designed and developed
a distributed Grid Resource Usage Data management and Accounting system
(RUDA). RUDA collects resource usage data from local accounting systems
of individual resource entities and stores the data locally. For
projects, collaborations, sites, and other organizations to perform
accounting tasks, RUDA can build hierarchical structures while still
storing data in a distributed manner. A RUDA prototype has been
developed and deployed onto Solaris and AIX platform computers of the
DOE Science Grid test bed and Earth System Grid II (ESG) at ORNL and
NCAR, and was used within the ESG project to monitor resource loads
generated by large-scale file movement operations and provide accounting
information. A draft paper "Meet the challenges of Grid Resource Usage
Data Management and Accounting" has been written to present the RUDA
architecture, its major features, and the mechanisms behind them. The
paper also discusses the applications and performance of RUDA. **\
\
**In term of fault notification and response, ORNL has completed the
development of the prototype fault monitoring and notification system
(WBS. B.6.6). The system comprises a sensor process, a C API library,
and a Web-based user interface. The sensor is a light-weighted process
that monitors existence of grid jobs and services on a local computer.
It allows users and applications to define, at runtime, failure
situations they want to monitor and responses when the interested
situations occur.\
\
In the area of fault-tolerance, DSG designed and developed a
leader-based group membership protocol featuring mechanisms that enable
a group of processes to adapt group membership in presences of process
failure, hardware failure, network partition, and network recovery
situations. The protocol was used in a system called XVM that conducted
a number of experiments on a 64-node Linux cluster. Experimental results
demonstrate practical group adaptation performance for services
distributed across a grid. A recently finished a paper describing the
protocol is available at \[TBD\]

**User Services** **-** A production version of PNNL's online trouble
ticket system, ESHQ, has been installed and is in testing on
[http://DSG.emsl.pnl.gov](http://DOE-SG.emsl.pnl.gov/), and a web page
has been created to guide users to helpdesk support. ESHQ is a Java
application, enabling support staff to query and respond to support
queue items from any computer. ESHQ has a graphical user interface and
supports a plethora of techniques for database searching. This not only
helps the support staff, it also allows users to search for solutions
for similar problems. We have formed a working group to develop trouble
ticket interoperability requirements among the DSG sites, and with other
Grid organizations, including iVDGL and GGF.

**Grid System Administration Tools** **-** The LBNL Python-wrapped
Globus services toolkit (www-itg.lbl.gov/Grid/projects/pyGlobus/) was
developed in the DSG and is being used to build some experimental Grid
system administration tools to help support the production usage of Grid
resources across the DSG sites. These include graphical tools for
checking the status of Grid resources, managing Globus configuration,
adding new Grid users, etc. A monitoring infrastructure based on the
NetSaint framework ([www.netsaint.org](http://www.netsaint.org/)) has
been deployed. LBNL developed a series of plug-ins built using pyGlobus
to support the monitoring of Grid Services, including GRAM, GridFTP,
GIS, etc.

**Gird User Access to Resources** - A standard template has been created
to describe DSG resources for the scientific community. A sample
resource description can be found at
<http://doesciencegrid.org/LBNL_resource_diesel.html>.\
\
The DSG has a 200,000 hour allocation on the NERSC-3 system and an
allocation of 2,000 Storage Resource Units on the NERSC HPSS system.
Some users also have access to the NERSC PDSF systems. An initial
agreement with ORNL allows DSG users to access three cluster systems in
the Computer Science and Mathematics Division as well as a Sun E250 and
HPSS in the ORNL Center for Computational Sciences. Through the user
proposal system for the Environmental Molecular Sciences Laboratory
(EMSL), resources may be requested on production and development systems
in the Molecular Science Computing Facility. Further, authorized EMSL
users will be able to run Grid applications.

**Applications** - The Extensible Computational Chemistry Environment
(Ecce) has been upgraded to use version 2.2 of Globus to launch
computations and retrieve results and integrate more Grid capabilities
as Grid services evolve. Large-scale inverse modeling is one of the most
computationally demanding analyses performed by PNNL in the
characterization of subsurface contaminant behavior. Field-scale
three-dimensional models of flow and transport at the Hanford site are
being calibrated with inverse modeling technology that scales up from
coarse-grained parallel processing on clusters to massively parallel
processing on DSG resources.\
\
At LBNL several applications are being prototyped on DSG compute
resources. These include a regional air quality model that has been
converted to MPI so as to be able to use the DSG clusters, and a
supernova cosmology data analysis application that is using a Grid
scheduler.\
\
As collaboration between the Globus project and the Argonne
bioinformatics group, the GADU (Genome Analysis and Databases Update)
tool has been developed: an automated, high-performance, scalable
computational pipeline for data acquisition and analysis of the newly
sequenced genomes with DOE Science Grid backend. GADU allows efficient
automation of major steps of genome analysis: data acquisition, data
analysis by variety of tools and algorithms, as well as data collection,
storage and annotation.\
\
The STAR experiment is a regular user of the PDSF and HPSS resources at
NERSC. The collaborative effort between DOE Science Grid, Particle
Physics Data Grid and the Scientific Data Management ISIC has resulted
in STAR using datagrid tools in production at NERSC supporting the data
analysis and simulation computations at PDSF. In collaboration with PPDG
and iVDGL the PDSF resource is a participating site in the Grid2003
project (see below) and there are several continuing high-energy physics
applications use on PDSF resulting from this.

**Demonstrations**

-   SC2002 - DSG efforts supported the development of Globus job
    submission software used by the National Fusion Collaboratory in its
    SC2002 demonstration. This demo featured the integration of two
    existing fusion applications distributed between exhibit floor
    machines in the LBNL and Argonne booths and server machines at the
    Princeton Plasma Physics Lab and General Atomics. In another
    exhibit, PNNL demonstrated an 80 gigaflop version of their new HP
    parallel supercomputer running Globus 2.2 on the DOE Science Grid.
    ORNL demonstrated a fault monitoring system consisting of a number
    of daemon processes working together to monitor DOE Science Grid
    resources at ORNL.

-   SC2003 - At Supercomputing 2003, LBNL demonstrated
    "*wxPyGlobusJobGui*", a graphical user interface which integrated
    functionality from three projects to provide a securely
    authenticated, grid-enabled, and network monitored prototype job
    submission and monitoring system. *wxPyGlobusJobGui* demonstrated
    the utility of pyGlobus to very rapidly generate a graphical user
    interface which automated the steps necessary to stage (copy) AMBER
    input and control files from the exhibition floor to a cluster
    located at Berkeley Lab, run the AMBER application on the input
    files, and stage the result files back. The graphical user interface
    demonstrated two other technologies: NetLogger and Akenti. Akenti is
    an access control policy library which uses digitally signed
    certificates to define access policies for shared resources. DSG
    added Akenti authorization calls to the Globus jobmanager, and
    defined a policy that allowed specific applications to be run only
    by authorized users. NetLogger, a lightweight network logging
    toolkit, facilitates the analysis and debugging of distributed
    computing environments. NetLogger message logging was integrated
    with the pyGlobus file staging and remote executing modules, as well
    as the Globus gatekeeper and jobmanager, and the Akenti access
    control policy library. The NetLogger messages were used to
    visualize the file staging, remote execution, and access control
    steps of the demonstration. This tool presented a timeline of
    activities and events (Figure 2) and identified and helped diagnose
    errors.

-   ORNL demonstrated the Resource Usage Data Management and Accounting
    (RUDA) and the fault monitoring and notification systems at
    Supercomputing 03. As shown in Figure 3, the RUDA system reports
    accumulated resource usage information on a set of computers in the
    DOE Science Grid test bed. It also demonstrated the fault monitoring
    and notification systems that dynamically report status information
    of grid services and jobs to a web page.

-   The National Fusion Collaboratory (NFC) and the Particle Physics
    Data Grid (PPDG) have been active collaborators with the DOE Science
    Grid, sharing experience and expertise. One obvious result was the
    PDSF cluster at NERSC was integrated with and participated in the
    Grid2003 project
    \[[www.ivdgl.org/grid2003](http://www.ivdgl.org/grid2003)\].
    Grid2003 (27 sites, 7 running applications) is a large scale
    application demonstrator in the experimental high energy physics and
    astrophysics communities, coordinated by the NSF funded iVDGL
    project as a joint effort with PPDG, the US-ATLAS, US-CMS, LIGO and
    SDSS software and computing projects, and the 27 sites. In addition
    to the technical details of middleware and application software
    installation and configuration, this project provided a context for
    looking at the technical aspects of policies related to
    authentication (identifying users by X509 credentials rather than
    Unix login) as well as grid-wide resource accounting issues.

```{=html}
<!-- -->
```
-   ![](media/image4.png){width="3.252083333333333in"
    height="2.5458333333333334in"}

-   A timeline form the *wxPyGlovusJobGui* used the status and error
    codes from the NetLogger messages to draw a "timeline" of events
    that indicate the progression of the file staging and job execution
    components as well as determining if errors occurred and flagging
    events which failed.

![](media/image5.png){width="3.125in" height="2.651388888888889in"}

![](media/image7.png){width="2.625in" height="1.707638888888889in"}

2.  Figure 3: RUDA and fault monitoring web pages.

## Reviews 

The DOE Science is reviewed annually for progress and plans. At the 2003
Annual Review, an external team of reviewer noted satisfaction for the
following accomplishments of the project

-   Application support through the collaboration with application
    specific Grid efforts such as the Earth System Grid (ESG), the
    Particle Physics Data Grid (PPD), as well as supplication oriented
    services such as progress creating Visualization portal

-   Significant progress in Certificate Authority management and
    deployment activities, most recently leading the Tokyo accord. This
    includes the PMA approach.

-   Firewall/IDS investigations that identify how firewalls and IDS
    interact with Grid applications. The project also demonstrated
    progress in creating IDS capability to improve this interaction

-   Positive impact on grid integration with local site security

-   Investigation into performance of grid services (starting with MDS)

-   Python tools (pyGlobus) that can by used by scientists in quickly
    crafting Grid applications

-   •Authorization and Authentication work to lower entry barrier for
    new grid users such as methods of handling private key information

-   Significant industrial interaction, which the reviewers felt, was
    key for efficient data movement.

Overall the reviewers felt the DOE Science Grid project has the
potential to be very valuable to the science community. They commended
the members of the project for exerting strong leadership in resolving
issues in policies, standards, and best practices.

Of course, the review panel also had suggestions, some beyond the scope
of the original project plan. They suggested the DOE Science Project
plan should be revised to take into account the new impact of the Open
Grid Service Architecture (OGSA), which was introduced in late 2002. The
review committee desired "A Persistent DOE Grid" that allows easy
movement of work across the range of DOE environments. This includes
allocation and account agreements, training and outreach, and similar
user environments. The reviewers wanted DSG to work with other partners
in the DOE to help develop set of best practices, explore ways to
harmonize site policies and define rules of operation, eventually
leading to establishment of a DOE Grid Operations Center that supports
all grid efforts.

The review suggested the DSG project should test Grid software and tools
at scale. Specifically important were user tools, package "DOE grid
tools", building regression testing suite and prototyping the
information architecture. They recommended steps towards integrated
monitoring tools (NetLogger, NetSaint, etc.) Finally, they also
suggested closer and broader contact with the science customers for
DOE's Grids. While some applications are engaged, the review encouraged
the project to reach out to SciDAC applications in particular as a way
to help demonstrate the usefulness of the DSG. Further recommendations
were to establish processes to understand application needs for tools
and services (and adjust software deployment accordingly) and then
ultimately to demonstrate usefulness of the Grid tools deployed to
applications.

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

#  

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# **Next Phase** **of the DOE Science Grid** -- Creating the DOE Persistent Science Grid

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

The DOE Science Grid has been very successful at exploring the issues of
creating production Grid services for the DOE science community. This
effort includes enabling Grid services into full production on major
platforms. In responding to the comments of the review discussed above
as well as the overall needs of the DOE science community, the DSG
proposes a Phase-2 of the DOE Science Grid. The next step is to expand
the number of production Grid services until the goal of a Persistent
Grid for DOE Science spanning multiple DOE sites is realizable.

The DOE Science Grid had four initial goals

-   Demonstrate the ability to reliably and efficiently deploy a grid.
    This has been demonstrated.

-   Work with application partners application partners to evolve grid
    software and function. The DSG has done this with limited
    applications, but needs to do more to help enable the Grid use to
    become commonplace.

-   Do the research and development required to fill the functional gaps
    that exist in research software. These gaps need to be filled in
    order to make Grid Computing possible in production settings. Much
    work has been done, but more is still needed

-   Create a test bed to scale. Demonstrations at SC 2003 indicate this
    goal is achieved.

What remains to be done is still a major effort. In order for serious
science projects to use the DSG, they have to believe their efforts will
have long term benefits in a persistent Grid environment. Moving from
today's demonstrations and test beds to the point where it is possible
to implement and maintain a true large scale, persistent DOE Science
Grid for all DOE researchers is the next logical step.

This *Persistent* *Science* *Grid* (PSG) would operate at the service
levels that production sites normally provide for FTP and batch
scheduling services. Grid services would be integrated into the normal
administrative infrastructure of sites so support staff has expertise in
its operation. Standard tools will monitor the PSG, and uptime and
maintenance are similar to mainstream services. In addition, the
policies that are related to Grid services will be brought in line with
standard DOE policies -- or where standard DOE policies do not yet
exist, they will be defined in the context of a mature production
service instead of an experimental or beta test context.

The DSG Phase 2 has the goal of creating the first version of a
Persistent Science Grid, and preparing the necessary information and
best practices so that the operation and enhancement of the PSG can be
handled by some sustaining organization. Furthermore, there needs to be
more effort to bring large-scale science applications to the PSG. In
order to do this the DSG-2 has three overall goals.

-   Create the DOE Persistent Science Grid on production resources at
    the participating sites.

-   Enable targeted large-scale science to use the PSG. The science
    applications will be strategic applications in disciplines important
    to the Office of Science

-   Make it easier for scientists to use DSG/PSG

The end result of this effort is that by the end of 2006 the version 1
of the PSG should be ready to hand off to an on-going effort that will
support and probably continue enhancements. There will also be a track
record of improved science and a core set of applications using the DSG
that will be able to make full use of the PSG.

There is still much work to be done in order to accomplish this goal.
Specific areas include:

-   Research efforts focused on monitoring, credential management and
    security are key any success in grid environments.

-   The DSG-2 will continue deploying and enhancing of Grid functions as
    a set of production service across PNNL, ORNL and NERSC. Furthermore
    a research, development and testing grid will be set up for using
    resources across all the sites.

-   DSG-2 will place significant effort in facilitating the use of the
    DSG for selected large-scale science. We will focus with scientific
    teams working in Genomics, Climate and Astrophysics, and with others
    as well. DSG will continue its engagements with the HEP and NP
    focused grid efforts as well..

-   To "lower the bar" for both users and site system managers, the DSG
    will document experiences for "Best Practices" in creating, managing
    and using grids. This includes creating guidelines for configuration
    recommendations, performance tuning, and troubleshooting. If also
    includes deploying a central point for publishing DSG information
    will assist. DSG-2 will serve as an expert resource as DOE develops
    policies that will impact the feasibility and usability of Grids. We
    will also use this a central point for others to get information and
    learn about the DSG.

-   By the time the PSG is in place, the Globus Toolkit will have
    evolved into a web-services compliant software suite, based on the
    recently announce Web Services Resource Framework (WSRF). This means
    it is critical the DSG-2 transitioning to the new software at an
    appropriate time but before the PSG is established. However, since
    parts of the new software will not offer backward compatibility with
    GT 2.X, so it will take additional research, testing and occasional
    development to help the science community move forward.

-   It is expected that the PSG will have a sustaining Grid Operations
    Center. Assuming DOE desires and funds a sustaining Grid effort,
    DSG-2 will work with other grid efforts to help make the PSG and a
    Grid Operations Center functional by the end of 2006.

-   

We will cover each of these tasks in the following sections.

# **Long term strategy** 

The creation of a ***persistent*** grid for DOE science raises the
question of what happens when this project ends in 2006. Emphasis in
this proposal extension is on engaging key DOE science applications that
are empowered if not enabled by the existence of the Persistent Science
Grid. Our long-term strategy is three-fold. First, to become of such
utility to DOE science programs that these science programs request that
a production grid be available. Second, to create a steering committee
of users much like the ESnet steering committee or the NERSC user group.
This group would be responsible for making sure the DOE Persistent
Science Grid continues to meet the needs of the scientists. Third, to
work with and leverage other scientific grid efforts around the world.

The production DOE computer centers exist because science requires these
computational and storage resources in order to advance. Similarly, for
the DOE Persistent Science Grid to become a long-term, supported
infrastructure in the production DOE environment, scientists have to go
to the Office of Science and explain that they require the PSG in order
to advance their science. Science driven support for a persistent grid
supplied by the production DOE infrastructure is a key to long-term
survival.

To get this support we have to show scientists in a few key DOE programs
that the DOE Science Grid enables new ways to do their science or that
the DOE Science Grid provides utility to the existing ways they do
science.

Persistent, production resources such as NERSC and ESnet maintain their
utility by having a formal group give advice on what isn't working.
Similarly a Persistent Science Grid will need a means to stay relevant
to changing science requirements. As part of our long-term strategy we
propose to create a means of collecting and prioritizing community
requirements following the model of the NERSC users group. We will hold
an initial meeting of a prototypical DOE Science Grid user group.

One potential issue with production computer centers supporting grid
users is the problem of having to support many different grid software
stacks. This is not a viable long-term strategy. To avoid this problem
we propose to define a level of interoperability between grids so that a
computer center would only have to support a single persistent grid
software stack. This will involve working with other grids, such as the
Earth Sciences Grid and PPDG as well as other grid efforts around the
world. One question we will attempt to answer is how the DOE Science
Grid can peer with other grids \[what's peer mean in this context?\].

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

#  

# Research and Development

There are still several research topics related to putting Grid into
production. Often they revolve around extending or modifying existing
tools and techniques to work in a distributed, peered environment.

## 

## 

## Monitoring and diagnosis

In a persistent multi-site Grid, both users and site administrators will
need to be informed about the state of Grid services across all the
sites. Tools for efficiently handling this have not yet been identified
and promoted within the DSG, but there are several likely candidates.

The objective is to have a distributed monitoring system that is
integrated with the site monitoring tools, as well as a "Grid Operations
Center" so that all sites can have up to date access to service state
information. In addition, well-understood procedures for documenting
maintenance or other scheduled downtimes need to be developed.
Monitoring should go beyond the very simplistic "is there a listener on
the port?" to higher levels in the software stack -- verifying
functionality in addition to accessibility. B eyond merely identifying a
service as "broken" it may be possible to perform some level of remote
diagnosis. This may take the form of tools that run a small suite of
further tests, a document that explains diagnosis procedures or merely
well understood procedures between site administrators for problem
diagnosis. These methods and tools are part of the standard
infrastructure to support production service levels -- currently some of
these tools exist within a single site, but in a production Grid they
will need to be extended to work across multiple sites.

1.  Security

Security is an ongoing concern with the Grid -- the goals of security
and ubiquitous access can be antinomic. For production use, there is
still significant research work to be done.

### 

### 

### Authentication, Authorization and Accounts

Authentication based on GSI is relatively mature. However, key
management approaches, integration with site security procedures and
revocation processes are not sufficient. Several basic, viable
approaches have been mapped out but there is significant room for
improvement in terms of security and usability.

Experience has shown that one of the largest barriers towards getting
application scientists using the Grid has been the complexity of the
public key infrastructure (PKI) security environment. Typically end
users find the acquisition and management of their PKI credentials
difficult and not well understood. This is especially true for users who
may work at multiple sites, on multiple resources, and may need to have
certificates signed from more than one certificate authority

In addition, there is little assurance that once certificates have been
issued, the user will handle them properly. Many site security people
are uncomfortable with current PKI mechanisms that allow users to manage
private keys in ways that may not meet site security policies, for
example, leaving private keys unencrypted or using a dictionary word for
a password.

One DSG objective is to make GSI authentication and key management
transparent to a user logging onto a machines. That is all GSI related
tools would return to the level of "plumbing" that operates without
explicit user interaction.

Authorization is a topic that has made progress in terms of
specifications and development but this progress is not yet reflected in
many production environments. The default grid-mapfile authorization
will need to be either replaced with more powerful and scalable
approaches, or integrated into automated tools that manage them properly
as well as integrated with Virtual Organization management tools.

System administrators struggle to interface Grid PKI with their local
site security infrastructure, and at times find scalability issues with
the current Grid tools. For example, many projects have written their
own ad hoc methods for updating GridMap files across resources, but no
standard approach is common across projects.

Accounts for Grid users are a contentious issue and one of the places
where the antonomic relationship between ubiquitous access and security
becomes clear. The management of accounts for Grid users will require a
lot of work in defining policies as well as developing tools and
procedures for management. There will have to be significant
coordination with DOE policy development as this progresses. For
example, a user receiving an account at NERSC should automatically
receive a certificate as part of the account setup process. This should
also include automatic inclusion in the appropriate site GridMap files.
In addition, an important service would be an online CA that generates
proxy certificates based on standard site authentication methods (using
PAM). This would allow Grid authentication to be seamlessly merged into
normal login processes via PAM modules.

To help system administrators have more control over how users interact
with personal private keys, DSG is investigating improving MyProxy to
allow long term storage of private keys. This allows local
administrators to control policies such as password length, style
enforcement, and password renewal policies, and prevents users from
insecurely managing their private keys.

For system administrators, tools to aid in bringing new machines into a
Grid are being developed, and a better understanding of what is included
in the process will be documented. For example, we now see the
importance of bulk certificate issuance (for example when 100 machines
are added to a test bed) when setting up large clusters.

1.  Intrusion Detection

To support the flexible and dynamic environment of the Grid, modified
security policies and configurations will likely give intrusion
detection systems a more prominent role. Past work at NERSC on BRO has
put monitoring of Grid-FTP at the same level as conventional FTP.
Continuing work will include making Bro aware of Gatekeeper and
Grid-based services in addition to analysis of the PKI credentials used
for authentication.

At the host level, the use of host based intrusion detection systems
will become more important as hackers attempt to exploit Grid services
to gain access to hosts. This is an area that has received insufficient
attention to date and must receive more.

# Deploying Production Resources on the DOE Science Grid

In order to facilitate applications running on the DOE Science Grid and
to increase the impact of the DOE Science Grid one of the primary goals
of the second phase of this project will be to continue the deployment
of production resources at NERSC, ORNL, and PNNL on the DOE Science
Grid. The project will work with production staff at each of the sites
to integrate Grid operations into their production processes and
practices, and create documentation that will enable other sites to do
the same. The production resources will include:

-   [NERSC:Seaborg, PDSF, Escher , HPSS]{.mark}

-   [ORNL: Eagle, Phoenix, Ram, Cheetah, HPSS]{.mark}

-   PNNL: MPP2, NWfs

## Factors for success

There are several critical factors for successful deployment of the DOE
Science Grid at the member sites:

-   Consistent, stable, robust configuration at all sites

-   Cross-site user support

-   Integration of Grid tools and components into production system
    administration processes

-   Eventual integration with a DOE Science Grid Operations Center

## Common user environment 

One fundamental challenge for Grid users today is the lack of a stable
and consistent software environment throughout the participating sites
of their virtual organization. This lack of consistency imposes a high
overhead on adapting an application for use on a Grid; users must
manually log into each site, learn the environment, verify what is
working, and then configure their application specifically for that
site. It is critical for DSG to address this problem through the
definition of a common user environment, and publication of this data,
without affecting local site policy.

A common user environment consists of both software and environment
definitions. The software stack that is defined needs to include not
only Grid middleware, but also compiler, user, and OS level tools that
scientists can expect to find on a resource. This is not to say that all
sites will run the same software, but that versions of important
software will be published at a well-known location for ease of use.
*Our experience has found that publishing detailed versioning
information on important system components can greatly increase
cross-site interoperability,* for example minor C library revisions can
contain major incompatibles which a affect applications at runtime and
must be addressed by users. We plan to leverage current projects such as
TeraGrid, NMI GRIDS center, and VDT who are already defining software
stacks in their work.

In addition, we have found that defining associated environment
variables can simplify a users interaction across sites to a large
degree. In order to allow local site autonomy while still providing a
consistent user environment, this is most frequently done through a
level of indirection, for example using SoftEnv keys or Modules. We plan
to define a standard set of environment variables for use in the DSG
Grid to provide a consistent user view of the resources.

1.  Configuration Management

As applications begin to span multiple sites, compatibility will become
an important issue. In addition, in a Grid of heterogeneous systems some
conventions will need to be applied so that not only are services
compatible with each other, but that application code is managed
properly as well. Various groups have come up with approaches for this
problem, but they will need to be evaluated carefully and then
implemented in a way that supports the characteristics of the DOE
Science Grid and its divers set of system and applications.

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## Coordination across three sites 

The DSG project will provide coordination and guidance between the
member sites that versions and portions of the Globus toolkit are to be
required on those resources that are deployed on the DOE Science Grid.
It will be up to each site to independently act on this guidance. It is
expected that follow-on sites will adopt similar portions of the
toolkit.

-   **Support that spans multiple sites** - It is unreasonable to expect
    users to manage problems and troulb etickets across DSG sites. Many
    DSG users expect that they will not have to determine which member
    site they need to request help from. Indeed when they have a
    problem, they may not even know where the cause is. It is necessary
    to provide a support structure that crosses individual site
    boundaries and provides "one-stop shopping" for DSG support. This
    central DSG help desk will be provide access to DSG staff and have
    enough knowledge of the production workings of each of the DSG
    resources they can make a preliminary determination whether the
    problem is system or grid related. In other instances, a user who
    generally uses one site will access resources on remote sites via
    Grid protocols and encounter problems and report it to the local
    site. The administrators at this local site will need a method of
    bringing the problem to the attention of multiple sites. This
    support structure will be valuable to both end users and site
    administrators.

-   **Cross site Trouble Ticket System** - In order to facilitate
    transferring help requests between sites, a trouble ticket exchange
    mechanism has been created. This system allows top-level support
    personnel to directly submit trouble tickets into a site's existing
    trouble ticket system, helping to smoothly integrate Grid operations
    into existing site processes. The trouble ticket exchange will be
    implemented at all 3 DOESG production sites and the iVDGL iGOC.

-   **Integration with System Administration Practices** - A site will
    not persist with quality services on the Grid unless grid operations
    are integrated with the site's everyday administrative. The site's
    staff must understand the Grid itself, the configuration of the
    toolkits it is based upon, and how they interact with the site's
    systems and users. Some areas that require further work are:

-   *Process for account maintenance includes Globus Certificates* -
    Each site's grid users will have grid credentials that will need to
    be associated with the user's account, and the site will need a
    process to keep the grid credentials associated with the user's
    account.

-   *Process to manage grid-mapfiles & add/remove users as
    appropriate* - Each site will have unique grid-mapfiles for each
    system that allows Globus access, and will therefore need a process
    to manage those files. Users must be added to the file(s) when they
    are granted access to the system, and should also be removed from
    the files when Globus access is revoked.\
    \
    Alternatively, extensions to GSI allow sites to eliminate
    grid-mapfiles entirely and integrate with site
    authentication/authorization tools like LDAP or NIS. This is an
    important development path that needs to be explored.

-   *Transition from Grid specialists to general system administration
    staff* - Just as with prior technologies (e.g. networking, web
    servers), administration of grid components should become
    commonplace for rank-and-file system administrators, and not the
    domain of a few experts. A goal of the DSG project is to effect that
    transition at the 3 sites, creating the processes and documentation
    to do so.

-   *Integration with site specific resource managers and filesystems* -
    A critical component of the Globus toolkit that will be unique to
    each site will be the interface between Globus and a site's Resource
    Management systems, namely the GRAM job-manager interfaces. The
    job-manager interfaces will need to be configured and maintained by
    the staff at each site that is responsible for the job manager.

    Resource Managers currently at the 3 sites:

```{=html}
<!-- -->
```
-   PNNL: LSF, NWfs migration tools

-   [NERSC: Seaborg - LoadLeveller, PDSF -- LSF/GridEngine]{.mark}

-   [ORNL: LoadLeveler]{.mark}

```{=html}
<!-- -->
```
-   *Exchange of account information* - It will be necessary to exchange
    accounting information between sites and for other reasons. To this
    end, DSG-2 will develop an exchange methodology/format for
    accounting records.

## Testbed 

Shortly after the DSG project began we found that we had an urgent need
for a functional test Grid to be able to thoroughly examine out proposed
solutions without impacting production resources. We needed a platform
where we could examine operation issues that would impact large
multi-site Grids at scale, and that could be used to experiment with new
solutions without affecting current production use.

As part of DSG-2, we will continue the deployment of a rid testbed at
all DSG project sites in order to allow testing to understand future
deployment needs, to test solutions in isolation from production work,
and to further our research and development goals.

1.  

2.  A sustaining testbed

```{=html}
<!-- -->
```
2.  **Large Scale Science running over the Grid**

Science applications have been an integral part of the DOE Science Grid
pilot project from its inception, not only because they provide an
opportunity to demonstrate the power of Grid in the DOE environment, but
also because working directly with applications teams exposes areas that
need improvement to advance Grid capabilities and operations to meet the
needs of the DOE science community and its partners. This approach
worked well in the first phase of this project, helping DSG to focus our
efforts on the foremost issues to enable the DSG to be a useful and
usable resource. Work with application and tool projects has been
instrumental in their progress, especially with troubleshooting problems
using Grid services in multi-site environments and defining best
practices. Also, DSG efforts in central Certificate Authority services
and policies saved the DOE applications community a great deal of effort
in setting up CAs and trust policies. However, as many DOE applications
move from demonstration to early production and new applications come on
line, it is clear that the applications focus of the DSG pilot needs to
grow.

Therefore DSG is increasing effort with applications in the second phase
of this project, with the intent of not only enabling applications to
use DSG resources, but also to enable science that is vitally enabled by
the DSG. In addition to providing DSG information and troubleshooting
services to the broad DOE community, we are establishing close alliances
with a small number of representative DOE science projects that use the
Grid to address new classes of problems. The many opportunities far
exceed our resources even with considerable leverage from the projects,
so we have selected a few strategic projects that span different styles
of distributed computing, including major efforts with projects from the
Genomes To Life program, astrophysics and the climate modeling program,
both described below. These are complemented by smaller efforts with
projects in particle physics, nuclear physics, fusion and power grid
modeling.

## ***Genomes to Life*** 

The DOE Genomes To Life (GTL) program is a highly visible, high impact
effort combining advanced genomic technologies and computation with the
information found in microbial genomes to establish a foundation for
understanding living systems. Over the next five years, this program
will produce an estimated hundreds, possibly thousands, of terabytes of
data from dozens of experiments conducted at laboratories and GTL
facilities across the nation. The short-term and long-term value of this
data to project participants, to the DOE, and to the nation depends on
being able to access the distributed data and being able to run analyses
across the databases. The DOE Persistent Science Grid is an ideal
infrastructure to enable the access and analysis of the widely
distributed GTL data.

Genomes To Life also makes an ideal application to utilize the PSG
because all the DOE Science Grid sites: LBNL, PNNL, ORNL, and ANL are
heavily involved in GTL and related biosciences programs. While the
seamless integration of GTL data and analysis across the sites is well
beyond the financial scope of the DOE Science Grid project,
computational biology funds already at the sites will be heavily
leveraged along with DOE Science Grid funds to enable this capability
for new science by the Genomes to Life program.

### Application requirements

In particular, we propose to create a GTL Data Grid service across the
DOE Science Grid sites. It is increasingly evident that a paradigm shift
is needed in the development of a data infrastructure, away from
dispersed "data collecting" repositories towards a conceptually
integrated "knowledge enabling" repository. The GTL Data Grid service
will be more than a storage and retrieval capability. Rather, in the
long term, it will support fundamentally new ways of doing science.

Leveraging both the DOE Persistent Science Grid infrastructure and the
GTL efforts at LBNL, PNNL, and ORNL, we propose to a distributed but
integrated suite of GTL databases, through the cooperative development
of data models and database schema prototypes being led by Adam Arkin
and Victor Markowitz at LBNL. This process will seek to identify shared
or common data elements, objects, concepts and identifiers that can lead
to metadata types that are sharable across the GTL facilities. In this
way at the highest level the independent systems can evolve to meet the
local conditions of domain experts effectively, while being able to
share a common intellectual layer of process and information. In
collaboration with all the DOE Science Grid sites, this will permit the
unique knowledge acquired at each site to be used across the DOE complex
and eventually permit users to mine data from the combined sites. This
effort will involve regular discussions between the GTL researchers
across DOE to define extensible standards for data models and schemas
that meet the needs of the planned GTL science.

### DOE Science Grid Impact

DSG will develop a prototype shared data infrastructure based on the
data models and data object schemas being defined in the data standards
discussions described above. Initially it will permit data sharing
between the databases within the existing GTL centers located at ORNL
(Geist), LBNL (Arkin & Markowitz), and PNNL (Studham). Once standard
data schemas and access interfaces are established then the GTL Data
Grid service will be expanded to allow data mining across the GTL
knowledge base as well as providing data input for large-scale
biological modeling efforts. The latter will involve computational
resources of the DOE Science Grid as well as the GTL Data Grid service.

The proposed distributed data infrastructure for GTL, based on the DOE
Science Grid, will be hierarchical, spanning (in a coordinated fashion)
multiple layers ranging from local data management (with efficient
storage and access) and information retrieval (with efficient indexing
and flexible querying capabilities) through global data integration
(with distributed annotation systems and standards for defining,
querying, and transmitting common data objects) towards distributed data
mining and eventually knowledge discovery.

A key component of this infrastructure will be biological data
integration, which is in its infancy. Seamless integration of such
dispersed and heterogeneous resources will present a lot of other
challenges for GTL including leveraging the computational resources of
the Persistent Science Grid to provide high-performance analysis of the
exponentially growing volumes of data, leveraging the ESnet networking
infrastructure for high-speed interconnects between geographically
distributed resources, and finally the creation of data mining and
visualization infrastructure that allows biologists to make inferences
from these massive amounts of heterogeneous and distributed biological
data.

To enable analysis of distributed biological data we propose to develop
a Biology Grid Application service. In the first year this service will
be developed and a prototype deployed at Pacific Northwest National
Laboratory and Oak Ridge National Laboratory but accessible by
biologists at all the sites and across the nation. In the second year
the service will be deployed across all the DOE Science Grid sites.

### GTL Companion Projects

Three biological application services will be integrated with the GTL
Data Grid services in order to provide access to potential input data as
well as potential storage location for important results: Rosetta, GIST
and GADU.

As part of the DOE Science Grid efforts PNNL's Molecular Science
Computing Facility, in conjunction with the Institute for Systems
Biology, is developing a Biology Grid Application service for harnessing
the unusable cycles of the MSCF HP supercomputer. One of the key
applications of this service is a code called Rosetta that predicts
protein structure in the absence of sequence-homology to proteins of
known structure. Rosetta is an ab-initio protein structure prediction
program developed at the University of Washington by David Baker's
group. Predicted protein structures are used in combination with other
bioinformatics tools, and global experimental observations, to uncover
the functional roles of these proteins of unknown function. The Rosetta
structure prediction procedure is a computationally intensive Monte
Carlo procedure for finding low-energy conformations for a given protein
sequence.

In addition to the Rosetta application the Biology Grid Application
service will also run the "Grid Based Genome Integrated Supercomputing
Toolkit" (GIST) developed in the Life Sciences division at ORNL and
being deployed at NERSC, ANL, and PNNL. Many major biology applications
are available through this grid-based toolkit including MPP-BLAST, HMMER
and PROSPECT. A central feature of the web services based GIST framework
is the concept of drop-in tools i.e. each site can customize the
available applications with site-specific research applications .

The GADU/Gnare project is an ongoing genome analysis effort in
collaboration with the DOE Science Grid that uses LCRC computational
resources at ANL, the MSCF at PNNL, and the Teragrid. The Persistent
Science Grid is key to future GADU developments being proposed for DOE's
Microbial Cell program, a companion to GTL. In the next phase, an
analysis portal is being constructed that will enable a much broader
community to access the GADU/Gnare and provide more sophisticated
analysis and annotation workflows within the entire set of sequenced
genomes. This has exposed challenges in Grid credential propagation and
data storage management, which can also be addressed with GTL Data Grid
services.

1.  Climate Modeling

DOE Climate Change Prediction Program **(**Warren Washington, Jerry
Meehl, Jeff Kiehl, Jim Hack, and Peter Gent, National Center for
Atmospheric Research; Bert Semtner, Naval Postgraduate School; John
Weatherly, U.S. Army Cold Regions Research and Engineering Lab
Laboratory) is a strategic project that carries out a substantial
portion of the climate change scenarios for the US contribution to the
Intergovernmental Panel of Climate Change (IPCC) fourth assessment
report.

The FY03 research saw a continuation of the benchmark 1000 year CCSM
control run, and an extension of the PCM control run to nearly 1500
years. These scenarios run with the Community Climate System Models
(CCSM) at a T85 resolution. The IPCC scenarios are categorized into two
groups: a set of three \"commitment scenarios\" to be completed by a
July 1 2004 deadline and a number of \"science scenarios\" that explore
alternative scenarios.

The IPCC commitment scenarios agreed upon at the IPCC meeting in July
2003 contain three different scenarios: SRES A1B, SRES B1 and a 20th
century emissions stabilization scenarios. The international climate
modeling community will carry out each of these agreed upon scenarios
with their respective climate models. The A1B and B1 scenarios will
consist of five-member ensembles run from the year 2000 to the year 2200
with one of the ensemble members being continued for an additional 100
years at constant emissions amounts. The 20th century emissions
stabilization scenario freezes atmospheric emissions at levels observed
in the year 2000. This emissions stabilization scenario will be carried
out at NERSC while the A1B and B1 scenarios will be run ORNl and NCAR.

These simulations assume emissions of greenhouse gases and
sulfate/carbon aerosols. In the most recent IPCC document, the PCM and
CCSM simulations have been highlighted in many parts of the document,
particularly, chapter 9, which discusses the climate model projections.
The computational approach of each CCSM2[^1] component is briefly
described below:

6.  Ocean Model - Based upon the Parallel Ocean Program (POP) code, the
    ocean model is designed to run efficiently on massively parallel
    machines. Horizontal resolution averages less than 1 degree, with 40
    vertical levels.

7.  Sea Ice Model - This model component is completely new. The
    thermodynamic part of the model uses the physics from the Bitz
    (University of Washington) ice model and the elastic-viscous-plastic
    dynamic code is utilized to simulate ice motion, using the Hunke and
    Dukowicz (LANL) approach to the solution of the ice dynamics.

8.  Atmospheric Model - The atmospheric component is the massively
    parallel version of the NCAR Community Atmospheric Model (CAM). This
    model includes solar and infrared radiation, boundary physics,
    precipitation physics.

9.  Land Model - The land model contains completely new biogeophysical
    and hydrological parameterizations, and demonstrates improved
    simulation of snow cover and seasonality of runoff (which transports
    water to the oceans at 0.5 degree resolution).

10. Coupler - The method of tying the components together and allowing
    the exchange of fluxes and state variables is the coupler. The
    coupler\'s design allows the component models to execute
    concurrently as separate executables with the information exchange
    achieved by message passing (MPI). The interpolation algorithm has
    been designed to run efficiently on distributed memory
    architectures. Chris Ding and collaborators at Lawrence Berkeley
    National Laboratory have improved the performance of the coupler on
    parallel computers. This is being further developed by a cooperative
    effort between DOE SciDAC program and NSF. The new version of the
    coupler has now been installed in the most recent version of CSM2.x.

One of the main uncertainties with regards to forcing is the role of
aerosols. The IPCC 20th century simulations will include volcanic and
direct effect sulfate aerosols. However, in the new CCSM2.x, four
aerosol types will be included: sea salt, carbon, sulfate indirect
effect, and mineral with differing atmospheric effects. The team will
run complex ensemble simulations, varying the aerosol effects one by one
(four 130 year experiments), and other experiments.

### Computational and Storage Requirements

The project requirement is for 2,304 IBM SP Power-3 Hours and 800 GB of
storage per Climate Model Year.

  ---------- ------------- -------------- --------------- ------------- -------------
  **Year**   **Goal**      **Run Size**   **Computation   **Yearly      **Storage**
                                          (NERSC SP       Computation   **(WTCK -
                                          Hours)**        Need**        need to
                                                                        verify)**

  2004       IPCC: Run     600 Model      1,382,400                     
                           Years                                        

             All forcings  4 runs \* 130  1,198,080       2,580,480     0.9 TB
                           Model Years                                  

  2005       Carbon        4 runs \* 130  1,198,080                     
             Aerosols      Model Years                                  

             Greenhouse    4 runs \* 130  1,198,080                     
             Gas           Model Years                                  

             Sulfate       4 runs \* 130  1,198,080       3,594,240     1.25 TB
             indirect      Model Years                                  

  2006       Sulfate       4 runs \* 130  1,198,080                     
             direct        Model Years                                  

             Ozone         4 runs \* 130  1,198,0801                    
                           Model Years                                  

             Solar         4 runs \* 130  1,198,080                     
                           Model Years                                  

             Volcanoes     4 runs \* 130  1,198,080       4,792,320     1.7 TB
                           Model Years                                  

  Totals                                                  10,967040     3.8 TB
  ---------- ------------- -------------- --------------- ------------- -------------

The results of this project will be submitted as a substantial portion
of the US simulations of the scenarios for the IPCC fourth assessment
report. The specific data products from these simulations will be used
for climate change impact studies and as boundary conditions to drive
regional-scale models by a large number of researchers who need access
to the large data sets. The integrated picture formed by these scenarios
will be the basis for long-term energy and resource use policies. By the
use of higher resolution studies, the data provided to the climate
impacts community that will be more useful than previous studies.

### DOE Science Grid Impact

This effort represents a multi-institutional distributed effort of
scientists and software engineers at the National Center for Atmospheric
Research (NCAR), Los Alamos National Laboratory (LANL), Oak Ridge
National Laboratory (ORNL), the Naval Postgraduate School (NPS), the
Cold Regions Research and Engineering Laboratory (CRREL), and the
Lawrence Berkeley National Laboratory (LBNL). It is expected 1/3 of the
computational will be done at NERSC, 1/3 at ORNL and 1/3 at NCAR. Data
will be permanently stored at NERSC (and possibly NCAR).

Currently, the research team is using "brute force" to accomplish their
aggressive schedule. They log in to each site to submit and check jobs.
While each job it parallel, the runs have to be done with careful
sequencing because each simulated year relies on the output of the
previous year. The team uses ftp mechanisms to transfer files. They have
created scripts and other tools to check the status and page on-call
team members if errors are detected.

The DOE Science Grid will work with the team to enhance the productivity
through the use of Grid software and functions. While the DSG only has
direct collaboration with NERSC and ORNL, there is an excellent working
relationship between the DSG and the Earth System Grid that included
LBNL, ORNL and NCAR. The areas of most value that will be priority for
DSG are

1.  Deploying a working Grid infrastructure on the NERSC and ORNL
    systems.

2.  Improving the ability to control workflow (jobs submission and
    monitoring, file transfer, etc) at the sites.

3.  Improving data transfer rates and recoverability between sites

4.  Developing and deploying improved end-to-end monitoring software for
    Grid jobs.

![](media/image8.png){width="4.864583333333333in" height="4.65625in"}

3.  Two hundred years of modeling El Niño events and surface
    temperatures on the Community Climate System Model (CCSM2) closely
    correlate with 50 years of actual climate data.

## Astrophysics

DOE has many highly visible, important efforts in astrophysics
including; understanding the fundamental origins of the universe,
understanding the large-scale structure of galaxies, probing the
properties of black holes, and determining the final outcome of the
universe. In the coming years, astronomy faces a data avalanche.
Breakthroughs and advancements in telescope, detector, and associated
computer technology allow astronomical surveys to produce terabytes of
images and catalogs. These datasets will cover large fractions of the
sky spanning the spectrum from gamma rays to radio and will probe
fundamental questions in physics and astronomy that were unimaginable
just a few years ago. Because this data is coming from a variety of
sources it is vitally important that it be accessible in a consistent
way to researchers around the world. The National Virtual Observatory
(NVO) is already building a framework to link the archival data sets
with ongoing space and ground surveys. Although the NVO begins to
address some of the issues involved in large-scale astrophysics, it does
not go far enough to enable the seamless integration of observation and
simulation that is necessary for the next generation of astrophysics.
Due to the presence of many important astrophysics researchers at DOE
labs participating in the proposal, the DOE Science Grid can play an
important role in providing computer science expertise in building a
framework for astrophysics data management and analysis, while also
leveraging current astrophysics funding.

The DSG will collaborate closely with three applications, enabling them
to take full advantage of both the current DOE Science Grid
infrastructure and preparing them for coming changes to Grid middleware.
One application area involves studying the early history of the universe
by looking at the cosmic background radiation left over from 200,000
years after the Big Bang. Another important area is the study of
supernova. Recent discoveries in this field have radically changed our
understanding of the make up of the universe. The final area involves
the study of black holes, and what happens when two large black holes
collide. Each of these areas is important to DOE, and their usage of
Grid computing to more tightly couple observation and simulation could
offer a whole new set of fundamental breakthroughs in astrophysics*.*

The current picture of cosmology has primordial density perturbations
generated in one or more quantum field phase transitions in the very
early Universe. They evolved in first a radiation- and then a
matter-dominated Universe, ultimately acting as the seeds around which
the large-scale structure of the Universe was formed. This picture is
subject to three classes of observational constraint. The first comes
from what we know of the very early Universe -- for example any quantum
field theory must satisfy the limits on particle types and numbers
inferred from light element abundances and primordial nucleosynthesis.
The second are the tiny anisotropies in the temperature and polarization
of the Cosmic Microwave Background radiation, whose angular power
spectra encode the amplitude and scale of the primordial perturbations
at the epoch of recombination, when the Universe first became
electrically neutral. The third is in the detailed statistical
properties of the resulting galaxies, and their clusters and
superclusters, at the endpoint of this evolution.

The Cosmic Microwave Background (CMB) is the earliest detectable image
of the Universe, being the surface at which the primordial photons last
scattered only 300,000 years after the Big Bang. As such it provides our
most powerful probe of early Universe cosmology -- which itself is our
only test-bed for theories of ultra-high energy physics beyond the
standard model. Since its serendipitous detection in 1967, generations
of experimentalists have struggled to develop instruments with
sufficient sensitivity to measure the minute fluctuations in the CMB -
typically one part in one hundred thousand in its temperature and one
part in ten million to one billion in its polarizations - whose detailed
statistical distributions carry the tell-tale imprint of this
fundamental physics.

In recent years this effort has resulted in the first resolved maps of
the primordial CMB temperature anisotropies, and the first unambiguous
detections of a harmonic sequence of peaks in the angular power spectrum
of these maps. These results alone have determined the flatness of
large-scale geometry of space-time and ruled out an entire class of
models of Grand Unified physics. Together with the type 1A supernova
results, they have also led us to the extraordinary conclusion that 95%
of the Universe is made up of unknown kinds of matter and energy.

These achievements have required significant computational resources,
including millions of CPU-hours and terabytes of both short- and
long-term data storage. However, they mark only the beginning of an
avalanche of ever larger and more complex CMB observations that will
force us to change not only the way that we analyze CMB datasets, but,
more fundamentally, the way that we work as distributed collaborations.

### Application Requirements

The archetype for this is the PLANCK project; scheduled to launch in
2007, this ESA/NASA satellite mission will carry 100 detectors, each
taking data at up to 200Hz for 14 months or more. The result will be
tens of terabytes of raw data measuring the anisotropies in both the CMB
temperature and polarization, incorporating complex cross-correlations
between multiple, frequency-dependent, signal sources (both CMB and
contaminating microwave foregrounds) and the dominant instrument noise.
Disentangling these will be an extraordinary data analysis challenge,
second only to the challenge of coordinating the efforts of the hundreds
of mission scientists from tens of countries around the globe.

### DOE Science Grid Impact

A significant step towards this goal would be the development of a
PlanckGrid, coupling the various computational resources -- both cycles
and storage---available to the Planck collaboration in a transparent
manner. A first step towards this goal would be the development of a
prototype PlanckGrid which the DSG can help facilitate.

## Other Applications \[Fusion-Keith, Power Grid-Scott\]

In addition to our three primary applications, several associated
applications will be using the Persistent Science Grid in order to
further their science needs.

### **Fusion**

### **High Energy Physics:** **Star,** **ATLAS, and ALICE**

Of the facilities participating in the DOE Science Grid, the PDSF
facility at NERSC has a form of data intensive computational services
appropriate for the experimental high-energy and nuclear physics
communities. While a number of experiment collaborations use PDSF, there
are three in particular that plan to significantly expand their scope of
grid computing over the next few years, STAR, ATLAS and ALICE.

The STAR experiment is located at Brookhaven National Lab along with the
other RHIC experiments and is focused on measuring the properties of
matter produced at the high energy density frontier by the collisions of
atomic nuclei at extremely high energies (200 A-GeV). It is postulated
that the quark and gluon constituents of the protons and neutrons in
normal atomic nuclei are "liberated" briefly during the high-density
stage of a collisions between two large nuclei (such as gold, used at
RHIC) and that region of similar size to the colliding nuclei contains a
soup (plasma) of quasi-free quarks and gluons, called the Quark Gluon
Plasma. As the hot-dense plasma expands and cools it forms a large
number of various elementary particles (baryons, mesons, ...) and the
STAR experiment detects many of these final-state particles. By making
measurements with various mass and energy of beam particles as well as
events with varying total multiplicity of produced particles, the
experiment can determine some properties of the matter produced during
the high-density stage of the collision. STAR is in its fourth year of
data taking in what is likely a fifteen-year program. Recent results
indicate that the energy loss of a parton traversing the collision
region for the highest mass systems (Au+Au) is much greater than that
for partons traversing ground-state nuclear matter, as occurs in
collisions of d+Au. Data replication between the computing facilities at
Brookhaven and NERSC is managed by STAR using grid middleware so that
terabytes can be moved in an automated fashion. In the coming year STAR
is planning to move it's computational workload for data analysis as
well as simulations onto the grid, an effort being carried out as part
of the PPDG project.

ATLAS is an experiment being prepared at CERN at the Large Hadron
Collider (LHC) accelerator and will begin data taking in 2008. It is a
very large collaboration (\~2000 physicists) with worldwide
participation including several hundred scientists in the U.S. The
primary goal of the ATLAS experiment is to measure properties of the
Higgs boson, predicted to give rise to the mass of other elementary
particles, and hence the matter in the universe. Like all such
experiments at a new energy frontier there will be numerous measurements
and discoveries leading to a better understanding of nature in addition
to the primary motivation. In the years leading up to data taking, the
computing program consists of a series of ever larger data challenges
carried out across larger scale data grids. Several goals of these data
challenges include validating the world-wide distributed computing
model, establishing the means to carry out data analysis on the grid to
enable scientists to be effective and productive members of the
collaboration at their home institutions, and also to carry out the
simulations and data analysis for algorithm development and determining
the sensitivity of the detector for the measurements to be performed.
The PDSF facility is one of the sites participating in the grid-based
data challenges for ATLAS.

The ALICE experiment, also located at the CERN LHC, is designed to
measure collisions of lead beams and so has similar physics to the STAR
experiment, but at higher energies. The U.S. effort on ALICE is quite
small at the moment but a proposal to DOE in 2004 is expected to
establish a program of participation in the US at approximately 10% the
size of the STAR experiment (50 scientists, compared to 500 in STAR).
The PDSF facility is being used by ALICE now in their data challenges
and is planned at the main center in the U.S. supported by DOE while OSC
at Ohio State University is pursuing support from NSF and the state of
Ohio. The distributed computing framework being used by ALICE (called
AliEn for Alice Environment) is planned as a prototype framework for all
the LHC experiments.

### Power grid simulations on DOE Science Grid 

The reliable and relatively inexpensive electricity supplied by today's
power grid is a foundation of our nation's prosperity. However, to keep
costs low, the nation has deferred infrastructure investment -
especially in transmission - by increasing competition and
inter-regional power transfers to squeeze more and more through the
existing system. The latest indicators of the consequences of forcing
the power grid to do more than it was designed to do include the August
14, 2003 blackout on the East Coast, the slow recovery from Hurricane
Isabel, and the extensive Western U.S. blackout in 1996. These events
demonstrate the vulnerability of the power grid to disruption. The
Office of Electric Transmission and Distribution (OETD) was created to
conduct the research and development needed to define and deploy
technologies that improve the reliability of today's power grid and to
define the power grid of the future.

**Application Requirements:** New processes that update corrective
action plans (both manual and automatic sequences) take as long as 30
minutes and are seldom used in practice because operators discount the
currency and accuracy of the models upon which they are based. This
collection of imperfect analyses tools taxes the control system
computers that, during critical moments, must process sensor data and
alarms at the expense of these processes.

**DOE Science Grid Impact:** Advanced computational hardware and
algorithms are a key mid-term objective to support fast power grid
simulation, a critical need in today's system, which is operated with
tools that cannot provide the real-time information needed. OETD has
identified improving the speed and accuracy of these simulations as the
key to improving reliability and control of the power grid. PNNL staff
currently leads the planning for fast simulation research for OETD's
Transmission Reliability Program.

With existing compute resources, 20-30 Distribution stations have been
simulated, each with thousands of homes with 5 major appliances. PNNL
proposes to grid-enable their existing power grid simulation codes so
that they can use the resources of the DOE Science Grid. The resources
of the DOE Science Grid would enable simulations of thousands of
stations with longer simulation times and finer resolution, which will
provide vital insights into the dynamic nature of the load on our aging
power transmission infrastructure

# **Lowering** **the Bar to** **Users and Managers of the** **Grid** 

One of the fundamental goals of the DSG project is to lower the bar to
newcomers to the Grid environment. The Grid has been said to be "a
solution looking for a problem", and many scientists have the impression
that it doesn't provide advantages commensurate with the level of effort
needed to use this approach, even though they have problem well suited
to Grid computing. Although this view of the Grid may not be correct,
clearly more work is needed by the middleware developers to increase the
usability for a wider user community. If there were mechanisms in place
to ease transition to Grid usage, more scientists would take advantage
of Grid technologies.

Scientists are not the only newcomers who need assistance in moving to
the Grid environment. System administrators also face new challenges
(and benefits) from Grid computing. Current installation and management
of Grid middleware can be challenging and error prone, especially as it
scales to larger installations. Many system administrators need help
from the Grid community in becoming better versed in Grid technologies
and while simple education is the first step, additional work is needed.
We see the need to create better tools to allow software installs,
certificate management, etc. in order to scale to the sizes of the major
DOE compute facilities, as well as additional tools to allow
production-level stability.

One of the areas that DSG has offered significant advantage to the DOE
community in the course of the project to date is in defining best
practice and standards for deployment and use of Grid technologies. This
is evident in the widespread use of the DOE-Grids CA and firewall
interactions. DSG will continue this approach, and to expand the best
current practice information to include documentation on monitoring,
large file transfers, and additional security concerns.

We plan to address the usability concerns for new scientists and system
administrators by concentrating on the following areas:

## 

## 

## 

## 

## Best Practices Recommendations and Documentation

DSg will continue our work in defining best practice recommendations and
publishing documentation to assist the broader Grid community. Examples
of previous work in this area are the DOE Grids CA and the Firewall
document that describes the interaction between all major Globus Toolkit
components and firewalls XX ref XX. This document has been used in
numerous Grid projects, including DOE SciDAC National Fusion
Collaboratory, the DOE SciDAC PPDG Collaboratory, the NSF TeraGrid
project, and others. We have identified the following as documents
needed to aid the community. Others are likely to become evident as our
work progresses.

1.  Monitoring Data Guidelines

> We will document the monitoring information that sites should publish
> to be considered part of the DOE Science Grid. This will include basic
> resource data as well as service status data needed by current
> application scientists. This work will be done in conjunction with the
> application community to identify further sources of data specific to
> their needs.

2.  

3.  Configuration recommendations

While the Globus project has invested a lot of effort in packaging
tools, there are still significant questions that administrators have to
answer then installing Globus. In some cases, the default configuration
may had unintended side effects that should be documented.

4.  Advice to set GridFTP transfer parameters\
    In order to achieve good performance with large file transfers,
    several parameters must be set. This document will describe those
    parameters, and how to evaluate them for your Grid, and we will also
    include wrappers to GridFTP that automatically set these for the DSG
    sites. We also plan to include information about other DOE lab
    software (ref Web100, matthis, Tierney TCP tuning doc work) for TCP
    tuning information.

5.  Performance Tuning

In addition, as more expertise is developed in other Grid services, we
intend to publish information about tuning them as well.

6.  Firewall interactions\
    While the current documentation address firewall interactions with
    the pre-OGSI Globus Toolkit, other Grid services must also interact
    with these security devices. The first software package we plan to
    address is the most recent stable release of Globus at the time,
    either the OGSI-compatible pieces of the Globus Toolkit 3.x release
    or the WSRF-compatible pieces of the Globus Toolkit 4.x release. We
    will also describe best practices for those developers working on
    writing OGSI-compatible (or possibly WSRF-compatible) services in
    how their work can most easily interact with firewalls in their own
    Grids.

7.  DOE Grid certificate authority\
    We will continue documenting and refining the documentation for end
    users and system administrators for the acquisition and management
    of certificates and Grid security infrastructure. This will include
    more detailed sections on signing policies and Certificate
    Revocation Lists (CRLs), among other topics of discussion.

8.  Troubleshooting

A broad base of organizational knowledge in dealing with problems is the
hallmark of a mature production service. DSG can jump start this process
by publishing information about common troubleshooting procedures.

## 

## 

## 

## Virtual Organization management

In the U.S., the DOE is the primary supporter of large scale science,
which means large numbers of scientist must come together in order to
achieve the goals of these big science projects. As in any large complex
organization, a large science collaborations have many activities, each
one fitting into an organizational structure with people assigned
various roles and scope of responsibilities. The amount of resources
available or assigned to the different activities is decided by some
management procedures, and the time dependent effects of various goals
and milestones are taken into account.

For scientific collaborations the main external factors leading to time
dependence are conferences, data taking run periods of experiments, and
funding cycles. The computing activities of a collaboration have these
features as well and the details of roles, responsibilities and
priorities are reflected in the authorization, accounting and
allocations mechanisms of the computing facilities being used. Each site
supporting the computing activities of the collaboration has some
mechanisms developed to handle these, but the methods are typically ad
hoc and different from one site to another.

As these collaborations move their computing activities into the Grid
environment there must be standard means to express and enforce these
authorization, accounting and allocation features. This mechanism to
describe and maintain the structure of the collaboration independent of
any single computing site is called a Virtual Organization Management
System (VOMS). The Akenti authorization service and the Globus Community
Authorization Service (CAS) are two examples of services that partially
fulfill the desired features of a VOMS. An effort called VOMS from the
EDG is another that is getting some testing and use in the experimental
HENP community. It has no authorization service in it but provides
attribute information about groups and roles that can be used by an
authorization service as part of enforcing authorization.

The DOE Science Grid will investigate the CAS, Akenti and VOMS plans and
collaborate with those groups to provide an appropriate VOMS service for
the DOE Science Grid sites and applications. The DOE Science Grid
project will work with the VOs in it' user community to develop and
support an authorization infrastructure that interfaces to and meets the
needs of the VOs' management services. This includes reviewing the
Virtual Organization requirements in order to determine an appropriate
solution that supports multiple VOs. For example, the PPDG user
community at NERSC/PDSF is working with the VOMS server and the DOE
Science Grid will collaborate with the PPDG effort and determine an
appropriate authorization service at PDSF that will correctly interpret
the extended X.509 proxies generated by VOMS.

## Outreach and training

A needed component to lower the bar for Grid use in the DOE is outreach
and training to the various Grid users, not only the scientists, but
also the system administrators, help desk staffers, and software
developers. Given the limited resources of the DOS-SG project, we plan
to focus outreach and training activities (outside of best practice
recommendations) on the need of the application scientist community, as
described above. When possible, we plan to record training sessions
offered over the AccessGrid for further use by the sites.

We plan to continue to offer additional tutorials out of the scope of
this project, for example at meetings such as GlobusWorld,
Supercomputing and HPDC, as we have done in the past, and to advertise
the availability of this training broadly.

## A Central Resource for Discovering DSG Services & Policies

In order to promote the use of the Grid by applications, a centralized
resource that explains the services available, service level
descriptions and policies regarding their use should be created and kept
up to date. In addition, to facilitate account creation, this site
should be a collection point for documents and forms needed by users to
apply for accounts and allocations for all of the participating sites.

This site will also contain dynamic status information such as current
service availability, software package names and versions, scheduled
downtimes as well as contact information for Grid related work at each
site.

## 

## 

## 

## 

##  {#section-250 .unnumbered}

##  {#section-251 .unnumbered}

##  {#section-252 .unnumbered}

##  {#section-253 .unnumbered}

## DOE Regulations

-   

Engagement with applications communities as well as mainstream site
security staff, makes it clear that there are points of friction between
the various visions of the Grid and the existing body of DOE and site
specific policies and business practices. Bringing these two into
alignment will require significant effort, including major effort from
the science community to articulate their actual needs and bringing it
to the attention of DOE headquarters. The DOE Science Grid will play a
role in facilitating this interaction, but any major changes will be
truly credible only if articulated and championed by the scientific
community.

###  {#section-254 .unnumbered}

## 

## 

## 

## 

## 

### 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## 

## Transitioning the DOE Science Grid to OGSA

Recent work in the Grid community has been focused on the integration of
emerging Web Services technologies with Grid technologies. This has led
to the development of the Open Grid Services Architecture (OGSA) and the
Open Grid Services Infrastructure (OGSI). OGSA provides a Grid
architecture based on Web Services technologies, and OGSI provides the
base infrastructure for OGSA. At the core of OGSI is the "Grid Service
Specification" which extends the Web Service base and defines the
standard interfaces and behaviors of a Grid service. Current work is
ongoing in the Global Grid Forum (GGF) to standardize the Grid Service
Specification, and other OGSA/OGSI related specifications.

In the longer term OGSA/OGSI will provide the basic infrastructure for
building production Grid environments, but currently does not always
offer the full functionality or performance of the pre-OGSI Globus
Toolkit implementations. In addition to technology maturity issues,
upgrading core Grid middleware on a production Grid is a difficult
process. This is exacerbated by the lack of backwards compatibility
between OGSI and pre-OGSI Globus software. Thus we must be very careful
in designing a transition strategy so as to minimize the inconvenience
to our users.

One major aspect of this transition strategy involves the usage of our
Grid testbed machines. As described elsewhere in the proposal, an
important part of running a production Grid is having a series of
testbed machines that can be used for software validation, testing, and
development before being used on the production resources. We plan to
use these testbed machines to deploy a full set OGSI based services for
testing purposes before rolling out OGSI services onto the production
Grid resources.

Another aspect of our transition strategy is to use a phased approach to
deploying OGSI based services on the DOE Science Grid. We will begin by
focusing on services that do not exist in the pre-OGSI Grid middleware.
In particular, we plan on researching and deploying a monitoring system
based on OGSI. OGSI provides a framework that should allow us to build
an end-to-end monitoring system that can be used for troubleshooting,
fault detection, and performance analysis. Researchers at ANL and LBNL,
in consultation with both the DSG community and the wider Grid
community, will be responsible for developing this system. After
deployment and testing on the testbed machines, the monitoring system
will be integrated into the current DOE Science Grid production
middleware.

A third important aspect of our transition plan involves running both
OGSI and pre-OGSI services where appropriate. Given the complete shift
in protocols it is relatively easy to support both OGSI and pre-OGSI
versions of major services, e.g., resource management (GRAM), data
management (GridFTP, RLS, etc.), and information services (MDS). The
ability to run both will be essential to ensuring a smooth transition
for our user base. We foresee the need to support pre-OGSI services for
a minimum of 6 months to a year after we have fully rolled out OGSI
services. It will take Grid users and developers at least that long to
port their applications to using OGSI services.

## Integration with a Potential Grid Operations Center

###  {#section-346 .unnumbered}

### 

### 

### 

### 

### 

### 

### 

### 

### During the review process, and in discussions with other Grids such as TeraGrid and IVDGL, the notion of a DOE Science *Grid Operations Center* was advanced. As part of the development of a persistent Grid, we are assuming that a separately funding Grid Operations Center will be developed in parallel and that at the close of this round of funding, the products and expertise of the DOE Science Grid will be integrated into this "GOC". As a consequence we expect to collaborate with the nascent GOC over the course of the next 2 years, to ensure that the DOE Science Grid enjoys full integration and support for this facility.

The DOE SG is collaborating with the DOEGrids CA effort in ESnet as will
as the iVDGL grid operations effort (iGOC) at Indiana University. This
collaboration is developing a model of distributed grid operations
center services. The DOEGrids CA operations are one example with the
distributed registration authorities. We are identifying additional
services, such as improved identity management that can support incident
response procedures, Virtual Organization management services and
grid-level monitoring that should become part of a distributed grid
operations center. In collaboration with DOEGrids and the iGOC we will
determine appropriate modes of operation for these services to achieve
necessary levels of reliability and availability, including replication
where necessary.

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

## 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

## 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

# 

### 

### 

### 

### 

### 

### 

### 

### 

###  {#section-450 .unnumbered}

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

## 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

## 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### 

### **Relation to PPDG, iVDGL, OSG, LCG**

The grid vision shared across much of the high-energy and nuclear
physics community is that during the next two or three years of "grid
project" effort (PPDG, iVDGL, DSG) we need to make the transition from
grid as primarily a development effort to the point that real persistent
grids are in production and available as a functional resource for
scientists. This real persistent grid is being promoted in the HENP
community (and beyond) via the Open Science Grid (OSG) roadmap
([www.opensciencegrid.org](http://www.opensciencegrid.org/)). One aspect
of the strategy to bring this vision to reality is to establish a
persistent shared grid infrastructure with use by real applications. The
PPDG is adjusting its focus to establish a shared grid infrastructure
across five DOE labs (BNL, FNAL, JLab, NERSC/PDSF, SLAC) to be shared by
the PPDG experiments. iVDGL coordinated the Grid2003 demonstration grid
of 27 sites including NERSC/PDSF with 10 applications participating. The
Grid2003 shared infrastructure is continuing (perhaps called Grid2004,
Grid2005, ...) and several groups are continuously running their
production applications on this infrastructure. This experience over the
past two years has demonstrated that the most effective means to achieve
a shareable grid infrastructure is to simply start sharing distributed
resources by several applications in an iterative fashion with
increasing scope and these iterations drive the detailed choices for
technologies and configurations. The most effective way for the DOE
Science Grid project to interact with these other grid projects is to
take advantage of the points of overlap at the DSG facilities in order
to engage in the complete range of issues, discussions and choices from
the fabric layer up through the middleware to the interface to
application software. In this way, DSG can play the necessary role of
achieving a greater degree of standardization in grid deployment, which
will also help in making a grid more supportable.

3.  -   
    -   
    -   
    -   
    -   
    -   
    -   

    ```{=html}
    <!-- -->
    ```
    -   

    ```{=html}
    <!-- -->
    ```
    -   

    ```{=html}
    <!-- -->
    ```
    -   
    -   
    -   
    -   
    -   
    -   
    -   

    ```{=html}
    <!-- -->
    ```
    -   
    -   

    ```{=html}
    <!-- -->
    ```
    -   
    -   

-   

-   

-   

-   -   
    -   

-   

-   

-   -   
    -   

4.  **Management plan**

The management plan for the Phase 2 is an extension of the management
plan for Phase1. DSG will continue the very successful work of the DOE
Science Grid (DSG) Engineering Working Group, with systems and research
staff from each of the sites. This weekly meeting is used to coordinate
progress and address issues across the sites.

In order to insure the DSG is meeting the needs of application partners,
we propose to expand the Co-PI Management Council into a semi-annual
meeting of all the participants in the DOE Science Grid -- A Grid User
Forum. This meeting will be modeled after the NERSC User Group and ESnet
Coordinating Committee. It will include all the DSG PIs and workers,
representatives for the application teams, representatives of the sites
involved with the DSG and other interested people, such as
representatives of other Grid efforts. This meeting will be designed to
discuss the needs of the Grid users, the plans of the DSG and the sites
and other issues. Special work groups will be occasionally identified to
work on specific issues.

There are several staff changes coming with Phase 2. Bill Johnston has
taken over the management of the ESnet Program. His focus on ESnet means
that the Principle Investigator will be Bill Kramer, and the LBNL
Co-Investigator will be Keith R Jackson. Further, Ian Foster's
involvement in many project involved in a number of Grid projects,
including the OGSA effort, limits his time for involvement so, Jennifer
Schopf will be Argonne Co-Investigators. Since Ray Bair moved from PNNL
to ANL, Scott Studham will serve as Co-Investigator from PNNL. Al Geist
remains the Co-Investigator at ORNL. All this changes will have a
positive effect on the second phase of the project. Furthermore, because
of the close working relationship between DSG and ESnet, and between DSG
and the OGSA effort, Bill Johnston and Ian Foster will continue to serve
as advisors to the DSG PI Management Council.

The DSG Phase 2 has the goal of creating the first version of a
Persistent Science Grid, and preparing the necessary information and
best practices so that the operation and enhancement of the PSG can be
handled by some sustaining organization. Furthermore, there needs to be
more effort to bring large-scale science applications to the PSG. Our
metrics of success include:

-   Create the DOE Persistent Science Grid on production resources at
    the participating sites.

-   Enable targeted large-scale science to use the PSG. The science
    applications will be strategic applications in disciplines important
    to the Office of Science

-   Make it easier for scientists to use DSG/PSG

The end result of this effort is that by the end of 2006 the version 1
of the PSG should be ready to hand off to an on-going effort that will
support and probably continue enhancements. There will also be a track
record of improved science and a core set of applications using the DSG
that will be able to make full use of the PSG.

5.  Budget

The staffing level for Phase 2 if the same as in Phase 1, 7.5 FTEs. The
FTE counts at each lab will remain the same.

address any left over funds from phase 1 (per Mary Anne)

# Milestone (for long term strategy)

Define a DSG user forum and host an initial meeting of this forum to
talk about issues, community needs, and science applications on the
persistent DOE Science Grid.

Research and Development

-   Development, deployment and operation of a distributed monitoring
    system for Grid services

-   Development and deployment of production quality, scalable
    authorization systems

-   Development and deployment of extended network intrusion detection
    systems that are aware of all Globus 2.x and OGSA protocols

-   Specification and deployment of standards for configuration
    management across DSG sites

-   Define a set of documented issues raised by the DOE Grid user
    community that conflict with DOE regulations

Deploying production resources on the DSG

-   Documented problem escalation procedures

-   Development and deployment of a cross site trouble ticket system

-   Site account generation system includes handling of Grid
    certificates

-   Integration of the Grid authorization system into site account
    management processes

-   Grid administration training material for mainstream systems
    administrators

-   Grid services are managed, monitored and supported via normal site
    processes

-   Grid services are integrated with production batch schedulers and
    storage systems

```{=html}
<!-- -->
```
-   Define a set of standard interface definitions for exporting site
    accounting information to a distributed accounting system

Fostering DOE science applications over the Grid

-   Genomes to Life

    -   Production PNNL and ORNL databases with GTL-developed schemas
        are Grid-accessible

    -   Biology Grid Application service is deployed on DOE SG

    -   DOE SG computing resources are used in production Rosetta, GIST
        and GADU analyses

-   Climate

    -   TBD

-   Astrophysics

    -   Seamless integration of DSG resources into the NVO

    -   Development of a prototype PlanckGrid, coupling resources at
        NERSC and NCSA

-   Other

    -   With STAR, ATLAS and ALICE, establish the means to carry out
        data analysis and simulations on the Grid

    -   Grid-enable OETD power grid simulation codes

A central resource for discovering DSG services and policies

-   Web site that contains status information, notices, service
    descriptions, policy documents, and forms needed by users

-   Documenting experiences for information sharing

-   Publication of a series of documents describing recommended on DSG
    configuration, tuning and troubleshooting

Transitioning to OGSA

-   All production Grid services are OGSA compliant

Integration with the future Grid Operations Center

-   Trouble ticketing, monitoring, escalation procedures are handed off
    to DOE Science Grid Operations Center

-   Demonstrate methods of replicating high-availability services

Lower the bar to newcomers to Grid

-   Define and implement a common software stack for DSG resource
    providers, with managed versioning

-   Produce best practices documents covering optimal GridFTP
    parameters, Firewall interactions (updated for OGSI), and a Users
    and Administrators guide to DOE Grid CA

-   Deploy tools that integrate certificate generation with standard
    account generation practices at NERSC

-   Develop and deploy a VOMS service to DSG users

-   Deliver Access Grid tutorials on use of DSG to DOE applications
    scientists

```{=html}
<!-- -->
```
-   Long term strategy

    -   Define a Science Grid user forum and host an initial meeting of
        this forum

    -   Hold a workshop to define a desired level of interoperability
        between grids

6.  Facilities Descriptions

    The following facilities across multiple labs will be integrated
    into the Production DOE Science Grid.

Compute Systems to be integrated into Cross Site Production Grid

  ---------- -------------- ---------------- ------------- ------------ ----------------
  **Site**   **System       **Processors**   **Operating   **Memory**   **Theoretical
             Name**                          System**                   Performance
                                                                        (Teraflops)**

  NERSC      Seaborg        6555             AIX           170TB        9.8 Tflops

  NERSC      PDSF           412              Linux         412GB        .75 Tflops

  NERSC      Newton         8                Linux         8GB          

  NERSC      Escher         12               IRIX          24GB         

                                                                        
  ---------- -------------- ---------------- ------------- ------------ ----------------

***Data Storage*** Systems to be integrated into Cross Site Production
Grid

+------+------------+---------+-----------+-----------+-------------+
| **Si | **System   | **Files | **        | **Onli    | **Offline   |
| te** | Name**     | ystem** | Operating | neStorage | Storage**   |
|      |            |         | System**  | C         |             |
|      |            |         |           | apacity** | *           |
|      |            |         |           |           | *Capacity** |
+------+------------+---------+-----------+-----------+-------------+
| N    | Seaborg    | GPFS    | AIX       | 44TB      |             |
| ERSC |            |         |           |           |             |
+------+------------+---------+-----------+-----------+-------------+
| N    | PDSF       | NFS     | Linux     | 55TB      | .           |
| ERSC |            |         |           |           |             |
+------+------------+---------+-----------+-----------+-------------+
| N    | HPSS       | HPSS    | AIX       | 35TB      | 10          |
| ERSC |            |         |           |           | Petabytes   |
+------+------------+---------+-----------+-----------+-------------+
|      |            |         |           |           |             |
+------+------------+---------+-----------+-----------+-------------+
|      |            |         |           |           |             |
+------+------------+---------+-----------+-----------+-------------+

7.  

[^1]: http://www.ccsm.ucar.edu