# Presentation: 160417

## Prototyping anEarth System GridA Department of Energy -- Office of Science Next Generation InternetApplication, Network Technology, and Network Testbed Partnership

- Wu Feng ...................................................... Los Alamos National Laboratory
- Ian Foster ......................................................... Argonne National Laboratory
- _Steve Hammond_ (PI) .........   National Center for Atmospheric Research
- Bill Hibbard ............................................................... University of Wisconsin
- Carl Kesselman .................................  USC Information Sciences Institute
- Arie Shoshani, Brian Tierney ...... Lawrence Berkeley National Laboratory
- _Dean Williams_ (PI) ..................  Lawrence Livermore National Laboratory

## Overview

- Background and motivation
- Earth System Grid approach
- Project milestones
- Highlights of Progress to date
- SC99 demo
- Other issues

## Context: The Accelerated Climate Prediction Initiative (SSI)

- Model development and evaluation
- Atmosphere/ocean/land surface/sea ice components, coupled models, process parameterizations, simulation design, algorithm development
- Climate projections (PBs of data)
- Optimization of computer performance, multi-decadal & multi-century ensemble simulations, narrow uncertainty in predictions, archives of petabyte datasets
- Impact assessment, analysis (1000s users)
- Policy evaluation, impacts research, regional/national/international assessments, specialized tools and datasets, information dissemination

## Why Climate ChangePresents a Networking Problem

- Data sources: high data rates
- Multiple, high-resolution, multi-component runs: 100 MB/s (@5 TF/s) => 3 PB/yr
- Data analyses: sophisticated, intensive
- Intercomparisons with observed data & other models
- Regional impacts studies => downscaling
- User community: large and diverse
- O(100+) laboratories, centers, universities
- Highly multidisciplinary

**Notes:**
EOS: 3 MB/s -> 15 MB/s derived; 7 PB total

## Requirements: Summary

-  “Enable a geographically distributed community [of thousands] to perform sophisticated, computationally intensive analyses on Petabytes of data”
-   *Virtual Proximity - *work in a distributed environment as easily as if everything was located on your laptop supercomputer.

## Approach: Earth System Grid

- Integrate data archives into a distributed data management and analysis “Grid”
- More than storage & network, also e.g.
- Caching and mirroring to exploit locality
- Intelligent scheduling to determine appropriate replica, site for (re)computation, etc.
- Coordinated resource management for performance guarantees, QoS.
- Embedded security, policy, agent technologies for effective distributed analysis

## Illustrative Scenario

- “How do midwest flood
-  frequencies under 2xCO2
-  scenario compare with
-  historical data?”

## Approach

- Construct _Grid-enabled analysis clients_
- Develop _intelligent middleware services_
- Distributed cache management service
- High-performance data transfer service
- Remote execution service
- Leverage and enhance existing Grid services providing resource management, security, instrumentation

## Earth System Grid Architecture

## Building Blocks Include

- PCMDI software system: data analysis
- Globus toolkit: Grid middleware, including security, resource management, etc.
- DPSS: high-speed network cache
- STACS storage access coordination system
- High-speed data transfer libraries
- Large databases of climate model output and observational data

## Project Goals: Year 1

- Prototype manually operated Grid; use for transfers from LANL to other sites
- Prototype high-speed data transfer library
- Characterize DOE-NCAR connections
- Create online climate data database
- Address security issues: firewalls, smartcards
- Quasi-real-time distribution of output from climate simulations to multiple centers
- Prototype analysis client access to remote data

## Project Goals: Year 2

- Start automation of Grid operations
- Remote execution service
- Locate data replicas via Metadata Catalog
- Single-sign-on capability; SAN/WAN gateway
- Prototype tape scheduling service.
- Address TCP 16-bit checksum problem, if real
- Diffserv support for high-speed data transfers
- Deliver products to clients from dynamically selected remote datasets, computers
- Prototype use of thin clients (VisAD & Vtk)

## Project Goals: Year 3

- Demonstrate full scale Earth System Grid
- Support real-time distribution of larger datasets, more grid-enabled analysis clients
- Distributed cache management service
- Visualization on thin clients (VisAD & Vtk)
- Automatic migration from tape to disk, automatic distributed cache management.
- Introduction of collaboration technologies.
- Broker that selects data and computers according to needs and availability

## Current View from the Trenches

- Single FTP from LANL to NCAR, sustains 200-400 KB/s.
- Manual transfer of multiple, concurrent FTPs can do nearly 4 MB/s - very tedious.
- Need to sustain *O*(10)MB/s to keep pace with compute rate on current machines.
- Issues:
- fire walls
- multiple networks
- OS, server load, MTU size and packet loss

## Earth System Grid Progress- Bulk data xfer LANL to NCAR

- _What’s Done_
- Data server in the green, i.e., “open network”
- Beta version of optimized FTP
- Globus v1.1 beta 16
- Compiled and running at LANL
- CryptoCard/SecureID certificate authority (CA)

- _To Do_
- Network testing and tuning to/from data server in the green, i.e., optimized ftp between firewalls.
- System Integration
- CA / Gatekeeper
- Gatekeeper / LSF
- Globus ssh
- Installation into Production
- Generalization of Overall System
- LDAP, LDAP, LDAP!

## Earth System Grid Progress 

- Prototype high-speed data transfer service
- Specification for a “Grid-Storage API”, and three prototype implementations
- local file I/O
- remote FTP and HTTP servers
- DPSS server (work at LBNL)
- Prototype Replica Catalog
- An LDAP-based replica catalog & a set of operations for:
- listing the replicas associated with a logical file
- examining the metadata asssociated with a logical file
- creating new replicas (vis Grid Storage API operations)
- deleting replicas

## Earth System Grid Progress

- Prototype Data Browser Gui
- Significant progress in developing a Java-based Windows-like browser for browsing the contents of a logical file catalog, examining the replicas associated with a file, requesting the creation of new replicas, and viewing the progress of transfers.
- Ordered a Data Grid Node
- Ordered 20-node linux cluster, 4TB disk to run DPSS and Globus security mechanism, ETA early October.
- High Speed data transfer experiments
- ANL-LBNL: very good
- ANL-LLNL: much worse

## Earth System Grid Progress

- More API definitions ...
- Near completion of an API between PCMDI  components  that use the metadatabase and the “query execution component” in STACS. This includes the function of query planning, request and monitoring of file transfers, as well as management of local disk cache.  File transfers will be done through the Globus Storage Client API.

## PCMDI data cache to serve the diagnostic community

**PERFORMANCE/CAPACITY:**
**Maximum I/O rate (burst I/O per sec.) 30,800 **
**Maximum I/O rate (sustained I/O per sec.) 12,000 **
**Maximum transfer rate (MB/sec.) 154.36**
**Number of drives 40 - 100**
**50 GB drives 7,200 rpm drives 3.0 TB - 5.0**

**MetaStor Scalable Server-Attached Storage **

- 4 processor Sun Enterprise 450

**RAID**

**Fiberchannel**

**ESnet**

**ENSO**

## Earth System Grid Progress

- OpenLDAP server is up and running at PCMDI
- OpenLDAP schema for CDMS (based on GDT 1.3) is done.
- OpenLDAP/CDMS database is up and running at PCMDI
-  Integrated Metadata Catalog database
-  OpenLDAP/CDMS Python API

## Earth System Grid Progress

- Vtk Python interface tested
- OpenLDAP/CDMS data be passed to Vtk Python Modules (can generate nice 3D images)
- Python Netlogger interface is now available
- Coupled climate model (PCM) data in netCDF format is now being cataloged and moved over to HPSS at NERSC about 170 GB.

## Supercomputing ’99 Demo

**Supercomputing ’99 Demo**

**RAID**

**Site Mgr **
**LLNL**

**Network**

**Site Mgr **
**SC99**

**Site Mgr **
**LBNL**

**Site Mgr **
**ANL**

**Local disks**

## Distributed Databases - Inquire & Ingest Data

*Client-Side*
*(application)*

*Server-Side*
*(remote-site)*

*NCAR*

*(Network)*

*LLNL*

*LBNL*

*ANL*

## Other Issues

- Technologies we will produce which may be useful to others:
- Data Grid Infrastructure
- High-speed data transfer libraries
- replica management libraries
- optimized ftp for WAN bulk data transfers
- visualization on light weight clients
- Requirements for milddleware services
- Authentication infrastructure
- Resource mgmt: computers, network
- Directory services
- Instrumentation services

## Other Issues, cont

- Requirements for testbeds
- Support for high bandwidth management
- high bandwidth connections between core sites - NOT just NTON connected sites, but also to ANL and NCAR
- Thoughts on maximizing NGI impact/relevance
- emphasize identification of requirements for shared “middleware” and other services
- Use this opportunity to drive creation of persistent infrastructure across DOE.

## DOE’s NGI Program: BuildingAn Integrated Grid Architecture

- Archives, networks, computers, display devices, etc.;
- associated local services

- Protocols, authentication, policy, resource
- management, instrumentation, discovery, etc., etc.

- Grid
- Fabric

- Grid
- Services

- Appln
- Toolkits

- Applns

**...**

- ... a rich variety of applications ...

- Remote
- viz
- toolkit

- Remote
- comp.
- toolkit

- Remote
- data
- toolkit

- Remote
- sensors
- toolkit

- Async.
- collab.
- toolkit

## The Earth System Grid andDOE’s NGI Program

- viz
- toolkit

- Remote
- comp.
- toolkit

- The Earth System Grid andDOE’s NGI Program

- Archives, networks, computers, display devices, etc.;
- associated local services

- Protocols, authentication, policy, resource
- management, instrumentation, discovery, etc., etc.

- Remote
- data
- toolkit

- Grid
- Fabric

- Grid
- Services

- Appln
- Toolkits

- Applns

**...**

- Remote
- sensors
- toolkit

- Async.
- collab.
- toolkit

- ... rich variety of applications ...

- Grid Services Package:
- Authentication, resource
- management, information,
- instrumentation, fault
- detection, data access,
- other services as and
- when available

## Summary

- We can develop a prototype, but an operational infrastructure to provide scientists with seamless virtual proximity to geographically distributed models, computers, data, and analysis will only be possible with high performance networks.
- In addition, we need the middleware and applications capable of effectively running over high bandwidth, distributed resources.