# Presentation: 334761

## Extensible Terascale Facility Workshop

- Friday, September 20, 2002Arlington, VA**(including participation from****22+ Access Grid Nodes)**

## Overview

## Inventing the Future:The NSF TeraGrid

- Ian Foster
- Associate Director, Mathematics and Computer Science Division, Argonne National Lab
- Professor of Computer Science, University of Chicago
- foster@mcs.anl.gov
_**Collaborators**_
**Rick Stevens (ANL), Fran Berman (SDSC), Charlie Catlett (ANL), Mike Levine (PSC),**
**Paul Messina (Caltech), Dan Reed (NCSA), Ralph Roskies (PSC)**

## Overview

- The need for distributed science infrastructure
- Application drivers
- Technology requirements
- The NSF TeraGrid
- Physical structure
- Grid software and services model
- TeraGrid and the national community
- Resource consumers: application scientists
- Resource providers: new sites
- TeraGrid sites and network
- NCSA, SDSC, Caltech, ANL, PSC; TeraGrid backplane

## Overview

- The need for distributed science infrastructure
- Application drivers
- Technology requirements
- The NSF TeraGrid
- Physical structure
- Grid software and services model
- TeraGrid and the national community
- Resource consumers: application scientists
- Resource providers: new sites
- TeraGrid sites and network
- NCSA, SDSC, Caltech, ANL, PSC; TeraGrid backplane

## New Modes of Science

- Technology drivers: exponential growth in
- Sensor & data storage capabilities
- Unprecedented instrumentation & science data archives
- Computational capabilities
- Emergence of simulation science as a first-class citizen
- Network capacity
- Transition from obstacle to opportunity
- Discipline drivers:more complex problems
- Larger and more multidisciplinary approaches & teams
- Need for distributed science infrastructure
- Enable access to, and integration of, remote resources
- Enable (inter)national-scale communities

## New Modes of Science

**LIGO**

**ATLAS and CMS**

**NVO and ALMA**

***The number of nation-scale (or larger) projects is growing rapidly!***

**Climate Change**

## Required Science Infrastructure

- Resources
- Computing, storage, data

## Required Science Infrastructure

- Resources
- Computing, storage, data

- Services
- Authentication, discovery, ...

## Required Science Infrastructure

- Resources
- Computing, storage, data

- Services
- Authentication, discovery, ...

## Required Science Infrastructure

- Resources
- Computing, storage, data

- Services
- Authentication, discovery, ...

- Connectivity
- Reduce tyranny of distance

## Required Science Infrastructure

- Resources
- Computing, storage, data

- Services
- Authentication, discovery, ...

- Connectivity
- Reduce tyranny of distance

- Technologies
- Build services & applications

## Required Science Infrastructure

- Resources
- Computing, storage, data

- Services
- Authentication, discovery, ...

- Connectivity
- Reduce tyranny of distance

- Technologies
- Build services & applications

- A

- A

- A

## Required Science Infrastructure

- Resources
- Computing, storage, data

- Services
- Authentication, discovery, ...

- Connectivity
- Reduce tyranny of distance

- Technologies
- Build services & applications

- A

- A

- A

- Communities
- Operational procedures, ...

## Overview

- The need for distributed science infrastructure
- Application drivers
- Technology requirements
- The NSF TeraGrid
- Physical structure
- Grid software and services model
- TeraGrid and the national community
- Resource consumers: application scientists
- Resource providers: new sites
- TeraGrid sites and network
- NCSA, SDSC, Caltech, ANL, PSC; TeraGrid backplane

## TeraGrid Objectives

-  Create significant enhancement in capability
- Beyond capacity, provide basis for exploring new application capabilities
-  Deploy a balanced, distributed system
- Not a “distributed computer” but rather
- A distributed “system” using Grid technologies
- Computing and data management
- Visualization and scientific application analysis
-  Define an open and extensible infrastructure
- An “enabling cyberinfrastructure” for scientific research
- Extensible beyond the original four sites

## Distributed Terascale Facility (DTF) TeraGrid

**ANL:** Visualization

**Caltech:** Data collection analysis

**SDSC**: Data Intensive

**IA64**

**IA32**

**IA64**

**4 TF IA-64**
**220 TB Storage**
**Sun Server**

**Visualization nodes**
**1 TF IA-64**
**20 TB Storage**

**0.4 TF IA-64**
**IA-32 Datawulf**
**80 TB Storage**

**LA **
**Switch/Router**

**Chicago**
**Switch/Router**

**IA32**

- Storage Server

- Disk Storage

- Cluster

- Visualization
- Cluster

_**LEGEND**_

**IA64**

**40 Gb/s**

**30 Gb/s**

**NCSA**: Compute Intensive

**IA64**

**6 TF IA-64**
**120 TB Storage**

**30 Gb/s**

**30 Gb/s**

**30 Gb/s**

- Focus on Homogeneity:
- Linux, Globus Toolkit, Itanium2

- TeraGrid Lite up now
- Network up Dec 02
- DTF production mid-03

## Proposed Extensible TeraGrid Facility

- PSC integrated Q3 03

## ETF TeraGrid Management Structure

_**Site Coordination Committee**_
- Site Leads

_**Project Director**_
- Rick Stevens (UC/ANL)

_**Technical Coordination Committee**_
- Project-wide Technical Area Leads

- _Clusters_
- Pennington
- (NCSA)

- _Networking_
- Winkler
- (ANL)

- _Grid Software_
- Kesselman (ISI)
- Butler (NCSA)

- _Data_
- Baru
- (SDSC)

- _Applications_
- WIlliams
- (Caltech)

- _Visualization_
- Papka
- (ANL)

- _Performance Eval_
- Brunett
- (Caltech)

_**Chief Architect**_
- Dan Reed (NCSA)

_**Executive Committee**_
- Fran Berman, SDSC (Chair)
- Ian Foster, UC/ANL
- Michael Levine, PSC
- Paul Messina, CIT
- Dan Reed, NCSA
- Ralph Roskies, PSC
- Rick Stevens, UC/ANL
- Charlie Catlett, ANL

- _Technical Working Group_
*Are we creating an extensible cyberinfrastructure?*

- _External Advisory Committee_
*Are we enabling new science?*
*Are we pioneering the future?*

- _Institutional Oversight Committee_
- TBD, UCSD
- Richard Herman, UIUC
- Mark Kamlet, CMU
- Dan Meiron, CIT (Chair)
- Robert Zimmer, UC/ANL

- _User Advisory Committee_
*Are we effectively supporting good science?*

- NSF MRE Projects

- Internet-2
- McRobbie

- Alliance UAC
- Sugar, Chair

- NPACI UAC
- Kupperman, Chair

_**NSF ACIR**_

- _NSF Technical Advisory Committee_

- Policy Oversight

- Objectives/Eval

- Architecture

**Currently being formed**

_**Executive Director /**_
_**Project Manager**_
- Charlie Catlett (UC/ANL)

_**ANL**_
- Evard

_**CIT**_
- Bartelt

_**NCSA**_
- Pennington

_**SDSC**_
- Andrews

_**PSC**_
- Scott

- _Operations_
- Sherwin
- (SDSC)

- _User Services_
- Wilkins-Diehr (SDSC)
- Towns (NCSA)

- Implementation

- _NSF Program Advisory Team_

- PSC UAG

## TeraGrid Application Targets

- Multiple classes of user support
- Each with differing implementation complexity
- Minimal change from current practice
- New models, software, and applications
- Usage exemplars
- “Traditional” supercomputing made simpler
- Remote access to data archives and computers
- Distributed data archive access and correlation
- Remote rendering and visualization
- Remote sensor and instrument coupling
- Partnerships with GriPhyN, iVDGL, PPDG, EU DataGrid, NEESgrid, others

## Building a Grid

- Resources implement _standard Grid interfaces and protocols_ to enable secure remote access
- Discovery, characterization, access, accounting, etc.
- Leverage GGF and NSF Middleware Initiative, etc.
- Resources also implement _standard runtime environments_ to enable application portability
- For various resource functions: compute, data, etc., etc.
- Exploit site homogeneities when possible
- Sites collaborate on _distributed operations_ to enable use of TeraGrid as a facility
- Problem determination, support

## Layered Grid Services

| Service Layer | Functionality | TeraGrid Implementation |
| --- | --- | --- |
| Advanced Grid Services | super schedulers, resource discovery services, repositories, etc. | SRB-G, MPICH-G2, distributed accounting, etc. |
| Core Grid Services | TeraGrid information service, advanced data movement, job scheduling, monitoring | Globus Toolkit GASS, MDS; Condor-G, NWS |
| Basic Grid Services | Authentication and access
Resource allocation/Mgnt
Data access/Mgmt
Resource Information Service, Accounting | Globus Toolkit GSI-SSH, GRAM, GridFTP, GRIS; Condor,  |

**Collective**

**Resource**

## TeraGrid Runtime Environment

- Globus
- Credential

- Site 2

- Certificate

- Authenticated
- interprocess
- communication

**CREDENTIAL**

- Mutual
- user-resource
- authentication

- Mappingtolocal ids

- Assignment of
- credentials to
- “user proxies”

- Single sign-on via “grid-id”

## Example Application Services

| Service | Objective |
| --- | --- |
| Basic Batch Runtime | Supports running static-linked binaries |
| High Throughput Runtime (Condor-G) | Supports running naturally distributed applications using Condor-G |
| Advanced Batch Runtime | Supports running dynamic-linked binaries |
| Scripted Batch Runtime | Supports scripting (including compile) |
| On-Demand / Interactive Runtime | Supports interactive applications |
| Large-Data | Supports very large data sets, data pre-staging, etc. |
| File-Based Archive | Supports GridFTP interface to data services |

## Example Application Service Definition

| Specification | Basic Batch Runtime |
| --- | --- |
| Objective | To support execution of compiled binaries |
| Assumptions / Policy | 1 user per node, dedicated during runtime
nodes accessible for login (minimal environment)
static linked binaries
attributes and values published in MDS (using XML schema currently being developed) |
| System Software | Query for values (e.g. OS release, kernel level) |
| Node Hardware | Query for values (e.g. IA64 nodes, n CPU/node) |
| Software and Libraries | MPICH-G (min version) |
| I/O | Local space: $TG_Local
Home space: $TG_Home
Global (shared) space: $TG_Global
Node communication (protocol(s), naming)
Note in/out streaming (to other notes within cluster, to other clusters, outside clusters)
GridFTP |
| System Commands & Utilities | Path $TG_UTIL
Min set (e.g. tar, gzip, gsi-scp, gsi-ssh, etc.) |

## TeraGrid Distributed Accounting

- Globus
- Credential

- Site 1

- Site 2

- to/from
- TeraGrid
- Acct Service

- Database API

**.**
**.**
**.**

- Local Acctg

- TG Acctg

- Local Acctg

- TG Acctg

- Standard Usage Record(normalized units and “exchange rates” are already used in PACI, but only for reporting)
- Protocol for authorization, account debit (similar to credit card transaction)

- Protocol and record format work is happening within GGF.  Expected completion by 1q2003

## Overview

- The need for distributed science infrastructure
- Application drivers
- Technology requirements
- The NSF TeraGrid
- Physical structure
- Grid software and services model
- TeraGrid and the national community
- Resource consumers: application scientists
- Resource providers: new sites
- TeraGrid sites and network
- NCSA, SDSC, Caltech, ANL, PSC; TeraGrid backplane

## Community Access to TeraGrid

- TeraGrid resources to be allocated via a modified NRAC process
- In addition, each site has some discretionary time
- Strong interest in nontraditional application modalities
- May include linkages to other Grid infrastructures, such as International Virtual Data Grid Laboratory (iVDGL)
- Application development can proceed via Grid interfaces, or via higher-level tools
- E.g., GriPhyN Virtual Data Toolkit, MPICH-G2 wide area MPI, ...

## Integrating New Resource: Principles

- TeraGrid is designed for extension
- Focus on integrating “resources” not “sites”
- Provide clear specifications for what a resource must do in order to participate
- Adding resources will require significant, but not unreasonable, effort
- Exploit homogeneity when possible
- Synergy: whole is more than sum of the parts
- New resources should add new capabilities, enable new communities, and/or enhance existing capabilities

## Integrating New Resource: Details

- Support key protocols, interfaces, and behaviors on the new resource(s)
- (To be) documented in TeraGrid specifications
- Supported by standard implementations: NMI, etc.
- Integrate with TeraGrid backplane
- Virtual machine room network, not traffic transit network
- Contribute to TeraGrid resource pool
- Involves TeraGrid service level agreements
- Participate in TeraGrid operational structure
- Site lead + POCs for Grid, net, resource, user services
- Engage in early application experiments to debug and refine infrastructure

## Standards  Cyberinfrastructure

- Standards  Cyberinfrastructure

- Runtime

- File-based Data Service

- Collection-based Data Service

- Visualization Services

- Interactive Collection-Analysis Service

- Interactive Development

- IA-64 Linux Clusters

- Alpha Clusters

- IA-32 Linux Clusters

- Visualization Services

_*Data/Information*_

_*Compute*_

_*Analysis*_

- Relational dBase Data Service

- Certificate Authority

- Certificate Authority

- Certificate Authority

- Grid Info Svces

**If** done _*openly*_ and well...
- other IA-64 cluster sites will adopt TeraGrid service specifications, increasing users’ leverage in writing to the specification
- others will adopt the framework for developing similar services on different architectures

- Grid Applications

## For More Information

- “The TeraGrid:   A Primer”
- Accessible at www.teragrid.org

## Overview

- The need for distributed science infrastructure
- Application drivers
- Technology requirements
- The NSF TeraGrid
- Physical structure
- Grid software and services model
- TeraGrid and the national community
- Resource consumers: application scientists
- Resource providers: new sites
- TeraGrid sites and network
- NCSA, SDSC, Caltech, ANL, PSC; TeraGrid backplane