# Presentation: 536001

## The NSF TeraGrid:A Pre-Production Update

**2****nd**** Large Scale Cluster Computing Workshop**
**FNAL**
** ****21 Oct 2002**
**Rémy Evard, evard@mcs.anl.gov**
**TeraGrid Site Lead**
**Argonne National Laboratory**

## A Pre-Production Introspective

**Overview of The TeraGrid**
- For more information:
- www.teragrid.org
- See particularly the “TeraGrid primer”.
- Funded by the National Science Foundation
- Participants:
- NCSA
- SDSC
- ANL
- Caltech
- PSC, starting in October 2002
**Grid Project Pondering**
- Issues encountered while trying to build a complex, production grid.

## Motivation for TeraGrid

**The Changing Face of Science**
- Technology Drivers
- Discipline Drivers
- Need for Distributed Infrastructure
**The NSF’s Cyberinfrastructure **
- “provide an integrated, high-end system of computing, data facilities, connectivity, software, services, and sensors that ...”
- “enables all scientists and engineers to work on advanced research problems that would not otherwise be solvable”
- Peter Freeman, NSF
**Thus the Terascale program**
**A key point for this workshop: **
- TeraGrid is meant to be an infrastructure supporting _many_ scientific disciplines and applications.

## Historical Context

**Terascale funding arrived in FY00**
**Three competitions so far:**
- FY00 – Terascale Computing System
- Funded PSC’s EV68 6TF Alpha Cluster
- FY01 – Distributed Terascale Facility (DTF)
- Initial TeraGrid Project
- FY02 – Extensible Terascale Facility  (ETF)
- Expansion of the TeraGrid
**An additional competition is now underway for community participation in ETF**

## Distributed Terascale Facility (DTF) TeraGrid

**ANL:** Visualization

**Caltech:** Data collection analysis

**SDSC**: Data Intensive

**IA64**

**IA32**

**IA64**

**4 TF IA-64**
**220 TB Storage**
**Sun Server**

**Visualization nodes**
**.1 TF IA-64**
**20 TB Storage**

**0.4 TF IA-64**
**IA-32 Datawulf**
**80 TB Storage**

**LA **
**Switch/Router**

**Chicago**
**Switch/Router**

**IA32**

- Storage Server

- Disk Storage

- Cluster

- Visualization
- Cluster

_**LEGEND**_

**Distributed Terascale Facility (DTF) TeraGrid**

**IA64**

**40 Gb/s**

**30 Gb/s**

**NCSA**: Compute Intensive

**IA64**

**6 TF IA-64**
**120 TB Storage**

**30 Gb/s**

**30 Gb/s**

**30 Gb/s**

- Focus on Homogeneity:
- Linux, Globus Toolkit, Itanium2

- TeraGrid Lite up now
- Network up Dec 02
- DTF production mid-03

## Extensible TeraGrid Facility

**NCSA**: Compute-Intensive

**SDSC**: Data-Intensive

**PSC**: Compute-Intensive

**IA64**

**IA64**

**Pwr4**

**EV68**

**IA32**

**EV7**

**IA64**

**Sun**

- 10 TF IA-64
- 128 large memory nodes
- 230 TB Storage

- 4 TF IA-64
- DB2, Oracle Servers
- 500 TB Disk Storage
- 6 PB Tape Storage
- 1.1 TF Power4

- 6 TF EV68
- 71 TB Storage
- 0.3 TF EV7 shared-memory
- 150 TB Storage Server

- .5 TF IA-64
- 96 Visualization nodes
- 20 TB Storage

- 0.4 TF IA-64
- IA32 Datawulf
- 80 TB Storage

**Extensible Backplane Network**

**LA**
**Hub**

**Chicago**
**Hub**

**IA32**

- Storage Server

- Disk Storage

- Cluster

- Shared Memory

- Visualization
- Cluster

_**LEGEND**_

**30 Gb/s**

**IA64**

**30 Gb/s**

**30 Gb/s**

**30 Gb/s**

**30 Gb/s**

**Sun**

**Sun**

**ANL**: Visualization

**Caltech**: Data collection analysis

**40 Gb/s**

- Backplane Router

- PSC integrated Q3 03

## Extensible TeraGrid Facility

**NCSA**: Compute-Intensive

**SDSC**: Data-Intensive

**PSC**: Compute-Intensive

**IA64**

**IA64**

**Pwr4**

**EV68**

**IA32**

**EV7**

**IA64**

**Sun**

- 10 TF IA-64
- 128 large memory nodes
- 230 TB Storage

- 4 TF IA-64
- DB2, Oracle Servers
- 500 TB Disk Storage
- 6 PB Tape Storage
- 1.1 TF Power4

- 6 TF EV68
- 71 TB Storage
- 0.3 TF EV7 shared-memory
- 150 TB Storage Server

- .5 TF IA-64
- 96 Visualization nodes
- 20 TB Storage

- 0.4 TF IA-64
- IA32 Datawulf
- 80 TB Storage

**Extensible Backplane Network**

**LA**
**Hub**

**Chicago**
**Hub**

**IA32**

- Storage Server

- Disk Storage

- Cluster

- Shared Memory

- Visualization
- Cluster

_**LEGEND**_

**30 Gb/s**

**IA64**

**30 Gb/s**

**30 Gb/s**

**30 Gb/s**

**30 Gb/s**

**Sun**

**Sun**

**ANL**: Visualization

**Caltech**: Data collection analysis

**40 Gb/s**

- Backplane Router

- PSC integrated Q3 03

- 96 Dual PIVs64 Dual Madison

- 32 Dual PIVs66 Dual Itanium2

- 32 Dual PIVs256 Quad Itanium2
- 128 Dual Power4

- 256 Dual Itanium2
- 700 Dual Madison

- 128 EV7
- 750 Quad EV68

## Argonne ETF Cluster Schematic

**Argonne ETF Cluster Schematic**

- 8

- STARLIGHT - GbE

***Existing******Facilities***

-    Gigabit Enet
-    10 Gigabit Enet
-    Myrinet
-    Fibre Channel
-    Direct Video Link

- Myrinet 2000CLOS switch

- 2 storage nodes

- Display WallsAG Nodes
- CAVE

- 1

- 10

- 3xIBM FASTt700
-  20 TB FC Storage

- 2

- 2

- 4 login nodes

- 64 compute nodes.5 TF IA-64

- 4 mgmt nodes

- 96

- 96

- 96

**DTF Backplane****Connection**

- CiscoGSR

- MREN – OC-12

- HSCC – OC-48

- ESNet – OC-12

- Linux Cluster574p IA-32

- HPSS and ADSM
- 120 TB archive

- 2

- 1

- 1 per

- 96 viz nodesPentium4.9 TF
- GeForce4 graphics

- 64

- Linux Cluster350p Pentium4

- 8

**Juniper Border Router**

- Cisco6509

- Charlie Catlett

## TeraGrid Objectives

** ****Create significant enhancement in capability**
- Beyond capacity, provide basis for exploring new application capabilities
** ****Deploy a balanced, distributed system**
- Not a “distributed computer” but rather
- A distributed “system” using Grid technologies
- Computing and data management
- Visualization and scientific application analysis
** ****Define an open and extensible infrastructure**
- An “enabling cyberinfrastructure” for scientific research
- Extensible beyond the original four sites

## Where We Are

**Early access**
**to McKinley**
**At Intel**

**Jan ‘01**

**DTF Proposal **
**Submitted**
**To NSF**

**Jan ‘02**

**Jan ‘03**

**Early McKinleys **
**at TG sites for**
**Testing/benchmarking**

**TeraGrid clusters**
**installed**

**TeraGrid Operations Center**
-   Prototype                Day Ops                   Production

**Basic Grid svcs**
-   Linux clusters
-   SDSC SP
-   NCSA O2K

**Core Grid services deployment **

**Initial apps**
**on McKinley**

**TeraGrid Networking Deployment**

**TeraGrid prototype**
**at SC2001, 60 Itanium**
**Nodes, 10Gbs network**

***IA-64 ***
***systems ***

***TeraGrid***
***Prototypes ***

***Grid Services on***
***Current Systems***

*** ******Networking***

*** ******Operations***

**10Gigabit Enet testing**

**“****TeraGrid Lite” ****systems and Grids**
**testbed**

**Advanced Grid services testing **

***Applications***

**Clusters upgraded**
**to Madison**

**DTF TeraGrid in****production**

**ETF Approved**

**Apps deployed at**
**SC2001 on prototype.**

**Early apps being tested**
**on TG-Lite.**

## Challenges and Issues

**Technology and Infrastructure**
- Networking
- Computing and Grids
- Others (not covered in this talk):
- Data
- Visualization
- Operation
- ...
**Social Dynamics**
**To Be Clear...**
- While the following slides discuss problems and issues in the spirit of this workshop, the TG project is making appropriate progress and is on target for achieving milestones.

## Networking Goals

**Support high bandwidth between sites**
- Remote access to large data stores
- Large data transfers
- Inter-cluster communication
**Support extensibility to N sites**
-   4 <= N <= 20 (?)
**Operate in production, but support network experiments. **
**Isolate the clusters from network faults and vice versa.**

## NSFNET 56 Kb/s Site Architecture

**VAX**

**Fuzzball**

- 1024MB

- Across the room

- Across the country

- Bandwidth in terms of **burst** data transfer
- and user **wait time**.

## 2002 Cluster-WAN Architecture

**OC-48 Cloud**
**(2.4Gb/s)**

**2002 Cluster-WAN Architecture**

- 1 TB

- n x GbE (small n)

- OC-12
- (622Mb/s)

- Across the room

- Across the country

## To Build a Distributed Terascale Cluster...

**Interconnect**

**To Build a Distributed Terascale Cluster...**

- Big Fast Interconnect
- (OC-800ish)

- 10 TB

- 10 TB

- n x GbE (40 <= n <= 200)

## TeraGrid Interconnect: Qwest Partnership 

- 1

- 2

- Caltech/JPL

- SDSC/UCSD

- NCSA/UIUC

- ANL

**Physical**
- # denotes λ count

**Light Paths**
**(Logical)**

- La Jolla

- San Diego

- LA

- Pasadena

- Chicago

*Urbana*

*Argonne*

**Phase 0 (June 2002)**

**Phase 1 (November 2002)**

- 2

- 1

- 4

- 3

- 3

- 3

- 3

## Extensible TeraGrid Facility

**NCSA**: Compute-Intensive

**SDSC**: Data-Intensive

**PSC**: Compute-Intensive

**IA64**

**IA64**

**Pwr4**

**EV68**

**IA32**

**EV7**

**IA64**

**Sun**

- 10 TF IA-64
- 128 large memory nodes
- 230 TB Storage

- 4 TF IA-64
- DB2, Oracle Servers
- 500 TB Disk Storage
- 6 PB Tape Storage
- 1.1 TF Power4

- 6 TF EV68
- 71 TB Storage
- 0.3 TF EV7 shared-memory
- 150 TB Storage Server

- .5 TF IA-64
- 96 Visualization nodes
- 20 TB Storage

- 0.4 TF IA-64
- IA32 Datawulf
- 80 TB Storage

**Extensible Backplane Network**

**LA**
**Hub**

**Chicago**
**Hub**

**IA32**

- Storage Server

- Disk Storage

- Cluster

- Shared Memory

- Visualization
- Cluster

_**LEGEND**_

**30 Gb/s**

**IA64**

**30 Gb/s**

**30 Gb/s**

**30 Gb/s**

**30 Gb/s**

**Sun**

**Sun**

**ANL**: Visualization

**Caltech**: Data collection analysis

**40 Gb/s**

- Backplane Router

- PSC integrated Q3 03

## Teragrid Logical Network Diagram

**Teragrid Logical Network Diagram**

**Los Angeles**

- 2200mi

- 140mi

- 25mi

- 115mi

- 15mi

**Caltech Cluster**

**SDSC Cluster**

**NCSA Cluster**

**ANL Cluster**

**Starlight**

**Chicago**

***I-WIRE***

**Notes:**
See Q&A text on Phase I and Phase II.  The DTF site network experts have discussed this architecture with a number of networking experts including Bill St.Arnaud (CA*Net), the Abilene backbone team (led by Steve Corbato), the I-WIRE/Startap/Starlight team (led by Argonne staff), Qwest (Wes Kaplow), and Indiana University’s NOC staff.
Architectural options include:
Large routers (phase 1, illustrated).  Traffic segregation, including dedicated 10 GbE paths, can be accomplished via available means such as MPLS.  Concern is whether the routers (or switch/routers) can handle the load.  Hence testing will begin in October 2001 with multiple router vendors.
Optical mesh (phase 2).  Most optical switching architectures assume many paths, whereas the DTF backbone is only 4 wavelengths.  It would be possible to architect this network as 16 wavelengths, each running at 2.5 Gb/s, but it is unclear that partitioning the bandwidth in this way would provide benefits offsetting the additional complexity.  Nevertheless these options exist as fall-back architectures should we find that the phase 1 approach introduces bottlenecks. 

## I-WIRE Geography

**UI-Chicago**

**Illinois Inst. Tech**

**Argonne Nat’l Lab**
- (approx 25 miles SW)

**Northwestern Univ-Chicago**
**“****Starlight”**

**U of Chicago**

- I-55

- Dan Ryan Expwy
- (I-90/94)

- I-290

- I-294

**UIUC/NCSA**
- Urbana (approx 140 miles South)

**I-WIRE Geography**

- Status:Done: ANL, NCSA, StarlightLaterals in process: UC, UIC, IIT
- Investigating extensions to Northwestern Evanston, Fermi, O’Hare, Northern Illinois Univ, DePaul, etc.

## State of Illinois I-WIRE

**Starlight**
**(NU-Chicago)**

**Argonne**

**UChicago**

**IIT**

**UIC**

- State/City Complex
- James R. Thompson Ctr
- City Hall
- State of IL Bldg

**4**

**12**

**4**

**2**

**2**

**4**

**18**

**4**

**10**

**12**

**2**

- Level(3)
- 111 N. Canal

- McLeodUSA
- 151/155 N. Michigan
- Doral Plaza

- Qwest455 N. Cityfront

- UC Gleacher Ctr
- 450 N. Cityfront

**State of Illinois I-WIRE**

** ****I-WIRE timeline**
- 1994: Governor interest
- schools and networks
- 1997: task force formed
- 1999: I-WIRE funding approved
- 2002: fiber in place and operational
** ****Features**
- fiber providers
- Qwest, Level(3)
- McLeodUSA, 360Networks
- 10 segments
- 190 route miles and 816 fiber miles
- longest segment is 140 miles
- 4 strands minimum to each site

- Numbers indicate fiber count (strands)

**NU-Evanston**

**FNAL**

## I-Wire Transport

**UIUC/NCSA**

**Starlight**
**(NU-Chicago)**

**Argonne**

**UChicago**

**IIT**

**UIC**

- State/City Complex
- James R. Thompson Ctr
- City Hall
- State of IL Bldg

- UC Gleacher Ctr
- 450 N. Cityfront

**I-Wire Transport**

- Each of these three ONI DWDM systems have capacity of up to 66 channels, up to 10 Gb/s per channel
- Protection available in Metro Ring on a per-site basis

- _TeraGrid Linear_
- 3x OC192
- 1x OC48
- First light: 6/02

- _Metro Ring_
- 1x OC48 per site
- First light: 8/02

- _Starlight Linear_
- 4x OC192
- 4x OC48 (8x GbE)
- Operational

- Qwest455 N. Cityfront

## Network Policy Decisions

**The TG backplane is a closed network, internal to the TG sites.**
- Open question: what is a TG site?
**The TG network gear is run by the TG network team.**
- I.e. not as individual site resources.

## Network Challenges

**Basic Design and Architecture**
- We think we’ve got this right.
**Construction**
- Proceeding well.
**Operation**
- We’ll see.

## Computing and Grid Challenges

**Hardware configuration and purchase**
- I’m *still* not 100% sure what we’ll be installing.
- The proposal was written in early 2001.
- The hardware is being installed in late 2002.
- The IA-64 line of processors is young.
- Several vendors, all defining new products, are involved.
- Recommendations:
- Try to avoid this kind of long-wait, multi vendor situation.
- Have frequent communication with all vendors about schedule, expectations, configurations, etc.

## Computing and Grid Challenges

**Understanding application requirements and getting people started before the hardware arrives.**
**Approach: TG-Lite**
- a small PIII testbed
- 4 nodes at each site
- Internet/Abilene connectivity
- For early users and sysadmins to test configurations.

## Computing and Grid Challenges

**Multiple sites, one environment:**
- Sites desire different configurations.
- Distributed administration.
- Need a coherent environment for applications.
- Ideal: binary compatibility
**Approach: service definitions.**

## NSF TeraGrid: 14 TFLOPS, 750 TB

**NSF TeraGrid: 14 TFLOPS, 750 TB**

- HPSS

- HPSS

- 574p IA-32 Chiba City

- HR Display & VR Facilities

- Myrinet

- Myrinet

- 1176p IBM SP
- Blue Horizon

- Sun E10K

- 1500p Origin

- UniTree

- 1024p IA-32
-   320p IA-64

- HPSS

- 256p HP
- X-Class

- 128p HP
- V2500

- 92p IA-32

**NCSA**: Compute-Intensive

**ANL**: Visualization

**Caltech**: Data collection analysis

**SDSC**: Data-Intensive

## Defining and Adopting Standard Services

- IA-64 Linux TeraGrid Cluster Runtime

- File-based Data Service

- Collection-based Data Service

- Volume-Render Service

- Interactive Collection-Analysis Service

- IA-64 Linux Cluster Interactive Development

- Finite set of TeraGrid services- applications see *standard services* rather than *particular implementations*...

- ...but sites also provide additional services that can be discovered and exploited.

- Grid Applications

## Strategy: Define Standard Services

**Finite Number of TeraGrid Services**
- Defined as specifications, protocols, API’s
- Separate from implementation (magic software optional)
**Extending TeraGrid**
- Adoption of TeraGrid specifications, protocols, API’s
- What protocols does it speak, what data formats are expected, what features can I expect (how does it behave)
- Service Level Agreements (SLA)
- Extension and expansion via:
- Additional services not initially defined in TeraGrid
- e.g. Alpha Cluster Runtime service
- Additional instantiations of TeraGrid services
- e.g. IA-64 runtime service implemented on cluster at a new site
**Example: File-based Data Service**
- API/Protocol:  Supports *FTP* and *GridFTP, GSI *authentication
- SLA
- All TeraGrid users have access to *N* TB storage
- available 24/7 with *M*% availability
- >= *R* Gb/s read, >= *W* Gb/s write performance

## Standards  Cyberinfrastructure

**Standards ******** Cyberinfrastructure**

- Runtime

- File-based Data Service

- Collection-based Data Service

- Visualization Services

- Interactive Collection-Analysis Service

- Interactive Development

- IA-64 Linux Clusters

- Alpha Clusters

- IA-32 Linux Clusters

- Visualization Services

_*Data/Information*_

_*Compute*_

_*Analysis*_

- Relational dBase Data Service

- TeraGrid Certificate Authority

- Certificate Authority

- Certificate Authority

- Certificate Authority

- Certificate Authority

- Grid Info Svces

**If** done _*openly*_ and well...
- other IA-64 cluster sites would adopt TeraGrid service specifications, increasing users’ leverage in writing to the specification
- others would adopt the framework for developing similar services on different architectures

- Grid Applications

## Computing and Grid Challenges

**Architecture**
- Individual clusters architectures are fairly solid.
- Aggregate architecture is a bigger question.
- Being defined in terms of services.
**Construction and Deployment**
- We’ll see, starting in December.
**Operation**
- We’ll see.  Production by June 2003.

## Social Issues: Direction

**4 sites tend to have 4 directions.**
- NCSA and SDSC have been competitors for over a decade.
- This has created surprising cultural barriers that must be recognized and overcome.
- Including PSC, a 3rd historical competitor, will complicate this.
- ANL and Caltech are smaller sites with fewer resources but specific expertise.  And opinions.

## Social Issues: Organization

**Organization is a big deal.**
- Equal/fair participation among sites.
- To the extreme credit of the large sites, this project has been approached as 4 peers, not 2 tiers.  This has been extremely beneficial.
- Project directions and decisions affect all sites.
- How best to distribute responsibilities but make coordinated decisions?
- Changing the org chart is a heavyweight operation, best to be avoided...

## The ETF Organizational Chart

_**Site Coordination Committee**_
- Site Leads

_**Project Director**_
- Rick Stevens (UC/ANL)

_**Technical Coordination Committee**_
- Project-wide Technical Area Leads

- _Clusters_
- Pennington
- (NCSA)

- _Networking_
- Winkler
- (ANL)

- _Grid Software_
- Kesselman (ISI)
- Butler (NCSA)

- _Data_
- Baru
- (SDSC)

- _Applications_
- WIlliams
- (Caltech)

- _Visualization_
- Papka
- (ANL)

- _Performance Eval_
- Brunett
- (Caltech)

_**Chief Architect**_
- Dan Reed (NCSA)

_**Executive Committee**_
- Fran Berman, SDSC (Chair)
- Ian Foster, UC/ANL
- Michael Levine, PSC
- Paul Messina, CIT
- Dan Reed, NCSA
- Ralph Roskies, PSC
- Rick Stevens, UC/ANL
- Charlie Catlett, ANL

- _Technical Working Group_
*Are we creating an extensible cyberinfrastructure?*

- _External Advisory Committee_
*Are we enabling new science?*
*Are we pioneering the future?*

- _Institutional Oversight Committee_
- TBD, UCSD
- Richard Herman, UIUC
- Mark Kamlet, CMU
- Dan Meiron, CIT (Chair)
- Robert Zimmer, UC/ANL

- _User Advisory Committee_
*Are we effectively supporting good science?*

- NSF MRE Projects

- Internet-2
- McRobbie

- Alliance UAC
- Sugar, Chair

- NPACI UAC
- Kupperman, Chair

_**NSF ACIR**_

- _NSF Technical Advisory Committee_

- Policy Oversight

- Objectives/Eval

- Architecture

**Currently being formed**

_**Executive Director /**_
_**Project Manager**_
- Charlie Catlett (UC/ANL)

_**ANL**_
- Evard

_**CIT**_
- Bartelt

_**NCSA**_
- Pennington

_**SDSC**_
- Andrews

_**PSC**_
- Scott

- _Operations_
- Sherwin
- (SDSC)

- _User Services_
- Wilkins-Diehr (SDSC)
- Towns (NCSA)

- Implementation

- _NSF Program Advisory Team_

- PSC UAG

## Social Issues: Working Groups

**Mixed effectiveness of working groups**
- The networking working group has turned into a team.
- The cluster working group is less cohesive.
- Others range from teams to just email lists.
**Why?**
- Not personality issues, not organizational issues.
**What makes the networking group tick:**
- Networking people already work together:
**The individuals have a history of working together on other projects.**
**They see each other at other events.**
**They’re expected to travel.**
**They held meetings to decide how to build the network before the proposal was completed.**
- The infrastructure is better understood:
**Networks somewhat like this have been built before.**
**They are building one network, not four clusters.**
**There is no separation between design, administration, and operation.**
**Lessons:**
- Leverage past collaborations that worked.
- Clearly define goals and responsibilities.

## Social Issues: Observations

**There will nearly always be four opinions on every issue.**
- Reaching a common viewpoint takes a lot of communication.
- Not every issue can actually be resolved.
- Making project-wide decisions can be tough.
**Thus far in the project, the social issues have been just as complex as the technical.**
- ... but the technology is just starting to arrive...
**It’s possible we should have focused more on this in the early proposal stage, or allocated more resources to helping with these.**
- We have, just appointed a new “Director of Engineering” to help guide technical decisions and maintain coherency.

## Conclusion

**Challenges abound!  Early ones include:**
- Network design and deployment.
- Cluster design and deployment.
- Building the right distributed system architecture into the grid.
- Getting along and having fun.
- Expansion.
**The hardware arrives in December, production is in mid-2003.**
**Check back in a year to see how things are going...**