# Presentation: 057624

## TeraGrid Applications WorkshopOn Demand SessionsTeraGrid Lite Overview and Update

**Rob Pennington, NCSA**
**robp@ncsa.uiuc.edu**

## On Demand  

**Linda Winkler (ANL), Cristina Beldica (NCSA), Steven Engelhardt, (NCSA), Tom Sherwin (SDSC), Amit Majumdar (SDSC), Sharon Brunett (Caltech), Lynn Ten Eyck, Michael Welge (NCSA), Tom Prudhomme (NCSA), Randy Butler (NCSA), Alan Blatecki (NSF), Carl Kesselman (ISI), Randy Heiland (NCSA), Rick Stevens (ANL), Richard Raymond (PSC) **
**Consumer: Mark Ellisman (UCSD)**
**Producer:Rob Pennington (NCSA)**

## On Demand  

** ****Computations, data and viz**
- Networking?
- Supercomputing, cycle scavenging
- Data requirements
**Real time/time critical**
- Medical community – image data analysis and response
**Short turnaround time**
- Interactive data analyses and manipulations
**Scheduling/reservation in advance vs. pre-emptive interrupt driven**
- Coordinated activity vs. spontaneous
- Capability (required) vs capacity (what’s available)
- Rollout vs overprovisioning
**Dynamic growth/shrinkage of an application **
- As resources become available or are no longer necessary

## On Demand  

**User level checkpointing?**
- Self preservation tactic – specify restart criteria
- Restart capability by scheduler
- Opens door to re-prioritization at checkpoint time
**System level checkpointing**
- Extra storage or ? As necessary to be able to nice the currently running job?

## On Demand   

**Define what could be done with infinite resources**
- How can we let some users *think* they have infinite resources?
- There is no way we can meet this requirement so we need to set priorities based on policies for resources.  For the right user, it should be possible to obtain all of the resources.
- Advance reservations
- Pre-define what is going to be required, need to support with policy
- Design the system to respond within some interval
- Response time guarantee?  Viz or steering?
- Multi-user, interactive, short response time.
- Portals, collaboration, think/compute time cycle with low utilization.
- Access to compute resources, data and networks
- Deadline driven scheduling
- Resource guarantee?
- Can this change dynamically?  Pre-emptive ability.
- Brain surgery or shake table experiments
- Provide a completion guarantee?
- Fault tolerance and/or failover to assure completion

## On Demand  

**First cut at a response**
- Implement advanced reservations across TG
- Requires foreknowledge of requirements
- Maintain a resource pool that is not pre-scheduled
- Load insensitive, treated as unlimited
- Saturation point
**Second cut at a response**
- Reserved domain filled with jobs that can be bumped?

## On Demand  

** ****What is the requirement for resources?**
- Advance reservation:  My problem is size X, when could I run this on resource Y?
- Specify resource and deadline requirements.
**Arbitration of requests, **
**Trade off of risk of non-completion and “cost” of the resource**

## On Demand  

** ****Need the software infrastructure for advance reservations**
- Is there software that provides advance reservations?
- Maui might be a prototype, starting point
- Need an admin in the loop to do this right now
- Install/prototype it across the sites
- Build an infrastructure on this to get started
- *** We need a consistent architecture across the sites***
**Next priority would be for deadline driven scheduling, followed by making an “interactive teraflop” type of system.**

## TeraGrid Timelines

**Early access**
**To McKinley**
**At Intel**

- Jan ‘01

- Proposal
- Submitted
- To NSF

- Jan ‘02

- Jan ‘03

**Early McKinleys **
** ****at TG sites for**
**Testing/benchmarking**

** ****TeraGrid clusters**

**TeraGrid Operations Center**
-   Prototype                Day Ops                   Production

**Basic Grid svcs**
-   Linux clusters
-   SDSC SP
-   NCSA O2K

**Core Grid services deployment **

**Initial apps**
**On McKinley**

** ****TeraGrid **
**Operational**

**TeraGrid Networking Deployment**

**TeraGrid prototype**
**At SC2001, 60 Itanium**
**Nodes, 10Gbs network**

***McKinley ***
***systems ***

***TeraGrid***
***Prototypes ***

***Grid Services on***
***Current Systems***

*** ******Networking***

*** ******Operations***

**10Gigabit Enet testing**

**“****TeraGrid Lite”**
**   ****Systems and Grids**
**   ****testbed**

**Advanced Grid services testing **

***Applications***

## TeraGrid Cluster Configuration

- Myrinet Fabric

- Ethernet
- Switch

- Common Storage

- SCALE THIS DOWN FOR
- TERAGRID LITE with IA-32

## TeraGrid Lite ®

**Clusters and Grids Working Groups project**
- IA-32 nodes
- All of the software should work on these nodes
- Minimize the software issues, focus on admin and apps
**Goals**
- Install small clusters at the 4 sites to work on:
- Linux software configuration across the 4 sites
- Grid software configuration across the 4 sites
- Accounts management differences
- Monitoring tools emplacement
- Platform for early TG test users
- Test and integration testbed
- “Feeder” for other platforms
- Larger production clusters at the sites, including IA-32 and IA-64

## Configuration

**4-5 IA-32 nodes at each site**
- Dedicated to this testbed
**Status snapshot**
- _[http://josie.ncsa.uiuc.edu/documents/tg-checkpoint.htm](http://josie.ncsa.uiuc.edu/documents/tg-checkpoint.htm)_
- _[http://www.ncsa.uiuc.edu/~jbasney/teragrid-status/](http://www.ncsa.uiuc.edu/~jbasney/teragrid-status/)_
- Work is on-going on these nodes to finish issues
**Accounts**
- http://accounts.teragrid.org/
**Next generation TG Lite**
- Initial McKinley systems have arrived at sites
- Up to 15 systems being installed

## IA-32 Prototypes TeraGrid

- File Staging
- 30GB

- Ethernet
- Switch

- IA-32
- Interactive
- Machine

- 100 Mb Ethernet

- 100 Mb
- Ethernet

- fileserver

- /home

- Testbeds deployed

- IA-32, RH 7.2

- IA-32, RH 7.2

- IA-32, RH 7.2

- IA-32, RH 7.2

## McKinley Prototypes TeraGrid

- File Staging
- 30GB

- Myrinet
- Switch

- Ethernet
- Switch

- Host
- Machine

- 100 Mb Ethernet

- 100 Mb
- Ethernet

- Internal
- fileserver

- /home

- Myrinet

- Testbeds in progress

## TeraGrid Lite

** ****Auto download and installation of a similar package?**
- Man power intensive, provide instead a config list
**IA-32 testbed for up to a year as a stable platform**
- Simple comms across the 4 testbeds?
- Math libs: MKL, PETSc,
- GridFTP, MPICH-G2
- Environment variables
- TG_sitename – to customize makefile, runscript
- Vendor compilers and paths in common or aliased
- Common doc set across the sites with basic information
- “lsof”, tools for what apps are using and dependencies
- Logical configuration for the resources to be used by the app to map to the execution environment.
**MDS information for monitoring in the future**
- Specification of the requirements for the node to be mapped to the app
- Tool for submitter to make reasonable resource selection

## TeraGrid Lite

** ****Coordinated reservations for multi-site jobs**
- Can I schedule N nodes at this site at this time for this long
- Resource specification and ability to make/release local reservation
- N nodes, start time, duration – reserve or cancel/supercede
- Publish existing reservations for people to view as a first attempt
- This might be possible using existing MDS scripts (Ian?)
- Separation of reservation from submission
**Policies: **
- Windows during which reservations are enabled for sites initially
**Need to specify an API to request reservation information from a system?**
- Can I make a reservation request to this machine?
- Can I make the following reservation on this machine?
- Need the right features in the GRAM to support reservation information request