# Presentation: 800208

## What the Users’ WantorThe NSF’s Terascale Computing System and Teragrid:Support for Scientific ResearchorMaking the Best of It

- Mike Levine
- Scientific Director, PSC
- SOS7 – Durango – 5 Mar 2003

## Choice of a Title  I.

- The first title, *What do the Users’ Want, *was proposed by Neil.  Thank you, Neil.
- It is a lovely title and a very important question, but
- I don’t really understand things, here, that most of you do not already understand:
- The users’ want to be able to get their work done!
- As efficiently as possible in their time and machine time
- On ever increasing problem sizes.
- (Usually, they are not picky about the method.)

## Choice of a Title  II

- The second title, *The NSF’s Terascale Computing System and Teragrid: Support for Scientific Research, *was my choice.
- Dan Reed might do a better job on this subject but went for something more relevant to this meeting.
- Perhaps not being as wise as he, I will say a few words on this subject.
- Then, I will abuse Neil’s hospitality and move to another topic.

## NSF’s Terascale Computing Systems

- I have already introduced the TCS in the panel *Machines Already Operational *.
- TCS was meant to very substantially increase the size of machine open to US scientists.
- This it has done*.*
- Soon to be joined by *DTF* (the Distributed Terascale Facility)
- Described, yesterday, by Dan Reed.
- A very large IBM/IA64 - Linux cluster
- Distributed between NCSA and SDSC with
- Additional capabilities at ANL and Caltech.
- Interconnected by a multi-lamba network, 30 Gb/s to each site.

## Teragrid

- Join TCS to DTF
- Upgrade network to be routed and extensible
- Extend it to PSC and into TCS
- Begin, shortly, to incorporate additional sites & resources.
- A basis for a National CyberInfrastructure
- To “revolutionize our efforts in scientific research”
- Incorporate computation, data intensive work, visualization, instruments, diverse facilities.
- Lots of software effort to provide a uniform, distributed environment for users.
-  One of the new resources is EV7/Marvel.

## EV7 (cpu)/Marvel (system)

- “... the greatest scientific processor, ever...”  (Bill Camp, yesterday)
- Pre-production systems at PSC, CEA, ... for several months.
- Jean Gonnord, yesterday, mentioned CEA’s work on EV7
- (He has a substantial body of benchmark information to be summarized, below)
- Production systems are now shipping (2 at PSC)
- PSC is building up towards 2 systems
- ~250 processors 		 ~1/2 TB memory/system.
- #1: NSF: large memory, high bw, SMP; ETF resource.
- #2: NIH: all of the above, specific data-intensive applications.
- We believe that Marvel will supply a good deal of *What the Users Want.*
- In addition to science output, we hope to
- Learn more about the application value of high bandwidth systems.
- Encourage vendors to match it or do better.

## NIH Marvel:

- Partner with four world leaders in three diverse fields:
- Eric Lander and the Whitehead group (genomics),
- Michael Klein et al from the University of Pennsylvania (structural biology)
- Klaus Schulten et al from the University of Illinois (structural biology)
- Terrence Sejnowski et al, UCSD and PSC (neuroscience).
- They present compelling examples of data-, memory-, and compute-intensive problems that can only realistically be attacked with the proposed architecture.

## Alpha EV7/Marvel

- Alpha EV7 = Alpha EV68 core (1 GHz)
- + *on-*chip 2D torus SMP interconnect
- + huge IO & memory bandwidth (per CPU)
- 12.8 GB/s (=6B/f! recall Buddy’s chart; ES45=2 GB/s)
- + low memory latency
- 80ns, local (ES45=140ns)
- Marvel = 2-128 proc SMP’s [HP has yet to promise systems >64p]
- Low intra-SMP memory latency
- ( ~350 ns, furthest node)
- Large aggregate memory (global, up to 8 GB/proc)
- 8P early test system testing at PSC
- Multi-week tests (local & remote users, incl ORNL)
- Multiple applications (& OS’s)
- Excellent McCalpin Streams performance
- 2*16P production systems now at PSC

## EV7 – The System is the Silicon....

***Building a System....***

_***EV7 + I/O + Memory = SYSTEM !***_

## Direct processor-processor interconnects (16P Torus)

## 128P Partitionable System using 8P Building Block Drawers  

- 128P Partitionable System using 8P Building Block Drawers

- Up to 128 processors
- Up to 4TB memory
- Loads of IO (PCI-X & AGP)

##      EV7 tests (November 2001)

- 1.1 Ghz, 1Ghz memory
- TERA
- 1753 : 640 Mflops (29% of peak)
- 1003 : 660 Mflops
- PUMA
- 272s (EV68@833 : 405s)
- 1.48 times better than ES45 (clock ratio is 1.32)
- PUMA with MPI, 16 processors (800 MHz)
- ES45 : 51.384 s
- EV7 : 33s (ratio : 1.56)

**Jean Gonnord, CEA/DAM **

## EV7 tests

- A
- V

**Jean Gonnord, CEA/DAM **

## Choice of a Title  III:Let’s Make the Best of It

- An important topic drifted in and out of several talks, yesterday but was not given direct attention.
- Bill Camp mentioned it as “reliability”
- Dan Reed mentioned it as “carefully engineered” clusters
- Dieter said we had heard enough of “fault tolerance”.
- (He, I think, was wrong.)
- Dan Katz mentioned it under “software management and configuration”.
- It is directly supportive of *What the Users Want.*

## Let’s Make the Best of It

- The issue is “system reliability & availability”, hardware and software.
- “*It”*, in the title,* *refers to our lovely, expensive systems.
- More specifically, I refer to issues that I would characterize as “good engineering” and not to theoretical issues (which are however more fascinating).
- In contrast to some of the fault tolerance discussion, I suggest
- there is a fair amount of “low-hanging fruit”
- requiring but small amounts of effort and
- having little or no impact on performance.
- Many of the comments in the panel of “Machines Already Operational” implied clear lack of sufficient attention tothese types of issues.

## (at this point I wish I had)The Cartoon from The New Yorker

- A picture of a man leaving Church,
- pausing at the door to say to the Minister:
*“**Thank you, Reverend, for not mentioning*
*me by name in your sermon”*

- (but, I was preparing this during Thomas’ talk, yesterday evening)

## Whose Baby is This?

- We save bundles by buying commodity components, either raw or from large vendors.
- At the component level, we still benefit from the substantial engineering that went into their design.
- At the system level, however, multiple forms of danger lurk.
- When no one else claims ownership of dealing with these dangers, it is “our baby”.
- (Applause for the efforts, described today, to strongly influence vendors.)

## What Forms of Danger?

- Issues of scale:
- We are integrating these components into systems in many cases well beyond anything imagined by the original designers.
- Issues of style of usage:
- We are often using these systems in modes different from that intended by the original designers and not understood by them.
- (e.g. very large-scale, highly synchronized applications may be peculiar to HPTC.)
- In addition to needing some things that they do not provide, we also often do not need things that they do provide and for which other system compromises have been made.
- These issues are terribly atypical of the vast majority of their customer base. (as has been mentioned, frequently)
- Not dealing with them limits our system scalability.

## What Don’t We Need?

- We do not run a life support system.  We can get along pretty well with the temporary failure of a fair amount of hardware.
- With increasing scale, attempts to totally prevent failure are insufficient.
- Then, work to prevent “splatter” is more important than slightly reducing the frequency of failure.
- We do not need “rapid response” to most failures.
- Once a node goes, the immediate job is toast.
- Here, too, “containment” is very important.

## Examples of Dealing With Such Issues

- If you have not, you should read the RAS requirements in the Red Storm solicitation. (Jim Tompkins mentioned that, briefly)
- From those requirements, you can readily infer the kinds of problems that Camp, Tompkins etal are working to avoid.
- I, at least, was impressed by the level of effort SNL is prepared to expend in this domain.
- At PSC, working with HP, we have implemented continuous monitoring of soft-fault errors.
- We are now doing true “preventative maintenance”.
- Contrary to historical practice, analysis need not be done on nodes optimized for computation.
- Dan Reed mentioned this sort of thing, yesterday.
- Particularly applicable to disks, memory & network.

## Solutions

- I am not here to propose a solution.
- My immediate goal is to call attention to the problem.
- Just as it has become clear that HPC makes special demands on Linux, it also makes special demands on system configuration and operation.
- Both might benefit from more coordinated attention.
- Perhaps at the next conference, Neil might consider some further attention to this issue.