# Presentation: 622435

- NERSC Status and Plans
- for the
- NERSC User Group MeetingFebruary 22,2001
- BILL  KRAMER
- DEPUTY DIVISION DIRECTOR
- DEPARTMENT HEAD, HIGH PERFORMANCE COMPUTING DEPARTMENT
- kramer@nersc.gov
- 510-486-7577

**Notes:**
1974	Founded at Lawrence Livermore Laboratory as the Controlled Thermonuclear Research Computer Center late 1970s	Began operating system development, name changed to Magnetic Fusion Energy Computer Center (MFECC)
1980s	Developed the Cray Time Sharing System (CTSS) for Cray 1, S-MP, Y-MP and Cray 2
1980s	Established MFEnet, including satellite uplinks
1985-90	Transition from MFEnet to Energy Science Network (ESnet)
1988	DOE broadened mission of the Center, name changed to National Energy Research Supercomputer Center (NERSC)
1990	Energy Science Network (ESnet) began T1 service
1991	Switch to vendor supplied operating systems
1995-96	DOE recompeted the NERSC Program, Lawrence Berkeley National Laboratory has been chosen as the new site		Grand Reopenning at LBNL Oct 22, 1996

## Agenda

- Update on NERSC activities
- IBM SP Phase 2 status and plans
- NERSC-4 plans
- NERSC-2 decommissioning

## ACTIVITIES AND ACCOMPLISHMENTS

## NERSC Facility Mission

**NERSC Facility Mission**

- To provide reliable, high-quality,state-of-the-art computing resourcesand client support in a timely manner—independent of client location–while wisely advancing the state of computational and computer science.

## 2001 GOALS

- PROVIDE RELIABLE AND TIMELY SERVICE
- Systems
- Gross Availability, Scheduled Availability, MTBF/MTBI, MTTR
- Services
- Responsiveness, Timeliness, Accuracy, Proactivity
- DEVELOP INNOVATIVE APPROACHES TO ASSIST THE CLIENT COMMUNITY EFFECTIVELY USE NERSC SYSTEMS.
- DEVELOP AND IMPLEMENT WAYS TO TRANSFER RESEARCH PRODUCTS AND KNOWLEDGE INTO PRODUCTION SYSTEMS AT NERSC AND ELSEWHERE
- NEVER BE A BOTTLENECK TO MOVING NEW TECHNOLOGY INTO SERVICE.
- INSURE ALL NEW TECHNOLOGY AND CHANGES IMPROVE (OR AT LEAST DOES NOT DIMINISH) SERVICE TO OUR CLIENTS.

## GOALS (CON’T)

- NERSC AND LBNL WILL BE A LEADER IN  LARGE SCALE SYSTEMS  MANAGEMENT **&** SERVICES.
- EXPORT KNOWLEDGE, EXPERIENCE, AND TECHNOLOGY DEVELOPED AT NERSC, PARTICULARLY TO AND WITHIN NERSC CLIENT SITES.
- NERSC WILL BE ABLE TO THRIVE AND IMPROVE IN AN ENVIRONMENT WHERE CHANGE IS THE NORM
- IMPROVE THE EFFECTIVENESS OF NERSC STAFF BY IMPROVING INFRASTRUCTURE, CARING FOR STAFF, ENCOURAGING PROFESSIONALISM AND PROFESSIONAL IMPROVEMENT

**CONSISTENT**
**SERVICE **
**&  SYSTEM**
**ARCHITECTURE**

**SUCCESS**
**FOR**
**CLIENTS**
**AND**
**FACILITY**

**TECHNOLOGY**
**TRANSFER**

**RELIABLE ****SERVICE**

**TIMELY****INFORMATION**

**INNOVATIVE ****ASSISTANCE**

**LARGE**
**SCALE**
**LEADER**

**CHANGE**

**RESEARCH FLOW**

**STAFF ****EFFECTIVENESS**

**NEW**
**TECHNOLOGY**

**WISE**
**INTEGRATION**

** ****MISSION**

**NEW**
**TECHNOLOGY**

## Major AccomplishmentsSince last meeting (June 2000)

- IBM SP placed into full service April 4, 2000 – more later
- Augmented the allocations by 1 M Hours in FY 2000
- Contributed to 11M PE hours in FY 2000 – more than doubling the FY 2000 allocation
- SP is fully utilized
- Moved entire facility to Oakland – more later
- Completed the second PAC allocation process with lessons learned from the first year

## Activities and Accomplishments

- Improved Mass Storage System
- Upgraded HPSS
- New versions of HSI
- Implementing Gigabit Ethernet
- Two STK robots added
- Replaced 3490  with 9840  tape drives
- Higher density and Higher speed tape drives
- Formed Network and Security Group
- Succeeded in external reviews
- Policy Board
- SCAC

## Activities and Accomplishments

- Implemented new accounting system – NIM
- Old system was
- Difficult  to maintain
- Difficult to integrate to new system
- Limited by 32 bits
- Not Y2K compliant
- New system
- Web focused
- Available data base software
- Works for any type of system
- Thrived in a state of increased security
- Open Model
- Audits, tests

## 2000 Activities and Accomplishments

- NERSC firmly established as a leader in system evaluation
- Effective System Performance (ESP) recognized as a major step in system evaluation and is influencing a number of sites and vendors
- Sustained System Performance measures
- Initiated a formal benchmarking effort to the NERSC Application Performance Simulation Suite (NAPs) to possibly be the next widely recognized parallel evaluation suite

## Activities and Accomplishments

- Formed the NERSC Cluster team to investigate the impact of SMP commodity clusters for High Performance, Parallel Computing to assure the most effective implementations of division resources related to cluster computing.
- Coordinate all NERSC Division cluster computing activities (research, development, advanced prototypes, pre-production, production, and user support).
- Initiated a formal procurement for mid-range cluster
- In consultation with DOE, decided not to award as part of NERSC program activities

## NERSC Division

**CENTER FOR BIOINFORMATICS**
**& COMPUTATIONAL GENOMICS**
*MANFRED ZORN*

**SCIENTIFIC DATA**
**MGMT RESEARCH**
*ARIE SHOSHANI*

**ADVANCED  SYSTEMS**
*TAMMY WELCOME*

**DIVISION  ADMINISTRATOR**
**& FINANCIAL MANAGER**
*WILLIAM FORTNEY*

**DISTRIBUTED SYSTEMS  DEPARTMENT**
*WILLIAM JOHNSTON*
- Department  Head
*DEB AGARWAL**, *Deputy

**HIGH PERFORMANCE**
**COMPUTING DEPARTMENT**
*WILLIAM KRAMER*
- Department  Head

**CHIEF TECHNOLOGIST**
*DAVID BAILEY*

**COMPUTATIONAL SYSTEMS**
*JIM CRAW*

**COMPUTER OPERATIONS & NETWORKING SUPPORT**
*WILLIAM HARRIS*

**MASS  STORAGE**
*NANCY MEYER*

**USER SERVICES**
*FRANCESCA VERDIER*

**FUTURE INFRASTRUCTURE NETWORKING & SECURITY**
*HOWARD WALTER*

**HENP  COMPUTING**
*DAVID QUARRIE*

**HIGH PERFORMANCE**
**COMPUTING RESEARCH DEPARTMENT**
*ROBERT LUCAS*
- Department  Head

**APPLIED NUMERICAL ALGORITHMS**
*PHIL COLELLA*

**CENTER FOR COMPUTATIONAL SCIENCE & ENGR.**
*JOHN BELL*

**FUTURE**
** ****TECHNOLOGIES **
** ***ROBERT LUCAS *(acting)

**IMAGING & COLLABORATIVE COMPUTING**
*BAHRAM PARVIN*

**SCIENTIFIC  **
**COMPUTING**
*ESMOND  NG*

**VISUALIZATION**
*ROBERT LUCAS *(acting)

**SCIENTIFIC  DATA**
**MANAGEMENT**
*ARIE SHOSHANI*

**COLLABORATORIES**
*DEB AGARWAL*

**DATA INTENSIVE DIST.**
**COMPUTING**
*BRIAN TIERNEY (CERN)*
*WILLIAM JOHNSTON (*acting)

**DISTRIBUTED SECURITY RESEARCH**
*MARY THOMPSON*

**NETWORKING**
*WILLIAM JOHNSTON** *
- (acting)

- Rev: 02/01/01

**NERSC Division**

## HIGH PERFORMANCE

**USER  SERVICES**
*FRANCESCA *
*VERDIER*
- Mikhail Avrekh
- Harsh Anand
- Majdi Baddourah
- Jonathan Carter
- Tom DeBoni
- Jed Donnelley
- Therese Enright
- Richard Gerber
- Frank Hale
- John McCarthy
- R.K. Owen
- Iwona Sakrejda
- David Skinner
- Michael Stewart (C)
- David Turner
- Karen Zukor

**HIGH PERFORMANCE**
**COMPUTING DEPARTMENT**
*WILLIAM KRAMER*
- Department Head

- Rev: 02/01/01

**ADVANCED**
**SYSTEMS**
*TAMMY *
*WELCOME*
- Greg Butler
- Thomas Davis
- Adrian Wong

**COMPUTATIONAL**
**SYSTEMS**
*JAMES CRAW*
- Terrence Brewer (C)
- Scott Burrow (I)
- Tina Butler
- Shane Canon
- Nicholas Cardo
- Stephan Chan
- William Contento (C)
- Bryan Hardy (C)
- Stephen Luzmoor (C)
- Ron Mertes (I)
- Kenneth Okikawa
- David Paul
- Robert Thurman (C)
- Cary Whitney

**COMPUTER OPERATIONS  & NETWORKING SUPPORT**
*WILLIAM HARRIS*
- Clayton Bagwell Jr.
- Elizabeth Bautista
- Richard Beard
- Del Black
- Aaron Garrett
- Mark Heer
- Russell Huie
- Ian Kaufman
- Yulok Lam
- Steven Lowe
- Anita Newkirk
- Robert Neylan
- Alex Ubungen

**MASS   STORAGE**
* **NANCY MEYER*
- Harvard Holmes
- Wayne Hurlbert
- Nancy Johnston
- Rick Un (V)

**(C)  Cray  (FB) Faculty UC Berkeley  (FD) Faculty UC Davis  (G) Graduate Student Research Assistant  (I) IBM  (M) Mathematical Sciences Research Institute  (MS) Masters Student  (P) Postdoctoral Researcher  (SA) Student Assistant  (V) Visitor  *  On leave to CERN**

**HENP  **
**COMPUTING**
*DAVID QUARRIE*
**  CRAIG TULL** *(Deputy)
- Paolo Calafiura
- Christopher Day
- Igor Gaponenko
- Charles Leggett (P)
- Massimo Marino
- Akbar Mokhtarani
- Simon Patton

**FUTURE INFRASTRUCTURE NETWORKING  & SECURITY**
*HOWARD WALTER*
- Eli Dart
- Brent Draney
- Stephen Lau

**APPLIED NUMERICAL ALGORITHMS**
*PHILLIP COLELLA*
- Susan Graham (FB)       Anton Kast                    Peter McCorquodale (P)       Brian Van Straalen
- Daniel Graves                Daniel Martin (P)          Greg Miller (FD)

**CENTER FOR BIOINFORMATICS  &  COMPUTATIONAL GENOMICS**
*MANFRED ZORN*
- Donn Davy                      Inna Dubchak                * Sylvia Spengler

**SCIENTIFIC COMPUTING**
*ESMOND NG*
- Julian Borrill	 Xiaofeng He (V)              Jodi Lamoureux (P)         Lin-Wang Wang
- Andrew Canning          Yun He 	      Sherry Li                          Michael Wehner (V)
- Chris Ding                    Parry Husbands (P)         Osni Marques	           Chao Yang
- Tony Drummond 	  Niels Jensen (FD)           Peter Nugent                     Woo-Sun Yang (P)

**CENTER FOR COMPUTATIONAL SCIENCE  &  ENGINEERING**
*JOHN BELL*
- Ann Almgren                   William Crutchfield       Michael Lijewski           Charles Rendleman
- Vincent Beckner              Marcus Day

**FUTURE TECHNOLOGIES**
*ROBERT LUCAS (acting)*
- David Culler (FB)              Paul Hargrove              Eric Roman                   Michael Welcome
- James Demmel (FB)          Leonid Oliker               Erich Stromeier            Katherine Yelick (FB)

**VISUALIZATION**
*ROBERT LUCAS (acting)*
- Edward Bethel                  James Hoffman (M)            Terry Ligocki             Soon Tee Teoh (G)
- James Chen (G)                David Hoffman (M)             John Shalf                 Gunther Weber (G)
- Bernd Hamann (FD)         Oliver Kreylos (G)

**IMAGING  &  COLLABORATIVE COMPUTING**
*BAHRAM PARVIN*
- Hui H Chan (MS)          Gerald Fontenay                   Sonia Sachs             Qing Yang
- Ge Cong (V)                  Masoud Nikravesh (V)         John Taylor

**SCIENTIFIC DATA MANAGEMENT**
*ARIE SHOSHANI*
- Carl Anderson                 Andreas Mueller              Ekow Etoo                      M. Shinkarsky (SA)
- Mary Anderson               Vijaya Natarajan              Elaheh Pourabbas (V)     Alexander Sim
- Junmin Gu                       Frank Olken                     Arie Segev (FB)             John Wu
- Jinbaek Kim (G)

**HIGH PERFORMANCE COMPUTING RESEARCH DEPARTMENT**
*ROBERT LUCAS*
*Department Head*

**                              ****(FB) Faculty UC Berkeley  (FD) Faculty UC Davis  (G) Graduate Student Research Assistant  (M) Mathematical Sciences Research Institute  (MS) Masters Student **
**                              ****(P) Postdoctoral Researcher  (S) SGI  (SA) Student Assistant  (V) Visitor   * Life Sciences Div. – On Assignment to NSF**

- Rev: 02/01/01

## FY00 MPP Users/Usage by Discipline

## FY00 PVP Users/Usage by Discipline

## NERSC FY00 MPP Usage by Site

## NERSC FY00 PVP Usage by Site

## FY00 MPP Users/Usage by Institution Type

## FY00 PVP Users/Usage by Institution Type

## NERSC System Architecture

- VIS LAB

- MILLENNIUM

- SYMBOLIC
- MANIPULATION
- SERVER

- REMOTE
- VISUALIZATION
- SERVER

- CRI T3E
- 900 644/256

- IBM SP
- NERSC-3
- Processors 604/304 Gigabyte
- memory

- CRI SV1

**FDDI/**
**ETHERNET**
**10/100/Gigbit**

- MAX
- STRAT

- IBM
- And STK
- Robots

- DPSS

- PDSF

**NERSC System Architecture**

- ResearchCluster

- IBM SP
- NERSC-3 – Phase 2a
- 2532 Processors/ 1824 Gigabyte
- memory

- LBNL Cluster

## Current Systems

| System | Description |
| --- | --- |
| IBM SP RS/6000 | NERSC-3/Phase 2a – Seaborg – a 2532 processor systems using 16 CPU SMP Nodes, with the “Colony” Double/Double Switch.  Peak performance is ~3.8 Tflop/s.  12 GB memory per computational node, 20 TB of usable globally accessible parallel disk.
NERSC-3/Phase 1 - Gseaborg- a 608-processor IBM RS/6000 SP system, with a peak performance of  410 gigaflop/s, 256 gigabytes of memory, and 10 terabytes of disk storage.  NERSC Production Macine |
| Cray T3E | NERSC-2 - Mcurie- a 696-processor, MPP system with a peak speed of 575 gigaflop/s, 256 megabytes of memory per processor, and 1.5 terabytes of disk storage.   A peak CPU performance of 900 MFlops per processor.  NERSC production machine. |
| Cray Vector Systems | NERSC-2 - Three Cray SV1 machines.  A total of 96 vector processors in the cluster, 4 gigawords of memory, and a peak performance of 83 gigaflop/s.  The SV-1  (Killeen) is used for interactive computing. The remaining two SV1 machines (Bhaskara and Franklin) are batch-only machines. |
| HPSS – Highly Parallel Storage System | HPSS is a modern, flexible, performance-oriented mass storage system, designed and developed by a consortium of government and commercial entities. It is deployed at a number of sites and centers, and is used at NERSC for archival storage. |
| PDSF - Parallel Distributed Systems Facility | The PDSF is a networked distributed computing environment -- a cluster of workstations -- used by six large-scale high energy and nuclear physics investigations for detector simulation, data analysis, and software development. The PDSF includes 281 processors in  nodes for computing, and eight disk vaults with file servers for 7.5 TB of storage. |
| PC Cluster Projects | A 36-node PC Cluster Project Testbed, which is available to NERSC users for trial use; a 12-node Alpha "Babel" cluster, which is being used for Modular Virtual Interface Architecture (M-VIA) development and Berkeley Lab collaborations.
Associated with the LBNL 160 CPU (80 node) cluster system for mid range computing.  Myrinet 2000 interconnect, 1 GB of memory per node, and 1 TB of shared, globally accessible disk. |

## Major Systems

- MPP
- IBM SP – Phase 2a
- 158 16-way SMP nodes
- 2144 Parallel Application CPUs/12 GB per node
- 20 TB Shared GPFS
- 11,712 GB swap space - local to nodes
- ~8.6 TB of temporary scratch space
- 7.7 TB of permanent home space
- 4-20 GB home quotas
- ~240 Mbps aggregate I/O - measured from user nodes (6 HiPPI, 2 GE, 1 ATM)
- T3E-900 LC with 696 PEs - UNICOS/mk
- 644 Application Pes/256 MB per PE
- 383 GB of Swap Space - 582 GB Checkpoint File System
- 1.5 TB /usr/tmp temporary scratch space - 1 TB permanent home space
- 7- 25 GB home quotas, DMF managed
- ~ 35 MBps aggregate I/O measured from user nodes - (2 HiPPI, 2 FDDI)
- 1.0 TB local /usr/tmp

- Serial
- PVP - Three J90 SV-1 Systems running UNICOS
- 64 CPUs Total/8 GB of Memory per System (24 GB total)
- 1.0 TB local /usr/tmp
- PDSF - Linux Cluster
- 281 IA-32 CPUs
- 3 LINUX and 3 Solaris file servers
- DPSS integration
- 7.5 TB aggregate disk space
- 4 striped fast Ethernet connections to HPSS
- LBNL – Mid Range Cluster
- 160 IA-32 CPUs
- LINUX with enhancements
- 1 TB aggregate disk space
- Myrinet 2000 Interconnect
- GigaEthernet connections to HPSS

- Storage
- HPSS
- 8 STK Tape Libraries
- 3490 Tape drives
- 7.4 TB of cache disk
- 20 HiPPI Interconnects, 12 FDDI connects, 2 GE connections
- Total Capacity ~ 960TB
- ~160 TB in use
- HPSS - Probe

## T3E Utilization95% Gross Utilization

- Systems Merged

- Allocation
- Starvation

- Allocation
- Starvation

- Checkpoint
- t - Start of
- Capability Jobs

- Full
- Scheduling
- Functionality

- 4.4% improvement
- per month

## SP Utilization

- In the 80-85% range which is above original expectations for first year
- More variation than T3E

## T3E Job Size

- More than 70% of the jobs are “large”

**T3E Job Size**

## SP Job Size

- ~ 60% of the jobs are > 1⁄4 the maximum size

- Full size jobs more than 10% of usage

**Notes:**
SP shows very similar “capability” profile

## Storage:HPSS

## NERSC Network Architecture

**NERSC Network Architecture**

## CONTINUE NETWORK IMPROVEMENTS

**CONTINUE NETWORK ****IMPROVEMENTS**

## LBNL Oakland Scientific Facility

**LBNL Oakland Scientific Facility**

## Oakland Facility

- 20,000 sf computer room; 7,000 sf office space
- 16,000 sf computer space built out
- NERSC occupying 12,000 sf
- Ten year lease with 3 five year options
- $10.5M computer room construction costs
- Option for additional 20,000+ sf computer room

## LBNL Oakland Scientific Facility

**LBNL Oakland Scientific Facility**

- Move accomplished between
- Oct 26 to Nov 4

- System		Scheduled	Actual
- SP		10/27 – 9 am	no outage
- T3E		11/3 – 10 am	11/3 – 3 am
- SV1’s		11/3 – 10 am	11/2 – 3 pm
- HPSS		11/3 – 10 am	10/31 – 9:30 am
- PDSF		11/6 – 10 am	11/2 – 11 am
- Other Systems	11/3 – 10 am	11/1 – 8 am

## Computer Room Layout 

- Up to 20,000 sf of computer space
- Direct Esnet node at OC12

## 2000 Activities and Accomplishments

- PDSF Upgrade in conjunction with building move

## 2000 Activities and Accomplishments

- netCDF  parallel support developed by NERSC staff for the Cray T3E.
- A similar effort is being planned to port netCDF to the IBM SP platform.
- Communication for Clusters: M-VIA and MVICH
- M-VIA and MVICH are VIA-based software for low-latency, high-bandwidth, inter-process communication.
- M-VIA is a modular implementation of the VIA standard for Linux.
- MVICH is an MPICH-based implementation of MPI for VIA.

## FY 2000 User Survey Results

- Areas of most importance to users
- available hardware (cycles)
- overall running of the center
- network access to NERSC
- allocations process
- Highest satisfaction (score > 6.4)
- Problem reporting/consulting services (timely response, quality, followup)
- Training
- Uptime (SP and T3E)
- FORTRAN (T3E and PVP)
- Lowest satisfaction (score < 4.5)
- PVP batch wait time
- T3E batch wait time
- Largest increases in satisfaction from FY 1999
- PVP cluster (we introduced interactive SV1 services)
- HPSS performance
- Hardware management and configuration (we monitor and improve this continuously)
- HPCF website (all areas are continuously improved, with a special focus on topics highlights as needing improvement in the surveys)
- T3E Fortran compilers

## Client Comments from Survey

- "Very responsive consulting staff that makes the user feel that his problem, and its solution, is important to NERSC"
- "Provide excellent computing resources with high reliability and ease of use."
- "The announcement managing and web-support is very professional."
- "Manages large simulations and data. The oodles of scratch space on mcurie and gseaborg help me process large amounts of data in one go."
- "NERSC has been the most stable supercomputer center in the country particularly with the migration from the T3E to the IBM SP".
- "Makes supercomputing easy."

## NERSC 3 Phase 2a/b

## Result: NERSC-3 Phase 2a

- System built and configured
- Started factory tests 12/13
- Expect delivery 1/5
- Undergoing acceptance testing
- General production April 2001
- What is different that needs testing
- New Processors,
- 	New Nodes, New memory system
- New switch fabric
- New Operating System
- New parallel file system software

## IBM Configuration

**	****Phase 1****	****Phase 2a/b**
**Compute Nodes****	****256****	****134***
**Processors****	****256x2=512****	****134x16=2144***
**Networking Nodes****	****8****	****2****	**
**Interactive Nodes****	****8****	****2**
**GPFS Nodes ****	****16****	****16**
**Service Nodes****	****16****	****4**
**Total Nodes (CPUs)****	****304 (604)****	****158 (2528)**
**Total Memory (compute nodes)****	****256 GB****	****1.6 TB**
**Total Global Disk ****(user accessible)****	****10 TB****	****20 TB**
**Peak (compute nodes)****	****409.6 GF****	****3.2 TF***
**Peak (all nodes)****	****486.4 GF****	****3.8 TF***
**Sustained System Perf ****	****33    GF****	****235+ GF/280+ GF**
**Production Dates****	****April 1999****	****April 2001/Oct 2001**
***is a minimum - may increase due to sustained system performance measure**

## What has been completed

- 6 nodes added to the configuration
- Memory per node increased to 12 GB for 140 compute nodes
- “Loan” of full memory for Phase 2
- System installed and braced
- Switch adapters and memory added to system
- System configuration
- Security Audit
- System testing for many functions
- Benchmarks being run and problems being diagnosed

## Current Issues

- Failure of two benchmarks need to be resolved
- Best case: indicate broken hardware – likely with the switch adaptors
- Worst case: indicate design and load issues that are fundamental
- Variation
- Loading and switch contention
- Remaining tests
- Throughput, ESP
- Full System tests
- I/O
- Functionality

## General Schedule

- Complete testing – TBD based on problem correction
- Production Configuration set up
- 3rd party s/w, local tools, queues, etc
- Availability Test
- Add early users ~10 days after successful testing complete
- Gradually add other users – complete ~ 40 days after successful testing
- Shut down Phase 1 ~ 10 days after system open to all users
- Move 10 TB of disk space – configuration will require Phase 2 downtime
- Upgrade to Phase 2b in late summer, early fall

## NERSC-3 Sustained System Performance Projections

- Estimates the amount scientific computation that can really be delivered
- Depends on delivery of Phase 2b functionality
- The higher the last number is the better since the system remains at NERSC for 4 more years

**Software lags hardware**

**Test/Config, Acceptance,etc**

## NERSC Computational Power vs. Moore’s Law

## NERSC 4

## NERSC-4

- NERSC 4 IS ALREADY ON OUR MINDS
- PLAN IS FOR FY 2003 INSTALLATION
- PROCUREMENT PLANS BEING FORMULATED
- EXPERIMENTATION AND EVALUATION OF VENDORS IS STARTING
- ESP, ARCHITECTURES, BRIEFINGS
- CLUSTER EVALUATION EFFORTS
- USER REQUIREMENTS DOCUMENT (GREENBOOK) IMPORTANT

## How Big Can NERSC-4 be

- Assume a delivery in FY 2003
- Assume no other space is used in Oakland until NERSC-4
- Assume cost is not an issue (at least for now)
- Assume technology still progresses
- ASCI will have a 30 Tflop/s system running for over 2 years

## How close is 100 Tflop/s

- Available gross space in Oakland is 3,000 sf without major changes
- Assume it is 70% usable
- The rest goes to air handlers, columns, etc.
- That gives 3,000 sf of space for racks
- IBM system used for estimates
- Other vendors are similar
- Each processor is 1.5 Ghz, to yield 6  Gflop/s
- An SMP node is made up of 32 processors
- 2 Nodes in a frame
- 64 processors in a frame = 384 Gflops per frame.
- Frames are 32 - 36" wide and 48” deep
- service clearance of 3 feet in front and back (which can overlap)
- 3 by 7 is 21 sf per frame

## Practical System Peak 

- Rack Distribution
- 60% of racks are for CPUs
- 90% are user/computation nodes
- 10% are system support nodes
- 20 % of racks are for switch fabric
- 20% of racks for disks
- 5,400 sf / 21 sf per frames = 257 frames
- 277 nodes that are are directly used by computation
- 8,870 CPUS for computation
- system total is 9,856 (308 nodes)
- Practical system peak is  53 Tflop/s
- .192 Tflop/s per node * 277 nodes
- Some other places would claim 60 Tflop/s

## How much use will it be

- Sustained vs Peak performance
- Class A codes on T3E samples at 11%
- LSMS
- 44% of peak on T3E
- So far 60% of peak on Phase 2a (maybe more)
- Efficiency
- T3E runs at a 30 day average about 95%
- SP runs at a 30 day average about 80+%
- Still functionality planned

## How much will it cost

- Current cost for a balanced system is about $7.8M per Tflop/s
- Aggressive
- Cost should drop by a factor of 4
- $1-2 M per Teraflop/s
- Many assumptions
- Conservative
- $3.5 M per Teraflop/s
- Added costs for install, operate and balance the facility is 20%.
- The full cost is $140 M to $250 M
- Too Bad

## The Real Strategy

- Traditional strategy within existing NERSC Program funding. Acquire new computational capability every three years - 3 to 4 times capability increase of existing systems
- Early, commercial, balanced systems with focus on
- 		- stable programming environment
- 	- mature system management tools
- 	- good sustained to peak performance ratio
- Total value of $25M - $30M
- 		- About $9-10M/yr. using lease to own
- Have two generations in service at a time
- 		- e.g. T3E and IBM SP
- Phased introduction if technology indicates
- Balance other system architecture components

## Necessary Steps

-  1) Accumulate and evaluate benchmark candidates
-  2) Create a draft benchmark suite and run it on several systems
-  3) Create the draft benchmark rules
-  4) Set basic goals and options for procurement and then create a draft RFP document
-  5) Conduct market surveys (vendor briefings, intelligence gathering, etc.) - we do this after the first items so we can be looking for the right information and also we can tell the vendors what to expect.  It is often the case that we have to "market" to the vendors on why they should be bidding - since it costs them a lot.
-  6) Evaluate alternative and options for RFP and tests.  This is also where we do a technology schedule (when what is available) and estimate prices - price/performance, etc.
-  7) Refine RFP and benchmark rules for final release
-  8) Go thru reviews
-  9) Release RFP
- 10) Answer questions from vendors
- 11) Get responses - evaluate
- 12) Determine best value - present results and get concurrence
- 13) Prepare to negotiate
- 14) Negotiate
- 15) Put contract package together
- 16) Get concurrence and approval
- 17) Vendor builds the system
- 18) Factory test
- 19) Vendor delivers it
- 20) Acceptance testing (and resolving issues found in testing)- (1st payment is 2 months after acceptance)
- 21) Preparation for production
- 22) Production

## Rough Schedule

- Goal – NERSC-4 installation in first half of CY 2003
- Vendor responses (#11) in early CY 2002
- Award in late summer/fall of CY2002.
- This is necessary in order to assure delivery and acceptance (# 22) in FY 2003.
- A lot of work and long lead times (for example, we have to account for review and approval times, 90 days  for vendors to craft responses, time to negotiate, ...)
- NERSC Staff kick off meeting first week of march,
-  Been doing some planning work already.

## NERSC-2 Decommissioning

- RETIRING NERSC 2 IS ALREADY ON OUR MINDS
- IF POSSIBLE WE WOULD LIKE TO KEEP NERSC 2 IN SERVICE UNTIL 6 MONTHS BEFORE NERSC 4 INSTALLATION
- Therefore, expect retirement at the end of FY 2002
- It is “risky” to assume there will be a viable vector replacement
- Team is working to determine possible paths for traditional vector users
- Report due in early summer

## SUMMARY

- NERSC does an exceptionally effective job delivering services to DOE and other researchers
- NERSC has made significant upgrades this year that position it well for future growth and continued excellence
- NERSC has a well mapped strategy for the next several years