# Presentation: 697545

## NERSC PVP Roadmap

***Jonathan Carter***
**NERSC User Services**
** ****jcarter@nersc.gov**
**510 486-7514**

## NERSC

- The Department of Energy, Office of Science Supercomputer Facility
- Unclassified, open facility; serving over 2000 users in all scientific disciplines relevant to the DOE mission
- 25th anniversary in 1999

## NERSC FY00 PVP Usage by Site

## NERSC System Architecture

- VIS LAB

- MILLENNIUM

- SYMBOLIC
- MANIPULATION
- SERVER

- REMOTE
- VISUALIZATION
- SERVER

- CRI T3E
- 900 644/256

- IBM SP
- NERSC-3
- Processors 604/304 Gigabyte
- memory

- CRI SV1

**FDDI/**
**ETHERNET**
**10/100/Gigbit**

- MAX
- STRAT

- IBM
- And STK
- Robots

- DPSS

- PDSF

**NERSC System Architecture**

- ResearchCluster

- IBM SP
- NERSC-3 – Phase 2a
- 2532 Processors/ 1824 Gigabyte
- memory

- LBNL Cluster

## Current Systems

## Transition to MPP

- Over the past 4 years, NERSC has completed a paradigm shift to Massively Parallel production computing
- 1996 - 100% of production computing resources were vector SMP
- 2001 - less than 15% of production computing resources are vector SMP
- The shift was completed in FY 00 when all major projects had more MPP time than vector time.

## FY00 MPP Users/Usage by Discipline

## FY00 PVP Users/Usage by Discipline

## Processor Design as Seen in the TOP500

## Vector SMP as seen in TOP500

## NERSC 2 Decommissioning

- 3 year procurement cycle
- Keep two generations in service
- NERSC 4 planned for early 2003
- If possible we would like to keep NERSC 2 in service until 6 months before NERSC 4 installation
- Expect retirement at the end of FY 2002

## Cray PVP Systems

- The line of Unicos PVP machines started with the XMP is coming to an end with the SV1ex
- SV2 inherits more of Unicos/mk and IRIX  architecture than Unicos architecture
- NERSC is actively involved in the providing requirements for the SV2

## PVP Roadmap Project

- Contact largest users from each office to characterize use of the Cray SV1 cluster. The responses so far can be grouped:
- amount of memory or disk
- shared-memory multi-processing
- software costs to go to a distributed-memory programming model would be prohibitive
- job management issues: better interactive availability, longer queues
- algorithms most suited to vector architecture
- dependency on Cray or NERSC software (compilers, libraries, etc.) only available on PVP

## SMP Clusters

- A possible solution to the first three points is to use individual nodes of an SMP cluster as a replacement
- Upcoming technology is equal to, or exceeds, memory, disk available on Cray SV1 on a per node basis
- Phase 2A IBM SP has 12GB memory per node, 16 375MHz Power3 CPU
- We intend to experiment with ‘PVP-like’ jobs on certain nodes of new SP

## 10 - 100 Tflop/s Cluster of SMPs

- The first ones are already being delivered
-  LLNL is installing a 10 Tflop/s now
-  LANL just ordered a 30 Tflop/s system
- Systems are large clusters
-  SMP nodes in US
-  Vector nodes in Japan
- Programming model:
-  ‘Shared-memory’ MPI, OpenMP, and/or vectors on node
-  MPI for global communication

## IBM SMP Roadmap

- Power3+
- 375 MHz

- L1 cache 64KB

- 8 GB/s

- 1.6 GB/s

- Power4
- 2 x >1 GHz

- L1 cache 2 x 32KB

- >100 GB/s

## Job Management

- MPP systems dictated a different style of management than PVP
- Due to lack or overhead of gang-scheduling,and somewhat more instability:
- queues are generally shorter,
- interactive use more limited
- With an SMP cluster, could manage some nodes in a similar style to PVP

## Vector Architecture

- Gap is closing between commodity processors and custom vector processors
- Undertake a short study to benchmark a selection of PVP codes, vector vs. latest scalar technology

## Software Migration Issues

- Legacy software with Cray Fortran extensions, Cray library calls, or assembly language
- some vendors provide compiler flags, wrappers to other libraries which mimic Cray
- UNICOS unformatted data
- currently in discussions with IBM
- CTSS file conversion routines
- Unsupported libraries, tv80, graflib, baselib