# Presentation: 951183

## NERSC Overview

**Richard Gerber**
**National Energy Research Scientific Computing Center**
**Lawrence Berkeley National Laboratory**
**NERSC User Services**
**ragerber@lbl.gov**
**Princeton Plasma Physics Laboratory**
**Geophysical Fluid Dynamics Laboratory**
**November 19-20, 2007**

## Credits

**Some slides and ideas taken from:**
**Jim Craw, Systems Group Leader**
**Jason Hick, Storage Group Leader**
**Helen He, User Services Franklin POC**
**Katie Antypas, User Services (Debuggers)**
**Shreyas Cholia, Grid Services**
**Luis DeRose and John Levesque, Cray Research**

## Outline

**NERSC Systems Overview**
**Franklin Overview**
**Franklin Programming Environment**
**Running Jobs on Franklin**
**File Transfer Options**
**Debugging with DDT**
**Seaborg Decommission**

## NERSC Systems 2007

**ETHERNET**
**10/100/1,000 Megabit**

- FC Disk

- STK
- Robots

**HPSS**
**100 TB of cache disk**
**8 STK robots, 44,000 tape slots, **
**max capacity 44 PB**

**PDSF****~1,000  processors ****~1.5 TF, 1.2 TB of Memory**
**~300 TB of Shared Disk****Ratio = **_**(0.8, 20)**_

**Ratio = (RAM Bytes per Flop, Disk Bytes per Flop)**

- Testbeds and servers

**Visualization and Post Processing Server- davinci**
**64  Processors**
**.4 TB Memory**
**60 Terabytes Disk**

**NCS-b – Bassi**
**976 Power 5+ CPUs**
**SSP5 - ~.83 Tflop/s**
**6.7 TF, 4 TB Memory**
**70 TB disk**
**Ratio = (0.5, 9)**

**NERSC Global File System**
**300 TB shared usable disk**

**Storage**** Fabric**

**OC 192 – 10,000 Mbps**

**IBM SP**
**NERSC-3 – “Seaborg”**
**6,656 Processors (Peak 10 TFlop/s)**
**SSP5 – .98 Tflop/s **
**7.8 Terabyte Memory**
**55 Terabytes of Shared Disk**** Ratio = **_**(0.8,4.8)**_

**10 Gigabit,**
** ****Jumbo 10 Gigabit ****Ethernet**

**NCS-a Cluster – “jacquard”**
**650 CPU **
**Opteron/Infiniband 4X/12X **
**3.1 TF/ 1.2 TB memory**
**SSP - .41 Tflop/s**
**30 TB Disk**
**Ratio = (.4,10)**

**Cray XT4**
**NERSC-5 - “Franklin”**
**SSP ~16 Tflop/s**

## NERSC Systems

| Name | Type | Cores
SMP Sz | Total Mem 
Mem/Core | Peak TFlop/s | SSP TFlop/s |
| --- | --- | --- | --- | --- | --- |
| Franklin | Cray XT Opteron 2.6 GHz | 19,320
2 | 39 TB
1.8 GB | 101.5 | 16.1 |
| Seaborg | IBM SP
P3 375 MHz | 6,080
16 | 7.2 TB
1.2 GB | 10  | .9 |
| Bassi | IBM p575
P5 1.9 GHz | 888
8 | 3.6 TB
4 GB | 7.4 | .85 |
| Jacquard | LNXI Opteron
2.2 GHz | 712
2 | 2.1 TB
3 GB | 3.1 |  |
| DaVinci | SGI Altix Itanium 1.4 GHz | 32
32 | 192 GB
6 GB | 0.179 |  |
| PDSF | Various Linux | 550 | 550 GB
1 GB |  |  |

## Archival Storage (HPSS)

**Archival Storage (HPSS)**

**3.5 Petabytes stored**
**61 Million Files**
**Disk Cache 100 TB**
**3.2 GB/sec max transfer**
**User system: “archive”**
**Backups system: “hpss”**
**hsi command line interface**

## NERSC Global File System (NGF)

**Shared file system available from NERSC compute platforms**
**/project/projectdirs/your_project created and allocated upon request**
**150 TB currently available**
**Default project quota of 1 TB**
**Use to share files among members of your group**

## Outline

**NERSC Systems Overview**
**Franklin Overview**
**Franklin Programming Environment**
**Running Jobs on Franklin**
**File Transfer Options**
**Debugging with DDT**
**Seaborg Decommission**

## Franklin

- Benjamin Franklin,
- one of America’s first scientists, performed ground breaking work in energy efficiency, electricity, materials, climate, ocean currents, transportation, health, medicine, acoustics and heat transfer.

## About Franklin

**Cray XT4, No. 9 in Top 500, large XT4**
**9,740 dual-core nodes (19,480 cores)**
**AMD Opteron 2.6 GHz, 5.2 GFlops/sec peak**
**102 node cabinets **
**16 KWs per cabinet (~1.7 MWs total)**
**39.5 TBs aggregate memory**
**16.1+ Tflop/s Sustained System Performance (SSP)**
**>16 times that of Seaborg or Bassi        **** **
**101.5 Tflop/s theoretical system peak performance**
**Cray SeaStar2 / 3D Torus interconnect (17x24x24)**
**~366 TBs of usable shared disk**** **

## XT4 Architecture

## Cray XT4 Compute Blade

**Cray XT4 Compute Blade**

**4 DIMM Slots****with Chipkill**

**CRAY**
*SeaStarTM*

**CRAY**
*SeaStarTM*

**CRAY**
*SeaStarTM*

**CRAY**
*SeaStarTM*

**Redundant VRMs**

**Blade Backplane Connector (>100 GB/sec)**

**Notes:**
This is the basic compute module.
There are 4 independent PEs on this module, each connected via traces in the board and HyperTransport to a Seastar.
One question that could be discussed here is the ability of the Opteron to run as a glueless SMP.  That is, we could design traces between the 4 Opterons to carry coherent HT traffic and make a 4-way SMP.  Why didn’t we do this?
Two reasons, price and performance.
The 800 series Opterons cost about 3X more than the 100 series Opterons we use in Red Storm.
In addition, the memory latency on the SMP parts is longer than on the 100 series parts.  The 100 series parts have a 58 ns memory latency, 70 ns for the 200 (2-way) series parts and 102 ns for the 4-way parts.  This is a best case latency assuming zero hops for the SMP cases.  Every memory reference also initiates cache coherency snoop traffic on the SMP design. Overall, we believe that MPI work running on SMP nodes would lose about 20-25% of the performance and it could cost 20-25% more for the customer.  This leads to an overall price/performance drop of over 1.5X and we felt that the flexibility did not outweigh the sustained price/performance disadvantages of this approach.
We are the only MPP on the market, and the only solution in the market based on 100-series Opterons, the fastest X86 Chip made.

## Franklin Overview

- Compute
-  Node

- Compute
-  Node

- Compute
-  Node

- Compute
-  Node

- Compute
-  Node

- Compute
-  Node

- Compute
-  Node

- etc....

- No
- local
- disk

- Login
-  Node

- Login
-  Node

- Login
-  Node

- etc....

**/home**

- Login
-  Node

- Login
-  Node

**/scratch**

- Login
-  Node

- etc....

**/project**

**HPSS**

- Full Linux OS

- CNL (no logins)

## Outline

**NERSC Systems Overview**
**Franklin Overview**
**Franklin Programming Environment**
**Running Jobs on Franklin**
**File Transfer Options**
**Debugging with DDT**
**Seaborg Decommission**

## Compute Node Linux (CNL)

**Most HPC systems run a full OS on all nodes.**
**To increase performance you need to minimize the effect of the OS on the running of your application.**
**CNL is a lightweight operating system **
**CNL can be considered as a full Linux operating system with components that increase the OS interventions removed.**

## CNL Limitations

**CNL has the following features missing:**
**NFS – you cannot launch jobs from an NFS mounted directory (e.g., /usr/common, /project on Franklin)**
**Dynamic libraries**
**A number of services may not be available **
**If you are not sure if something is supported, try man pages, e.g.:**

- > man mmap
- NAME
-        mmap, munmap - map or unmap files or devices into memory
- IMPLEMENTATION
-        UNICOS/lc  operating  system  -  **not supported on Cray XT series**
**       ****compute nodes**

## Cray XT4 programming environment overview 

**PGI compiler suite (default)**
**Pathscale compiler suite (limited number of licenses)**
**Optimized libraries:**
**     ****- 64 bit AMD Core Math library (ACML): Level 1,2,3 of BLAS, LAPACK, FFT**
**     ****- SciLib: Scalapack, BLACS, SuperLU  **
**MPI-2 message passing library for communication between nodes**
**     ****(derived from MPICH-2, implements MPI-2 standard, except for support of dynamic process functions) **
**SHMEM one-sided communication library**

**Notes:**
ScaLAPACK  contains parallel versions of a set of LAPACK (Linear Algebra)routines.
The BLACS package is a set of communication routines used by ScaLAPACK.
The SuperLU library is designed to solve large, sparse nonsymmetric systems of linear equations.
Library written in C, but can also be called from Fortran programs.
See mpi limitations on page 22.

## Cray XT4 programming environment overview 

**GNU C library, gcc, g++**
**aprun command to launch jobs; similar to mpirun command. **
**Torque (PBS-like) batch system**
**Performance tools: CrayPat, Apprentice2**
**DDT debugger**

**Notes:**
ScaLAPACK  contains parallel versions of a set of LAPACK (Linear Algebra)routines.
The BLACS package is a set of communication routines used by ScaLAPACK.
The SuperLU library is designed to solve large, sparse nonsymmetric systems of linear equations.
Library written in C, but can also be called from Fortran programs.
See mpi limitations on page 22.

## Compiler drivers to create CNL executables

**Compiler drivers take care of loading appropriate libraries****(-lmpich, -lsci, -lacml, -lpapi)**
**Available drivers :**
**Fortran 90/95 programs      ftn**
**Fortran 77 programs           f77**
**C programs                          cc**
**C++ programs                     CC  **
**Cross compiling environment**
**Compiling on a Linux service node **
**Generating an executable for a CNL compute node**
**Do not use pgf90, pgcc unless you want a Linux executable for the service node**
**Information message:**
- 		ftn: INFO: linux target is being used

## Sample PGI Compiles

**Compile MPI code**
**% *****ftn mpi_program.f***
**% *****cc mpi_program.c***
**% *****CC mpi_program.C***
**Compile Fortran Shmem code**
**% *****ftn shmem_program.f*****  **
**Compile Fortran OpenMP code**
**% *****ftn -mp=nonuma  openmp_program.f***
**Compile Fortran Mixed MPI/OpenMP code**
**% *****ftn -mp=nonuma  mixed_program.f***

## Recommend Compiler Flags

**-fastsse  ****(NERSC recommends)**
**      ****implies       ****-O2 -Munroll=c:1 -Mnoframe –Mlre -Mvect=sse, -Mscalarsse, -Mcache_align –Mflushz**
**-Munroll=c specifies completely unroll loops with this loop count or less **
**-Mnoframe does not set up a stack frame `**
**-Mlre is loop-carried redundancy elimination **
**-Mvect=sse,-Mscalarsse optimized for SSE hardware and vectorization**
**-Mcache_align aligns the top level arrays and objects on cache-line boundaries **
**-Mflushz flushes SSE denormal numbers to zero **

## Outline

**NERSC Systems Overview**
**Franklin Overview**
**Franklin Programming Environment**
**Running Jobs on Franklin**
**Separate presentation**
**File Transfer Options**
**Debugging with DDT**
**Seaborg Decommission**

## Outline

**NERSC Systems Overview**
**Franklin Overview**
**Franklin Programming Environment**
**Running Jobs on Franklin**
**File Transfer Options**
**Debugging with DDT**
**Seaborg Decommission**

## bbcp

**Faster alternative to scp**
**Bulk data not encrypted**
**Auto resume on interrupted transfers**
**Availability**
**Franklin test version in /usr/common/osp/bin/bbcp**
**Currently only outbound**
**Jacquard and DaVinci in /usr/common/usg/bin/bbcp**
**Quick test**
** ****F->D 440 MB file: scp 20sec, bbcp 5 sec **
**Must use**
**[-S | -T]  "ssh -x -a -oFallBackToRsh=no %I -l %U %H path_to_bbcp”**

## HSI and HTAR

**HPSS command-line transfer utilities**
**HSI**
**FTP-like interface (but better and more powerful)**
**HTAR**
**Enhanced method of archiving tar files to HPSS**
**Does not need local space to stage tar file**
**Creates separate index file so you can query tar contents without retreiving tar files from HPSS**
**NERSC will build and give you binaries for your local machines upon request**
**Check to see if your platform/OS is already available at ****https://www.nersc.gov/projects/download/hpss/ **

## Changes to HPSS

**HPSS will be upgraded from version 5.1 to 6.2**
**Eliminates our dependence on DCE**
**Reduces need to take frequent planned outages**
**New HPSS account/password management process (details still under review)**
**No more ssh to auth.nersc.gov to change passwords**
**NERSC NIM/LDAP passwords will be used**
**We will continue to allow ftp access with encrypted string in .netrc file**
**Password expiration**
**Contact NERSC user services if HPSS password expiration would present significant challenges to you/your project.**

## NERSC Grid Services

**Globus Toolkit 4.0.3**
**GridFTP for data**
**GRAM for remote job submission and batch access.**
**GSI and X509 certificates for security.**
**Using OSG Software Stack based on the Virtual Data Toolkit (VDT).**

## NERSC Grid Services

**GRAM Gatekeeper**
**Allows you to run / submit, monitor and query jobs on any NERSC machine remotely.**
**Interfaces with various backend batch systems (PBS, Loadleveler, SGE) while presenting a uniform interface to the user (Globus RSL).**
**GridFTP**
**Provides high performance data movement between NERSC and remote site / client system. **
**Supports striped, third party and parallel transfers.**

## Certificate Services

**All Grid services at NERSC use X509 Grid certificates for authentication and access.**
**NERSC Users can acquire a grid certificate from the DOEGrids site (**_**[https://pki1.doegrids.org/ca/](https://pki1.doegrids.org/ca/)**_**).**
**User must register their certificate with NERSC in NIM.**
**Once the certificate has been approved in NIM, user can access any of the NERSC Grid Services with this.**

## Coming Soon...

**Grid Services on Franklin**
**Planned for early 2008**
**NERSC Online CA**
**Simplify the process of acquiring cert**
**User will be given a short lived cert based on LDAP password**
**MyProxy Server**
**Allows users to store their certificates in a secure fashion, and generate short term proxy certificates that can be accessed from anywhere**

## Vision for Grid Services

**To be able to submit, manage and coordinate jobs across NERSC systems (and across other grid sites) using a common interface directly from a client machine.**
**To make it easy to manage workflows across multiple systems / sites.**
**To be able to move data across sites using high performance parallel transfers. **

## Example

- HPSS

- Bassi

- NGF

- DaVinci

- Remote
- Web Server

- MyProxy

-  Get Certificate
-  Submit Job to Bassi
-  GridFTP to HPSS
-  Do analysis on DaVinci
-  Move results to web server

## Outline

**NERSC Systems Overview**
**Franklin Overview**
**Franklin Programming Environment**
**Running Jobs on Franklin**
**File Transfer Options**
**Debugging with DDT**
**Seaborg Decommission**

## DDT Debugger

**DDT (Distributed Debugging Tool)**
**Some HPC Customers**
**LLNL, TACC, Barcelona, Leibniz LRZ, Stuttgart HLRS, CEA (France), CINECA (Italy), RAL (UK), etc.**
**Spring 2007 tested DDT on NERSC platforms**
**Low learning curve for Totalview users**
**Basic debugging functionality worked as expected**
**Found some bugs; responsive developers**
**Viable alternative to Totalview**
**Available on Franklin in the “ddt” module**
**https://www.nersc.gov/nusers/resources/franklin/software/ddt.php**

## DDT Licensing Model and Price

**Flexible model**
**1024 processors**
**Can be divided any way**
**One 1024 processor job**
**Two 512 processor jobs**
**One 512, one 256, four 64 processor jobs**
**Significantly cheaper than Totalview**

## DDT Functionality

**Parallel Debugger**
**Support for MPI, OpenMP, pthreads**
**Fortran, C, C++**
**Typical serial debugging features **
**set breakpoints and watches, step through program, dive into arrays, evaluate expressions, analyze core files**
**Parallel debugging features**
**Step through processors**
**View variables across processors**
**Grouping processors Parallel Stack View**
**Other Features**
**Memory Debugging**
**Visualization Tools**

## Parallel Stack View

**Allows user to see position of each processor in the code in the same window**
**Essentially groups processors by location in code -- only reasonable strategy at high concurrencies**

## Outline

**NERSC Systems Overview**
**Franklin Overview**
**Franklin Programming Environment**
**Running Jobs on Franklin**
**File Transfer Options**
**Debugging with DDT**
**Seaborg Decommission**

## Seaborg Decommission

**Yes, Seaborg is really going away:**
**Users will be informed and reminded of the decommission schedule.**
**There will be no more scheduled maintenance for software or hardware.**
**Only emergency maintenance will be performed.**

## Saving Your Seaborg Data

**As part of the decommission process: **
**/home directories will be archived into HPSS.**
**Ongoing, order based on date of last access**
**/scratch files will not be saved.**
**Archive your important files.**
**Please remove unneeded files in /home so we can save space in HPSS.**

## Seaborg Decommission

**Schedule:**
**01-Nov-2007 -****	****No new user accounts created**
**	****                      ****-****	****No new software     **
**31-Dec-2007 - ****	****End of IBM maintenance ****	****(hardware and software)**
**08-Jan-2008 - ****	****End of Allocation Year (AY) ****	****Seaborg retired from service**

## Seaborg Decommission

**Schedule (continued):**
**09-Jan-2008 -****	****All /home directories archived**
**15-Jan-2008 - ****	****Hardware ****	****	****	****	****	****removal begins; ****	****	****“Power Off”**
**15-Feb-2008 - Seaborg out of OSF & ****	****	****excessed according to DOE ****	****Rules**

## Seaborg

** **

- January 2001 - January 2008