# Presentation: 048376

## National Energy Research 

**National Energy Research **
**Scientific Computing Center **
**(NERSC)**
**NERSC Site Report**
**Shane Canon (canon@nersc.gov)**
- NERSC Center Division, LBNL
- 10/15/2004

## NERSC Outline

**PDSF**
**Other Computational Systems**
**Networking**
**Storage**
**GUPFS**
**Security**

## PDSF – New Hardware

**PDSF – New Hardware**

**49 Dual Xeon Systems**
**10 Dual Opteron Systems**
**All nodes are using native SATA controller (SI 3112 and SI 3114)**
**All nodes are gigE**
**Upgraded hard drives on 14 nodes (Added ~14 TB formatted**
**Foundry FES48 – 2 10G, 48 1G ports**

## PDSF – Other Changes

**New hardware will run SL (3.03)**
**CHOS already installed and will help  ease transition to SL for users**
**New nodes will run under Sun GridEngine**
** ****PDSF did not renew LSF maintenance**
**LSF nodes will slowly  be transitioned over to SGE**

## PDSF Projects

**Exploratory work has been hampered by involvement with NCS procurement, GUPFS project (and bike accidents)**
**Recent focus has been **
**CHOS**
**Deployment of new hardware**
**SL**
**Lustre**

## PDSF - Lustre

**Still not tested with users**
**Newer versions seem much more robust**
**Good at spot lighting flakey hardware**
**Older hardware is being reconfigured for use as a Lustre pool.  Roughly 10 TB of total space.**

## NERSC - IBM SP

**Upgraded to 5.2**
**Serious problems at first**
**IBM dispatched team to diagnose and fix problems**
**Added FibreChannel disk**
**~13 TB**
**FAStT 700 based**

## NERSC Systems - NCS

**Award has been made**
**No formal announcement until acceptance is completed**

## NERSC Systems - NVS

**New Visualization System**
**Small Altix System (4 nodes)**
**Some early issues**
**Channel bonded Ethernet Jumbo not supported**
**Using a Apple Xserve raid on it until O3k is decommissioned**

## Networking – 10G

**NERSC is building up a 10G infrastructure**
**Two MG8s provide core switching and routing for 10G network**
**Jumbo frames**
**Initially focused on core, mass storage, and visualization system.  Exploring ways to extend to Seaborg.  PDSF provided its own 10G Layer 3 switch.**

## NERSC - WAN

**10 G upgrade to WAN is in the works**
**Waiting on Bay Area Metropolitan Area Network deployment by ES Net.  Procurement is already under way**

## Mass Storage

**Latest Hardware**
**New Movers will have 10G links (testing is starting)**
**LSI based storage**
**Other projects**
**DMAPI work**
**Portals and other web interfaces into HPSS**

## Security - OTP

**Project on hold while funding is explored**
**To date various tokens have been evaluated**
**Focus is on products that are extensible and can be integrated fully in to NERSC and DOE infrastructures**
**Testing of cross RADIUS delegation**
**Should integrate into Grid using MyProxy or KCA approach**

## Bro Lite

**DOE Funded**
**Simplify Bro**
**Configuration (GUI)**
**Output filters****Available: Soon**
**Beta slots available**
**Contact:  security@nersc.gov**

## GUPFS

**Planned deployment late 2005**
**Unified filesystem spanning all NERSC systems (NCS, Seaborg, PDSF)**
**Possible candidates**
**GPFS, ADIC, Lustre, Panasas, Storage Tank**
**Results:  http://www.nersc.gov/projects/GUPFS**
**Contact:  gupfs@nersc.gov**

## GUPFS Tested

**File Systems**
**Sistina GFS 4.2, 5.0, 5.1, and 5.2 Beta**
**ADIC StorNext File System 2.0 and 2.2**
**Lustre 0.6 (1.0 Beta 1), 0.9.2, 1.0, 1.0.{1,2,3,4}, 1.2.1**
**IBM GPFS for Linux, 1.3 and 2.2.  Beta 2.3.**
**SANFS starting soon**
**Panasas**
**Fabric**
**FC (1Gb/s and 2Gb/s): Brocade SilkWorm, Qlogic SANbox2, Cisco MDS 9509, SANDial Shadow 14000**
**Ethernet (iSCSI): Cisco SN 5428, Intel & Adaptec iSCSI HBA, Adaptec TOE, Cisco MDS 9509**
**Infiniband (1x and 4x): InfiniCon and Topspin IB to GE/FC bridges (SRP over IB, iSCSI over IB), **
**Inter-connect: Myrinnet 2000 (Rev D)**
**Storage **
**Traditional Storage: Dot Hill, Silicon Gear, Chaparral**
**New Storage: Yotta Yotta GSX 2400, EMC CX 600, 3PAR, DDN S2A 8500**

## Procurements

**Several Procurements are starting up**
**GUPFS**
**Global Filesystem for NERSC**
**Deployment targeted for Spring 2005**
**NERSC5 – **
**Follow on to Seaborg**
**Likely target is 2005/2006**
**NCSe**
**Second year of funding for new capability at NERSC (NCS was first block)**
**Target Workload still being determined**

## PDSF - Utilization

**STAR has steadily picked up production over past months primary reason**
**Continued to encourage use of SGE pool for smaller groups and Grid projects**