# Presentation: 223913

## National Energy Research 

**National Energy Research **
**Scientific Computing Center **
**(NERSC)**
**The GUPFS Project at NERSC**
- GUPFS Team
- NERSC Center Division, LBNL
- November 2003

## Global Unified Parallel File System (GUPFS)

- Global/Unified
- A file system shared by major NERSC production systems
- Using consolidated storage and providing unified name space
- Automatically sharing user files between systems without replication
- Integration with HPSS and Grid is highly desired
- Parallel
- File system providing performance that is scalable as the number of clients and storage devices increase

## GUPFS Project Overview

- Five year project introduced in NERSC Strategic Proposal
- Purpose to make advanced scientific research using NERSC systems more efficient and productive
- Simplify end user data management by providing a shared disk file system in NERSC production environment
- An evaluation, selection, and deployment project
- May conduct or support development activities to accelerate functionality or supply missing functionality

## Current NERSC Storage Configuration

- Each system has its own separate direct-attached storage
- Each system has its own separate user file system and name space
- Data transfer between systems is over the LAN
- Include Large computational systems, small systems, and support systems

## NERSC Storage Vision

- Single storage pool, decoupled from NERSC computational systems
- Diverse file access - supporting both home file systems and large scratch file system
- Flexible management of storage resource
- All systems have access to all storage – require different fabric
- Buy new storage (faster and cheaper) only as we need it
- High performance large capacity storage
- Users see same file from all systems
- No need for replication
- Visualization server has access to data as soon as it is created
- Integration with mass storage
- Provide direct HSM and backups through HPSS without impacting computational systems
- Potential Geographical distribution

**Notes:**
Flexible management – expansion and upgrade independent

## Envisioned NERSC Storage Configuration(GUPFS)

## Where We Are Now

- At the beginning of the 3rd year of the three-year technology evaluation phase
- Evaluation of technology components needed for GUPFS:
- Shared File System,
- SAN Fabric, and
- Storage
- Constructed complex testbed environment simulating envisioned NERSC environment
- Developed testing methodologies for evaluation
- Identifying and testing appropriate technologies in all three areas
- Collaborating with vendors to fix problems and influence their development in directions beneficial to HPC

## GUPFS Tesbed (FY2004)

## Technologies Evaluated 

- File Systems
- Sistina GFS 5.1, 5.2 Beta
- ADIC StorNext (CVFS) File System 2.0
- Lustre 0.6 (1.0 Beta 1)
- GPFS 1.3 for Linux
- Panasas
- Fabric
- FC (1Gb/s and 2Gb/s): Brocade SilkWorm and Qlogic SANbox2
- Ethernet (iSCSI): Cisco SN 5428, Intel iSCSI HBA, Adaptec TOE
- Infiniband (1x and 4x): InfiniCon and Topspin IB to GE/FC bridges (SRP over IB, iSCSI over IB),
- Inter-connect: Myrinnet 2000 (Rev D)
- Storage
- Traditional Storage: Dot Hill, Silicon Gear, Chaparral
- New Storage: Yotta Yotta NetStorager, EMC CX 600, 3PAR

## Parallel I/O Tests

- I/O Tests used for file system and storage evaluation:
- Cache Write
- Cache Read
- Cache Rotate Read
- Disk Write
- Disk Read
- Disk Rotate Read

## Storage Performance

## Storage Scalability

- Traditional Storage: DotHill, 1 1Gb/s port, 6-way R0
- New Storage: YottaYotta. 4 blades, 2 FC ports per blade, 32-way R0
- Client: Dual P4 Xeon, 1GB Mem, Linux 2.4.18-10smp
- HBA: Qlogic QLA2340 2Gb HBA

## Comparison of Fabric Technologies

- Storage: YottaYotta (1 2Gb/s FC Raid-0 LUN)
- GE: Intel E1000
- FC: Qlogic QLA2340 2Gb/s HBA
- IB: InfiniCon Infin7000 Gen 1 switch & 4x HCA

## Software iSCSI vs. iSCSI HBA

- Storage: Chaparral (1 2Gb/s FC Raid-0 LUN)
- iSCSI: Intel PRO/1000 IP Storage Adapter
- Switch: Cisco SN5428 iSCSI Switch (1 1Gb/s iSCSI port)
- Normalized CPU Utilization=%SYS/Throughput (sec per KB/s)
- Date: 07/2003

**Notes:**
Note the LOG scale for File Creation
Note that there is no Performance difference on ADIC for same-dir vs separate dir
Note that in all cases, the number of processes was one greater than the number of clients, the extra one performing test setup (dir and file creation).

## Performance Impact of Kernel Changes

- Storage: Chaparral (1 2Gb/s FC Raid-0 LUN)
- iSCSI HBA: Intel PRO/1000 IP Storage Adapter
- Switch: Cisco SN5428 iSCSI Switch (1 1Gb/s iSCSI port)
- Date: 07/2003

**Notes:**
Note the LOG scale for File Creation
Note that there is no Performance difference on ADIC for same-dir vs separate dir
Note that in all cases, the number of processes was one greater than the number of clients, the extra one performing test setup (dir and file creation).

## Infiniband SRP Performance

- Storage: 3PARdata & EMC 600 (2Gb/s FC Raid-5 LUNs)
- FC: Qlogic QLA2340 2Gb/s HBA
- IB: Topspin 90 switch & 4x HCA
- Date: 08/2003

**Notes:**
Note the LOG scale for File Creation
Note that there is no Performance difference on ADIC for same-dir vs separate dir
Note that in all cases, the number of processes was one greater than the number of clients, the extra one performing test setup (dir and file creation).

## When Storage is a Bottleneck

- Storage: DotHill (2 1Gb/s FC Raid-5 LUNs)
- FC: Qlogic QLA2340 2Gb/s HBA
**Date: 03/2003 – 04/2003**

## ADIC StorNext File System

- Storage: YottaYotta, Single 28-way RAID-0 LUN (exported on 8 2Gb/s ports)
- Clients: Dual P4 Xeon, 1GB Mem, Linux 2.4.8-10smp
- FC: Qlogic QLA 2340

**Notes:**
Note the LOG scale for File Creation
Note that there is no Performance difference on ADIC for same-dir vs separate dir
Note that in all cases, the number of processes was one greater than the number of clients, the extra one performing test setup (dir and file creation).

## GPFS/Linux 1.3 Performance

- SAN: 3PARdata (8 2Gb/s FC ports over 1 Raid-5 LUN)
- Clients: Dual P4 Xeon, 1GB Mem, Linux 2.4.18-10smp
- FC: Qlogic QLA2340
- Interconnect: Myrinet (Rev D) LAPI

- NSD: 8 YottaYotta 4-way RAID-0 LUNs (8 2Gb/s ports)
- Clients: Dual P4 Xeon, 1GB Mem, Linux 2.4.8-10smp
- FC: Qlogic QLA 2340
- Interconnect: GigE

**Notes:**
Note the LOG scale for File Creation
Note that there is no Performance difference on ADIC for same-dir vs separate dir
Note that in all cases, the number of processes was one greater than the number of clients, the extra one performing test setup (dir and file creation).

## GPFS/Linux I/O Scalability

- Linux Cluster: NERSC Alvarez (87 nodes)
- Storage (ALV): 2 storage nodes (IBM Raid-5 LUNs, 100MB/s max)
- Storage (GUS): YottaYotta 8 Raid-0 LUNs, 8 x 2Gb FC ports
- Interconnect: Myrinet 2000 (Rev C)
- Date: 08/2003 – 10/2003

**Notes:**
Note the LOG scale for File Creation
Note that there is no Performance difference on ADIC for same-dir vs separate dir
Note that in all cases, the number of processes was one greater than the number of clients, the extra one performing test setup (dir and file creation).

## Panasas File System

- Host: Linux 2.4.21
- Panasas: 1 Shelf, 4 x 1 GigE with trunking, 1 DirectorBlade, 10 StorageBlade, with Direct Flow
- Interconnect: GigE
- File Sizes per I/O process/client: 128 MB, 1GB, 4 GB

**Notes:**
Note the LOG scale for File Creation
Note that there is no Performance difference on ADIC for same-dir vs separate dir
Note that in all cases, the number of processes was one greater than the number of clients, the extra one performing test setup (dir and file creation).

## File System - Metadata Performance

- Storage: YottaYotta (Raid-0 LUNs, 8 2Gb/s FC ports)
- FC: Qlogic QLA2340 2Gb/s HBA
- Date: 03/2003

**Notes:**
Note the LOG scale for File Creation
Note that there is no Performance difference on ADIC for same-dir vs separate dir
Note that in all cases, the number of processes was one greater than the number of clients, the extra one performing test setup (dir and file creation).

## Near Term Evaluation Activities

- Continue GFS, ADIC StorNext, GPFS File, and Lustre evaluations
- Conduct test of additional file system, storage and fabric technologies as possible.  Potential targets:
- Panasas object based file system (with Direct Flow)
- IBM SAN File System (StorageTank)
- IBRIX File System
- DataDirect Networks storage
- Cisco MD950x FC and iSCSI director class switch
- iSCSI HBAs
- Quadrics interconnect (with Lustre evaluation, etc).
- Myrinet Gigabit Ethernet blade (fabric bridge)
- 10Gb Ethernet
- 4Gb and 10Gb Fibre Channel
- 12x Infiniband

## Thank You

- GUPFS Project Web Site
- http://www.nersc.gov/projects/gupfs
- GUPFS FY02 Technical Report
- http://www.nersc.gov/aboutnersc/pubs/GUPFS_02.pdf
- Contact Info:
- Project Lead: Greg Butler (gbutler@nersc.gov)

## GUPFS Team

- Greg Butler (GButler@nersc.gov, Project Lead)
- Will Baird (WPBaird@lbl.gov)
- Rei Lee (RCLee@lbl.gov)
- Craig Tull (CETull@lbl.gov)
- Michael Welcome (MLWelcome@lbl.gov)
- Cary Whitney (CLWhitney@lbl.gov)