# Presentation: 417816

## Storage Efficient Access Status:PVFS2

- Rob Ross
- Mathematics and Computer Science Division
- Argonne National Laboratory

## SEA Talks

- Four talks
- Myself: PVFS2
- Wei-keng Liao: MPI-IO and PnetCDF
- Alok Choudhary: Futures
- Ekow Otoo: MPI-IO to Mass Storage Systems
- My outline
- Quick overview of PVFS2
- PVFS2 on BG/L

## The PVFS2 Parallel File System

- _Parallel_ file system
- Distributed data and metadata
- Tuned for performance and concurrency
- Production ready
- 1.0 release at SC2004
- In use at ANL, OSC, Univ. of Utah CHPC, others
- Open source and open development
- LGPL license on all but kernel module, GPL on kernel module
- Current CVS is anonymously accessible
- Mailing lists where developers can track and initiate discussions
- Community research vehicle
- Heterogeneous system support
- Predominantly user-space code
- Rapid porting via network and storage abstractions
- Many labs and universities extend or modify PVFS2 to explore new ideas

## Who is PVFS2?

- PVFS2 is an open, collaborative effort
- Core development
**Argonne National Laboratory**
**Ross, Latham, Gropp, Thakur**
- Clemson University
- Ligon, Settlemyer
- Ohio Supercomputer Center
- Wyckoff, Baer
- Major collaborators
**Northwestern University**
**Choudhary, Ching**
- Ohio State University
- Panda, Yu
- Penn State University
- Sivasubramaniam, Kandemir, Vilayannur

## PVFS2 at OSC

- 506 total clients
- 116.8 TByte file system

- ...

- GigE Switch

- IB Switch

- ...

- ...

- 112 dual P4 nodes

- 144 dual P4 nodes

- 250 IA64 nodes

- 16 dual P4 servers- 7.3 TB each- multi-home

- IB

- IB

- GigE

- GigE

- FastE

## OSC cluster performance

- Data sizes are per-client
- Achieving ~2.8GB/sec write, ~3.8GB/sec read
- No network optimization (memory registration or pipelining)

## PVFS2 and IBM BG/L

## IBM BG/L in a nutshell

- 32K processors, 0.7GHz PowerPC 440
- 70.72 TFlops (Linpack)
- 1⁄4 size of final LLNL deployment

- 5.6/11.2 GF/s
- 1.0 GB

- 90/180 GF/s
- 16 GB

- 2.8/5.6 TF/s
- 512 GB

- 180/360 TF/s
- 32 TB

- System

## View of I/O on BG/L

- Storage nodes
- Local access to disks
- GigE connections to loginand IO nodes
- Login nodes
- Interactive machines
- _Place where data staging will__occur_
- IO nodes
- Aggregators for compute node I/O
- 1:8 to 1:64 ratio of IO nodes to compute nodes
- Tree connection to compute nodes
- Compute nodes
- _Source/sink of runtime I/O_

- Login Nodes

- Storage Nodes

- Compute Nodes

- IO Nodes

## Why put PVFS2 on BG/L?

- It’s fun 
- It provides another data point for I/O performance
- Lustre and GPFS also being ported
- Most importantly, PVFS2 addresses three key scalability problems for parallel file systems:
- I/O performance (especially for noncontiguous data)
- Metadata performance (in particular open/close)
- Failure tolerance
- Because of these advantages, we believe that PVFS2 has the best chance of extracting the highest possible I/O performance from BG/L

## First steps in running PVFS2 on BG/L

**Goal:** **Enable data staging and runtime I/O to a PVFS2 file system**
- Run PVFS2 servers on storage nodes
- dual Xeon nodes running SLES Linux and 2.6.5 kernel
- Mount PVFS2 file system on login nodes
- PowerPC 970 nodes running SLES Linux and 2.6.5 kernel
- Mount PVFS2 file system on IO nodes
- BG/L PowerPC nodes running 2.4.19 kernel (no longer MontaVista)
**This only took two weeks to accomplish!**
- Mostly learning/creating build environment
- Minimal patching to PVFS2 (all in CVS)
- 12 PVFS2 servers providing a single coherent file system
- (Assuming 900mbit/sec network to each, peak of ~1.3GB/sec raw BW)

## Write performance (the bad news)

- Simple pattern:
- Single file
- Independent MPI-IO
- One big access each
- We have more work to do here!
- There are many BG/L tuning parameters that we haven’t yet tweaked

## Read performance (the good news)

- Peak of 600MB/sec (44% of raw BW, also no tuning)

## I/O research in BG/L

**Plan:** Experiment with new MPI-IO algorithms
- Control of access mapping to IO nodes
- Caching of data at IO nodes (to what extent possible)
- New GPFS, Lustre, PVFS2 optimizations
- To do this, we must be able to rebuild ROMIO and link to IBM MPI
**Next Tasks:**
- ANL ensures that ROMIO builds cleanly against IBM MPI
- IBM provides MPI without ROMIO

## Summary

- SEA group is actively involved in providing I/O components for newest DOE platforms
- PVFS2 is filling an important role as research platform and production-ready parallel file system
- Same attributes that make PVFS2 ideal for research help us port to new systems like BG/L
- This early success is buying us greater credibility with IBM
- Hopefully will lead to even more open collaboration
- Work at MPI-IO layer (next talk) will further improve performance on systems such as BG/L