# Presentation: 732978

## ASC PI MeetingI/O and File SystemsR&D and Deployment

**Lee Ward, Sandia**
**Bill Boas, LLNL**
**Gary Grider, LANL**

- 02/2005

## The Primary Driver:  A Balanced System Approach

- Programs

**Application**
**Performance**

**Computing Speed**

**Parallel I/O**

**Network Speed**

**Memory**

**Archival**
**Storage**

- FLOP/s

- Gigabytes/sec

- GigaBytes/sec

- Gigabits/sec

**1**

**0.05**

**0.5**

**5**

- TeraBytes

**50**

**0.01**

**0.1**

**10**

**1**

**0.1**

**1**

**10**

**100**

**8**

**800**

**80**

**0.8**

- ‘96

- ‘97

- ‘00

- 2003

**Year**

**Disk**

- TeraBytes

**2.5**

**25**

**250**

**2500**

**Computational Resource Scaling for ASCI Physics Applications**
- 1 	FLOP/s	Peak Compute
- 0.5	Byte/FLOP/s	Memory
**0.001****	****Byte/s/FLOP/s****	****Peak Parallel I/O**
- 0.008	bit/s/FLOP/s	Peak Network
- 0.0001 Byte/s/FLOP/s	Archive

##    FS Requirements Summary

- From Tri-Lab File System Path Forward RFQ (which came from the Tri-labs file systems requirements document) ftp://ftp.lanl.gov/public/ggrider/ASCIFSRFP.DOC
**POSIX-like Interface, Works well with MPI-IO, Open Protocols, Open Source (parts or all), No Single Point Of Failure , Global Access**
*Global name space, ...*
*Scalable*** ***bandwidth, metadata, management, security ...      1  GB/sec per Tflop, 1000’s of metadata ops/sec*
*WAN Access, Global Identities, Wan Security, ...*
*Manage, tune, diagnose, statistics, RAS, build, document, snapshot, ...*
*Authentication, Authorization, Logging, ... *

## Enterprise Wide Scalable Sharing

**Native**

**Native**

**NFS**

**CIFS**

**Linux Compute Clusters**

**Central Global**
**Parallel File System**

**Analysis & Visualization Tools**

**Linux Viz Clusters**

**Non Linux Viz/analysis Clusters and workstations**

**Native**

**Altix**

**Native**

**Archive**

## Vendor Collaborations

**Solution for Linux clusters and enterprise class heterogeneous global parallel file systems**
- HP/CFS/Intel Lustre Path Forward for object based secure global parallel file system  2002-present
- Very scalable bandwidth, good non scaled metadata
- Being deployed/used at LLNL and Sandia
- Starting metadata scaling work  finishing CMD-1 (4 metadata servers, starting CMD-2 (>15 metadata servers)
- Panasas 2000-present
- Very scalable bandwidth, good non scaled metadata
- Being deployed/used at LANL and LLNL
- Multi-metadata servers by volume done, arbitrary multiple metadata servers this year

## With only vendor relationships there are still            gaps in our Strategy

- May not be a native client to new OBSD file systems (more heterogeneous access), need WAN access to OBSD file systems,
- We need to deal with the fact that file systems are based on a geometry at the application may not be able to easily conform its memory to that geometry
- Scalable metadata and security has not been done before in the most general sense
- How would we integrate Archive/HSM into this common enterprise class OBSD FS

## Our University Partnerships

**Heterogeneous access and WAN access  (University of Michigan)**
- Secure file sharing with access control and need to know as replacement for DFS which fits into ICSE infrastructure for LAN and WAN
- Provide heterogeneous near native client to OBFS
- Showing up in Linux 2.6 first Kerberos/gss file system access shown Jan 2005
**Aligning application memory with file system geometry (Northwestern University and ANL)**
- Move coordination of collaborative caching to MPI-IO, out of the file system for overlapped, unaligned, small I/O
- This is a good position to be in, as we have been building “plumbing” for years, now we are working with applications to enable exploitation of these new file systems
- Showing up in ROMIO/MPI-IO
**Metadata and Security scaling  (UC Santa Cruz, Lustre, Panasas, and Sandia)**
- Squashing/replicating permissions and directory decomposition caching studies
- Panasas and Lustre working on metadata decomposition and scaling with current ideas
- Scaling matadata performance into the petaflops range looks problematic!
**Minnesota (Intelligent Storage Consortia) and STK **
- Can we leverage both the object based parallel file system technologies and commercial (non parallel) archive products to provide a scalable parallel archive that is or could be integrated tightly into a globally shared scalable parallel file system?
- What would need to change in these global parallel file systems and commercial backup/archive/hsm products to enable this capability
- Work just beginning.

## POSIX IO Additions

**Most of the current issues with POSIX IO semantics lie in the lack of support for distributed/parallel processes**
- Concepts that involved implied ordering need to have alternative verbs that do not imply ordering
- Vectored I/O read/write calls that don’t imply ordering
- Extending the end of file issues
- Group opens
- Etc.
- Concepts that involve serialization for strict POSIX metadata query/update need to have lazy POSIX alternatives
- Last update date (mtime,atime,ctime), Size of file
- Active storage
**Status:  **
- Pursuing Labs joining “The Open Group” which holds the current “One Unix” charter which merges IEEE, ANSI, and POSIX standards and evolves the POSIX standard for UNIX
- Next Step is to write up proposed new changes and begin discussion process within the POSIX UNIX IO API working group in the Open Group forum.

## Government multi-agency I/O coordination 

**Authored joint R&D needs document for next five years of needed I/O and file systems R&D work (DOE NNSA/Office of Science, DOD NSA)**
**Working with DARPA on how to spin up more I/O R&D for HPC  (as part of high productivity project)**
**Working with NSF on how to spin up more I/O R&D for HPC  (  under HECURA umbrella or not  )                                                                                                                                                                                                   **

## Are we headed towards our vision?

- Lustre in use, in programmatic role at LLNL and Sandia
- Panasas in use, in programmatic and supports roles LANL, Sandia and LLNL
- Academic and other partnerships paying off
- ANL and Northwestern stuff going into ROMIO, aggregation, two phase I/O, sieving, better caching coming
- NFSv4 getting into Linux
- We showed multi-realm kerberized controlled access to a file system via NFSv4
- UCSC ideas beginning to become appealing to our file system partners
- Implementing first generation enterprise class common parallel file systems, shared between multiple clusters (Linux natively, others via NFSv3)
- pNFS entering IETF to allow for non Linux clients to access object file systems natively
- MPI-IO beginning to be used in more apps
- Future works spinning up, POSIX api, object archive, multi-agency I/O R&D investments, object archive

## LANL Deployments

**Turquoise – Institutional and Alliance Computing**
**Yellow – Unclassified programmatic computing**
**Red – Classified programmatic computing**

## Current Turquoise Configuration 

*Current Turquoise Configuration**** ***

- Pink 1916 Xeon procs

- Myrinet

- 64 IO nodes

- Gig-E

- Panasas      16 Storage Shelves      80 TB
- ~6.4 GB/s

- 16 IO nodes

- 112 Compute Nodes

- Myrinet

- Mauve         Altix Cluster  256 procs

- TLC      224 AMD64 procs

- Future Institutional machine fy05

## Current Yellow Configuration 

*Current Yellow Configuration**** ***

- Flash 728 AMD64 procs

- Myrinet

- 64 IO nodes

- Gig-E

- Panasas        8 Storage Shelves      40 TB
- ~3.2 GB/s

- Future Viz cluster FY05/06

- Future Direct HPSS movement agents FY05

## Red Configuration in a couple of weeks

*Red Configuration**** ****in a couple of weeks*

- Lightning 3072 AMD64 procs

- Myrinet

- 128 IO nodes

- Gig-E

- Panasas      48 Storage Shelves      200 TB
- ~20 GB/s

- Future Viewmaster Viz cluster fy05/06

- Future Capacity machine fy05/06

- Future direct HPSS movement agents        (FY05 ASAP)

## PaScalBB – Parallel Scalable Backbone

- To Site Network

- Core switches

- I/O nodes

- Compute Nodes

- Cluster A,  interconnect A

- Cluster B,       interconnect B

- Cluster C,    interconnect C

- Archive

*PaScalBB – Parallel Scalable Backbone*

- Layer 2 ethernet switches

- Single file system

## Pink Data Throughput

## Pink Metadata Throughput

## Lustre tested in two environments and in GA or imminent
ICC & NWCC capacity machines
Feynman
Lustre intended for Red Storm and in test on the hardware
Panasas in production test on Feynman

- ICC & NWCC capacity machines
- Feynman
**Lustre intended for Red Storm and in test on the hardware**
**Panasas in production test on Feynman**

- 02/2005

*Sandia Deployments*

## Sandia Capacity Machines

**EA version of SFS on Liberty and Spirit**
- Successful limited availability trial
- Withdrawn to upgrade hardware and software
**Hardware now in place and being verified for all (ICC & NWCC)**
**Installing SFS 1.1.-1 on all**
- Based on CFS 1.2.3 Lustre
- Sandia pre-production verification currently in progress

## Sandia Feynman

**CFS 1.4 Lustre deployed**
- GA on this machine and has supported science runs
- Adding additional disk and OSS nodes to expand capability
- Intended as primary “scratch” space
**Four shelves of Panasas deployed**
- GA on this machine
**At least one of these will be used as the environment morphs into ROSE**

## Sandia Red Storm

**CFS 1.4 server and a, heavily, modified client tested on Red Squall**
**Cray beginning to bring it up on Red Storm hardware in place at Sandia now**

## LLNL AC and Open Compute Resources

*LLNL AC and Open Compute Resources*

## LLNL Open Computing Facility - 2005

**64**
**OSS**
**180 TB**
**SATA**

**BB/MKS Version 9, Feb3 2005**

**2 Login nodes**

**OSS**

**OSS**

**OSS**

**OSS**

**OSS**

**OSS**

**OSS**

**OSS**

**OSS**

**OSS**

**OSS**

**OSS**

**PVC - 128 Port Elan3**

**52 Dual P4**

**6 Dual P4**

**HPSS**
**Archive**

**LLNL**
**External**
**Backbone**

**1,004 Quad Itanium2 Compute Nodes**

**B451**

**GW**

**MDS**

- 4 Login nodes
- with 6 Gb-Enet

- 16 Gateway nodes @ 350 MB/s
- delivered Lustre I/O over 4x1GbE

**Thunder – 1,024 Port QsNet Elan4**

**924 Dual P4 Compute Nodes**

**B439**

**GW**

**GW**

**MDS**

**MDS**

- 2 Login nodes
- with 4 Gb-Enet

- 32 Gateway nodes @ 190 MB/s
- delivered Lustre I/O over 2x1GbE

**ALC - 960 Port QsNet Elan3**

**1,114 Dual P4 Compute Nodes**

**B439**

**GW**

**GW**

**MDS**

**MDS**

- 4 Login nodes
- with 4 Gb-Enet

- 32 Gateway nodes @ 190 MB/s
- delivered Lustre I/O over 2x1GbE

**MCR – 1,152 Port QsNet Elan3**

**NAS**
**Home**
**Dirs**

**OSS**

**OSS**

**OSS**

**OSS**

**OSS**

**OSS**

**192**
**OSS**
**258 TB**
**FC**

**224 OSS**
**900 TB SATA**

**12.5**
**GB/s**

**36**
**GB/s**

**18.7**
**TFLOPs**
**LP**

**7.6**
**TFLOPs**
**LP**

**6.5**
**TFLOPs**
**LP**

**VIZ**

**6509 Ethernet Switches with 1 or 10 Gig ports**

**65,536 Dual PowerPC 440 Compute Nodes**

**1024 I/O**
**Nodes**
**PPC440**

**B453**

**BG/L torus, global tree/barrier**

**360**
**TFLOPs**
**peak**

**45**
**GB/s**

**GW**

**GW**

**MDS**

**MDS**

**B439**

**25**
**GB/s**

**19.2**
**GB/s**

**9.6 GB/s**

**1152 GigE Ports**

**386 GigE**
** ****Ports**

**128 GigE**
**Ports**

**128 GigE**
**Ports**

**B453**

**448 GigE  Ports**

**LLNL Open Computing Facility - 2005**

**MDS**

**MDS**

## Steps to SWGFS in OCF 1 of 2

**Deploy all necessary network connections for BG/L IONs, OSS, and “network” MDS**
**Migrate all MDS from Elan 3/4 in MCR, ALC-P, Thunder to Federated Gigabit Ethernet infrastructure**
**FY 05 CFS Development SOW includes one generation of backward compatibility, i.e Version 1.2 to 1.4 for protocols and file formats**
**Finalize features in Lustre Version 1.4 for testing in Testbed and on ALC-T**
**Finalize plan for CHAOS migration to Linux 2.6**
**     ****First deploy CHAOS 2.6 and Lustre 1.4 on ALC-T and IO Testbed.**
**  **

## Steps to SWGFS in OCF - 2 of 2

**6. Proposal to Users for cross-mounting:**
- ALC-P sees /p/gm1, gm2, gT1
- MCR sees /p/gm1, gT1
- Thunder sees /p/gm1, gm2, gA1
- PVC/Sphere sees /p/gm1, gm2, gA1, gT1
- Review with Users to gain approval and prepare scripts
**7. Deploy SWGFS Phase 1 on MCR, PVC/Sphere, ALC-P and Thunder**
**8. Migrate Lustre from Version 1.2 to 1.4**
**9. Migrate CHAOS in production to Linux 2.6**
**10. Cross mount BG/L and  SWGFS Phase 1 with as above to complete SWGFS Phase 2 to include BG/L **
**11. pNFS is maybe the direction**
**	**

## Strawman SWGFS Testing and Deployment Schedule

**FY’04**

**FY’05**

**FY’06**

**OCF**

**Q1**

**Q2**

**Q3**

**Q4**

**Q1**

**Q2**

**Q3**

**Q4**

**Q3**

**Q4**

**SCF**

**MCR**

**ALC-P**

**Thunder**

**BG/L-O**

**White**

**White**

**Lilac**

**Purple**

**-**

**Strawman SWGFS Testing and Deployment Schedule**

**Key**

**Test & Dev.**

**Science Runs**

**Limtd Avail Prod**

**Gen Avail Prod**

**ALC-T**

**PVC**

**BG/L Vis**

**Purpura**

**SWGFS**

**Sphere**

**UM**

**UV**

**Q1**

**Q2**

**Q3**

**Q4**

**Q1**

**Q2**

**Q3**

**Q3**

**Q4**

- 1

- 2

- 2

**Step No.**

- 3

**CHAOS**

- 4

- 5

- 6

- 7

- 8

- 9

- 10

**	**

- _Steps for SWGFS Deployment in OCF_
- 1. Deploy all necessary network connections for BG/L IONs, OSS, and “network” MDS
- 2. Migrate all MDS from Elan3/4 in MCR, ALC-P, Thunder to Federated Gigabit Ethernet infrastructure
- 3. FY 05 CFS Development SOW includes one generation of backward compatibility, i.e Version 1.2 to 1.4 for protocols and file formats
- 4. Finalize features in Lustre Version 1.4 for testing in Testbed and on ALC-T
- 5. Finalize plan for CHAOS migration to Linux 2.6, first deploy CHAOS 2.6 and Lustre 1.4 on ALC-T and IO Testbed.

- 6. Proposal to Users for cross-mounting:
- ALC-P sees /p/gm1, gm2, gT1 **(remove root FN?)**
- MCR sees /p/gm1, gT1
- Thunder sees /p/gm1, gm2, gA1
- PVC/Sphere sees /p/gm1, gm2, gA1, gT1
- Review with Users to gain approval and prepare scripts
- 7. Deploy SWGFS Phase 1 on MCR, PVC/Sphere, ALC-P and Thunder
- 8. Migrate Lustre from Version 1.2 to 1.4
- 9. Migrate CHAOS in production to Linux 2.6
- 10. Cross mount BG/L and  SWGFS Phase 1 with as above to complete SWGFS Phase 2 to include BG/L

## LLNL Open Computing Facility

**64**
**OSS**
**180 TB**
**SATA**

**BB/MKS Version 9, Feb3 2005**

**2 Login nodes**

**OSS**

**OSS**

**OSS**

**OSS**

**OSS**

**OSS**

**OSS**

**OSS**

**OSS**

**OSS**

**OSS**

**OSS**

**PVC - 128 Port Elan3**

**52 Dual P4**

**6 Dual P4**

**HPSS**
**Archive**

**LLNL**
**External**
**Backbone**

**1,004 Quad Itanium2 Compute Nodes**

**B451**

**GW**

**MDS**

- 4 Login nodes
- with 6 Gb-Enet

- 16 Gateway nodes @ 350 MB/s
- delivered Lustre I/O over 4x1GbE

**Thunder – 1,024 Port QsNet Elan4**

**924 Dual P4 Compute Nodes**

**B439**

**GW**

**GW**

**MDS**

**MDS**

- 2 Login nodes
- with 4 Gb-Enet

- 32 Gateway nodes @ 190 MB/s
- delivered Lustre I/O over 2x1GbE

**ALC-P - 480 Port QsNet Elan3**

**1,114 Dual P4 Compute Nodes**

**B439**

**GW**

**GW**

**MDS**

**MDS**

- 4 Login nodes
- with 4 Gb-Enet

- 32 Gateway nodes @ 190 MB/s
- delivered Lustre I/O over 2x1GbE

**MCR – 1,152 Port QsNet Elan3**

**NAS**
**Home**
**Dirs**

**OSS**

**OSS**

**OSS**

**OSS**

**OSS**

**OSS**

**192**
**OSS**
**258 TB**
**FC**

**224 OSS**
**900 TB SATA**

**12.5**
**GB/s**

**36**
**GB/s**

**18.7**
**TFLOPs**
**LP**

**7.6**
**TFLOPs**
**LP**

**6.5**
**TFLOPs**
**LP**

**VIZ**

**6509 Ethernet Switches with 1 or 10 Gig ports**

**65,536 Dual PowerPC 440 Compute Nodes**

**1024 I/O**
**Nodes**
**PPC440**

**B453**

**BG/L torus, global tree/barrier**

**360**
**TFLOPs**
**peak**

**45**
**GB/s**

**GW**

**GW**

**MDS**

**MDS**

**B439**

**25**
**GB/s**

**19.2**
**GB/s**

**9.6 GB/s**

**1152 GigE Ports**

**386 GigE**
** ****Ports**

**128 GigE**
**Ports**

**128 GigE**
**Ports**

**B453**

**448 GigE  Ports**

**LLNL Open Computing Facility**
** ****SWGFS Deployed**

**MDS**

**MDS**

## SWGFS in SCF

**White – AIX/GPFS**
**UM (magenta) – AIX/GPFS**
**UV (violet) – AIX/GPFS**
**Purple GPFS by Q1 FY06**
**Lilac – Linux/Lustre Version 1.2**
**GViz – Linux/Lustre in FY 05 Q2**
**Some hints that IBM may “open up” GPFS**

## backup

## Example of  poor alignment and extremely              small write problems

- Parallel file

-  23

- 33

- 43

- 13

-  44

- 34

- 24

-  14

- 11

- 12

- 13

-   14

- Process 1

- 21

- 22

-  23

- 24

- Process 2

- 31

- 32

- 33

- 34

- Process 3

- 41

- 42

- 43

-  44

- Process 4

- RAID Group 1

- RAID Group 2

- Very small, unbalanced, and unaligned writes, much smaller than a full RAID stripe, every process queues up to read/update/write a partial block, all serialized on RAID Group 1, then everyone moves and does it again, eventually they all move to serialize on RAID Group 2, etc.

- RAID Group 3

## Apps can utilize smart S&CS                           investments in Middleware (MPI-IO)

- Parallel file

-  23

- 33

- 43

- 13

-  44

- 34

- 24

-  14

- 11

- 12

- 13

-   14

- Process 1

- 21

- 22

-  23

- 24

- Process 2

- 31

- 32

- 33

- 34

- Process 3

- 41

- 42

- 43

-  44

- Process 4

- RAID Group 1

- RAID Group 2

**Thanks to the S&CS Argonne and Northwestern work, collective two-phased buffered I/O helps align, balance, and aggregate.  **

- RAID Group 3

- CB procs

- RAID Group 4

## Historical Time Line for ASC I/O R&D

**2002**

**2000**

**2001**

**1999**

- proposed Path Forward activity for SGPFS

- propose initial architecture

- SGPFS workshop **“You are Crazy**”

- build initial requirements document

- Path Forward project to pursue RFI/RFQ, analysis, recommend funding open source OBSD development and NFSv4 projects

- Lets re-invent Posix I/O ?

- Path Forward proposal with OBSD vendor, Panasas born

- Tri-Lab joint requirements document complete

- Path Forward Lustre project born

- Alliance contracts placed with universities on OBSD, small/unaligned/unbalanced/overlapped  I/O, and NFSv4

**2003**

**2004**

- U of Minn Object Archive begins

- pNFS effort begins

## Why Panasas?

**Needed a true production file system for Pink FY2003 and Lightning (early) FY2004**
**Request for Proposal for File System**
- RFP in early 2003, based on file systems Path Forward RFP
- Panasas strongest proposal
- No real Lustre bidders (HP said too early for Lustre to step up to production role)
**Acceptance on Pink FY2003, Acceptance on Lightning early FY2004**
- Passed all Acceptance Tests
- Maximums – File size, # of files, etc.
- I/O Performance – N to N & N to 1 read/write bandwidth
- Metadata Performance – Creates, Deletes, stats/sec.
- RAS – Graceful reaction to catastrophic events
- Uptime
**Will we re-evaluate – yes, when it makes sense,  that is our job**