# Presentation: 775726

## LLNL SGPFS Requirements, Expectations and Experiences*

**LLNL SGPFS Requirements, Expectations and Experiences*******

- Dr. Mark K. Seager
- ASCI Tera-scale Systems PI
- Lawrence Livermore National Laboratory
- POBOX 808, L-60
- Livermore, CA 94550
**seager@llnl.gov**
**925-423-3141**

- *Work performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract W-7405-Eng-48.

**Notes:**
GOALS
1) encourage each vendor to have a SPGFS for their platform just as we have done in the interconnect to encourage the main vendors to have an interconnect with the characteristics ASCI wants
2) encourage industry to work together (what ASCI help would be needed if any?) to to create the standards required for an interoperable distributed, heterogeneous SPGFS and each implement one or cooperate on creating a portable one and then port it to their platforms. Do we have a preference for one or both?

My view of the realities
1. Creating a single vendor SPGFS is non trivial and a heterogeneous interoperable multi-vendor one is even more difficult.
2. For interoperability there first there has to be agreement on requirements across vendors. Then standards are required in several areas such as formats, synchronization, naming, file system management, API, protocol stack, I/O model,error recovery, interface to HSM, etc. This could take 2-5 years to make happen. Where does vendor SAN work fit in all this since currently vendors are taking a generally proprietary view on many aspects of SAN and can't even get interoperability at the lowest Fiber Channel level for example and there are multiple several man year efforts at just the sys management level.
3. There is a lot of code involved. I would estimate order 500K lines of C, based on NetApps and HPSS examples being of this order (does anyone know how big IBM's GPFS is? or even just the ordinary Unix or NT FS?). Assuming 15 lines of debugged code per day per programmer (includes the whole software engineering load requirements,design, coding unit test, integration testing reviews etc), (our experience roughly on HPSS) that implies order 170 person years of work to get a production quality system working. For a 30 person project that means 5-6 years and a cost of $40M +.   To get running across multiple vendors requirements additional porting or what have you/support, documentation, training adding to the cost. What in 2/3 could be done in parallel. It seems to me very optimistic to have something for 100TF timeframe. Are there smaller steps that could be taken that would be useful and lead to something for ASCI in shorter timeframes?
4. The notion LANL has been tossing around of a quick reference implementation ala ANL's MPICH does not really apply as the order of complexity of such a file system is a couple orders of magnitude more complex. We are talking an operating system level of complexity. It is easy to get a prototype working. We had a good prototype of HPSS working in a year with parallel I/O etc but it has taken several more to really make it production ready. I think IBM has had similar experiences with GPFS and more needs doing. There may also be an assumption that a vendor or univ such as Minn as something that could quickly be extended to meet above. Is there any basis for such a hope?
5. What are the requirements really driving this. Is PFTP an option across platforms and viz/compute etc engines? Can HPSS which has a good parallel I/O disk performance be an interim solution?
6. I'm not sure what the business case for the vendors is, would they expect ASCI to do it all or what percentage?

7. Other realities?

## Some Experiences with Global or Parallel FS

**SCF/OCF NFS environment at LLNL very challenging**
**Number of NFS OP/s huge**
**Anti-social usage patterns that cause NFS crashes not uncommon**
**Delivered performance not scalable with network bandwidth pipes**
**IBM PIOFS ******** GPFS experience**
**performance scaling brings out bugs**
**capacity scaling brings out bugs**
**performance can be tuned for reads or writes for a small class of applications (sweet spot of performance curves narrow)**
**user load brings out huge number of bugs**
**FLOW CONTROL VERY CRITICAL**
**Namespace and file allocation policies can take a great deal of time and kill performance**
**Getting performance from a file system requires holistic system knowledge**
**Parallel file system layout and parameters**
**Communication parameters**
**OS and device driver parameters (e.g., coalescing device driver)**
**RAID5 setup and parameters**
**Disk setup and parameters**

## Slide 3

## File system usage varies widely

## Monthly transfers to HPSS are large, vary significantly and are growing over time

## Monthly transfers to HPSS are large, vary significantly and are growing over time

## Computer center networking environments are very complex, change rapidly and must provide reliable service to a large number of customers

## OCF NFS Services provide global home directories and /nfs/tmp services, but don’t scale

****

- Private
- Gigabit
- Switch

- c3

- po

- q9

- x2

- y0

**OCF Center-Wide NFS Services**

- Network Appliance F760
_*Home Directory Services*_
- c3,po,q9,x2  504 GB each
-                          2 TB  total
- Clustered for HA/failover
_*NFS Tmp Services           *_
- y0                 504 GB total

- FE
- Switch

*Compass Cluster*
*(Digital 8400)*

*Blue**(IBM SP)*

*Riptide*
*(SGI Onyx)*

- OCF Backbone

*Tera Cluster*
*(Compaq ES40)*

- External
- Gigabit
- Switch

**OCF NFS Services provide global home directories and /nfs/tmp services, but don’t scale**

## Why SPGFS is HARD

**Scalable - ASCI scaling (~6.4 GB/s for 3.9 TF ******** 12.8 GB/s on 10.2 TF) on a single platform is very hard work.**
**Scalability requires striping at multiple levels**
**File servers within a file system must be highly balanced (number of RAID adapters/server, RAID configuration/adapter, interconnect, workload)**
**Cluster file system model required for applications scaling**
**Global - huge distributed highly available namespace with scalable performance is an unsolved problem**
**Parallel - performance for wide parallel application mix extremely hard**
**Highly balanced client/server ratio for each app (FLOW CONTROL)**
**Scalable interconnect bandwidth without contention **
**Minimal latency to keep performance application block size requirements reasonable.  Software stack, networking and distance all contribute**
**Fast - scheduling dynamic mix very hard problem (class of service)**
**Secure - authorization scheme can add huge latencies and most schemes not scalable**
**Manageability - file system setup, management (du on 1.6 PB?), quotas, FS reservation and clean up for production job, class of service**

## Why SGPFS development will be hard

**Multiple OS’s**
**What networking infrastructure**
**What NAP model**
**Standards for protocols and interfaces**
**Design is hard, development is harder, testing is hardest of all and is the key issue**
**Based on the NetApps and HPSS examples, we estimate 500K lines of C**
**Assuming 15 lines of debugged code/day/programmer = 170 person years**
**includes the whole software engineering load requirements,design, coding unit test, integration testing reviews etc.**
**For a 30 person project that means 5-6 years and a cost of $40M+.   **
**To get running across multiple vendors requirements additional porting or what have you/support, documentation, training adding to the cost. **

## Final Thoughts

**Seager’s Second Law Of Parallel Programming:**
**It is infinitely easy to get parallel I/O to run arbitrarily slow**

**File system stability is a sacred covenant between vendor, service provider and users.**