# Presentation: 338510

## Achievements and Challenges for I/O in Computational Science

- Rob Ross
- Mathematics and Computer Science Division
- Argonne National Laboratory

**Notes:**
25 minutes.

## I/O in Computational Science

- I/O is an increasingly important part of computational science
- Lots of different I/O needs from applications
- Initialization
- Input datasets vary widely in size and format
- Checkpointing (defensive I/O)
- Lots of data written all at once
- Visualization
- Subset of checkpoint data
- More frequent writes during runtimethan with checkpoints
- Probably read many times
- Data movement
- Wide-area data access

| Application | Reading and Generation | Post-processing, Checkpointing | Analysis |
| --- | --- | --- | --- |
| Astrophysics | 20-200 | 20-200 | 20 |
| Supernova | 20 | 2 | 2 |
| Climate Modeling | 2 | 2 | 1 |
| Cosmology | 5 | 1 | 1 |
| Fusion | 1,000 | 1 | 0.5 |

**All values are for a single run;** **units are TBytes**.  Data primarily from workshop on Requirements for Ultrascale Computing in Washington, DC, June 2003.

**Notes:**
I/O I imagine has come up as an issue for every application represented in the room, whether it is for reading large input datasets, checkpointing during the run, outputting visualization data, or moving data to a remote site for further processing.  Either you’ve run a problem where I/O just took too long or you can imagine such a problem.  Usually when we think of I/O as a problem, we are thinking about performance issues.

## Parallel I/O

- Parallel I/O is simply using many I/O resources in a coordinated way to solve a single problem more quickly
- Example: storing a checkpoint into a single file
- Same thing we do in parallel processing
- Parallel I/O is becoming mandatory for applications
- “It’s not working like it used to?”
- A single BG/L compute node has no more than 60 Mbyte/sec of I/O bandwidth
- But the whole machine might have 30 Gbyte/sec of I/O bandwidth (e.g. LLNL)!
- I/O software determines how well we can make use of the available I/O hardware
- Especially at scale

**Notes:**
BG/L machine makes a good case

## What Drives I/O in HPC?

- Not just providing performance with parallel I/O
- Three metrics on which we measure success:
- _Usability_ – How well I/O interfaces map to application data models and access patterns
- Solutions are unique to HPC
- _Performance and scalability_ – How well our I/O systems are tuned for common application patterns (e.g. concurrent access, noncontiguous access) and metadata access
- _Reliability and management_ – How much maintenance our parallel I/O systems require, how well they handle failures
- This talk covers all three areas, pointing out both successes and challenges

**Notes:**
How well the software works when applications use it; fraction of peak.

## Usability

## Application View of I/O

- _It doesn’t matter how fast the I/O system is if apps can’t use it well_
- Applications internally use complex data structures to organize data
- Ideally data would be stored in a similar format
- Canonical representation
- Typed data
- Multidimensional, unstructured datasets
- Attributes of data, of the run
- More domain or data model specificity leads to more convenience for applications
- But we can’t afford to rewrite everything for each application...

**Notes:**
Usability is always with respect to your *users*.  For computational science, our users are very different from “general purpose” users.

## Organization of I/O Software

- I/O components layeredto provide neededfunctionality (I/O stacks)
- Common APIs allowcombination of components
- Parallel file systemorganizes hardware intosingle, fast storage space
- I/O middleware matches to programming model, provides optimizations
- Example: collective I/O operations in MPI-IO
- High-level I/O libraries (HLLs) provide usability

## High-level I/O Libraries

- Provide structured data storage
- Multidimensional, typed datasets
- Attributes of data, provenance
- Metadata is placed in the file itself,simplifying data movement, archiving
- Two good examples
- _HDF5_ – first to use MPI-IO, widely used
- _PnetCDF_ – parallel API for netCDF data
- Compelling alternative to POSIX, MPI-IO
- Both of these are too low-level
- Important step, but still somewhat general...

## Challenge: Bridging the Usability Gap

- Applications still struggle touse this infrastructure
- Build new layers on top ofexisting I/O software stack
- Maximize code reuse
- Benefit from optimizations
- Match I/O interfaces to data models or domains
_**Must be a collaborative effort!**_
- Application people know the models
- I/O system people know the optimizations

## Challenge: Standard APIs to Wide-Area Data Access

- Recent trend: Accessing data between sites
- Tools for moving data across the wide area
- GridFTP
- Storage Resource Managers
- Logistical Networking
- Storage Resource Brokers
- Groups are developing MPI-IOinterfaces to various wide-areadata transfer tools
- SRM, GridFTP, SRB,Logistical Networks
- HDF5, PnetCDF between sites
- Performance can vary even morewidely than local file systems!

## Performance and Scalability

## Performance and Scalability

- Goal: Minimize the time applications spend performing I/O-related operations
- Maximize time applications spend computing
- End-to-end I/O performance includes
- Concurrent access to files
- For real application access patterns
- Metadata operations
- Creating files, traversing directories, etc.
- Overhead of all I/O software layers
- Features aren’t free

## Parallel File Systems

- Three popular parallel file system solutions
- GPFS
- Lustre
- PVFS/PVFS2
- All three being actively developed and deployed
- Competition in this space is good
- No “one size fits all” solution at this time
- All three already in use on BG/L systems!
- All capable of 10GByte/sec+ I/O rates, given adequate storage hardware and easy access patterns

- Updated results from “Shared Parallel Filesystems in Heterogeneous Linux Multi-Cluster Environments” by Cope, Oberg, Tufo, and Woitaszek of Univ. of Colorado at Boulder, using caggreIO benchmark.

## Complication: I/O Access Patterns

- Application I/O is often complex, not just big blocks
- Ignoring ghost cells, extracting subarrays
- Additional data stored by high-level I/O libraries
- These result in noncontiguous I/O
- I/O interfaces determine ability to extract performance
- Define the knowledge that the I/O system has to work with
- Standard (POSIX) file system interface does not allow for efficient noncontiguous access

## Supporting Noncontiguous I/O

- Three approaches for noncontiguous I/O
- Use POSIX and suffer
- Perform optimizations at the MPI-IO layer as work-around
- Augment the parallel file system
- Augmenting the parallel file system API is most effective

- Results from “Datatype I/O” prototype in PVFS1 with tile example

- POSIX I/O

- MPI-IOOptimizations

- PFS Enhancements

## Creating Files

- Even creating files can take significant time on very large machines!
- Why?
- It’s complicated 
- ...but it mostly has to do with the interface we have to work with and implications on synchronization
- What happens if we change this interface?

## Creating Files Efficiently

- Improving the file system interface improves performance for computational science
- Leverage communication in MPI-IO layer

- Time to Create Files Through MPI-IO

**Notes:**
Change in the interface allows us to take advantage of capabilities of other layers in I/O stack.

## High-Level I/O Library Performance

- High-level I/O libraries cost performance
- Second-generation high-level I/O libraries are showing promise
- Better leveraging features of MPI-IO
- Using simpler file models that allow for greater concurrency
- Still, performance is only a fraction of peak!
- Applications must make tough decisions in some cases between functionality/usability and performance

- The FLASH I/O benchmark, shows PnetCDF performance to be competitive with and in some cases significantly higher than HDF5 performance.  This is due to the light-weight, low-overhead nature of PnetCDF and its tight coupling to MPI-IO (results from ASCI Frost machine at LLNL, rates in MB/sec).  This work performed in collaboration with Alok Choudhary and Jianwei Li of NWU.

- Mbytes/sec

**Notes:**
Need to think about the performance of the I/O stack as a whole.

## Challenge: Minimizing I/O Costs

- Need other parallel file systems to adopt API enhancements
- Currently available in PVFS2 file system
- Standardizing extensions to POSIX I/O for HPC
- High-level I/O libraries need more work
- Caching components integrated into HLLs
- Or maybe I/O middleware?
- New file formats, tuned for performance

## Reliability and Management

## I/O System Complexity

- Sheer number of devices is an issue
- Administration (configuration and tuning)
- Reliability

## File System Administration

- It is the role of the parallel filesystem to organize and managethe I/O resources
- PFSes are themselves difficultto manage!
- Failure tolerance
- Tuning
- Installation and configuration
- Similar technologies (e.g. RDBs, networking) now need experts to manage them
- New software solutions can alleviate many of these problems for I/O systems

**Notes:**
What is ironic here...

## Autonomic Storage

- Self-healing, self-maintaining, self-tuning
- Adapts to device failures transparently
- Automatically integrates new storage devices
- Balances data to preserve performance
- Not a reality for parallel I/O, yet.
- New PFS designs integrate communication between servers
- Exchange information about health, load, allocated space
- Prototyping in PVFS2 parallel file system
- Next step will be to integrate policies, enforce them
- Moving data in response to policy decisions is the easy part!

## Impact of Hardware Failures

- More components usually means more failures
- Disk failures may be tolerated with RAID-like concepts
- Server failures may be tolerated with high availability approaches
- Client failures can be a real problem, especially at scale
- Clients will not all be online
- 99.99% up indicates ~6 nodes down at any time on a 64K node system
- 99.9% up indicates ~65 down at any time on same
- MTTFs of 6-8 hours on large DOE machines (e.g. ASCI Q)
- Need approaches that minimize impact of client failures

**Notes:**
Client failures are a real problem as we move into a world of systems with tens of thousands of nodes.

## NFS Did Get This Right...

- NFS (v3) doesn’t store important data on clients
- Known as “stateless” clients
- Client failures don’t impact servers or other clients
- Parallel file systems may be built similarly
- PVFS2 takes this approach
- But we lose traditional performance enhancements
- Such as client-side caching
- No room for cache on BG/L nodes anyway

**Notes:**
Notorious NFS did get this right.

## Challenge: Reliability, Manageability, and Performance

- Autonomic storage concepts are not yet reality for parallel file systems
- Maintaining predictable I/O performance in autonomic storage will be tricky!
- Getting both reliability and performance is a challenge
- Start with simple, stateless clients
- Analog to smaller OSes being used on clients
- Very difficult if we want to minimize cost!

**Notes:**
An added complication in bringing autonomic concepts into parallel I/O is the need for predictable I/O performance.
Traditional techniques for reliability require lots of additional hardware...expensive!

## Conclusions

## Summary

- Many recent successes in I/O for computational science
- Multiple file system options
- Multiple high-level interfaces available for applications
- Remote data access capabilities
- Usability, performance, management, and reliability of existing parallel I/O systems can all be improved
- Application interfaces aren’t convenient to use
- Observed performance rarely reaches peak performance
- Parallel file systems are difficult to manage, require too much expertise, and are “reliability challenged”
- Development and adoption of solutions to these issues are critical to the future success of HPC systems

**Notes:**
For this group, I think that usability is the #1 problem.

## It’s (Almost) All About Interfaces

- APIs play a fundamental role in I/O system software development and use
- Organization of components into I/O stacks using common APIs
- Development of new, domain- or model-specific I/O libraries for better usability
- Extensions to traditional parallel file system interfaces to increase performance
- Common interfaces for wide-area data access
- More database-like interfaces for finding data in file systems
- Changing interfaces is never easy!

**Notes:**
Both interfaces that we as I/O system software people use internally and interfaces that you as application groups see and use to interact with these systems.

## Looking Forward

- Efforts are underway to revitalize I/O system software to tackle problems for current and future HPC systems
- Deployment and adoption of these solutions will enable new and more data-oriented applications
- It has to be a team effort
**Scientific Data Management SciDAC is actively pursuing these collaborations**
- 	If you can’t get enough I/O, attend our “Parallel I/O in Practice” tutorial at SC2005.

## Acknowledgements

- The Scientific Data Management Center
- Colleagues at ANL
- W. Gropp, R. Thakur, S. Lang, R. Latham, J. Lee
- Members of the I/O and data management community and their respective teams
- A. Choudhary, Northwestern University
- W. Ligon, Clemson University
- P. Wyckoff, Ohio Supercomputer Center
- A. Shoshani, Lawrence Berkeley National Laboratory
- N. Samatova, Oak Ridge National Laboratory
- G. Grider, Los Alamos National Laboratory
- L. Ward, Sandia National Laboratories
- T. Critchlow and W. Loewe, Lawrence Livermore National Laboratory
- D.K. Panda, Ohio State University
- G. Gibson, Panasas
- R. Haskin, IBM
-       This work was supported by the Mathematical, Information, and Computational Sciences Division subprogram of the Office of Advanced Scientific Computing Research, Office of Science,  U.S. Department of Energy, under Contract W-31-109-ENG-38.