# Presentation: 192956

## IO File Striping

**IO File Striping**
**Katie Antypas**
**User Services Group**
**Kantypas@lbl.gov**
**NERSC User Group Meeting**
**October 2, 2007**

## Outline

**File Striping**
**Definition**
**Default Striping**
**Pros and Cons**
**One file-per-processor**
**Shared Files**
**Gotchas**

## We wish users didn’t have to deal with striping
Ideally it would be hidden from users
Unfortunately however, performance is too critical to ignore it

**Ideally it would be hidden from users**
**Unfortunately however, performance is too critical to ignore it**

**Motivation**

## What is File Striping?

**Lustre file system on Franklin made up of an underlying set of parallel I/O servers **
**OSSs (Object Storage Servers) - nodes dedicated to I/O connected to high speed torus interconect**
**OSTs (Object Storage Targets) software abstraction of physical disk (1 OST maps to 1 LUN)**
**File is said to be striped when read and write operations access multiple OSTs concurrently**
**Striping can increase I/O performance since writing or reading from multiple OSTs simultaneously increases the available I/O bandwidth**

## Franklin Configuration

**Franklin Configuration**

**20 OSS**
**80 OST**

**Franklin Compute and Interactive Nodes**

**“****The Torus”**

**5 DDN**
**80 LUN**

**FC Network**

*...*

*Connectivity and configuration set in a “good” way for parallelism. Using 20 OSTs will spread evenly over the 5 DDN appliances.*

*...*

## Default Striping on /scratch

**3 parameters characterize striping pattern of a file **
**Stripe count**
**Number of OSTs file is split across**
**Default is 4**
**Stripe size **
**Number of bytes to write on each OST before cycling to next OST**
**Default is 1MB**
**OST offset **
**Indicates starting OST**
**Default is round robin across all requests on system**

- I/O Servers

- OSTs

- OSS 0

- OSS 1

- OSS 2

- OSS 3

- OSS 4

- OSS 5

- OSS 19

- 0,20,40,60

- 1,21,41,61

- 2,22,42,62

- 3,23,43,63

- 4,24,44,64

- 5,25,45,65

- 19,39,59,79

## Max Bandwidth of OST and OSS

- 4 OSTs

- Storage Servers

- Compute Nodes

- Max Bandwidth to single OST ~350 MB/sec

- Torus Network

- 4 OSTs

- Max Bandwidth to single OSS ~700 MB/sec

## Default Stripe Count of 4 on /scratch

**Pros**
**Get 4 times the bandwidth you could from using 1 OST**
**Max bandwidth to 1 OST ~ 350 MB/Sec**
**Using 4 OSTs ~1,400 MB/Sec**
**Cons **
**For better or worse your file now is in 4 different places**
**Metadata operations like ‘ls -l’ on the file could be slower**
**For small files (<100MB) no performance gain from striping**

- I/O Servers

- OSS 1

- OSS 2

- OSS 3

- OSS 4

- OSS 5

- OSS 6

- OSS 19

- OSTs

- 0,20,40,60

- 1,21,41,61

- 2,22,42,62

- 3,23,43,63

- 4,24,44,64

- 5,25,45,65

- 19,39,59,79

## Why a stripe count of 4?

**Balance **
**With a few important exceptions, should work decently for most users**
**Protection **
**Each OST is backed up by a physical disk (LUN)**
**Stripe count of 1 leave us vulnerable to single user writing out huge amount of data filling the disk**
**Striping of 4 is a reasonable compromise, although not good for large shared files or certain file-per-proc cases**

- OSS 1

- OSS 2

- OSS 3

- OSS 4

- OSS 5

- OSS 6

- OSS 19

- OSTs

- 0,20,40,60

- 1,21,41,61

- 2,22,42,62

- 3,23,43,63

- 4,24,44,64

- 5,25,45,65

- 19,39,59,79

## Changing the Default Stripe Count

**A number of applications will see benefits from changing the default striping**
**Striping can be set at a file or directory level**
**When striping set on a directory: all files created in that directory with inherit striping set on the directory**
**Stripe size - # bytes written on each OST before cycling to next OST**
**OST offset - indicates starting OST **
**Stripe count - # of OSTs file is split across**

**lstripe <directory|file> <stripe size> <OST Offset> ****<stripe count>**** **

**lstripe mydirectory 0 -1 ****X**** **

## Parallel I/O Multi-file 

- 0

- 1

- 2

- 3

- 4

- File

- File

- File

- File

- File

- processors

**Each processor writes its own data to a separate file**
**Advantages**
**Simple to program**
**Can be fast -- (up to a point)**
**Disadvantages**
**Can quickly accumulate many files**
**Hard to manage**
**Requires post processing**
**Difficult for storage systems, HPSS, to handle many small files**

- 5

- File

## One File-Per-Processor IO with Default Striping

***With greater than roughly 20 processors/files, each striping over 4 OSTs, files begin to overlap with each other on OSTs***
***Stripe count of 4 is not helping application get more overall bandwidth***
***Stripe count of 4 can lead to contention  ***

- OSTs

- OSS 0

- OSS 1

- OSS 2

- OSS 3

- OSS 4

- OSS 5

- 0

- 1

- 2

- 3

- 4

- 5

- Torus Network

- 40,000

- 0,20,40,60

- 1,21,41,61

- 2,22,42,62

- 3,23,43,63

- 4,24,44,64

- 5,25,45,65

## One File-Per-Processor IO with Stripe Count of 1

***Use all OSTs but don’t add more contention than is necessary***

- 4 OSTs

- OSS 0

- OSS 1

- OSS 2

- OSS 3

- OSS 4

- OSS 5

- Torus Network

- 0

- 1

- 2

- 3

- 4

- 5

- 40,000

- OSS 19

## Recommendations for One-File-Per Processor IO

| Use stripe count of 1
“lstripe mydir 0 -1 1” | Use stripe count of 1
“lstripe mydir 0 -1 1” |
| --- | --- |
| Don’t worry too much
Default striping fine
Maybe a small performance benefit from a stripe count of 1
Do nothing or 
“lstripe mydir 0 -1 1” | Try to use all 80 OSTs
A stripe count of 1, 2 or 4
80p “lstripe mydir 0 -1 1”
40p “lstripe mydir 0 -1 2” 
20p “lstripe mydir 0 -1 4” |

- > 1GB

- <100s MB

- < 80

- > 80

- File Size Per Processor

- Number of Processors

## Parallel I/O Single-file 

- 0

- 1

- 2

- 3

- 4

- File

- processors

**Each processor writes its own data to the same file using MPI-IO mapping**
**Advantages**
**Single file**
**Manageable data**
**Disadvantages**
**Lower performance than one file per processor at some concurrencies**

- 5

## Shared File I/O with Default Stripe Count 4

***All processors writing shared file will write to 4 OSTs***
***No matter how much data the application is writing, it won’t get more than ~1,400 MB/Sec (4 OSTs * 350 MB/Sec)***
***Less sophisticated than you might think - no optimizations for matching processor writer to same OST***
***Need to use more OSTs for large shared files***

- OSS 0

- OSS 1

- OSS 2

- OSS 3

- OSS 4

- OSS 5

- Torus Network

- 0

- 1

- 2

- 3

- 4

- 5

- 40000

- OSS 19

## Shared File I/O with Default Stripe Count 80

***Now Striping over all 80 OSTs***
***Increased available bandwidth to application***
***Theoretically (700 MB/Sec (OSS Max) * 20 OSSs)***
***In practice 11-12 GB/Sec***

- OSS 0

- OSS 1

- OSS 2

- OSS 3

- OSS 4

- OSS 5

- 0

- 1

- 2

- 3

- 4

- 5

- 40000

- OSS 19

- Torus Network

## Reduced Writers to Single-file 

- 0

- 1

- 2

- 3

- 4

- File

- processors

**On Franklin best performance when writers matches # of OSTs (80)**
**Subset of processors writes data to single file**
**Functionality not yet available in XT4 MPI-IO**
**Advantages**
**Single file; manageable data**
**Better performance than all tasks writing for high concurrency jobs**
**Disadvantages**
**Requires changes to code**
**Application may not have enough memory to handle data from other processors**

- 5

## Recommendations

- >4096

- 2048

- 1024

- <512

- <1 GB

- 10 GB

- 100 GB

- 1 TB

- Aggregate File Size

- Processors

- Single Shared File, Default Striping

- Single Shared File, Stripe over all OSTs

- Try fewer writers

- Single Shared File, Stripe over some OSTs (~11)

*Legend*

|  |  | Stripe over some OSTs
“lstreip mydir 0 -1 11”
 | Stripe over all OSTs
“lstripe 0 -1 80”
Try fewer writers |
| --- | --- | --- | --- |
| 
 |  |  |  |
|  |  | Stripe over all OSTs
“lstripe mydir 0 -1 80”
 |  |
| 
 |  |  | N/A |

**Don’t do anything **
**Default striping fine**

**Stripe over some OSTs**
**“****lstripe dir 0 -1 11”**

## Striping Summary

**Franklin Default Striping**
**Stripe size - 1MB (enter ‘0’ for default)**
**OST offset - round robin starting OST (enter ‘ -1’ for default)**
**Stripe over 4 OSTs (Stripe count 4)**
**One File-Per-Processor**
**“****lstripe mydir 0 -1 1”**
**Large shared files **
**“****lstripe mydir 0 -1 80”**
**Medium shared files**
**Experiment a little 10-40 OSTs**
**“****lstripe mydir 0 -1 11”**

## Gotchas

**File system is a shared resource and a heavy I/O job can slow the system down for everyone**
**Write fewer large blocks of data rather than many small chunks of data**
**Some MPI-IO features under optimized**
**Collective I/O**
**2 Phase I/O**
**Lustre file system sensitive write size and write offset**

***Please contact consultants if you have low performing I/O.  There may be something simple we can do to increase performance substantially***

- OSS 2

- OSS 3

- OSS 4

- OSS 5

## Cray/NERSC I/O Initiatives

**Working to address and characterize I/O variation**
**Benchmarking runs**
**Lustre Monitoring Tool**
**Improving MPI-IO layer**
**Multiple people at Sun and Cray working on this**
**Discussions forming about improving HDF5 performance on Lustre**

## Best Practices

**Add striping line to batch script so you don’t forget it!**
**Do large I/O: write fewer big chunks of data (1MB+)  rather than small bursty I/O**
**Do parallel I/O. **
**Serial I/O (single writer) can not take advantage of the system’s parallel capabilities.**
**Stripe large files over many OSTs.**
**If job uses many cores, reduce the number of tasks performing IO (experiment with this number, 80, 160, 320)**
**Use a single, shared file instead of 1 file per writer, esp. at high parallel concurrency.**
**Use an IO library API and write flexible, portable programs.**

## Extra Slides

## Parallel I/O: A User Perspective

**Wish List**
- Write data from multiple processors into a single file
- File can be read in the same manner regardless of the number of CPUs that read from or write to the file. (eg. want to see the logical data layout... not the physical layout)
- Do so with the same performance as writing one-file-per-processor (only writing one-file-per-processor because of performance problems)
- And make all of the above portable from one machine to the next

## I/O Formats

## Common Storage Formats

**ASCII:  **
**Slow**
**Takes more space!**
**Inaccurate**
**Binary**
**Non-portable (eg. byte ordering and types sizes)**
**Not future proof**
**Parallel I/O using MPI-IO**
**Self-Describing formats**
**NetCDF/HDF4, HDF5, Parallel NetCDF**
**Example in HDF5: API implements Object DB model in portable file**
**Parallel I/O using: pHDF5/pNetCDF (hides MPI-IO)**
**Community File Formats**
**FITS, HDF-EOS, SAF, PDB, Plot3D**
**Modern Implementations built on top of HDF, NetCDF, or other self-describing object-model API**

## HDF5 Library

**Can store data structures, arrays, vectors, grids, complex data types, text**
**Can use basic HDF5 types integers, floats, reals or user defined types such as multi-dimensional arrays, objects and strings**
**Stores metadata necessary for portability - endian type, size, architecture**

*HDF5 is a general purpose library and file format for storing scientific data*

## HDF5 Data Model

**Groups**
**Arranged in directory hierarchy**
**root group is always ‘/’**
**Datasets**
**Dataspace**
**Datatype**
**Attributes**
**Bind to Group & Dataset**
**References**
**Similar to softlinks**
**Can also be subsets of data**

- “/”
- (*root*)

- “Dataset0”
*type,space*

- “Dataset1”
*type, space*

- “subgrp”

- “time”=0.2345

- “validity”=None

- “author”=Jane Doe

- “Dataset0.1”
*type,space*

- “Dataset0.2”
*type,space*

- “date”=10/24/2006

## A Plug for Self Describing Formats ...

**Application developers shouldn’t care about about physical layout of data**
**Using own binary file format forces user to understand layers below the application to get optimal IO performance**
**Every time code is ported to a new machine or underlying file system is changed or upgraded, user is required to make changes to improve IO performance**
**Let other people do the work**
**HDF5 can be optimized for given platforms and file systems by HDF5 developers**
**User can stay with the high level**
**But what about performance?**

## IO Library Overhead

**IO Library Overhead**

- Data from Hongzhang Shan

*Very little, if any overhead from HDF5 for one file per processor IO compared to Posix and MPI-IO*

## Ways to do Parallel IO

## Serial I/O 

- 0

- 1

- 2

- 3

- 4

- File

- processors

**Each processor sends its data to the master who then writes the data to a file**
**Advantages**
**Simple**
**May perform ok for very small IO sizes**
**Disadvantages**
**Not scalable**
**Not efficient, slow for any large number of processors or data sizes**
** ****May not be possible if memory constrained**

- 5

## Parallel I/O Multi-file 

- 0

- 1

- 2

- 3

- 4

- File

- File

- File

- File

- File

- processors

**Each processor writes its own data to a separate file**
**Advantages**
**Simple to program**
**Can be fast -- (up to a point)**
**Disadvantages**
**Can quickly accumulate many files**
**Hard to manage**
**Requires post processing**
**Difficult for storage systems, HPSS, to handle many small files**

- 5

- File

## Flash Center IO Nightmare...

**Large 32,000 processor run on LLNL BG/L**
**Parallel IO libraries not yet available**
**Intensive I/O application**
**checkpoint files .7 TB, dumped every 4 hours, 200 dumps**
**used for restarting the run**
**full resolution snapshots of entire grid**
**plotfiles - 20GB each, 700 dumps**
**coarsened by a factor of two averaging**
**single precision**
**subset of grid variables**
**particle files 1400 particle files 470MB each**
**154 TB of disk capacity**
**74 million files!**
**Unix tool problems**
**2 Years Later still trying to sift though data, sew files together**

## Parallel I/O Single-file 

- 0

- 1

- 2

- 3

- 4

- File

- processors

**Each processor writes its own data to the same file using MPI-IO mapping**
**Advantages**
**Single file**
**Manageable data**
**Disadvantages**
**Lower performance than one file per processor at some concurrencies**

- 5

## Parallel IO single file

- 0

- 1

- 2

- 3

- 4

- 5

- processors

- array of data

***Each processor writes to a section of a data array.  Each must know its offset from the beginning of the array and the number of elements to write***

## Trade offs

*It isn’t hard to have speed, portability or usability. It is hard to have speed, portability and usability in the same implementation*

**Ideally users want speed, portability and usability**
**speed - one file per processor**
**portability - high level IO library**
**usability**
**single shared file and **
**own file format or community file format layered on top of high level IO library**

## Good I/O Performance with Simple I/O Patterns

**File system capable of high performance for shared files**
**Large block sequential I/O**
**Transfer size multiple of stripe size**
**No metadata**

## More complicated I/O patterns

**Harder for file system to handle**
**Smaller amounts of data, (MBs/proc)**
**Transfer size not multiple of stripe width**
**Start offset doesn’t match stripe width**
**Strided data**
**Can result in lower shared file performance**

## Description of IOR

**Developed by LLNL used for purple procurement**
**Focuses on parallel/sequential read/write operations that are typical in scientific applications**
**Can exercise one file per processor or shared file access for common set of testing parameters **
**Exercises array of modern file APIs such as MPI-IO, POSIX (shared or unshared), HDF5 and parallel-netCDF**
**Parameterized parallel file access patterns to mimic different application situations**

## Benchmark Methodology 

***Focus on performance difference between single shared and one file per processor***

## Small Aggregate Output Sizes 100 MB - 1GB

*One File per Processor vs Shared File - GB/Sec*

*Aggregate File Size 100 MB*

*Clearly the ‘one file per processor’ strategy wins in the low concurrency cases correct?*

*Aggregate File Size 1 GB*

- Peak performance line - Anything greater than this is due to caching effect or timer granularity

## Small Aggregate Output Sizes 100 MB - 1GB

***One File per Processor vs Shared File - Time***

*Aggregate File Size 100 MB*

*But when looking at absolute time, the difference doesn’t seem so big...*

*Aggregate File Size 1 GB*

## Aggregate Output Size 100GB

**Aggregate Output Size 100GB**

***One File per Processor vs Shared File***

*Rate: GB/Sec*

*Time: Seconds*

*Is there anything we can do to improve the performance of the 4096 processor shared file case ?*

-  2.5 mins

- 390 MB/proc

- 24 MB/proc

- Peak performance line

## Aggregate Output Size 1TB

***One File per Processor vs Shared File***

*Rate: GB/Sec*

*Time: Seconds*

- ~ 3 mins

*Is there anything we can do to improve the performance of the 4096 processor shared file case ?*

- 976 MB/proc

- 244 MB/proc

## Recommendations

**Think about the big picture**
**Run time vs Post Processing trade off**
**Decide how much IO overhead you can afford**
**Data Analysis**
**Portability**
**Longevity**
**H5dump works on all platforms**
**Can view an old file with h5dump**
**If you use your own binary format you must keep track of not only your file format version but the version of your file reader as well**
**Storability**

## Recommendations

**Use a standard IO format, even if you are following a one file per processor model**
**One file per processor model really only makes some sense when writing out very large files at high concurrencies, for small files, overhead is low**
**If you must do one file per processor IO then at least put it in a standard IO format so pieces can be put back together more easily**
**Splitting large shared files into a few files appears promising**
**Option for some users, but requires code changes and output format changes**
**Could be implemented better in IO library APIs**
**Follow striping recommendations**
**Ask the consultants, we are here to help!**

## Questions?