# Presentation: 432443

## Investigation of Leading HPC I/O Performance Using a Scientific Application Derived Benchmark

- Julian Borrill, Leonid Oliker, John Shalf, Hongzhang Shan
- Computational Research Division/
- National Energy Research Scientific Computing Center (NERSC)
- Lawrence Berkeley National Laboratory

## Overview

- Motivation
- Demands for computational resources growing at rapid rate
- Racing toward very high concurrency petaflop computing
- Explosion of sensor & simulation data make I/O critical component
- Present MADbench2: lightweight, portable, parameterized I/O benchmark
- Derived directly from CMB analysis package
- Allows study under realistic I/O demands and patterns
- Discovered optimizations can be fed back into scientific code
- Tunable code allows I/O exploration of new and future systems
- Examine I/O performance across 7 leading HEC systems
- Luster (XT3, IA-64 cluster), GPFS (Power5, AMD-cluster)BG/L (GPFS and PVFS2), CXFS (SGI Altix)
- MORE???? (what is different about this from other I/O benchmarks)
- Schtick: this is application driven (cannot understand results of other benchmarks in the context of application requirements)

## Cosmic Microwave Background

- After Big Bang, expansion of space cools the Universe until it falls below the ionization temperature of hydrogen when free electrons combine with protons
- With nothing to scatter off, the photons then free-stream; the CMB is therefore a snapshot of the Universe at the moment it first becomes electrically neutral about 400,000 years after the Big Bang
- Tiny anisotropies in CMB radiation aresensitive probes of cosmology

***Cosmic**** - primordial photons filling all space*
***Microwave**** - red-shifted by the continued expansion of the Universe from 3000K at last scattering to 3K today*
***Background**** - coming from “behind” all astrophysical sources.*

**Notes:**
Early Universe too hot to be completely ionized, so photons ping around bouncing off the free electrons. Whencooled, hydrogen is formed and the photons free-stream. The CMB is an image of this last-scattering surface(really a moment in time but appears as a surface in space to an observer)
Continued expansion of Universe stretches photons’ wavelengths from 3000K black-body at last scattering to3K black body today. Unlike other photons detected by astronomers, the CMB consists of primordial photons

## CMB Science

- The CMB is a unique probe of the very early Universe
- Tiny fluctuations in its temperature (1 in 100K) and polarization (1 in 100M) encode the fundamental parameters of cosmology, including the geometry, composition (mass-energy content), and ionization history of the Universe
- Combined with complementary supernova measurements tracing the dynamical history of the Universe, we have an entirely new “concordance” cosmology: 70% dark energy + 25% dark matter + 5% ordinary matter
- Nobel prizes: 1978 (Penzias & Wilson) detection CMB, 2006 (Mather & Smoot) detection CMB fluctuations.

- Dark Energy

**Notes:**
Supernovae tells us about the dynamics of the Universe, and hence the difference between dark energy (whichpushes) and dark matter and ordinary matter (which pull)
CMB says y + x = 1; SN1a says y - x = 0.4; therefore y = 0.7 and x = 0.3From other measurements, we know x_ordinary = 0.05. Thus we are in the dark about 95% of the Universe!

## CMB Data Analysis

- from the time domain:precise high-resolution measurements of the microwave sky - *O*(1012)
- to the pixel domain:pixelized sky map - *O*(108)
- and finally to the multipole domain:angular power spectrum (most compact sufficient statistic for CMB) - *O*(104)
- 	calculating the compressed data and their reduced error bars (data correlations for error/uncertainity analysis) at each step
- Problem exacerbated by an explosion in dataset sizes as cosmologists try to improve accuracy
- HEC has therefore become an essential part of CMB data analysis

**CMB Data Analysis**

**Notes:**
Data sizes quoted here applicable to the ESA/NASA Planck satellite mission due to launch in 2007
Angular power spectrum captures the strength of the correlations on various angular scales
With the advent of E- and B-mode polarization measurements, we are moving from a single temp spectrum(top black line) to 3 auto (TT, EE, BB) and 3 cross (TE, TB=0, EB=0) spectra
Spectral plot shows the 4 non-zero spectra, and notes some of the key physics we hope to measure:Reionization history of Universe shows up as a TE bump on large scales (low spherical harmonic multipoles)while the holy grail BB spectrum has the potential to show us signals from gravity waves generated duringinflation (10^{-35} secs after Big Bang) as well as from gravitational lensing by intervening masses (e.g. clustersof galaxies)

## MADbench2 Overview

- Unlike most I/O benchmarks, MADbench2 is derived directly from important app
- Benchmark retains operational complexity and integrated system requirements of the full science code
- Eliminated special-case features, preliminary data checking, etc.
-  Out-of-core calculation because of large size of pixel-pixel correlation matrices
- Holds at most three matrices in memory at any one time
- MADbench2 used for
- Procuring supercomputers and filesystems
- Benchmarking and optimizing performance of realistic scientific applications
- Comparing various computer system architectures

**MADbench2 Overview**

## Computational Structure

- Derive spectra from sky maps by:
- _Compute, __Write __(Loop):_ Recursively build sequence of Legendre polynomial based CMB signal pixel-pixel correlation component matrices
- _Compute/Communicate_: Form and invert CMB signal & noise correlation matrix
- _Read__, Compute, __Write __(Loop)_: Read each CMB component signal matrix, multiply by inverse CMV data correlation matrix, write resulting matrix to disk
- _Read__, Compute/Communicate (Loop):_ In turn read each pair of these result matrices and calculate trace of their product
- Recast as benchmarking tool: all scientific detail removed, allows varying busy-work component to measure balance between computational method and I/O

## MADbench2 Parameters

- IOMETHOD - either POSIX of MPI-IO data transfers
- IOMODE - either synchronous or asyncronous
- FILETYPE - either unique (1 file per proc) or shared (1 file for all procs)
- BWEXP - the busy work exponent 
- _Command-Line Arguments:_
- NPIX - number of pixels (matrix size)
- NBIN - number of bins (matrix count)
- BScaLAPCK - ScaLAPACK blocksize
- FBLOCKSIZE - file Blocksize
- MODRW - IO concurrency control (only 1 MODRw procs does IO simultaneously)

**MADbench2 Parameters**

| CPU | BG/L CPU | NPIX | NBIN | Mem (GB) | DISK (GB) |
| --- | --- | --- | --- | --- | --- |
| ---  | 16 | 12,500 | 8 | 6 | 9 |
| 16 | 64 | 25,000 | 8 | 23 | 37 |
| 64 | 256 | 50,000 | 8 | 93 | 149 |
| 256 | --- | 100,000 | 8 | 373 | 596 |

## Parallel Filesystem Overview

| MachineName | Parallel FileSystem | ProcArch | InterconnectCompute to I/O Node | MaxNode BW
to IO | MeasuredNode
BW (GB/s) | I/OServers/Clients | MaxDisk
BW(BG/s) | TotalDisk 
(TB) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Jaguar | Lustre | AMD | SeaStar-1 | 6.4 | 1.2 | 1:105 | 22.5 | 100 |
| Thunder | Lustre | IA64 | Quadrics | 0.9 | 0.4 | 1:64 | 6.4 | 185 |
| Bassi | GPFS | Pwr5 | Federation | 8.0 | 6.1 | 1:16 | 6.4 | 100 |
| Jacquard | GPFS | AMD | Infiniband | 2.0 | 1.2 | 1:22 | 6.4 | 30 |
| SDSC BG/L | GPFS | PPC | GigE | 0.2 | 0.2 | 1:8 | 8 | 220 |
| ANL BG/L | PVFS2 | PPC | GigE | 0.2 | 0.2 | 1:32 | 1.3 | 7 |
| Columbia | CXFS | IA64 | FC4 | 1.6 | N/A | N/A | 1.6 | 600 |

- Lustre, GPFS, PVFS2

- CXFS

## Jaguar Performance

- Highest synchronous unique read/write performance of all evaluated platforms
- Small concurrencies insufficient to saturate I/O
- Seastar max throughput  1.1 GB/s
- System near theoretical I/O peak at P=256
- Reading is slower than writing due to buffering
- Unlike *unique* files, *shared *files performance is uniformly poor:
- Default I/O traffic only uses 8 of 96 OSTs
- OST restriction allows consistent performance, but limits single job access to full throughput
- Using 96 OSTs (lstripe) allows comparable performance between unique and shared
- OST 96 is not default due to:
- Increase risk job failure
- Exposes jobs to more I/O interference
- Reduce performance of *unique* file access

- Default

- With Striping

- Lustre 5,200 dual-AMD node XT3 @ ORNL
- Seastar-1 via HyperTransport in 3D Torus
- Catamount: compute PE, Linux: service PEs
- 48 OSS, 1 MDS, 96 OST, 22.5 GB/s I/O peak

## Thunder Performance

- Second highest overall *unique* I/O performance
- Peak and sustained a fraction of Jaguar
- I/O  trend very similar to Lustre Jaguar system
- Writes outperform reads (buffering)
*Shared* significantly slower than *unique*
- Unlike Jaguar, attempts to stripe did not improve performance
- Difference likely due to older hw and sw
- Future work will examine performance on updated sw environment

- Lustre 1,024 quad-Itanium2 node @ LLNL
- Quadrics Elan4 fat-tree, GigE, Linux
- 16 OSS, 2 MDS, 32 OST, 6.4 GB/s peak

## Bassi & Jacquard Performance

- Unlike Lustre, Bassi and Jacquard’s attain similar *shared *and *unique* performance
*Unique *I/O significantly slower than Jaguar
- Bassi and Jacquard attain high *shared* performance with no special optimization
- Bassi quickly saturates I/O due to high BW node to I/O interconnect
- Higher read behavior could be result of GPFS prefetching
- Jacquard continues to scale at 256 indicating that GPFS NFS have not been saturated
- Bassi outperforms Jacquard due to superior node to I/O BW (8 vs 2 GB/s)

- _Bassi_ GPFS 122 8-way Power5, AIX, Federation, fat-tree
- 6 VSD, 16 FC links, 6.4 GB/s peak @ LBNL
- _Jacquard_ GPFS 320 dual-AMD, Linux, Infiniband, fat-tree
- IB4X,12x (leaves, spine), peak 4.2 GB/s (IP over IB) @ LBNL

- Bassi

- Jacquard

## SDSC BG/L Performance

- BG/Ls have lower performance but are the smallest systems in our study (1024 node)
- Original SDSC configuration rather poor I/O and scaling
- Upgrade (WAN) comparable with Jacquard, and continues to scale at P=256
- Wan system: many more spindles and NSDs and thus higher available bandwidth
- Like other GPFS systems: *unique* and *share* show similar I/O rate with no tuning required

- GPFS 1,024 dual-PPC @ SDSC
- Global Tree, CNK (compute), Linux (service)
- 1:8 I/O servers to compute, forwarding via GigE
- Original: 12 NSD, 2 MDS, Upgrade: 50 NSD, 6 MDS

## ANL BG/L Performance

- Low I/O throughput across configurations
- Drop off in read performance beyond P=64
- Attempts to tune I/O performance did not succeed
- Raid “chunk” size, striping
- Future work will continue exploring optimizations
- Normalized compute-server ratio (8:1 vs 32:1) w/ SDSC by using 4x ANL procs with 3 of 4 idle
- Improved ANL 2.6x, still 4.7x slower vs SDSC
- Ratio of I/O nodes is only 1 of many factors

- PVFS2 1,024 dual-PPC @ ANL
- Global Tree, CNK (compute), Linux (service)
- 1:32 I/O servers to compute (vs 1:8 at SDSC)
- Peak I/O BW 1.3 GB/s

## Columbia Performance

- Default I/O rate lowest of evaluated systems
- Read/*Shared* peaks at P=16I/O interface of Altix CCNUMA shared across node
- Higher P does not increase BW potential
- With increasing concurrency:
- Higher lock overhead (access buffer cache)
- More contention to I/O subsystem
- Potentially reduced coherence of I/O request
- DirectIO bypasses block-buffer cache, presents I/O request directly to disk subsystem from memory
- Prevents block buffer cache reuse
- Complicated I/O, each transaction must be block-aligned on disk
- Has restrictions on mem alignment
- Forces programming in disk-block sized I/O as opposed to arbitrary size POSIX I/O
- Results show DirectIO significantly improves I/O
- Saturation occurs at low P (good for low P jobs)
- Columbia CCNUMA also offers option of using idle procs for I/O buffering for high-priority jobs

- CXFS, 20 Altix3700, 512-way IA64 @ NASA
- 10,240 procs, Linux, NUMAlink3
- Clients to FC without intervening storage server
- 3MDS via GigE, max 4 FC4, peak 1.6 GB/s

- Default

- Direct I/O

## Comparative Performance

- Summary text here

## Asynchronous Performance

- Most examined systems saturate at only P=256 - concern for ultra-scale
- Possible to hide I/O behind simultaneous calculation in MADbench2 via MPI-2
- Only 2 of 7 systems (Bassi and Columbia) support fully asynchronous I/O
- We develop busy-work exponent corresponds to O(N) flops
- Bassi and Columbia improve I/O by almost 8x for high peak improvement)
- Bassi now shows 2x the performance of Jaguar
- As expected small reproduced synchronous behavior
- Critical value for transition is between 1.3-1.4 ie algorithms > O(N2.6)
- Only BLAS3 computations can effectively hide I/O. If balance between computational and I/O rate continue to decline, effective will increase:However, we are quickly approaching the practical limit of BLAS3 complexity!

## Conclusions

- I/O critical component due to exponential growth sensor & simulated data
- Presented one of most extensive I/O analyses on parallel filesystems
- Introduced MADbech2 derived directly from CMB analysis:
- Lightweight, portable, generalized to varying computations ()
- POSIX vs MPI-IO, shared vs unique, synch vs async
- Concurrent accesses work properly with modern POSIX API (same as MPI-IO)
- It is possible to achieve similar behavior between shared and unique file access!
- Default for all systems except Lustre which required trivial mod
- Varying concurrency can saturate underlying disk subsystem
- Columbia saturates at P=16, while SDSC BG/L did not saturate even at P=256
- Asynchronous I/O offers tremendous potential, but supported by few systems
- Defined amount of computation by I/O data via 
- Showed that computational intensity required to hide I/O is close to BLAS3
- Future work: continue evaluating latest HEC, explore effect of inter-processor communication on I/O behavior, conduct analysis of I/O variability.

## Acknowledgments

- We gratefully thank the following individuals for their kind assistance:
- Tina Bulter, Jay Srinivasan (LBNL)
- Richard Hedges (LLNL)
- Nick Wright (SDSC)
- Susan Coughlan, Robert Latham, Rob Ross, Andrew Cherry (ANL)
- Robert Hood, Ken Taylor, Rupak Biswas (NASA-Ames)