![](media/image1.jpeg){width="1.4791666666666667in"
height="0.4791666666666667in"} **Advanced Scientific Computing
Research**

**Computer Science**

**FY 2006 Accomplishment**

**Enabling Efficient Access to Scientific Data**

## 

## Robert Ross\*, Rajeev Thakur, William Gropp, Argonne National Laboratory

## Alok Choudhary, Wei-keng Liao, Northwestern University

Arie Shoshani, Ekow Otoo, Lawrence Berkeley National Laboratory

Jeffrey Vetter, Oak Ridge National Laboratory

Summary

![](media/image3.wmf){width="4.3125984251968505in"
height="4.665354330708661in"}***Today's scientific applications demand
that high-performance I/O be part of their operating environment. These
applications access datasets of many gigabytes or terabytes, checkpoint
frequently, and create large volumes of visualization data. Such
applications are hamstrung by bottlenecks anywhere in the I/O path,
including the storage hardware, file system, low-level I/O middleware,
application-level interface, and in some cases the mechanism used for
Grid I/O access. Our work addresses inefficiencies in all the software
layers by carefully balancing the needs of scientists with
implementations that allow the expression and exploitation of
parallelism in access patterns.***

**Parallel NetCDF**

Scientific application developers desire flexible file formats that map
closely to the data structures used in the applications and store the
data together with its associated attributes. One of such standard file
formats commonly used in scientific community is netCDF. While very
successful, the netCDF interface is proving inadequate for parallel
applications because of its lack of a parallel access mechanism.

To provide the broad community of netCDF users with a high-performance,
parallel interface for accessing netCDF files, we have defined an
alternative parallel API, Parallel netCDF (PnetCDF). The parallel API
maintains the look and feel of the serial netCDF interface and preserves
the file format for backward compatibility to legacy codes.

MB/s

*Figure 1: Using the FLASH I/O benchmark, we see the benefits of the
PnetCDF API and format over its competitor HDF5 on the IBM Power system
at LLNL.*

The implementation is built on top of MPI-IO. The use of MPI-IO allows
us to benefit from many existing optimizations built into the MPI-IO
implementations and provides portability to all current HEC systems.
Performance is as much as a factor of 10 faster than competing libraries
for application benchmarks such as the FLASH I/O kernel (Figure 1). Most
recently we have augmented the PnetCDF API to allow for asynchronous I/O
and to better describe user's data buffers to eliminate unnecessary data
copying.

**SRM/MPI-IO**

A growing number of computational science applications operate on
distributed datasets. These applications suffer because there is no
standard API for accessing data located both remotely and locally, so
either these applications have to adopt proprietary APIs, or they must
copy data locally before jobs execute.

The goal of the SRM/MPI-IO effort is to enable access to remote data
sources through the standard MPI-IO interface. Applications using this
software can transparently access data held by Storage Resource Managers
(SRMs), entities that manage access to large data repositories stored on
tape or disk. Enhancements to SRM/MPI-IO enable applications to describe
file sets that will be subsequently accessed so that this data may be
prefetched to local storage prior to access, enabling local access rates
to remote data.

**MPI-IO and Parallel File Systems**

The performance of libraries such as PnetCDF is dependent on the
software underneath, particularly the MPI-IO implementation and the file
system. In addition to providing support for the ROMIO MPI-IO
implementation and the PVFS2 parallel file system, we seek to improve
these packages through optimizations designed specifically to help
computational science.

One current thrust involves development of caching infrastructure in the
MPI-IO layer. By placing the cache in this layer, we are able to more
efficiently manage consistency than we would be able to at the file
system layer, and we can better leverage the limited memory resources
available in petascale systems as well through collaborative caching. We
have developed prototypes of this system and are evaluating them for
future integration in ROMIO.

**Performance Analysis and Tuning**

Application I/O performance often does not match performance seen with
benchmarks. As a result, performance analysis and tuning are a critical
component of providing effective parallel I/O solutions to the
computational science community.

Our latest performance analysis efforts have focused on performance of
the Lustre file system on large clusters and on the Cray XT3. With
greater understanding we hope to improve the MPI-IO infrastructure to
better address performance quirks with this and other cluster file
systems.

**Outreach**

In addition to developing and supporting tools that enable parallel I/O
for DOE applications, we also strive to educate potential users of the
advantages of our software. In the past year we have presented four
tutorials on parallel I/O topics at major conferences, including SC2005,
Cluster 2005, and CCGrid 2005. Tutorials have also been submitted for
presentation at Cluster 2006, EuroPVM/MPI 2006, and SC2006.

**For further information on this subject contact:**

Rob Ross

Mathematics and Computer Science Division

Argonne National Laboratory

Email: rross@mcs.anl.gov

Phone: (630) 252-4588