![](media/image1.png){width="1.375in" height="0.74375in"}SDM Center
Report

October-December 2007

http://sdmcenter.lbl.gov

**[Highlights in this quarter]{.underline}**

-   Developed a real-time monitoring of simulation progress though
    dashboard

    -   Demonstrated flash-based dashboard visualization tools at CPES
        meeting. These tools use a workflow to generate graphs that are
        commonly used by scientists and a browser-based tool to examine
        and annotate these graphs. This is the *first* tool (to our
        knowledge) to provide real-time monitoring of simulation
        progress.

-   Two tutorials were given by SDM center at SC07

    -   \"Parallel I/O in Practice\", educating attendees on the
        state-of-the-art in parallel I/O techniques for computational
        science.

    -   Kepler tutorial. Included examples from various application
        projects. Also, A module on remote R enabled Kepler actor was
        presented.

-   Two students working with the SDM center graduated

    -   Avery Ching. Dissertation title: *Optimizing File Systems
        Techniques for Large-Scale Scientific Applications.* October 5,
        2007.

    -   Kenin Coloma. Dissertation title: *Optimizations for High
        Performance I/O Libraries.* November 2, 2007.

-   Performance evaluation completed

    -   SDM team has performed an evaluation of the parallel I/O system
        on the Jaguar Cray XT system at ORNL, to be published at IPDPS08
        in April.

-   Region finding algorithm for Fusion application developed

    -   A new algorithm was developed for finding regions of interest in
        toroidal mesh structures of fusion simulation data produced by
        GTC. The new algorithm is dozens of times faster than fastest
        version based on alternative approaches.

-   Kepler workflow used in biology application

    -   High-throughput homology workflow delivered to ScalaBLAST team
        at PNNL.

-   Developed framework for workflow provenance and dashboard support

    -   Completed provenance and dashboard requirements, architecture
        and prototypes of almost all integrated system components.
        Production prototype is running at ORNL.

**[All-Hands-Meeting Structure]{.underline}**

The 2.5 days All-Hands-Meeting took place near Seattle on Nov. 28-30,
2007. This year we organized the All-Hands-Meeting around three
themes: 1) presentations by SDM center members with emphasis on progress
in applying SDM center technology in various application projects,
advancement towards production use, and new technology advancements, 2)
mini-workshops that focus on specific topics of scientific data
management needs in several application domains and identifying and
delineating tasks to be performed jointly in the short run, and 3)
mini-workshops whose purpose is to identify joint tasks in collaboration
with other SciDAC centers and institutions. The application domains we
focused on were: Fusion, Climate, and Groundwater. The
centers/institutions we focused on were: Ultravis Institute, VACET
center, and PDSI institute. We also invited talks from the CEDPS center
to help identify technologies that the other center can use/benefit
from. All the presentations, as well as summaries of the mini-workshops
can be found on the SDM center web site (under "meetings" or directly at
the URL: http://sdm.lbl.gov/sdmcenter/pub/2007.11.SDM.AHM.htm.)

***[Publications this quarter]{.underline}***

\[LCC+07\] Wei-keng Liao, Avery Ching, Kenin Coloma, Arifa Nisar, Alok
Choudhary, Jackie Chen, Ramanan Sankaran, and Scott Klasky. Using MPI
File Caching to Improve Parallel Write Performance for Large-Scale
Scientific Applications. In the Proceedings of the ACM/IEEE Conference
on Supercomputing, November 2007.

\[CRL+07\] Avery Ching, Robert Ross, Wei-keng Liao, Lee Ward, and Alok
Choudhary. Noncontiguous Locking Techniques for Parallel File Systems.
In the Proceedings of the ACM/IEEE Conference on Supercomputing,
November 2007.

\[YVO07\] Weikuan Yu, Jeffrey S. Vetter, H. Sarp Oral, *Performance
Characterization and Optimization of Parallel I/O on the Cray XT*. 22nd
IEEE International Parallel and Distributed Processing Symposium
(IPDPS\'08), 2008. Miami, FL.

\[OOW07\] Elizabeth O\'Neil, Patrick O\'Neil and Kesheng Wu. Bitmap
Index Design Choices and Their Performance Implications. Eleventh
International Database Engineering & Applications 2007 Symposium
(IDEAS'07), Banff, Canada September, 2007.

\[PLK07\] N. Podhorszki, B. Ludäscher, S. Klasky. Archive Migration
through Workflow Automation, Intl. Conf. on Parallel and Distributed
Computing and Systems (PDCS), November 19--21, 2007, Cambridge,
Massachusetts.

\[BCK+07\] Roselyne Barreto, Terence Critchlow, Ayla Khan, Scott Klasky,
Leena Kora, Jeffrey Ligon, Pierre Mouallem, Meiyappan Nagappan, Norbert
Podhorszki, Mladen Vouk, \"Managing and Monitoring Scientific Workflows
through Dashboards,\" Poster \# 93, at Microsoft eScience Workshop
Friday Center, University of North Carolina, Chapell Hill, NC, October
13 - 15, 2007, pp. 108.

\[ACC+07\] Ilkay Altintas, George Chin, Daniel Crawl, Terence Critchlow,
David Koop, Jeff Ligon, Bertram Ludaescher, Pierre Mouallem1, Meiyappan
Nagappan, Norbert Podhorszki, Claudio Silva, MladenVouk, \"Provenance in
Kepler-based Scientific Workflow Systems," Poster \# 41, at Microsoft
eScience Workshop Friday Center, University of North Carolina, Chapell
Hill, NC, October 13 - 15, 2007, pp. 82.

\[SVK+07\] Carlos E. Scheidegger, Huy T. Vo, David Koop, Juliana Freire,
and Claudio Silva, Querying and Creating Visualizations by Analogy,
Awarded "best paper" at IEEE Visualization 2007.

\[DEL+07\] S. Davidson, S. Boulakia, A. Eyal, B. Ludäscher, T.
McPhillips, S. Bowers, M. Anand, and J. Freire. Provenance in scientific
workflow systems, IEEE Data Eng. Bull 30(4):44-50, 2007.

Papers abstracts and pointers to the full papers are available at the
sdmcenter.lbl.gov web site, under "publications".

Details of progress during this quarter are reported next.

**[Introduction]{.underline}**

Managing scientific data has been identified as one of the most
important emerging needs by the scientific community because of the
sheer volume and increasing complexity of data being collected.
Effectively generating, managing, and analyzing this information
requires a comprehensive, end-to-end approach to data management that
encompasses all of the stages from the initial data acquisition to the
final analysis of the data. Fortunately, the data management problems
encountered by most scientific domains are common enough to be addressed
through shared technology solutions. Based on the community input, we
have identified three significant requirements. First, more efficient
access to storage systems is needed. In particular, parallel file system
improvements are needed to write and read large volumes of data without
slowing a simulation, analysis, or visualization engine. These processes
are complicated by the fact that scientific data are structured
differently for specific application domains, and are stored in
specialized file formats. Second, scientists require technologies to
facilitate better understanding of their data, in particular the ability
to effectively perform complex data analysis and searches over large
data sets. Specialized feature discovery and statistical analysis
techniques are needed before the data can be understood or visualized.
To facilitate efficient access it is necessary to keep track of the
location of the datasets, effectively manage storage resources, and
efficiently select subsets of the data. Finally, generating the data,
collecting and storing the results, data post-processing, and analysis
of results is a tedious, fragmented process. Tools for automation of
this process in a robust, tractable, and recoverable fashion are
required to enhance scientific exploration.

Our approach is to employ an evolutionary development and deployment
process: from research through prototypes to deployment and
infrastructure. Accordingly, we have organized our activities in three
layers that abstract the end-to-end data flow described above. We
labeled the layers (from bottom to top):

-   Storage Efficient Access (SEA)

-   Data Mining and Analysis (DMA)

-   Scientific Process Automation (SPA)

The SEA layer is immediately on top of hardware, operating systems, file
systems, and mass storage systems, and provides parallel data access
technology, and transparent access to archival storage. The DMA layer,
which builds on the functionality of the SEA layer, consists of
indexing, feature identification, and parallel statistical analysis
technology. The SPA layer, which is on top of the DMA layer, provides
the ability to compose scientific workflows from the components in the
DMA layer as well as application specific modules.

# This report consists of the following sections, organized according to the three layers, as follows:

-   Storage Efficient Access (SEA) techniques

    -   Task 1.1: Low-Level Parallel I/O Infrastructure

    -   Task 1.2: Collaborative File Caching

    -   Task 1.3: File System Benchmarking and Application I/O Behavior

    -   Task 1.4: Application Interfaces to I/O

    -   Task 1.5: Disk Resident Extendible Array Libraries

    -   Task 1.6: Active Storage in the Parallel Filesystem

    -   Task 1.7: Cray XT I/O Stack Optimization

    -   Task 1.8: Characterization and Optimization of Parallel I/O on
        the Cray XT

```{=html}
<!-- -->
```
-   Data Mining and Analysis (DMA) components

    -   Task 2.1 High-performance statistical computing for scientific
        applications

    -   Task 2.2: Feature Selection in Scientific Applications

    -   Task 2.3: High-dimensional indexing techniques

-   Scientific Process Automation (SPA) tools

    -   Task 3.1: Dashboard Development

    -   Task 3.2: Provenance Tracking

    -   Task 3.3: Outreach

The reports by each of the three areas, SEA, DMA, and SPA, follow.

**[1. Storage Efficient Access (SEA)]{.underline}**

***Participants:*** Rob Ross, Rajeev Thakur, Sam Lang, and Rob Latham
(ANL), Alok Choudhary, Wei-keng Liao, Arifa Nisar (NWU), Ekow Otoo and
Doron Rotem (LBNL), Jeffrey Vetter and Weikuan Yu (ORNL), Jarek
Nieplocha and Juan Piernas Canovas (PNL)

The goal of this project is to provide significant improvements in the
parallel I/O subsystems used on today\'s machines while ensuring that
the capabilities available now will continue to be available as systems
increase in scale and technologies improve. A three-fold approach of
immediate payoff improvements, medium-term infrastructure development,
and targeted longer-term R&D is employed.

Two of our keystone components are the PVFS parallel file system and the
ROMIO MPI-IO implementation. These tools together address the
scalability requirements of upcoming parallel machines and are designed
to leverage the technology improvements in areas such as
high-performance networking. These are both widely deployed and freely
available, making them ideal tools for use in today's systems. Our work
in application I/O interfaces, embodied by our Parallel NetCDF
interface, also promises to provide short-term benefits to a number of
climate and fusion applications.

In addition to significant effort on PVFS, we recognize the importance
of other file systems in the HPC community. For this reason our efforts
include improvements to the Lustre file system, and we routinely discuss
both Lustre and GPFS during tutorials. Our efforts in performance
analysis and tuning for parallel file systems, as well as our work on
MPI-IO, routinely involve these file systems.

At the same time we continue to push for support of common,
high-performance interfaces to additional storage technologies. Our work
combining the Storage Resource Manager (SRM) with MPI-IO, in conjunction
with PVFS, will provide a single solution for the problems of
high-performance parallel storage access, integration of file system
with tertiary storage, and remote data access. Our work in Active
Storage will provide common infrastructure for moving computation closer
to storage devices, an important step in tackling the challenges of
petascale datasets.

**[Highlights this quarter]{.underline}**

-   SDM team presented \"Parallel I/O in Practice\" tutorial at SC07,
    educating attendees on the state-of-the-art in parallel I/O
    techniques for computational science.

-   SDM team has performed an evaluation of the parallel I/O system on
    the Jaguar Cray XT system at ORNL, to be published at IPDPS08 in
    April.

-   Two NWU graduate students (Avery Ching, Kenin Coloma) working with
    the SDM center completed their dissertations in the area of parallel
    I/O and have moved on to work in industry.

    **[Task 1.1: Low-Level Parallel I/O Infrastructure (Ross, Thakur,
    Lang, Latham)]{.underline}**

The objective of this work is to improve the state of parallel I/O
support for high-end computing (HEC) and enable the use of high
performance parallel I/O systems by application scientists. The Parallel
Virtual File System (PVFS) and ROMIO MPI-IO implementations, with
development lead by ANL, are in wide use and provide key parallel I/O
functionality. This work builds on these two components by enhancing
them in order to ensure these capabilities continue to be available as
systems continue to scale.

***Progress Report***

PVFS version 2.7.0 was released this quarter. This release included many
of the new developments in the system, including Myricom MX and Portals
communication drivers, the 2-dimensional file distribution system
developed by Kyle Schochenmaier last quarter, support for caching of
immutable files on clients, and some improvements designed to help our
colleagues at PNNL in their active storage development efforts.

At Supercomputing 2007 we held the PVFS Birds of a Feather session to
spread the news about PVFS to attendees, and we presented material in
the ANL booth on PVFS on the BlueGene/P system. We also presented two
full-day tutorials, one on parallel I/O that includes material on PVFS,
MPI-IO, and Parallel netCDF, and one on advanced MPI techniques that
includes material on MPI-IO.

Our collaboration with Garth Gibson's group at CMU continues. The work
on distributed directories was presented at the PDSI storage workshop at
Supercomputing 2007, and we look forward to working with Garth's
students again in 2008.

We held two mini-workshops at the SDM all-hands meeting to discuss I/O.
The first was centered around the role of I/O in visualization
algorithms. The second looked at how we might more effectively store
scientific data on a parallel storage system, breaking the "stream of
bytes" paradigm for more efficiency and expressiveness.

Our collaboration with the Argonne Leadership Computing Facility (ALCF)
continues to go well. PVFS testing continues on the single rack and 8
rack systems, and we are working through the few problems remaining
before it can be accepted.

We added a new feature to the ROMIO MPI-IO implementation to allow users
to specify hints from environment variables. This provides an
alternative to changing these hints from within the application,
allowing users to experiment with hints without recompilation.

***Plans for next quarter***

The first quarter of 2008 will be significant in that the ALCF 133TF
system will go into production in that time frame. Final preparations
for putting PVFS into production on this system are expected to
monopolize our time early in the quarter.

We will also be presenting a half-day tutorial at the FAST conference on
February 26. This tutorial will discuss parallel I/O in the context of
HPC. Our hope is to educate the FAST audience, that includes a large
number of academic and industry researchers interested in storage in
general, about some of the specific challenges in HPC.

Finally, we expect to continue our collaborations with the CMU storage
team, hopefully producing a paper on the distributed directory work and
possibly (eventually) incorporating this technology into future PVFS
production releases.

**[Task 1.2: Collaborative File Caching (Choudhary, Liao,
Nisar)]{.underline}**

The objective of this work is to develop a layer of user-level
client-side file caching system for MPI-IO library. The design uses an
I/O thread in each MPI process and collaborates all threads to perform a
coherent file caching.

***Progress Report***

The locking policy used in the distributed lock manager to enforce the
I/O atomicity has been relaxed such that some of the locks for write
requests can be sharable. Tests have been done to ensure the metadata
integrity.

The approach that dedicates a separate group of MPI processes as cache
servers has been prototyped. The creation of the cache servers is
implemented through MPI dynamic process management functionality. The
same caching policies and distributed lock management from the
I/O-thread approach are used in this design. Testing and development are
conducted on several parallel machines, including TeraGrid Tungsten
running Lustre file system and the IBM TeraGrid machine running GPFS
file system at NCSA, Jazz running PVFS file system at ANL, and Ewok
running Lustre at ORNL. We use a few I/O benchmarks for testing,
including the NASA BTIO benchmark, the FLASH application I/O kernel, and
S3D application I/O kernel.

***Plans for next quarter***

We plan to profile the cache-server implementation in details to
understand the potential performance bottlenecks.

***Publications***

-   Wei-keng Liao, Avery Ching, Kenin Coloma, Arifa Nisar, Alok
    Choudhary, Jackie Chen, Ramanan Sankaran, and Scott Klasky. *Using
    MPI File Caching to Improve Parallel Write Performance for
    Large-Scale Scientific Applications.* In the Proceedings of the
    ACM/IEEE Conference on Supercomputing, November 2007.

    **[Task 1.3: File System Benchmarking and Application I/O Behavior
    (Choudhary, Liao, Nisar)]{.underline}**

The objective of this work is to evaluate the relative performance of
the file systems available to important SciDAC applications on DoE
compute platforms, and, in particular the sustained speed of reading and
writing large datasets. The performance, functionality, and scalability
of MPI-IO, parallel netCDF, and HDF5 are critical for many applications.
Two groups in the SDM center, NWU and ORNL, are performing complementary
efforts in this area.

***Progress Report***

We continue the collaboration work on the S3D I/O kernel with Jacqueline
Chen at Sandia National Laboratories, Ramanan Sankaran and Scott Klasky
at ORNL. The I/O kernel now contains three I/O methods, including
MPI-IO, parallel netCDF, and HDF5. The performance evaluation of using
up to 11,000 process cores has been obtained on the Cray XT at ORNL. The
results show that both MPI-IO and the original one-file-per-process
method had very close write bandwidths to the system maximum bandwidth
when more than 2000 cores are used.

We have revised the pnetCDF method for FLASH I/O kernel benchmark to
avoid the limitation of 2GB array sizes. This problem has been reported
by Katie Antypas from NERSC, LBNL. To accommodate arrays of size larger
than 2 GB, we replaced the fixed-size arrays by declaring their most
significant dimensions as "unlimited". Hence, even though the subarrays
are still bound by the 2GB limitation, the global arrays comprised of
the subarrays can grow beyond 2 GB.

***Plans for next quarter***

We plan to evaluate the S3D I/O kernel using GPFS on the Mercury at
NCSA. As the Cray XT parallel machine at ORNL is upgrading to new
quad-core CPUs and a different operating system, we also plan to
evaluate the S3D I/O kernel on the new machine.

***Publications***

-   Avery Ching, Robert Ross, Wei-keng Liao, Lee Ward, and Alok
    Choudhary. *Noncontiguous Locking Techniques for Parallel File
    Systems.* In the Proceedings of the ACM/IEEE Conference on
    Supercomputing, November 2007.

    **[Task 1.4: Application Interfaces to I/O (Ross, Thakur, Latham,
    Liao, Choudhary, Liao, Nisar)]{.underline}**

The objective of this work is to improve the observed I/O throughput for
applications using parallel I/O by enhancements to or replacements for
popular application interfaces to parallel I/O resources. This task was
added in response to a perceived need for improved performance at this
layer, in part due to our previous work with the FLASH I/O benchmark.
Because of their popularity in the scientific community we have focused
on the NetCDF and HDF5 interfaces, and in particular on a parallel
interface to NetCDF files.

***Progress Report***

We released version 1.0.2 of Parallel netCDF in November. This release
included a number of bug fixes and build environment improvements to
allow PnetCDF to be built and to operate on a wider variety of systems,
including the new SiCortex system at Argonne. Performance improvements
were also made in how we perform attribute accesses.

We have investigated the poor performance of the pnetCDF method of IOR
benchmark on GPFS that was original reported by Hongzhang Shan at LLNL.
This problem only happens when running in pnetCDF independent I/O mode.
The cause is the file synchronization at the time the program switches
from the independent mode to collective mode. Explicitly calling file
synchronization has a significantly large cost on GPFS (also on all
other file systems). In general, switching between collective and
independent modes is not recommended.

We have started collaboration with the Adaptive IO System (ADIOS) team
lead by Scott Klasky at ORNL. ADIOS is an I/O system that separates the
selection and implementation of any particular I/O routines from the
scientific code offering unprecedented flexibility in the choices for
processing and storing data. ADIOS currently handles I/O on per-process
base. We have completed the initial work of implementing ADIOS interface
for FLASH I/O benchmark.

We have been working with Annette Koontz of PNNL to port their global
cloud resolving model I/O routines from a serial approach to a parallel
approach. The PNNL active storage team is also working with this group.

***Plans for next quarter***

We plan to work with the ADIOS team on developing pnetCDF interface and
designing new interface for global data structures and shared-file I/O.

***PhD Students Graduated***

-   Avery Ching. Dissertation title: *Optimizing File Systems Techniques
    for Large-Scale Scientific Applications.* October 5, 2007.

-   Kenin Coloma. Dissertation title: *Optimizations for High
    Performance I/O Libraries.* November 2, 2007.

    **[Task 1.5: Disk Resident Extendible Array Libraries (Otoo,
    Rotem)]{.underline}**

This work focuses on implementing a library for storage management
access of Disk Resident Multidimensional Extendible Arrays for parallel
applications. The library is called pDRXTA. It is the parallel
counterpart of a Disk Resident Multidimensional Extendible Array library
referred to simply as DRXTA. The pDRXTA library currently is only
partially completed and is being implemented to work with PVFS2 and the
MPI-2 RMA methods and also as an alternative to the Disk Resident Array
(DRA) that is associated with the Global Array (GA) library. Work is
ongoing to provide equivalent APIs that are callable from the Global
Array library. There is a corresponding ongoing work to evaluate and
compare its access functions with the use of skip-list access method for
array chunks in HDF5.

***Progress Report***

We have implemented the DRXTA functions, in C, to create, read, write
and manipulate out-of-core arrays stored in Unix file systems. Array
chunks are cached in and out of memory using the BerkeleyDB cache pool.
We have also partially implementated parallel multidimensional
extendible array files, using PVFS2 and the remote memory access (RMA)
functions of MPICH2.

***Plans for next quarter***

We plan to continue work on implementing and testing of extendible array
file for dense chunked arrays. We will also implement skip-list access
methods for chunked dense arrays, and we will perform a comparison study
of the skip-list access method versus the DRXTA access method for
chunked dense arrays and HDF5.

We will also implement a multi-threaded version of the DRXTA library for
multi-core architectures, and implement parallel out-core
multidimensional extendible array functions (pDRXTA) for dense and
sparse arrays.

**[Task 1.6: Active Storage in the Parallel Fileystem (Nieplocha,
Canovas)]{.underline}**

We are developing Active Storage, a promising technology for reducing
bandwidth requirements between the storage and compute elements of
current supercomputing systems, and leveraging the processing power of
the storage nodes used by some modern file systems. To achieve both
objectives, Active Storage allows certain processing tasks to be
performed directly on the storage nodes, near the data they manage.

***Progress Report***

Our recent work has focused on developing support for striped files. Our
previous implementations have been unable to deal with striped files,
i.e., files whose data is spread across several nodes. Striping files is
typically performed in parallel file systems to improve aggregate I/O
bandwidth. We completed the effort initiated in the previous quarter. In
the current design we launch a processing component per storage node
used by the matching file, and make every processing component process
only the file chunks stored in its own node. If the processing
components write to an output file, the output file will have to be
created with the same striping pattern as the input file, and every
processing component will have to write to only the file chunks stored
in its node. Otherwise, the I/O operations will not be entirely local.
There are applications which fulfill the above requirements. For
example, there are applications which process chunk-aligned,
fixed-length records, and save the resulting records, with the same
length and alignment, in another file (or even in the same input file).
Using several environment variables, the Active Storage Runtime
Framework passes to the libas library (a part of the Active Storage
implementation) striping information about the striped files, which must
transparently be accessed. When a processing component opens a file,
libas checks if the file is one of the files it must manage. If so,
libas will intercept any operation on that file in order to only read
and write the local chunks of the file. The experimental results on a
Lustre file system show that our implementation for striped files can
reduce the network traffic to near zero.

In addition, we have been working on adopting Active Storage to deal
with data files with specific complex formats, such as netCDF or HDF5,
which are very common for data exchange in some scientifc applications.
In particular, in collaboration with the Scidac GCRM project we have
been working with the netCDF climate data.

We held initial discussions with IBM about collaboration aiming at
supporting Active Storage for their GPFS parallel file system.

***Plans for next quarter***

In the next quarter, we will continue to develop and improve our user
framework for AS in preparation for the upcoming release.

**[Task 1.7: Cray XT I/O Stack Optimization (Vetter, Yu)]{.underline}**

SciDAC applications make use of a variety of different IO interfaces,
including MPI-IO, Fortran I/O, and Parallel netCDF. We have carried out
a series of studies to examine the parallel I/O subsystem at ORNL's
Leadership Computing Facility (LCF), which uses the Lustre file system.

***Progress Report***

We continued our efforts in developing the OPAL library, a feature-rich
and high-performance MPI-IO library for Lustre. In the past quarter, we
have introduced the direct and lockless I/O support. This is developed
to alleviate the problem as aforementioned in this report. Using a Linux
cluster, we have observed performance improvement with large streaming
of I/O requests over Lustre, avoiding data copies in the kernel buffer
cache. The Jaguar Cray XT system is in the process of. When that is
complete, we plan to explore the benefits of direct and lockless I/O to
scientific applications at ORNL further.

***Plans for next quarter***

In the coming months, we will continue to optimize and extend OPAL
library with more features to improve the performance of parallel I/O on
the Cray XT. We will also demonstrate the performance benefits of OPAL
to other applications running over Jaguar. Furthermore, we plan to
introduce new features into the Lustre file system for better
scalability and performance assurance under heavy load.

**[Task 1.8: 1. Characterization and Optimization of Parallel I/O on the
Cray XT (Vetter, Yu)]{.underline}**

To prepare for future exascale computing, it is important to gain a good
understanding on what impacts a hierarchical storage system would have
on the performance of data-intensive high performance computing (HPC)
applications, and accordingly, how to leverage its strengths and how to
mitigate possible risks. To this aim, we have performed a parallel I/O
performance analysis of the storage system on Jaguar. An
application-oriented perspective is adopted to reveal the implications
of storage organization to common I/O patterns in scientific
applications running over Jaguar.

***Progress Report***

We have extensively characterized the parallel I/O performance on the
Jaguar supercomputer. Our characterization covered the performance and
scalability of the individual storage units, as well as the entire
system. We have examined the best stripe sizes over Jaguar, and showed
that the file distribution pattern across the DDN storage couplets can
dramatically impact the aggregated performance. In addition, we have
also examined the scalability of metadata- and data-intensive
operations. Our results have demonstrated that, for parallel file open,
the shared file mode has the best scalability compared to the separated
file mode. Moreover, we have investigated the performance impacts of
parallel I/O techniques for handling small and non-contiguous I/O,
including data sieving and collective I/O. We have also documented that,
with overlapped file segments, data sieving from concurrent processes
can lead to performance degradations. Increasing the size of data
sieving buffer can improve the performance, but the performance of
writes is still hindered by the internal lock contentions at the Lustre
file system.

***Plans for next quarter***

We plan to parameterize and model the I/O performance over Jaguar and
project the expected performance for representative applications.

***Publications***

-   Weikuan Yu, Jeffrey S. Vetter, H. Sarp Oral, *Performance
    Characterization and Optimization of Parallel I/O on the Cray XT*.
    22nd IEEE International Parallel and Distributed Processing
    Symposium (IPDPS\'08). April 2008. Miami, FL.

**[2. Data Mining and Analysis (DMA)]{.underline}**

**Highlights this quarter:**

-   We have made significant progress in finding regions of interest in
    fusion simulation data produced by GTC. We have developed a version
    of the region finding algorithm by taking full advantage of the
    toroidal mesh structure. The key insight is that we can redefine the
    connectivity definition to better suit the particular mesh
    structure. The new algorithm is dozens of times faster than fastest
    version based on alternative approaches.

-   Co-taught the SPA's Supercomputing 2007 tutorial. A module on remote
    R enabled Kepler actor was presented. We worked with SPA group to
    conduct tutorials on using Remote R Actor. A total of \~20 attendees
    participated in the tutorial and the entire tutorial, technology and
    application was well received by everyone.

### [Task 2.1: High-performance statistical computing for scientific applications, ORNL]{.underline}

***Contact:*** Nagiza Samatova, ORNL

The following people have contributed to the project over this quarter:

Paul Breimier, Guru Kora, Jeff Ligon, Mladen Vouk, and Nagiza Samatova

## Goals

This quarter our main focus was to enable web-service data analytics in
Kepler through R. We worked closely with SDM\'s SPA team and Accelxon to
enable R web-services for Kepler.

## Accomplishments 

Web Services have become a critical part of today scientific research
domain. They support a loosely coupled service oriented architecture
that builds on previous distributed object architectures like CORBA,
Java RMI, and COM to provide scalable interoperable systems. This new
way of accessing the information resource is conducive for scientific
research as majority of data, computation and result analysis is carried
out by a group of closely working researchers. Correspondingly there are
a growing number of powerful tools that are available for building,
maintaining and accessing Web Service-based systems. These tools include
portals that allow user front-ends to Web Services. Our experience with
Scientific Data Management teams show that Kepler has been a very
effective data analysis tool for managing scientific workflow. Extending
the current data analysis capabilities of Kepler to web-services enabled
data analysis was a natural progression. This new feature of accessing
data analytics routines through a web-service enabled user to access
extensive data analysis functionality and high performance of parallel
computation.

We worked on two major components of the data analytics;

[1. Backend R web service:]{.underline}

       The remote R web server is a J2EE based Java web application. It
uses a customized version of R optimized to accepted streaming data and
output result in multiple data formats. Remote R web server program
operates by accepting HTTP GET/POST requests from the Kepler clients,
and provide an HTTP response back to the Kepler front-end. The HTTP
requests embed Kepler generated R scripts as well as data in them. The
HTTP response usually consists of an HTML document, but can also be a
raw file, an image, or some other type of document (defined by
MIME-types); if some error is found in Kepler request or while trying to
serve the request, the web server will send an error response which may
include some custom HTML or text messages to better explain the problem
to end users. Remote R web server also has the capability of logging
some detailed information, about Kepler requests and server responses,
to log files; this allows the researchers to collect statistics by
running log analyzers on log files. We worked with Accelxon\'s team and
used their software components to build remote R web application.

[2. Frontend remote R Kepler entity (Actor):]{.underline}

       Kepler includes an RExpression actor which interacts with a local
R installation.  We extended the RExpression actor to support remote R
by adding two options to the RExpression parameter window: a Boolean
parameter to specify intention to use remote R server and a string
parameter specifying the URL of remote R Server.  The first parameter
specifies whether the user wants to use remote R and defaults to False,
and the second allows the user to enter the location of the Remote R
server. Once the user wishes to use remote R, all R commands are
streamed to the remote R backend server that accepts it, processes the
script and returns the R output. The R output is returned in plain HTML.
If images are generated as part of the R output, then they are embedded
in the HTML output. It is the responsibility of the user to process the
R output with an appropriate display actor. The user doesn\'t have to
have a local copy of R installed. Since remote R has a centralized
deployment, it is much easier to update/extend/add R packages to the R
instance. It is also transparent to the operating system the Kepler is
running on. The backend R instance can be extended to use ParallelR
components at the backend. Since it is a managed deployment, we can give
user the power of running their scripts in Parallel using
shared/distributed memory computer clusters.

## Future Plans

-   After the SDM AHM in December, a task of making data management and
    analysis for the SciDAC climate group, and Marcia Brenstetter, has
    emerged. Our goal is to identify the requirements and design and
    prototype the solution to this problem.

-   Discussions with the fusion team, Dashboard project led by Scott
    Klasky, identified the needs for incorporating data anlytics
    capabilities from inside the Dashboard. We plan to continue these
    discussions and identify the requirements and roadmaps for future
    implementation.

-   Web Services based data analytics will be further extended to allow
    the addition of parallel analysis routines based on pR library.

### [Task 2.2: Feature Selection in Scientific Applications (LLNL)]{.underline}

# *Contact:* Chandrika Kamath, LLNL

## Accomplishments

My post-doc, Nicole Love, completed her 2-year term in early November.
As part of wrapping up her work on blob segmentation, she cleaned out
her codes and commented them, so I can continue the work by testing the
techniques on other image sequences. She also updated a report, which is
an expanded version of the SPIE paper we presented in August.

I resumed the work on the Poincare plot analysis. I wrote several
scripts to convert the new data from Josh Breslau at PPPL into a
suitable form for processing. I also started looking into more robust
features which can be extracted from the plots for use in
classification. In our earlier work, we had assumed that each orbit
would be represented by thousands of points. With far fewer points, the
orbits can appear very different, requiring a complete rethinking of
representative features for them. The new data also contains some
stochastic orbits. A key challenge with these orbits will be the correct
identification of incipient stochasticity.

## Plans for Next Quarter

I plan to use the features extracted for the different orbits in a
decision tree classifier. I expect I will need to iterate the process of
defining and extracting better and more robust features and evaluating
the performance of the decision trees. Also, I expect to see challenges
when the techniques are applied to data from a stellarator, where the
orbits appear more triangular than circular.

On the blob tracking work, I expect to pick up where Nicole left off. I
plan to go back to our four test sequences and see where the current
techniques are failing and why. This will help me to figure out the best
way to fix the techniques.

### [Task 2.3: High-dimensional indexing techniques (LBNL)]{.underline}

# Contact: John Wu, LBNL

Contributors: Arie Shoshani

## Accomplishments

-   In our interactions with FastBit users, we have fixed a number of
    bugs discovered by users and added two features requested: a Java
    API and a new logging functionality to output messages to any file
    specified by user. In this quarter we have also updated the
    reference document a number of times based on user feedback.

-   Working with the Visualization group of LBNL, we have updated our
    work on HDF5-FastQuery software and expanded the functionality to
    H5Part software developed by the Visualization group.

-   Helped Rishi Sinha of University of Illinois at Urbana-Champaign to
    develop a program to search for regions of interest on GTC data from
    fusion simulations. His program treats GTC mesh as irregular mesh
    and each point separately when trying to connect points into
    regions.

-   Developed our own algorithms for finding regions of interest on GTC
    data by taking advantage of the toroidal mesh structure. The key
    insight is that we can redefine the connectivity definition to fully
    match the toroidal mesh. This allows use to connect the points into
    regions much faster. We have observed average speedup of about 20
    for a test set of GTC data.

***Future Plans***

-   Continue to support FastBit users, particularly, developers at
    Yahoo, ICSI and University of Hamburg.

-   Continuing the work on feature identification in fusion simulation
    data, the next step is to work with some visualization people to
    develop a front-end for the underlying region finding algorithms.

-   Revisit the binning strategies in FastBit. Theoretical analysis of
    the new data structure is needed for "real-application data" not
    just the worst-case scenario.

***Publications***

\[OOW07\] Elizabeth O\'Neil, Patrick O\'Neil and Kesheng Wu. Bitmap
Index Design Choices and Their Performance Implications. Eleventh
International Database Engineering & Applications 2007 Symposium
(IDEAS'07), Banff, Canada September, 2007.

**[3. Scientific Process Automation]{.underline}**

PNNL (Terence Critchlow, George Chin),

ORNL (Scott Klasky, Roselyne Barreto)

NCSU (Mladen Vouk, Jeff Ligon, Pierre Mouallem, Mei Naggapan)

SDSC (Ilkay Altintas, Daniel Crawl)

UC Davis (Bertram Ludaescher, Norbert Podhorszki, Timothy McPhillips)

University of Utah (Steven Parker, Claudio Silva, Ayla Khan, David Koop)

Managing scientific data has been identified as one of the most
important emerging needs by the scientific community because of the
sheer volume and increasing complexity of data being collected.
Effectively generating, managing, and analyzing this information
requires a comprehensive, end-to-end approach to data management that
encompasses all of the stages from the initial data acquisition to the
final analysis of the data. One very significant requirement is
efficient generation and movement of data, collection and storing of the
results, data post-processing, and analysis of the results, i.e.,
facilitation of the scientific process. We are developing a suite of
tools as well as an integrated system for automation of this process in
a robust, tractable, and recoverable fashion so that scientific
exploration can be enhanced. Components are: workflow technologies for
run-time management of the processes, provenance collection and
management technologies, and end-user access technologies for display
and management of both workflows, provenance, analytics and results,
i.e., dashboards.

**Highlights this quarter:**

-   High-throughput homology workflow delivered to ScalaBLAST team at
    PNNL.

-   Completed provenance and dashboard requirements, architecture and
    prototypes of almost all integrated system components -- information
    available on-line
    ([[http://groups.google.com/group/spa-dev]{.underline}](http://groups.google.com/group/spa-dev)).
    Production prototype is running at ORNL.

-   SC07 Kepler tutorial. Full day tutorial on Kepler received good
    attendance.

-   Demonstrated flash-based dashboard visualization tools at CPES
    meeting. These tools use a workflow to generate graphs that are
    commonly used by scientists and a browser-based tool to examine and
    annotate these graphs. We have incorporated feedback and fixed bugs
    based on this interaction.

    **[Task 3.1: Dashboard Development]{.underline}**

ORNL (Scott Klasky, Roselyne Barreto)

NCSU (Mladen Vouk, Jeff Ligon, Pierre Mouallem, Mei Naggapan)

University of Utah (Steven Parker, Claudio Silva, Ayla Khan, David Koop)

**Current Status**

-   We have a fully operational (K)LAMP environment with pilot dashboard
    and provenance implementations.

-   Improved dashboard performance and scalability through flash video.
    Scalability improvements will be necessary in the future, but
    current version allows meaningful interaction for realistic runs.

-   Completed dashboard requirements, architecture and prototypes of
    almost all integrated system components -- information available
    on-line
    ([[http://groups.google.com/group/spa-dev]{.underline}](http://groups.google.com/group/spa-dev)).
    Production prototype is running at ORNL.

```{=html}
<!-- -->
```
-   The dashboard continues to be hardened and functionally extended,
    and ORNL is working with both Utah and NCSU to incorporate better
    flash routines for monitoring simulations, more integrated data base
    provenance and data tracking, and annotations..

-   Norbert Podhorszki has joined the ORNL team.

-   SDM representatives attended CPES AHM, December 3-4, 2007, at Oak
    Ridge National Laboratory. Dashboard graphing tools expanded based
    on user feedback from CPES meeting. Annotation tools are now more
    robust and easier to use.

    **Plans for next quarter**

-   Fully port Dashboard to NCSU. Develop initial version of a portable
    open/source version.

```{=html}
<!-- -->
```
-   Tie dashboard into provenance database.

-   Improve dashboard visualization tools with a 3-stage mechanism:

    -   1\. Use a browser-based flash tool for common interactions:
        scatter plots , x-y plots, and slices. Further exploration may
        require a more capable tool.

    -   2\. Integrate the visit visualization tool (LLNL) into the
        dashboard by running it on the server through a workflow.

    -   3\. Allow the user to download a visit configuration file for
        further interaction when they have visit installed.

```{=html}
<!-- -->
```
-   Improve Kepler file transfer, job submission and job monitoring
    capabilities.

-   Also, see AHM project plan

    **[Task 3.2: Provenance Tracking]{.underline}**

PNNL (George Chin),

SDSC (Ilkay Altintas, Daniel Crawl)

UC Davis (Bertram Ludaescher, Norbert Podhorszki, Timothy McPhillips)

University of Utah (Leena Kora)

**Current Status**

-   Completed provenance requirements, architecture and prototypes of
    almost all integrated system components -- information available
    on-line([[http://groups.google.com/group/spa-dev]{.underline}](http://groups.google.com/group/spa-dev)).
    Production prototype is running at ORNL.

```{=html}
<!-- -->
```
-   Continued updates to provenance database schema based on use-cases.

-   Based on Groundwater use-case, designed and implemented Kepler
    provenance architecture that supports saving to different types of
    underlying storage: SQL (using SPA schema), RDF (using Groundwater
    schema), and text files.

-   Developed draft version of Kepler actor for SRM lite (Mouallem)

    **Plans for next quarter**

```{=html}
<!-- -->
```
-   Implement compile-time and run-time code provenance for codes at
    ORNL.

-   Investigate buffering techniques to improve performance of Kepler
    provenance recording.

-   Continue work with end-users to identify improvements to the
    provenance usability and functionality framework, and implement the
    improvements.

-   Analysis and comparison of all current provenance approaches in
    Kepler (Manish Anand, UC Davis)

-   Also, see AHM project plan.

    **[Task 3.3: Outreach]{.underline}**

    **Current Status**

```{=html}
<!-- -->
```
-   The SC07 Kepler tutorial presentation completed.

-   CPES AHM meeting attended by SDM team.

-   Developed draft version of Kepler actor for SRM lite (Mouallem)

-   Hosted a group of visitors from the ITER organization for a month
    and organized a two-day meeting at SDSC with participants from ITER,
    SPA and CPES during their stay.

-   Helped PNNL Groundwater group to implement a production workflow by
    designing throttling features required for release of their system.

-   The CPES code coupling workflow now runs a parameter-study for
    different toroidal mode numbers in the ELITE step for each coupling
    cycle (This is necessary to provide an acceptable level of accuracy
    in determining when the XGC simulation becomes unstable).
    Demonstrated this capability at CPES AHM (ORNL, Dec 3-5)

-   Continuing development of generic transport actor for Kepler based
    on finite state machines, frames, and templates. Starting to work
    with Leena Kora at University of Utah on design of job scheduling
    and job monitoring generic actors.

    **Plans for next quarter**

```{=html}
<!-- -->
```
-   Extend interactions with Climate researchers

-   Continue to work on groundwater workflows. Submit paper with K.
    Schuchardt and J. Chase on concepts and requirements in the
    development of scientific workflows for subsurface transport. Paper
    will be submitted to a Special Issue of IEEE Trans. On Automation
    Science and Engineering on \"Scientific Workflow Management and
    Applications.\"

    ***[Publications and Related Presentations]{.underline}***

> Publications

-   N. Podhorszki, B. Ludäscher, S. Klasky. "Archive Migration through
    Workflow Automation", Intl. Conf. on Parallel and Distributed
    Computing and Systems (PDCS), November 19--21, 2007, Cambridge,
    Massachusetts.

-   Roselyne Barreto, Terence Critchlow, Ayla Khan, Scott Klasky, Leena
    Kora, Jeffrey Ligon, Pierre Mouallem, Meiyappan Nagappan, Norbert
    Podhorszki, Mladen Vouk, \"Managing and Monitoring Scientific
    Workflows through Dashboards,\" Poster \# 93, at Microsoft eScience
    Workshop Friday Center, University of North Carolina, Chapell Hill,
    NC, October 13 - 15, 2007, pp. 108.

-   Ilkay Altintas, George Chin, Daniel Crawl, Terence Critchlow, David
    Koop, Jeff Ligon, Bertram Ludaescher, Pierre Mouallem1, Meiyappan
    Nagappan, Norbert Podhorszki, Claudio Silva, MladenVouk,
    \"Provenance in Kepler-based Scientific Workflow Systems," Poster \#
    41, at Microsoft eScience Workshop Friday Center, University of
    North Carolina, Chapell Hill, NC, October 13 - 15, 2007, pp. 82.

-   \"Querying and Creating Visualizations by Analogy\", by Carlos E.
    Scheidegger, Huy T. Vo, David Koop, Juliana Freire, and Claudio
    Silva. Awarded "best paper" at IEEE Visualization 2007.

-   S Davidson, S Boulakia, A Eyal, B Ludäscher, T McPhillips, S Bowers,
    M Anand, and J Freire. Provenance in scientific workflow systems
    IEEE Data Eng. Bull 30(4):44-50, 2007.

> Presentations and other activities

-   SC07 complete materials available on-line at:
    [[http://sdm.ncsu.edu/tutorial/]{.underline}](http://sdm.ncsu.edu/tutorial/)
    or
    [[http://groups.google.com/group/spa-tutorial]{.underline}](http://groups.google.com/group/spa-tutorial).

-   A number of presentations were made at SDM and CPES AHMs.

-   Plan to attend ASCAC meeting in Denver (March 08).