![](media/image1.png){width="1.375in" height="0.74375in"}SDM Center
Report

April-June 2007

http://sdmcenter.lbl.gov

**[Highlights in this quarter]{.underline}**

-   Fine-grained locks were implemented internally in PVFS to help the
    system operate at peak efficiency on multi-core systems.

-   Garth Gibson of Carnegie Mellon University and the PDSI institute PI
    used PVFS in a graduate course as the basis for work in distributed
    directory storage in high-performance file systems.

-   Enhanced I/O efficiency for the Lustre file system by as much as
    400%, by introducing a novel technique called partitioned collective
    IO (ParColl). ParColl partitions parallel processes into subgroups,
    each carrying out smaller, yet aggregated IO operations.

-   Finalized the new multi-level bitmap index functions following the
    new insight gain from recent theoretical analyses. In performance
    measurements, the new implementation can actually be up to 10 times
    faster than our old software, but is never slower.

-   Produced Use-case document for Kepler outlining specific use-cases
    and data types for provenance information

-   Implemented a pilot version of the Dashboard operating with ORNL
    fusion codes.

-   A Kepler tutorial at SC07 conference was proposed and accepted. This
    is the second year in a row this tutorial will be offered.

Details and additional progress are reported next

**[Introduction]{.underline}**

Managing scientific data has been identified as one of the most
important emerging needs by the scientific community because of the
sheer volume and increasing complexity of data being collected.
Effectively generating, managing, and analyzing this information
requires a comprehensive, end-to-end approach to data management that
encompasses all of the stages from the initial data acquisition to the
final analysis of the data. Fortunately, the data management problems
encountered by most scientific domains are common enough to be addressed
through shared technology solutions. Based on the community input, we
have identified three significant requirements. First, more efficient
access to storage systems is needed. In particular, parallel file system
improvements are needed to write and read large volumes of data without
slowing a simulation, analysis, or visualization engine. These processes
are complicated by the fact that scientific data are structured
differently for specific application domains, and are stored in
specialized file formats. Second, scientists require technologies to
facilitate better understanding of their data, in particular the ability
to effectively perform complex data analysis and searches over large
data sets. Specialized feature discovery and statistical analysis
techniques are needed before the data can be understood or visualized.
To facilitate efficient access it is necessary to keep track of the
location of the datasets, effectively manage storage resources, and
efficiently select subsets of the data. Finally, generating the data,
collecting and storing the results, data post-processing, and analysis
of results is a tedious, fragmented process. Tools for automation of
this process in a robust, tractable, and recoverable fashion are
required to enhance scientific exploration.

Our approach is to employ an evolutionary development and deployment
process: from research through prototypes to deployment and
infrastructure. Accordingly, we have organized our activities in three
layers that abstract the end-to-end data flow described above. We
labeled the layers (from bottom to top):

-   Storage Efficient Access (SEA)

-   Data Mining and Analysis (DMA)

-   Scientific Process Automation (SPA)

The SEA layer is immediately on top of hardware, operating systems, file
systems, and mass storage systems, and provides parallel data access
technology, and transparent access to archival storage. The DMA layer,
which builds on the functionality of the SEA layer, consists of
indexing, feature identification, and parallel statistical analysis
technology. The SPA layer, which is on top of the DMA layer, provides
the ability to compose scientific workflows from the components in the
DMA layer as well as application specific modules.

# This report consists of the following sections, organized according to the three layers, as follows:

-   Storage Efficient Access (SEA) techniques,

    -   Task 1.1: Low-Level Parallel I/O Infrastructure

    -   Task 1.2: Collaborative File Caching

    -   Task 1.3 File System Benchmarking and Application I/O Behavior

    -   Task 1.4: Application Interfaces to I/O

    -   Task 1.5: SRM/MPI-IO Parallel File System Caching and Migration
        Service

    -   Task 1.6: Active Storage in the Parallel Filesystem

    -   Task 1.7: Cray XT I/O Stack Instrumentation and Optimization

```{=html}
<!-- -->
```
-   Data Mining and Analysis (DMA) components

    -   Task 2.1 High-performance statistical computing for scientific
        applications

    -   Task 2.2: Feature Selection in Scientific Applications

    -   Task 2.3: High-dimensional indexing techniques

-   Scientific Process Automation (SPA) tools

    -   Task 3.1: Dashboard Development

    -   Task 3.2: Provenance Tracking

    -   Task 3.3: Outreach

The reports by each of the three areas, SEA, DMA, and SPA, follow.

**[1. Storage Efficient Access (SEA)]{.underline}**

***Participants:*** Rob Ross, Rajeev Thakur, Sam Lang, and Rob Latham
(ANL), Alok Choudhary, Wei-keng Liao, Kenin Coloma, and Avery Ching
(NWU), Arie Shoshani and Ekow Otoo (LBNL), Jeffrey Vetter and Weikuan Yu
(ORNL), Jarek Nieplocha and Juan Piernas Canovas (PNNL)

The goal of this project is to provide significant improvements in the
parallel I/O subsystems used on today\'s machines while ensuring that
the capabilities available now will continue to be available as systems
increase in scale and technologies improve. A three-fold approach of
immediate payoff improvements, medium-term infrastructure development,
and targeted longer-term R&D is employed.

Two of our keystone components are the PVFS parallel file system and the
ROMIO MPI-IO implementation. These tools together address the
scalability requirements of upcoming parallel machines and are designed
to leverage the technology improvements in areas such as
high-performance networking. These are both widely deployed and freely
available, making them ideal tools for use in today's systems. Our work
in application I/O interfaces, embodied by our Parallel NetCDF
interface, also promises to provide short-term benefits to a number of
climate and fusion applications.

In addition to significant effort on PVFS, we recognize the importance
of other file systems in the HEC community. For this reason our efforts
include improvements to the Lustre file system, and we routinely discuss
both Lustre and GPFS during tutorials. Our efforts in performance
analysis and tuning for parallel file systems, as well as our work on
MPI-IO, routinely involve these file systems.

At the same time we continue to push for support of common,
high-performance interfaces to additional storage technologies. Our work
combining the Storage Resource Manager (SRM) with MPI-IO, in conjunction
with PVFS, will provide a single solution for the problems of
high-performance parallel storage access, integration of file system
with tertiary storage, and remote data access. Our work in Active
Storage will provide common infrastructure for moving computation closer
to storage devices, an important step in tackling the challenges of
petascale datasets.

**[Task 1.1: Low-Level Parallel I/O Infrastructure]{.underline}**

The objective of this work is to improve the state of parallel I/O
support for high-end computing (HEC) and enable the use of high
performance parallel I/O systems by application scientists. The Parallel
Virtual File System (PVFS) and ROMIO MPI-IO implementations, with
development lead by ANL, are in wide use and provide key parallel I/O
functionality. This work builds on these two components by enhancing
them in order to ensure these capabilities continue to be available as
systems continue to scale.

***Progress Report***

Functionality was added to the PVFS interfaces to allow control of
layout of files. This work facilitates research being performed in
active storage at PNNL and will be merged into an upcoming release. PVFS
startup times were improved for large volumes with optimizations at the
disk (Trove) layer. This layer was additionally enhanced to allow for
multiple I/O methods (e.g. direct, buffered) and storage of associated
metadata, enabling easier testing of multiple methods. Fine-grained
locks were implemented internally to help the system operate at peak
efficiency on multi-core systems.

Performance testing with PVFS on ALCF hardware continues, including
testing of the MX support recently contributed by Myricom. At the same
time, in conjunction with the ALCF team we have developed a strategy for
PVFS fault tolerance on ALCF hardware that maintains highest possible
performance. We have begun development of specific enhancements to PVFS
necessary to implement this strategy.

Garth Gibson of Carnegie Mellon University and the PDSI used PVFS in a
graduate course as the basis for work in distributed directory storage
in high-performance file systems. The PVFS team visited to provide
insight into the system to help the students, and currently one of the
student teams is completing this work, targeting a paper in the
prestigious FAST conference.

ANL has two student visitors working on PVFS this summer. One, David
Buettner of University of Heidelberg, is investigating efficient tracing
techniques for use in capturing behavior of the file system at a low
level. The second, Kyle Schochenmaier of Ames Laboratory, is
investigating the use of 2D distribution schemes to overcome the
"incast" phenomenon observed in other file systems.

***Plans for next quarter***

Effort in the next quarter on PVFS will be split between further
understanding and optimizing access with high-end storage arrays,
testing PVFS support over Myricom MX on their 10Gigabit hardware, and
testing and debugging on IBM BG/P hardware as it becomes available. This
is all in support of ALCF.

Major ROMIO changes may be undertaken in the following quarter as well,
depending on the outcome of the PnetCDF discussions (below).

**[Task 1.2: Collaborative File Caching]{.underline}**

The objective of this work is to develop a user-level client-side file
caching system layer for MPI-IO library. Our design uses an I/O thread
in each MPI process and makes all threads cooperate with each other to
perform a coherent file caching.

***Progress Report***

Most of the components of caching system have completed. They include
global cache metadata updates, read/write data from/to the underlying
file system, remote cache data GET/PUT, local cache data access, cache
metadata lock management, cache page eviction, migration, and a
two-phase data flushing mechanism. The implementation of our caching
system is placed at the ADIO layer and works will under both UFS and
PVFS1.

The majority of the I/O patterns presented in scientific applications
are write-only and these patterns do not require any software control
for parallel data consistency or concurrent access atomicity. Based on
this observation, we have developed a two-stage write-behind method for
improving the performance of parallel write-only operations. This work
also uses background threads in all MPI process to redistribute and
combine non-contiguous file requests among processes in order to
generate large contiguous I/O requests. We have also developed a method
to align the file access domains (file domains in ROMIO) such that they
are aligned with the file system's lock boundaries. This method has
shown a significant performance improvement for several I/O benchmarks.

We continue to exercise our design on the two parallel machines at NCSA:
a Dell Linux cluster running Lustre parallel file system and a TeraGrid
machine running an IBM GPFS file system. The current performance
evaluation focuses on the NASA BTIO benchmark, the FLASH application I/O
kernel, and S3D application I/O kernel.

***Plans for next quarter***

The current implementation of our caching system enforces the MPI-IO
atomicity, which is a strict semantics for file access. The MPI default
is non-atomic. We have started the investigation and developing an
alternative to relax the atomicity in our implementation. By doing so,
we anticipate significant reduction in lock contention as well as
coherence control communication.

**[Task 1.3 File System Benchmarking and Application I/O
Behavior]{.underline}**

The objective of this work is to evaluate the relative performance of
the file systems available to important SciDAC applications on DoE
compute platforms, and, in particular the sustained speed of reading and
writing large datasets. The performance, functionality, and scalability
of MPI-IO, parallel netCDF, and HDF5 are critical for many applications.
Two groups in the SDM center, NWU and ORNL, are performing complementary
efforts in this area.

***Progress Report***

The Northwestern University group has been using BTIO, FLASH I/O, NCIO,
and S3D I/O benchmarks to evaluate parallel shared-file MPI I/O
performance on the Lustre, and GPFS file systems. We have observed
significant performance degradation due to the file system lock
contention if the benchmarks ran on top of MPI I/O natively. Once each
I/O request is aligned with file system lock boundaries, significant
improvement is observed. This file lock contention issue has been
studied with the support of our MPI I/O caching layer. To further
address the drawback of using block-based file lock protocol, we have
implemented a scalable distributed lock management method that provides
true byte-range locking granularity. We also developed a hybrid
two-phase locking protocol to improve the non-contiguous I/O
performance. A synthetic I/O benchmark, named HPIO, has also been
developed and public available for evaluating the scalability of
parallel I/O with overlapping/non-overlapping, and
contiguous/non-contiguous patterns.

We continue the collaboration work on the S3D I/O kernel with Jacqueline
Chen at Sandia National Laboratories, Ramanan Sankaran and Scott Klasky
at ORNL. S3D is a parallel turbulent combustion application developed at
Sandia National Laboratories. The S3D's I/O is originally programmed in
Fortran I/O functions and each process writes all its sub-arrays to a
separate file at each checkpoint. This approach can result in thousands
to millions of files from a single production run. We have implemented
three I/O methods, including MPI I/O, parallel netCDF, and parallel
HDF5. All three methods write the arrays into a shared file in their
globally canonical order. This approach reduces the number of files to
one per checkpoint. The software has been provided to our collaborators
in June. We also evaluated the MPI I/O method on the Cray XT computers
at ORNL using up to 2000 processes and the preliminary results are
scalable.

***Plans for next quarter***

We plan to continue the work on developing the distributed lock manager
and test it using scientific I/O benchmarks, including the NASA BTIO,
FLASH I/O, NCIO, and S3D I/O. The evaluation will compare against a
block-based cache and locking Lustre and GPFS implementation to show how
false sharing negatively impacts I/O performance.

**[Task 1.4: Application Interfaces to I/O]{.underline}**

The objective of this work is to improve the observed I/O throughput for
applications using parallel I/O by enhancements to or replacements for
popular application interfaces to parallel I/O resources. This task was
added in response to a perceived need for improved performance at this
layer, in part due to our previous work with the FLASH I/O benchmark.
Because of their popularity in the scientific community we have focused
on the NetCDF and HDF5 interfaces, and in particular on a parallel
interface to NetCDF files.

***Progress Report***

John Shalf and Hongzhang Shan of LBL compared I/O performance on
machines with several I/O libraries using IOR and found Parallel-NetCDF
performance to be quite poor. We traced the cause to a particularly bad
interaction between the Lustre file system and ROMIO aggressively
locking and unlocking file regions for this particular file view. We
identified ROMIO hints which would reduce or eliminate the lock traffic
and observed improved Parallel-NetCDF performance.

The NWU team has also prototyped enhancements to PnetCDF to help users
employing hints, and they have been interacting with the IOR benchmark
development team to ensure that the appropriate hints will be passed in
subsequent IOR versions to help with performance.

***Plans for next quarter***

Rob Latham will be visiting UCAR for their \"NetCDF for Data Providers
and Developers\" workshop. While there, we will discuss the CDF-3 file
format proposal and give a brief presentation of Parallel-NetCDF and how
it relates to other high-level I/O libraries.

This discussion of modifications to the file format seems to be moving
towards consensus. Assuming that we can agree on how the file format
should change, we will prototype these changes in PnetCDF to allow for
experimentation and further discussion.

**[Task 1.5: SRM/MPI-IO Parallel File System Caching and Migration
Service]{.underline}**

The objective of this work is to combine the parallel I/O capabilities
of MPI-IO with the remote access and mass storage management
capabilities of the Storage Resource Manager (SRM), allowing
applications to access remote and local data and to stage data on local
parallel I/O resources. Functionality is being built to allow caching of
files on a local parallel file system (such as PVFS) and migrating data
between the cache and a remote tertiary storage system in the context of
a MPI application. For example, this capability would allow for movement
of data between HPSS and PVFS. LBNL leads this effort.

***Progress Report***

We designed and implemented a new technique for coupling parallel file
systems with file migration to and from HPSS. The main motivations for
this new approach are to avoid any modification of read/write statements
in existing parallel applications and to use the same technique for
remote file accesses irrespective of the programming languages and file
formats of applications.

We have also begun development and implementation of DREXTA: A Disk
Resident EXTendible Array using chunking and a parallel file system
(i.e., PVFS2).

***Plans for next quarter***

The main activities for the next couple of months to test the new access
library functions from both Fortran 90 and C/C++ applications reading
and writing data in various file formats, to modify programs to
demonstrate how the libmpiosrm library may be used for accessing remote
files and local parallel files for specific file formats, i.e., HDF5,
NetCDF and DREXTA, and to continue work on the implementation and
testing of DREXTA -- the disk resident extendible array file using
chunking and caching with BerkeleyDB at client nodes.

**[Task 1.6: Active Storage in the Parallel Filesystem]{.underline}**

The purpose of this work is to extend the active storage/disk concept to
parallel file systems, in particular Lustre and PVFS, and make it
practical for DOE applications. The PNL team leads this work.

***Progress Report***

During this second quarter we have ported the kernel-space
implementation of Active Storage to Lustre 1.6. The original
implementation was done on Lustre 1.4. We compared performance of Active
Storage over Lustre 1.6 with that of our new user-space approach that
was mentioned in the previous report. The experimental results have
shown that the user-space implementation of Active Storage is faster
than the kernel space implementation. Since it is also more flexible,
portable, and readily deployable than the kernel approach, we have
decided to focus our efforts on the new user-space approach.

The outcomes of this research appears in a paper titled \"Evaluation of
Active Storage Strategies for the Lustre Parallel File System\", which
has been accepted for publication in the forthcoming Supercomputing\'07
conference. Another paper titled \"Non-collective Parallel I/O for
Global Address Space Programming Models\" has been accepted for
publication in the IEEE Cluster 2007 conference. This paper presents
several approaches to run out-of-core applications in a cluster,
including Lustre over local disks of the compute nodes.

In addition, we developed a user framework to deploy Active Storage for
Lustre and PVFS. The framework is made up of a program to carry out
Active Storage jobs, and a library to provide transparent support for
processing striped files. The program receives as argument an XML file
containing a rule, which basically specifies the files to process and
the kind of processing to perform on them. We expect to release a first
version of this framework in the next weeks (more information at
http://hpc.pnl.gov/projects/active-storage).

***Plans for next quarter***

In the next quarter, we will continue to develop and improve our user
framework for AS in preparation for the upcoming release.

**[Task 1.7: Cray XT I/O Stack Instrumentation and
Optimization]{.underline}**

SciDAC applications make use of a variety of different IO interfaces,
including MPI-IO, Fortran I/O, and Parallel netCDF. We have carried out
a series of studies to examine the parallel I/O subsystem at ORNL's
Leadership Computing Facility (LCF), which uses the Lustre file system.

***Progress Report***

The Cray XT IO subsystem uses Lustre file system. To gain insights into
the deliverable IO efficiency on the Cray XT4 platform at ORNL, we are
performing an in-depth evaluation of its parallel IO software stack.
Through our evaluation, we have shown that it is important to make
correct choices on the number of stripes for parallel IO over Cray XT.
In addition, users need to be aware of the speed variance of IO accesses
with different IO request sizes. Also, we have demonstrated that it is
beneficial to limit IO accesses to a moderate number of processes,
typically less than 512 processes. To this end, we have described a
variety of tuning parameters for the parallel IO stack and have
demonstrated their effectiveness in enhancing the IO efficiency of a
common scientific IO benchmark, Flash IO. Our results suggest that by
tuning the collective buffer size and the number of IO aggregators, the
achievable bandwidth of Flash IO can be increased by as much as 30%.

The default MPI-IO package over Lustre is supplied from Cray. This
package contains a proprietary ADIO implementation built on top of
*libsysio*, which itself is an IO library for Catamount clients to
access Lustre file system. In order to instrument the internal MPI-IO
implementation, we built an alternative MPI-IO package for Cray XT,
using the default MPI-IO implementation from Argonne National
Laboratory. We validated this resulting open-source MPI-IO library
against the original package from Cray in terms of performance. Thus, it
provides a valid starting point for analysis and optimization of
Collective IO over XT. We are collaborating with the National Center of
Computational Sciences, Cray, and Cluster File Systems on the initial
deployment of this package.

Utilizing the same alternative package from the last study, we
introduced a novel technique called partitioned collective IO (ParColl)
to break this limitation. ParColl partitions parallel processes into
subgroups, each carrying out smaller, yet aggregated IO operations. In
doing so, this partitioning reduces the synchronization costs of global
collective operations, and improves the scalability of collective IO.
Our experimental results indicate that it can significantly enhance the
performance. In one case, our results show a 416% improvement on 1024
processes. We also show that scientific application benchmarks, such as
Flash IO, can benefit significantly from this technique. Moreover, our
experiments on an 80-node Linux cluster suggest that it is a general
technique and can be applied to a wide range of other platforms beyond
Cray XT.

***Plans for next quarter***

In the coming months, we will deploy our open-source MPI-IO library to
Jaguar at the National Center of Computational Sciences, Oak Ridge
National Laboratory. In addition, we will apply our MPI-IO optimizations
to representative SciDAC applications to demonstrate its benefits.

**[References]{.underline}**

\[1\] Wei-keng Liao, Kenin Coloma, Alok Choudhary, and Lee Ward,
"Cooperative Client-side File Caching for MPI Applications," In the
International Journal of High Performance Computing Applications, Volume
21, Number 2, pp. 144-154, 2007.

\[2\] Weikuan Yu, Sarp Oral, Jeffrey Vetter, Richard Barrett. Efficiency
Evaluation of Cray XT Parallel IO Stack. Cray User Group Meeting (CUG
2007), May 2007. Seattle, WA.

**[2. Data Mining and Analysis (DMA)]{.underline}**

### [Task 2.2: Feature Selection in Scientific Applications (LLNL)]{.underline}

# *Contact:* Chandrika Kamath, LLNL

## Accomplishments

## The work on Poincare plot analysis is still on hold. The hire of a software developer or post-doc is being constrained by various issues associated with the contract change at LLNL.

## We (my post-doc Nicole Love and I) continued our work on the problem of blob-tracking in NSTX. We have been having regular discussions with Stewart Zweben, Daren Stotler, and Ricky Maqueda from PPPL. Ricky provided us with additional data - longer sequences from a different camera on NSTX. These sequences are 8000 frames long in comparison with 300 frames from the PSI camera.

## We have applied our background subtraction software to remove the quiescent background intensity in the sequences. We are currently tracking down the presence of ghost pixels in two of the methods - it is unclear if these are due to the characteristics of the data or a bug in our codes. We expect that we will need these techniques for the longer sequences.

## We continue our experimentation with various segmentation techniques. We have implemented several improvements and are in the process of testing them. The performance of these methods is being summarized in an SPIE conference paper.

## I participated in the FSP math/cs panel from January through May, attending the committee meeting in May and contributing to the workshop report. I gave a talk on petascale data analysis at the Applied Math PI meeting in May.

## Further contacts with new applications is on hold until I can address the hiring situation. 

## Plans for Next Quarter

### For the NSTX data, we will be discussing our results with PPPL physicists and incorporating their suggestions and comments. We will apply our techniques to other sequences to evaluate the robustness of the segmentation methods, especially the settings of the parameters. We will also look into applying our tracking software to the blobs detected by segmentation. And the quest to hire a post-doc or software developer will continue. 

### 

### [Task 2.3: High-dimensional indexing techniques (LBNL)]{.underline}

# Contact: John Wu, LBNL

## Accomplishments

-   Add a new layer to FastBit to unify data access operations. This is
    the major addition planned before release to make the FastBit code
    more user-friendly.

-   Add a more consistent way to handle horizontal data partition in
    FastBit. Previously, the user of FastBit has to work directly with
    different horizontal data partitions. This new addition will
    eliminate this manual process and provide an easier way to aggregate
    results from multiple horizontal data partitions.

-   Finalize the new multi-level bitmap index functions following the
    new insight gain from recent analyses. In theory, the new
    implementation can be up to five times faster than the fastest
    indexing functions existed in the FastBit software before. In
    performance measurements, the new implementation can actually be up
    to 10 times faster than our old favorite, but is never slower. Looks
    like we have a winner here.

-   Work with JGI's QA group to use FastBit in their analyses of DNA
    sequence quality data. We have implemented a number of new functions
    to deal with string-valued variables for this collaboration effort,
    and have been working on automating the process of consolidating
    incoming data that are produced in small chunk. To allow these
    operations to precede more swiftly, we have make it possible for
    FastBit to access new data without using double buffering.
    Previously, all incoming records are double-buffered.

-   Continue to implement feature identification functions for data on
    toroidal meshes (with Rishi Sinha of UIUC). This extends our
    previous work on data from regular meshes. The target application is
    the analysis of fusion simulation data.

## 

## Plans for next quarter

-   Add documentation and tutorial to FastBit.

-   Continue the preparation to release FastBit.

-   Continuing the work on feature identification in fusion simulation
    data.

-   Revisit the binning strategies in FastBit. Theoretical analysis of
    the new data structure is needed for "real-application data" not
    just the worst-case scenario.

    **[3. Scientific Process Automation]{.underline}**

The Internet is becoming the preferred method for disseminating
scientific data from a variety of disciplines. This has resulted in
information overload on the part of the scientists, who are unable to
query all of the relevant sources, even if they knew where to find them,
what they contained, how to interact with them, and how to interpret the
results. Thus instead of benefiting from this information rich
environment, scientists become experts on a small number of sources and
use those sources almost exclusively. Enabling information based
scientific advances, in domains such as functional genomics, requires
fully utilizing all available information. We are developing an
end-to-end solution using leading-edge automatic wrapper generation,
mediated query, and agent technology that will allow scientists to
interact with more information sources than currently possible.
Furthermore, by taking a workflow-based approach to this problem, we
allow them to easily adjust the dataflow between the various sources to
address their specific research needs.

***Major Accomplishments***

-   Implemented new web service actor, WSWithComplexTypes, that permits
    sending and receiving complex web service types as parameters. The
    actor automatically creates composite actors for each complex
    parameter, within which are a series of connected XML actors that
    construct or decode the complex type. Since the contents of each
    composite actor are dependent on the complex type, they are
    generated on-the-fly each time the user changes the web service
    method.

-   Produced Use-case document outlining specific use-cases and data
    types for provenance information (completed - attached)

-   Implemented a pilot version of the Dashboard operating with ORNL
    fusion codes.

    ***Task 3.1: Dashboard Development***

    **Current Status**

```{=html}
<!-- -->
```
-   The dashboard is currently being hardened, and ORNL is working with
    both Utah and NCSU to incorporate flash routines for monitoring the
    simulation and data base routines for storing the data from the
    simulations. Our current work will be to allow users to organize and
    store notes for their simulations so that they can later query and
    search these simulations. We are also looking at methods to allow
    users to download files from the simulations from the dashboard to
    their local resources.

-   Created python scripts to produce the necessary information about
    the CPES workflow runs for the dashboard (meta-data about the
    simulation and workflow outputs, listings of variables for which
    images has been produced, actual timestep where the
    simulation/workflow processing is at) and integrated those scripts
    into the CPES workflows so that any CPES related workflow executions
    and all produced images are now visible on the dashboard.

-   Begun designing the workflows and dashboard for the GTC project,
    (using the circular version, and the shaped version which is one of
    the 3 Joule codes this year), the S3D code, and the CPES suite of
    codes.

    -   Worked with other members of the CPES team to create the first
        fully coupled edge/MHD simulation, looking at the Edge Localized
        Mode (ELM) crash. This workflow couples the XGC, M3D-OMP,
        M3D-MPP, and the Elite code.

    -   The S3D workflow is currently being designed and implemented at
        ORNL, and is based on the basic CPES monitoring workflow.
        Currently we can move the restart data from the Cray XT4 at ORNL
        to our end to end cluster, and then morph these files to a
        smaller set of files, tar these, place these on hpss.

    -   We have routines which also graph the netcdf files, and move
        them over to a location on the cluster where the dashboard can
        visualize these. We are also looking at multiple methods of
        moving the morphed files to Sandia for later analysis by S3D
        users.

    -   The GTC workflow is in its infancy, as the parallel I/O and
        asynchronous I/O are still being worked out. GTC will use a
        combination of binary files, hdf5 files, and netcdf files.

-   Implemented pilot version of the data-base for displaying data in
    the dashboard

-   Implemented pilot version of the visualizations for the dashboard
    display.

-   Identified a pilot of the provenance/dashboard operational
    environment (Kepler-Linux-Apache-MySQL-PHP), i.e., (K)LAPM

-   NCSU students made a number of visits to ORNL to work jointly with
    ORNL staff on the CPES Dashboard and Provenance project

-   Regular (bi-weekly) dashboard teleconferences are in place

-   Produced draft document on Dashboard/Provenance architecture
    (attached). This document includes

-   Literature Survey: A literature survey on data provenance was done.
    A bibliographic list is available in section 6 of the document
    SPA_Provenance_Dashboard_V3.doc

-   Comparison I: A comparison of the current capabilities of Kepler,
    Vistrails, And the Dashboard was done. The strengths and weaknesses
    of each tool were detailed. This can be found in section 4 of the
    document SPA_Provenance_Dashboard_V3.doc

    **Plans for next quarter**

```{=html}
<!-- -->
```
-   CPES. We plan on looking at adaptive workflows with UC Davis, and
    harden the current workflow for CPES users. We also plan on working
    with CPES users to allow more users to use the dashboard and
    workflow for XGC0-code coupling, and XGC1 simulations. We will also
    work with Utah and NCSU to incorporate their technology into the
    dashboard for ORNL users.

-   CPES. All workflow information is recorded in text files currently.
    The dashboard has been re-designed to take information from a
    database. We will modify the workflows and scripts to record
    everything in the database instead of the files.

-   S3D. We plan to finish their basic workflow during the summer, and
    allow S3D users to evaluate the basic workflow.

-   GTC. We plan on working with NWU to get higher performance I/O on
    the Cray XT3. Currently we are getting about 15GB/sec on 8K node
    simulations for writing the restarts not using MPI-IO, and are
    evaluating our parallel I/O routines (which should get slightly
    higher I/O rates using MPI-IO. We also plan on getting the basic GTC
    workflow running during the next quarter.

-   Have a fully operational (K)LAMP environment with pilot dashboard
    and provenance implementations and for the SC07 tutorial

-   Complete the Dashboard architecture document

    ***Task 3.2: Provenance Tracking***

    **Current Status**

```{=html}
<!-- -->
```
-   Organized and held Provenance/Dashboard design meeting in June at
    SDSC.

-   Created a merged schema from the previously worked on provenance
    schema, dashboard schema, and Kepler MoML. The new schema is under
    further construction and will provide a consolidated final view over
    the past work on the subject.

-   Produced Use-case document outlining specific use-cases and data
    types for provenance information.

    -   Defined the CPES workflow use-case for provenance tracking

    -   Worked with groundwater team to develop provenance use case for
        groundwater

        **Plans for next quarter**

```{=html}
<!-- -->
```
-   Finalize the database schema based on Use Case document.

-   Have a pilot implementation of the schema-based DB.

-   Design and document the Provenance architecture.

-   Implement provenance architecture including Kepler extensions and
    the related API components.

-   Complete the BPEL survey for different workflows (report)

    ***Task 3.3: Outreach***

    **Current Status**

```{=html}
<!-- -->
```
-   Continued active participation in Kepler consortium

    -   Investigated bugs with ExternalExecution actor and IOPort.

    -   New actors implemented for token type conversion:
        RecordToStrings, StringsToRecord, RecordMerger, StringsToXML,
        XMLToStrings.

    -   New actors implemented for XML processing: XMLAssembler,
        XMLDisassembler, XMLUpdater.

-   Identified opportunity for development of biology workflow. This
    will require coordinated effort between SDSC and PNNL, with the
    details still being worked out.

-   Developed several Kepler workflows for different simulation coupling
    scenarios in the CPES Fusion project with the co-ordination of Scott
    Klasky from ORNL. A full-coupled run using four different simulation
    codes (XGC-0, M3D-OMP, ELITE and M3D-MPP) has been recorded as a
    movie. However, the physics results from that run were not
    satisfactory yet, so this movie has not been made public.

-   Attended SciDAC PI meeting, solidified contacts with groundwater
    applications group, initiated discussions with VACET team (who
    expect to use Kepler as part of their work). We are investigating
    ways to either transition across or merge provenance systems
    utilized by the respective efforts.

-   Begun initial technology comparison between BPEL workflow engines
    and Kepler

    **Plans for next quarter**

```{=html}
<!-- -->
```
-   Extend WSWithComplexTypes actor with port parameters for constant
    inputs.

-   Initial prototype of biology workflow in place

-   New coupling scenarios will be tried to ensure precise computation
    that can lead to relevant physics results in the CPES project.

-   Re-new the "Blondin" workflow effort now that those codes are almost
    re-written.

-   Investigate what components of existing dashboard environment can be
    reused to establish a dashboard interface for groundwater.

-   Complete preliminary technology comparison with BPEL workflow
    engines

    ***New Publications and Related Presentations***

> Awards

-   Mladen Vouk and Claudio Silva both received IBM Faculty Awards

> Publications

-   Podhorszki, N., Ludaescher, B., Klasky, S.: Workflow Automation for
    Processing Plasma Fusion Simulation Data In 2nd Workshop on
    Workflows in Support of Large-Scale Science (WORKS\'07), June 25,
    2007, Monterey, California, U.S.A.

-   Bowers, S., McPhillips, T., Wu, M., and Ludaescher, B.: Project
    Histories: Managing Data Provenance Across Collection-Oriented
    Scientific Workflow Runs. In Data Integration in the Life Sciences
    2007 (DILS'07), June 27-29, 2007, Philadelphia PA.

-   Vouk, M. A., Altintas, I., Barreto, R., Blondin, J., Cheng, Z.,
    Critchlow, T., Khan, A., Klasky, S., Ligon, J., Ludaescher, B.,
    Mouallem, P. A., Parker, S., Podhorszki, N., Shoshani, A. and
    Silva, C. Automation of Network-Based Scientific Workflows. *Proc.
    of the IFIP WoCo 9 on Grid-based Problem Solving Environments:
    Implications for Development and Deployment of Numerical Software,
    IFIP WG 2.5 on Numerical Software,* Prescott, AZ, 2006.

-   Batchelor, D.A., Beck, M., Becoulet, A., Budny, R.V., Chang, C.S.,
    Diamond, P.H., Dong, J.Q., Fu, G.Y., Fukuyama, A., Hahm, T.S.,
    Keyes, D.E., Kishimoto, Y., Klasky, S., Lao, L.L., Li, K., Lin, Z.,
    Ludaescher, B., Manickam, J., Nakajima, N., Ozeki, T., Podhorszki,
    N., Tang, W.M., Vouk, M.A., Waltz, R.E., Wang, S.J., Wilson, H.R.,
    Xu, X.Q., Yagi, M. and Zonca, F. Simulation of Fusion Plasmas:
    Current Status and Future Direction. *Plasma Science and
    Technology,* 9:3, June 2007.

-   Klasky, Scott, Mladen A. Vouk, Ilkay Altintas, Jeff Ligon, Pierre
    Moualem. Scientific Data Workflows. Tutorial, National Center for
    Computational Sciences (NCCS) 2007 Users Meeting at Oak Ridge
    National Laboratory, March 27-29, 2007.

> Presentations and other activities

-   A Kepler tutorial at SC07 conference was proposed and accepted. This
    is the second year in a row this tutorial will be offered.

-   Norbert Podhorszki and David Koop attended the 2^nd^ Provenance
    Challenge workshop held in Monterey, CA on June 26. The challenge
    participants are now seeking for real-world workflow examples
    instead of the artificial workflow used in the first two challenges
    and the CPES coupling workflow developed by UCD is considered to be
    a candidate for the third challenge.

-   Tim McPhillips made a presentation at the Data Integration in the
    Life Sciences workshop in Philadelphia, PA on June 29. The
    presentation described how provenance of data used in and produced
    by scientific workflows can be tracked easily across multiple runs
    if the COMAD (Collection-Oriented Modeling and Design) paradigm is
    applied. Bertram Ludaescher has suggested that this approach be
    considered as a complement to the other ongoing provenance
    approaches in SPA.

-   Bertram Ludaescher attended the Principles of Provenance Workshop at
    Philadelphia, PA on June 27.

-   Regular postings on Wiki
    ([[http://www.vistrails.org/index.php/SDM_Provenance]{.underline}](http://www.vistrails.org/index.php/SDM_Provenance))

-   Use-cases and data requirements document -- V3a