![](media/image1.png){width="1.375in" height="0.74375in"}SDM Center
Report

January-March 2007

http://sdmcenter.lbl.gov

**[Introduction]{.underline}**

Managing scientific data has been identified as one of the most
important emerging needs by the scientific community because of the
sheer volume and increasing complexity of data being collected.
Effectively generating, managing, and analyzing this information
requires a comprehensive, end-to-end approach to data management that
encompasses all of the stages from the initial data acquisition to the
final analysis of the data. Fortunately, the data management problems
encountered by most scientific domains are common enough to be addressed
through shared technology solutions. Based on the community input, we
have identified three significant requirements. First, more efficient
access to storage systems is needed. In particular, parallel file system
improvements are needed to write and read large volumes of data without
slowing a simulation, analysis, or visualization engine. These processes
are complicated by the fact that scientific data are structured
differently for specific application domains, and are stored in
specialized file formats. Second, scientists require technologies to
facilitate better understanding of their data, in particular the ability
to effectively perform complex data analysis and searches over large
data sets. Specialized feature discovery and statistical analysis
techniques are needed before the data can be understood or visualized.
To facilitate efficient access it is necessary to keep track of the
location of the datasets, effectively manage storage resources, and
efficiently select subsets of the data. Finally, generating the data,
collecting and storing the results, data post-processing, and analysis
of results is a tedious, fragmented process. Tools for automation of
this process in a robust, tractable, and recoverable fashion are
required to enhance scientific exploration.

Our approach is to employ an evolutionary development and deployment
process: from research through prototypes to deployment and
infrastructure. Accordingly, we have organized our activities in three
layers that abstract the end-to-end data flow described above. We
labeled the layers (from bottom to top):

-   Storage Efficient Access (SEA)

-   Data Mining and Analysis (DMA)

-   Scientific Process Automation (SPA)

The SEA layer is immediately on top of hardware, operating systems, file
systems, and mass storage systems, and provides parallel data access
technology, and transparent access to archival storage. The DMA layer,
which builds on the functionality of the SEA layer, consists of
indexing, feature identification, and parallel statistical analysis
technology. The SPA layer, which is on top of the DMA layer, provides
the ability to compose scientific workflows from the components in the
DMA layer as well as application specific modules.

# This report consists of the following sections, organized according to the three layers, as follows:

-   Storage Efficient Access (SEA) techniques,

    -   Low-Level Parallel I/O Infrastructure

    -   Client-side file caching for MPI-IO

    -   File System Benchmarking and Application I/O Behavior

    -   Application I/O Stack Benchmarking and Optimizations

    -   SRM/MPI-IO Parallel File System Caching and Migration Service

    -   Active Storage in Parallel Fileystem

```{=html}
<!-- -->
```
-   Data Mining and Analysis (DMA) components

    -   High-performance statistical computing for scientific
        applications

    -   Feature Selection in Scientific Applications

    -   High-dimensional indexing techniques

-   Scientific Process Automation (SPA) tools

    -   Mediated Query Environment

    -   Outreach and User Interaction

    -   Incorporation of Agent / Workflow Technology

    -   SCIRun/Kepler integration

    -   Dashboard Generation

    -   Provenance Tracking and Outreach

During this period, the SDM center was invited to organize a session,
give a plenary talk and three other talks at the SIAM Computational
Science & Engineering (CSE\'07), In February 2007. All the talks are
posted at: http://sdmcenter.lbl.gov, under "Posters/talks".

-   Arie Shoshani (LBNL) presented the plenary talk entitled:
    "Scientific Data Management: Essential Technology for Data-Intensive
    Science".

-   Chandrika Kamath (LLNL) presented the talk entitled: "Mining Science
    Data".

-   Terence Crichlow (LLNL) presented a talk entitled: "Accelerating
    Scientific Exploration Using Workflow Automation Systems".

-   Rob Latham (ANL) presented a talk entitled: "HighPerformance
    Parallel Data and Storage Management"

Arie Shoshani was also invited to participate in a panel at the SciDAC-2
Organization Workshop on February 5-6, 2007 in Atlanta. He presented a
short talk on Storage and Data Challenges which is also posted at:
http://sdmcenter.lbl.gov, under "Posters/talks".

The reports by each of the three areas, SEA, DMA, ans SPA, follow.

**[1. Storage Efficient Access (SEA)]{.underline}**

***Participants:*** Rob Ross, Rajeev Thakur, Sam Lang, and Rob Latham
(ANL), Alok Choudhary, Wei-keng Liao, Kenin Coloma, and Avery Ching
(NWU), Arie Shoshani and Ekow Otoo (LBNL), Jeffrey Vetter and Weikuan Yu
(ORNL), Jarek Nieplocha and Juan Piernas Canovas (PNL)

The goal of this project is to provide significant improvements in the
parallel I/O subsystems used on today\'s machines while ensuring that
the capabilities available now will continue to be available as systems
increase in scale and technologies improve. A three-fold approach of
immediate payoff improvements, medium-term infrastructure development,
and targeted longer-term R&D is employed.

Two of our keystone components are the PVFS parallel file system and the
ROMIO MPI-IO implementation. These tools together address the
scalability requirements of upcoming parallel machines and are designed
to leverage the technology improvements in areas such as
high-performance networking. These are both widely deployed and freely
available, making them ideal tools for use in today's systems. Our work
in application I/O interfaces, embodied by our Parallel NetCDF
interface, also promises to provide short-term benefits to a number of
climate and fusion applications.

While we focus significant effort on PVFS, we recognize the importance
of other file systems in the HEC community. For this reason our efforts
include improvements to the Lustre file system, and we routinely discuss
both Lustre and GPFS during tutorials. Our efforts in performance
analysis and tuning for parallel file systems, as well as our work on
MPI-IO, routinely involve these file systems.

At the same time we continue to push for support of common,
high-performance interfaces to additional storage technologies. Our work
combining the Storage Resource Manager (SRM) with MPI-IO, in conjunction
with PVFS, will provide a single solution for the problems of
high-performance parallel storage access, integration of file system
with tertiary storage, and remote data access. Our work in Active
Storage will provide common infrastructure for moving computation closer
to storage devices, an important step in tackling the challenges of
petascale datasets.

**[Task 1.1: Low-Level Parallel I/O Infrastructure]{.underline}**

The objective of this work is to improve the state of parallel I/O
support for high-end computing (HEC) and enable the use of high
performance parallel I/O systems by application scientists. The Parallel
Virtual File System (PVFS) and ROMIO MPI-IO implementations, with
development lead by ANL, are in wide use and provide key parallel I/O
functionality. This work builds on these two components by enhancing
them in order to ensure these capabilities continue to be available as
systems continue to scale. Additional work is being performed on the
Lustre file system to improve its performance for certain critical
workloads.

***Progress Report***

PVFS version 2.6.2 was released on 1/17/2007, including a number of bug
fixes and new configuration options. Significant effort has been spent
in testing and configuring I/O on loaner DataDirect Networks storage
hardware in preparation for PVFS deployment on similar hardware in ALCF,
and we have begun discussions on early access to BG/P hardware for PVFS
testing purposes.

Plans were also made with Garth Gibson of Carnegie Mellon University for
PVFS to be used in a course as the basis for work in distributed
directory storage in high-performance file systems. We expect to make a
trip to CMU to discuss this project with his class and to discuss
further collaborations with the PDSI.

The ANL and NWU groups are working towards an improved two-phase
implementation for ROMIO. This implementation is complete and
preliminary testing has been performed. The ANL group has also completed
prototypes of MPI-IO atomic mode and shared file pointer operations
using MPI RMA and point-to-point communication, and a paper on this work
has been accepted for journal publication \[3\]. Overall these changes
in ROMIO are leading us to believe that some major overhaul in ROMIO is
necessary.

***Plans for next quarter***

Effort in the next quarter on PVFS will be split between further
understanding and optimizing access with high-end storage arrays,
testing PVFS support over Myricom MX on their 10Gigabit hardware, and
testing and debugging on IBM BG/P hardware as it becomes available. This
is all in support of ALCF.

Major ROMIO changes may be undertaken in the following quarter as well,
depending on the outcome of the PnetCDF discussions (below).

Finally, we have three students arriving in the coming quarter to work
on PVFS and ROMIO topics.

**[Task 1.2: Collaborative File Caching]{.underline}**

The objective of this work is to develop a user-level client-side file
caching system for

MPI applications that perform parallel I/O operations on shared files.
In our design, an I/O thread is created in each MPI process and all
threads cooperate with each other to perform a coherent file caching.

***Progress Report***

The prototype of the I/O thread mechanism has been completed. Its basic
operations include remote cache data GET/PUT, local cache data access,
global cache metadata updates, and read/write data from/to the
underlying file system. Some advance operations have also been built and
tested, including cache metadata lock management, cache page eviction
policy, page migration, and a two-phase data flushing mechanism.

We have studied the programming structure of ROMIO and ADIO. For the
testing and evaluation purpose, our caching system is placed at the ADIO
layer and currently works for UFS and PVFS1. A few MPI hints are added
for enabling the caching, adjusting cache page size, enabling the
two-phase flushing, changing the memory space upper bound for caching,
etc.

We continue to exercise our design on the two large machines at NCSA: a
Dell Linux cluster running Lustre parallel file system and a TeraGrid
machine running an IBM GPFS file system. The current performance
evaluation focuses on the NASA BTIO benchmark and the FLASH application
I/O kernel. We also studied the performance under direct I/O mode. The
preliminary results have been reported in two papers published in IPDPS
2007 conference \[1,2\].

***Plans for next quarter***

In many scientific applications, write-only I/O patterns such as data
checkpointing, dominate the overall I/O activities. The nature of these
patterns often needs no consistency or atomicity requirements specified
by POSIX standard. We will investigate the approach to relax such
requirements in our caching design. We also plan to develop a mechanism
based on our caching system to focus on improving the performance of
write-only operations. We believe the performance of our caching can be
further improved if the overhead for coherence and atomicity control is
removed.

**[Task 1.3 File System Benchmarking and Application I/O
Behavior]{.underline}**

The objective of this work is to evaluate the relative performance of
the file systems available to important SciDAC applications on DoE
compute platforms, and, in particular the sustained speed of reading and
writing large datasets. The performance, functionality, and scalability
of MPI-IO, parallel netCDF, and HDF5 are critical for many applications.
Two groups in the SDM center, NWU and ORNL, are performing complementary
efforts in this area.

***Progress Report***

The NWU group has been using BTIO, FLASH I/O, and NCIO benchmarks to
evaluate parallel shared-file I/O performance on the Lustre, and GPFS
file systems. We have observed significant performance degradation due
to the file system lock contention. Our experiments show that if the I/O
requests are carefully aligned with file system lock boundaries,
performance can be improved. We study the lock contention issue together
with the file caching design for MPI I/O. The preliminary results and
performance analysis are reported in the two papers published in IPDPS
2007 conference. To further address the drawback of using block-based
file lock protocol, we prototype a scalable distributed lock manager
architecture which has true byte-range granularity. Aiming to improve
the non-contiguous I/O performance, we developed novel list and datatype
locking methods and created hybrid two-phase locking protocols. We are
also designing a synthetic I/O benchmark to evaluate the scalability of
our method for non-overlapping and overlapping I/O patterns.

We also obtained the S3D application I/O kernel from our collaborators,
Jacqueline Chen at Sandia National Laboratories, Ramanan Sankaran and
Scott Klasky at ORNL. S3D is a parallel turbulent combustion application
using a direct numerical simulation solver developed at Sandia National
Laboratories. We have been studying the I/O patterns used in S3D. The
S3D's I/O is originally programmed in Fortran I/O functions and each
process writes all its sub-arrays to a separate file at each checkpoint.
We added the MPI I/O functionality to write the arrays into a shared
file in their globally canonical order. With this change, there is only
one file created per checkpoint, regardless of the number of MPI
processes used. The option to choose collective or independent MPI I/O
functions is also implemented. We are currently testing this work under
PVFS, Lustre and GPFS file systems.

***Plans for next quarter***

We plan to continue the work on developing the distributed lock manager
and test it using scientific I/O benchmarks, including the NASA BTIO,
FLASH I/O, NCIO, and S3aSim. The evaluation will compare against a
block-based cache and locking Lustre and GPFS implementation to show how
false sharing negatively impacts I/O performance. We plan to use the S3D
I/O kernel for performance evaluation on the Lustre and GPFS file
systems at NCSA and PVFS at SNL.

**[Task 1.4: Application Interfaces to I/O]{.underline}**

The objective of this work is to improve the observed I/O throughput for
applications using parallel I/O by enhancements to or replacements for
popular application interfaces to parallel I/O resources. This task was
added in response to a perceived need for improved performance at this
layer, in part due to our previous work with the FLASH I/O benchmark.
Because of their popularity in the scientific community we have focused
on the NetCDF and HDF5 interfaces, and in particular on a parallel
interface to NetCDF files.

***Progress Report***

We moved from using CVS internally to using SVN and Trac to manage the
Parallel netCDF source tree. This has enabled us to more easily share
the latest source with external parties, facilitating more community
involvement. We also packaged a pre-release, version 1.0.2pre2 and made
this available for comment on 2/16/2007. This version captures fixes to
a variety of problems seen on platforms when operating with datasets of
greater than 2GB that were related to overflow in internal variables.

Interest continues to grow in PnetCDF, and initial discussions on a
slight change to the file format (coordinated with the serial netCDF
team) are underway that would allow larger datasets to be stored.

Support for accessing Parallel Extendible Array files in Global Array
applications has been implemented as well, providing an alternative to
Disk Resident Arrays.

***Plans for next quarter***

This discussion of modifications to the file format seems to be moving
towards consensus. Assuming that we can agree on how the file format
should change, we will prototype these changes in PnetCDF to allow for
experimentation and further discussion. We expect to release version
1.0.2 in the near term.

We also plan on demonstrating and writing tutorials on how libmpiosrm
library is used to access remote files into local parallel files for
specific file formats (e.g. HDF5, BOXLib and Parallel Extendible Array
files) and continuing work on implementation and testing of parallel
extendible array files using chunking and block compaction.

**[Task 1.5: SRM/MPI-IO Parallel File System Caching and Migration
Service]{.underline}**

The objective of this work is to combine the parallel I/O capabilities
of MPI-IO with the remote access and mass storage management
capabilities of the Storage Resource Manager (SRM), allowing
applications to access remote and local data and to stage data on local
parallel I/O resources. Functionality is being built to allow caching of
files on a local parallel file system (such as PVFS) and migrating data
between the cache and a remote tertiary storage system in the context of
a MPI application. For example, this capability would allow for movement
of data between HPSS and PVFS. LBNL leads this effort.

***Progress Report***

Functions to read/write HDF5 files from/to remote mass storage system
from applications written in C/C++ are under development.

***Plans for next quarter***

The main activities for the next couple of months are providing wrapper
functions for accessing library functions from Fortran 90 applications
and completing the functions to read/write HDF5 files from/to remote
mass storage system from Fortran 90 and C/C++ applications.

**[Task 1.6: Active Storage in the Parallel Fileystem]{.underline}**

The purpose of this work is to extend the active storage/disk concept to
parallel file systems, in particular Lustre and PVFS, and make it
practical for DOE applications. The PNL team leads this work.

***Progress Report***

In the first quarter, we focused on bringing the Active Storage
prototype to work under the current version of Lustre and Linux. In the
second quarter, we completed the port of the original kernel-space
implementation to Lustre 1.4 and 1.6beta5 (the latest version used by
many Lustre sites). We identified several factors that degrade
performance (data copies between user and kernel space) and investigated
synchronization options between remote client and AS processing
component.

To address some of these performance issues with the kernel space
implementation, we developed a new implementation strategy for Active
Storage. It is done purely in the user space. Based on preliminary
experiments and experience, it appears to be faster and more flexible
than the original version. It is also more portable -- we implemented it
for both Lustre as well a PVFS2. These findings (for Lustre) are
described in a paper we submitted to SC'07.

***Plans for next quarter***

In the next quarter, we will pursue deployment of Active Storage in
biology or climate application and work on improving the user space
implementation.

**[Task 1.7: Cray XT I/O Stack Instrumentation and
Optimization]{.underline}**

SciDAC applications make use of a variety of different I/O interfaces,
including MPI-IO, Fortran I/O, and Parallel netCDF. One of the primary
foundations of these parallel I/O packages is Collective IO. However,
the performance of collective I/O on DOE Leadership platforms is
sometimes suboptimal. We are working to improve collective I/O on the
Cray XT system at ORNL's Leadership Computing Facility (LCF), which uses
the Lustre file system.

***Progress Report***

The Cray XT I/O subsystem uses Lustre file system. The default MPI-IO
package over Lustre is supplied from Cray. This package contains a
proprietary ADIO implementation built on top of *libsysio*, which itself
is an IO library for Catamount clients to access Lustre file system. In
order to instrument the internal MPI-IO implementation, we built an
alternative MPI-IO package for Cray XT, using the default MPI-IO
implementation from Argonne National Laboratory. The resulting
open-source MPI-IO library attains comparable performance to the
original package from Cray. Thus, it provides a valid starting point for
analysis and optimization of Collective I/O over XT.

Taking advantage of this alternative package, we have profiled the
internal processing of collective I/O operations. We found that the
amount of time spent in collective communications to prepare for the I/O
operations dominates over the time required to do the actual IO
operations themselves, imposing a 'collective wall' for I/O performance.
We implemented a novel technique for enhancing collective I/O in these
cases and have seen significant improvement in performance using this
technique, both on the Cray XT and on a Linux cluster. Results of this
work have been detailed in a paper submitted to SC'07.

***Plans***

In the coming months, we will continue our endeavors of parallel I/O
optimizations by instrumenting parallel I/O libraries, which are built
on top of these underlying file systems. In addition, we will apply
ParColl to representative SciDAC applications to demonstrate its
benefits. Furthermore, we plan to correlate and analyze the performance
benchmarking results, along with their IO access patterns, which will be
collected through the I/O profiling and trace tools. The aim of this
practice is to arrive at a predictive model that facilitates performance
analysis and optimization of SciDAC applications

**[References]{.underline}**

\[1\] Wei-keng Liao, Avery Ching, Kenin Coloma, Alok Choudhary, and Lee
Ward. An Implementation and Evaluation of Client-side File Caching for
MPI-IO. In the Proceedings of the 21th International Parallel and
Distributed Processing Symposium (IPDPS), Long Beach, California, March
2007.

\[2\] Wei-keng Liao, Avery Ching, Kenin Coloma, Alok Choudhary, and
Mahmut Kandemir. Improving MPI Independent Write Performance Using A
Two-Stage Write-Behind Buffering Method. In the Proceedings of the Next
Generation Software (NGS) Workshop, held in conjunction with the 21th
International Parallel and Distributed Processing Symposium (IPDPS),
Long Beach, California, March 2007.

\[3\] Robert Latham, Robert Ross, and Rajeev Thakur, \"Implementing
MPI-IO Atomic Mode and Shared File Pointers Using MPI One-Sided
Communication,\" to appear in Int\'l Journal of High Performance
Computing Applications, Summer 2007.

**[2. Data Mining and Analysis (DMA)]{.underline}**

### [Task 2.1: High-performance statistical computing for scientific applications, ORNL]{.underline}

***Contact:*** Nagiza Samatova, ORNL

The following people have contributed to the project over this quarter:

Jiangtian Li, Srikanth Yoginath, Guru Kora, Xiaosong, and Nagiza
Samatova

In collaboration with: Interactive Supercomputing Inc., John Drake,
Jackie Chen, George Ostrouchov, Sean Ahern, Phillip Rack and some of
their students.

## Accomplishments 

In this quarter most of the time was spent in interacting with different
application scientists and identifying their needs for high performance
statistical data analytics. The following four applications have been
explored:

-   *Nanoscience* (Dr. Philip Rack and Mr. Daryl Smith, DOE/BES-funded
    ORNL Center for Nanophase Materials, CNMS): This group is simulating
    an electron beam induced deposition process using Matlab library.
    They use a double Monte Carlo electron-solid code to track the
    electron trajectory in a solid. On top of this Monte Carlo code,
    they simulate electron stimulated decomposition of adsorbed gas
    molecules that condense onto the surface and grow a 3-dimensional
    structure. They also handle the adsorption/desorption using a
    langmuir isotherm approach and have routines to handle the surface
    diffusion. The process has been run in a so-called static mode
    (where the electron beam does not move), but have recently expanded
    it to a so-called raster mode, where they move the beam in x and y
    to grow extended structures. The rasterized code is where there is
    tremendous value in parallizing the code using parallel Matlab,
    though there might be ways of parallelizing computation in the basic
    static code as well. We have been working with this group to
    identify the strategy for parallelizing their Matlab code. We
    prepared an initial prototype and will need to benchmark it up and
    meet with the customer to identify the next steps and get their
    feedback.

-   *Climate statistical downscaling* (Drs. John Drake, George
    Ostrouchov): Assessment of global climate change impacts requires
    increasingly finer spatial and temporal resolutions from existing
    Earth Systems Modeling predictions. Downscaling is a valuable
    technique capable of achieving these scale requirements at a minimal
    computational cost. A feasible approach is to use experimentally
    observed regional climate data over a smaller time scale to learn
    statistical relationships that link large-scale simulation results
    with fina grane regional observations. The underlying statistical
    analysis framework includes the following major components:
    Empirical orthogonal function (EOF) analysis, along with multiple
    linear regression (MLR) in a trend analysis, and canonical
    correlation analysis (CCA) in an anomaly analysis. The current
    framework was written in sequential Fortran and has a problem of
    scaling to large data sets. We have been prototyping a modular
    library using parallel Matlab to address the scaling issue. The
    individual components (EOF, CCA, MLR, filters, etc) are routinely
    used in climate community. Hence, this library will bring not only a
    parallel framework for statistical downscaling but also a parallel
    library of routinely used data analysis components to the climate
    community. So far, the key components have been prototyped using
    parallel Matlab.

-   *Validation of climate modeling and simulations* (Dr. John Drake):
    The spherical harmonic transform is a critical computational kernel
    of the dynamics portion of spectral atmospheric weather and climate
    codes. A collection of MATLAB classes for computing spherical
    harmonic transforms have been develop by Drake et al and used to
    solve simple partial differential equations on the sphere. The
    spectral synthesis and analysis using fast Fourier transforms and
    Legendre transforms with the associated Legendre functions are used
    underneath. John Drake approached us with the problem of scaling
    this Matlab library. We have analyzed this library and identified
    the parts in which the underlying parallelism can be exploited: in
    the matrix multiplies of the transform itself and in the data
    decomposition of the gauss_grid and fields that derive from it. A
    few other components that could take advantage of vectorization have
    been identified. We produced a report on the strategy for
    parallelizing this Matlab library.

-   *Fusion simulation data analysis* (Drs. George Ostrouchov and Sean
    Ahern): Sean and George have approached us with the problem they
    have with the analysis of the fusion simulation output data produced
    by Dr. Zhihong Lin (UC Irvine). Their underlying coding framework is
    R and we analyzed how parallel R could be used for this purpose. The
    current parallel R framework for automatic parallelization of R
    scripts does not handle situations where the files are open in one
    for() loop but then these files pointers are used in reading the
    data in another for() loop. The current parallel R version could be
    applied to this code only if some re-writing of the code is
    permissible. This example also drives our parallel R develop work on
    handling file I/O in automatic script parallelization -- a problem
    that has not been addressed by parallel compiler technology before.

-   *Combustion simulation data analysis* (Dr. Jackie Chen and Mr. David
    Lignell): Jackie Chen has he in-house software called S3D that she
    used both for the simulation and analysis of her simulation output
    data. This is a Fortran 90 based software. She would like to improve
    the efficiency and efficacy of her data analysis capabilities by
    bringing her custom analysis codes in S3D to the parallel Matlab
    environment. At this stage we identified a small project and
    received a subset of her S3D library to explore a possibility of
    bringing this library into the parallel Matlab environment. If we
    demonstrate the proof-of-principle to transparently and efficiently
    to do it then a larger set of routines will be explored. I will have
    a summer student from Vanderbilt University (Andrew Jallouk) who
    will work on this problem with Jackie's PhD student, David.

## Plans for Next Quarter

We will benchmark statistical downscaling library, will modularize it
further so that individual components could be used as individual
function calls rather than as part of a monolithic system. We will also
explore the feasibility and our strategy to parallelizing and
integrating Jackie Chen's Fortran 90 S3D library with parallel Matlab.
We will start the prototype of parallelizing John Drake's Matlab library
based on the strategy identified in this quarter.

### [Task 2.2: Feature Selection in Scientific Applications (LLNL)]{.underline}

# *Contact:* Chandrika Kamath, LLNL

## Accomplishments

The work on Poincare plot analysis is on hold pending the hire of a
software developer to replace Cyrus Harrison, who left in mid-January.

We (my post-doc Nicole Love and I) continued our work on the problem of
blob-tracking in NSTX. We put together a short note summarizing our
analysis so far. This was used for discussion in a conference call with
Stewart Zweben, Daren Stotler, and Ricky Maqueda from PPPL. They are
impressed with our work. They suggested we remove the ambient intensity
from the images to accurately identify the blobs. We have been
investigating several ways of doing this. A particularly challenging
issue has been the scaling of the images so that the visual images
reflect the data.

I met with Jackie Chen to understand her requirements for the analysis
of data from combustion simulations.

I presented our work at the SIAM CSE meeting during the special session
organized by Arie Shoshani on the center\'s technology. I also presented
our work at an invited talk by the Bay Area SIG on Data Mining in March.

## Plans for Next Quarter

### For the NSTX data, I plan to schedule another conference call with PPPL to discuss the latest results. The next step is to characterize and track the blobs over time. In addition, we plan to get started with the data from combustion simulations and also pursue analysis opportunities identified recently in other application areas.

### [Task 2.3: High-dimensional indexing techniques (LBNL)]{.underline}

# Contact: John Wu, LBNL

Contributors: Kurt Stockinger

## Accomplishments

-   Added autoconf facilities to FastBit. This is the first step toward
    releasing the software. It makes the software easier to configure
    and build by end users.

-   Rewrote the multi-level bitmap index functions to following the new
    insight gain from recent analyses. The new implementation can be up
    to five times faster than the fastest indexing functions existed in
    the FastBit software before. We are in the process of finalizing the
    implementation and gathering performance statistics.

-   Working with JGI's QA group to use FastBit in their analyses of DNA
    sequence quality data. This work requires a number of new functions
    to deal with string-valued variables. This raw also comes in
    relative small chunks that require new functions to consolidate
    them.

-   Starting to implement feature identification functions for data on
    toroidal meshes. This extends our previous work on data from regular
    meshes. The target application is the analysis of fusion simulation
    data.

## Plans for next quarter

-   Add documentation and tutorial to FastBit.

-   Continue the preparation to release FastBit.

-   Continuing the work on feature identification in fusion simulation
    data.

-   Revisit the binning strategies in FastBit. A new data structure has
    been identified, we plan to implement the data structure and analyze
    the performance characteristics.

-   Start work on using FastBit for querying AMR data.

-   Start work on using FastBit for querying AMR data.

    **[3. Scientific Process Automation]{.underline}**

The Internet is becoming the preferred method for disseminating
scientific data from a variety of disciplines. This has resulted in
information overload on the part of the scientists, who are unable to
query all of the relevant sources, even if they knew where to find them,
what they contained, how to interact with them, and how to interpret the
results. Thus instead of benefiting from this information rich
environment, scientists become experts on a small number of sources and
use those sources almost exclusively. Enabling information based
scientific advances, in domains such as functional genomics, requires
fully utilizing all available information. We are developing an
end-to-end solution using leading-edge automatic wrapper generation,
mediated query, and agent technology that will allow scientists to
interact with more information sources than currently possible.
Furthermore, by taking a workflow-based approach to this problem, we
allow them to easily adjust the dataflow between the various sources to
address their specific research needs.

**[Task 3.1: Mediated Query Environment (SDSC / UC Davis)]{.underline}**

***Participants***: Ilkay Altintas, Bertram Ludaescher, Daniel Crawl.
*Contributing Staff and Students:* Norbert Podhorszki (UCD), Timothy
McPhillips (UCD), Shawn Bowers (UCD)

***Accomplishments:***

-   Completed the recruitment process of a new staff at SDSC.
    (March, 2007) Daniel Crawl, PhD was hired out of twelve candidates.

-   Worked with the Utah team on a consolidated data model for
    provenance framework in order to save workflow evolution,
    meta-information and execution provenance information in relational
    database. The evolving report on this can be found at:
    [[http://www.vistrails.org/index.php/SDM_Provenance]{.underline}](http://www.vistrails.org/index.php/SDM_Provenance)

-   Presented Kepler to Fermi Lab. (Feb, 2007) The LQCD team decided to
    use Kepler as a workflow environment after an evaluation of similar
    workflow systems.

-   Attended the Kepler tutorial at the NCCS users meeting in ORNL
    ([[http://nccs.gov/news/workshops/fy07usersmeeting/index.html]{.underline}](http://nccs.gov/news/workshops/fy07usersmeeting/index.html)
    ) as a presenter.

-   Submitted a tutorial proposal to SC2007 in collaboration with the
    rest of the SPA team.

-   Implemented a NEW RDBMS actor for inserting new entries in an
    existing database.

-   Continued work on CPES automation (Norbert P):

```{=html}
<!-- -->
```
-   We have built a new workflow for migrating an archive from one mass
    storage to another. This workflow enhances earlier work with
    concurrent transfers over the network. It was successfully used to
    migrate a 10TB INCITE archive from NERSC to ORNL within 11 days. The
    workflow used temporary disks at both sites to stage the files from
    and to the HPSS systems. It took care about not to stage too many
    files at once but enough to exploit the pipeline-parallel and
    concurrent processing capabilities of the Kepler environment. The
    execution has experienced (and survived) 63 operation failures (from
    the total of 726 operations), which were performed automatically
    again. The cumulative figure of the file operations (get=stage from
    HPSS to disk, transfer=transfer from NERSC to ORNL, put=stage to
    HPSS from disk) during the 11 days of execution shows how the
    bounded pipeline-parallel processing allowed the efficient transfer
    of files (Figure 1, below)

-   We have developed a workflow for coupling XGC-0 and M3D (see Figure
    2 below). The processing loop within the workflow transfers data
    regularly from XGC (from jaguar to ewok). On the current data, as a
    first step, the new equilibirium is computed which is immediately
    fed back to XGC and also a linear stability test is started. If the
    linear stability test fails, a job is prepared and submitted (on
    ewok) to perform nonlinear parallel M3D-MPP computation. Currently
    missing operations are the stopping of XGC and the submission of a
    new XGC job with inputs coming from the nonlinear computation.

```{=html}
<!-- -->
```
-   Continued work with Tim McPhillips et al. on Collection-Oriented
    Modeling and Design (COMAD) approach; prototyped workflows for
    phylogenetic applications

-   New Workflows and Scientific Communities and Outreach:

    -   **Adoption of various actors,** in particular command line, web
        service, data transformation, and browser actors in a variety of
        workflows from different communities, including in oceanography,
        phylogeny, seismology, computational chemistry, geosciences,
        ecology, digital preservation, text mining, meteorology and
        biology workflows.

        ***Publications and Presentations***

```{=html}
<!-- -->
```
-   Accepted papers and abstracts:

    -   Composing Different Models of Computation in Kepler and Ptolemy
        II, A. Goderis, C. Brooks, I. Altintas, C. Goble, E. A. Lee,
        *2nd International Workshop on Workflow systems in e-Science* in
        conjunction with ICCS 2007.

    -   Workflow automation for processing plasma fusion simulation
        data, N. Podhorszki, B. Ludaescher, S. Klasky, *2nd Workshop on
        Workflows in Support of Large-Scale Science* (WORKS'07).

-   Presentations:

    -   Presented the provenance framework in the Ptolemy/Kepler
        Miniconference, Feb 2007. "Provenance Framework in Kepler\",
        Norbert Podhorszki (University of California, Davis), Ilkay
        Altintas (San Diego Supercomputer Center):
        [[http://ptolemy.eecs.berkeley.edu/conferences/07/index.htm]{.underline}](http://ptolemy.eecs.berkeley.edu/conferences/07/index.htm)

    -   Presented the flexible scientific workflow approach in the
        Ptolemy/Kepler Miniconference, Feb 2007. \"Flexible Scientific
        Workflow with Dynamic Embedding\", A.H.H. Ngu, N. Haasch (Texas
        State University-San Marcos), T. Critchlow (Lawrence Livermore
        National Laboratory), S. Bowers, T. McPhillips, B. Ludaescher
        (University of California, Davis):
        [[http://ptolemy.eecs.berkeley.edu/conferences/07/index.htm]{.underline}](http://ptolemy.eecs.berkeley.edu/conferences/07/index.htm)

    -   Altintas presented Kepler to Fermi Lab (Feb, 2007) and to a
        large group in Monash University (March, 2007).

>  Ludaescher gave talks on scientific workflows for e-Science on March
> 26 at the School of Informatics in Edinburgh, and (heavily focused on
> GC needs) at the UCD Cyberinfrastructure workshop on April 5^th^.

***Plans for coming quarter***

-   Hiring of potential new students at UCD will happen for Fall 2007

-   Finalize the SPA build and build a new installer.

-   Attend design meetings as part of the dashboard team.

-   Attend design meetings and actively contribute to the design of the
    planned Kepler provenance feature extensions.

-   Define job regression workflow for Gygi (Bertram) Aug 07

-   Define advanced workflows for peta-scale applications for fusion
    (Ilkay) Aug 07

![](media/image2.png){width="5.990972222222222in"
height="2.1104166666666666in"}

**Figure 1: File migration workflow: measurements**

![](media/image3.png){width="5.996527777777778in"
height="4.279861111111111in"}

**Figure 2: Workflow for coupling XGC-0 and M3D**

**[Task 3.2: Outreach and User Interaction (LLNL)]{.underline}**

***Participants:*** Terence Critchlow

The primary implementation of this task has been transferred to PNNL
since there are no longer any SPA members working at LLNL.

**[Task 3.3: Incorporation of Agent / Workflow Technology
(NCState)]{.underline}**

***Participants***: Mladen Vouk, Jeff Ligon, Pierre Mouallem, Meiyappan
Nagappan

***Progress Report***

In accord with our plan, during the last 3 months of this reporting
period, we were working on activities that support two major goals:

(i) Semi-automatic generation of data-transforms that translate into web
    and other services and pro-active support functions for scientific
    workflows, and

(ii) Identification, assessment and implementation of general as well as
     domain-specific workflow extensions to Kepler/Ptolemy and similar
     frameworks that enable support of complex scientific workflow
     description and execution.

These activities included

a)  Dissemination of SDM related information and research -- this
    includes tutorials, work on papers, theses and reports, as well as
    documentation of existing SPA codes

> The most prominent element is the hands-on Kepler tutorial given at
> the NCCS meeting at ORNL on 29th March 2007.

b)  Work on provenance solutions

    -   There are 4 types of provenance that we have identified, namely,
        \'Data\', Process, Workflow, and System. We are in the middle of
        finalizing what each one means (and are assessing implementation
        pilots), where they should be stored and how they should be
        displayed. Information is posted on the Vistrails wiki.

    -   [[http://www.vistrails.org/index.php/SDM_Provenance]{.underline}](http://www.vistrails.org/index.php/SDM_Provenance)

c)  Further work on case-study workflows (TSI Blondin, Fusion
    workflows - Klasky).

d)  Work on dashboard prototypes

    -   High level Architecture for the \"Dashboard" is given below. It
        is the results of a continuous collaboration within SDM/SPA and
        with the ORNL End to End Solutions group. We are finalizing on
        the data flow among these components.

![](media/image4.png){width="6.26875in" height="4.701388888888889in"}

**Task 4: SCIRun/Kepler/VisTrails integration (Utah)**

***Participants:*** Steven Parker, Claudio Silva, Ayla Khan, Steven
Callahan

Our efforts have been centered around two main areas:

We continue to work on the provenance specification and implementation.
We now have a full draft of this specification that includes schemas for
workflow meta-data, workflow execution, and workflow evolution. Further
detail can be found on this wiki page:
[[http://www.vistrails.org/index.php/SDM_Provenance]{.underline}](http://www.vistrails.org/index.php/SDM_Provenance)

We explored the possibility of using SVG and JavaScript to create a
simple browser-based graphics tool, which was determined to be
impractical due to incomplete browser support for the SVG specification.
We are currently working on implementing a simple graphics tool using
Adobe\'s Flex SDK to build a flash-based interface for graphs and simple
2D visualization. We believe that this will be a powerful complement to
the workflow technologies and will allow scientists to monitor
simulations from any flash-enabled browser (98.3% of all users:
http://www.adobe.com/products/player_census/flashplayer/).

Accomplishments:

-   Created provenance schema document and prototype

-   Investigated SVG and Flash and chose Flash as the primary tool for
    dashboard visualization.

Publications/presentations:

-   We have submitted a paper on our multi-layer provenance architecture
    to a special issue of Concurrency\-\--Practice and Experience on the
    1st Provenance Challenge.

Plans for coming quarter:

-   Implement flash viewer for displaying graphs through dashboard

-   Finalize provenance for workflow schemas

-   Continue progress on CCA/Kepler bridge

    **[Task 3.5:Provenance Tracking and Outreach (PNNL)]{.underline}**

***Participants***: George Chin

***Accomplishments***

-   Conducted comparative study of workflow provenance approaches and
    techniques that had been presented by various academic and research
    institutions participating in the Provenance Challenge Workshop
    (associated with the IEEE International Symposium on High
    Performance Distributed Computing). We evaluated and compared the
    provenance concepts, schemas, and features available in a number of
    research provenance systems.

-   Working with San Diego Supercomputing (SDSC), University of Utah,
    North Carolina State University, and UC-Davis to design a provenance
    data model/schema that will support both Kepler and SciRun/Vistrials
    workflow management systems.

-   Continuing to work with computational and subsurface transport
    scientists on the SciDAC-2 Groundwater Reactive Transport Modeling
    and Simulation project on Hybrid Numerical Methods for Multiscale
    Simulations of Subsurface Biogeochemical Processes. We are
    collaborating to develop a scientific workflow based on continuum
    scale simulations targeted at quasi-2D experimental flow cell. We
    have implemented portions of the workflow in Kepler and are
    currently focused on developing embedded user interface capabilities
    to support more dynamic interactions and control of the transport
    model components. We will also explore implementing the workflow in
    SciRun to take advantage of its support for closely-coupled
    components.