![](media/image1.png){width="1.375in" height="0.74375in"}SDM Center
Report

April - September 2008

http://sdmcenter.lbl.gov

**[Highlights in this period]{.underline}**

-   FastBit indexing technology receives R&D 100 award

```{=html}
<!-- -->
```
-   FastBit has been proven theoretically to be optimal, it performs 10
    times faster than any known indexing method in practice based on it
    use of our a patented compression method, it can support
    multi-variable data search, and it can support large cardinality
    scientific data. These unique characteristics made it extremely
    useful in a variety of scientific applications. Our implementation
    was packaged this year and released under an open-source license,
    and has attracted a lot of interest in multiple scientific
    application, as well as new areas, such as network traffic data
    analysis, query-based visualization, and business application, such
    as Yahoo. The poster presented at the R&D 100 award event is posted
    at: http://sdmcenter.lbl.gov/posters/FastBit.Poster.RD100.pdf

```{=html}
<!-- -->
```
-   Parallel statistical routines made available through the dashboard

```{=html}
<!-- -->
```
-   An experimental set of \~15 analysis routines were added to the
    Dashboard web application. The initial set of routines was selected
    based on the most frequently used analysis functions that are
    currently used by climate and fusions communities. The set has been
    derived from the analysis of climate and fusion data analysis
    packages, such as CDAT, GKV, and vugyro.

```{=html}
<!-- -->
```
-   Parallel NetCDF for 64 bit machines developed

```{=html}
<!-- -->
```
-   A new netCDF file format to support larger than 4 GB array size was
    developed and is being tested extensively. The new format uses
    64-bit integers to accommodate array attributes. Our initial work
    includes changes of APIs that define the array dimensionalities and
    global attributes, and we have been in constant contact with the
    Unidata team that supports the serial netCDF code base to ensure
    that they are comfortable with this new format, called the "CDF-5"
    format.

```{=html}
<!-- -->
```
-   Automatic provenance capture have been developed

```{=html}
<!-- -->
```
-   Run-time provenance capture scripts and automatic data-base feed
    have been developed. This includes system provenances on the setup
    of simulation programs, the workflow provenance, and data provenance
    which captures the history of each data file. The data provenance is
    now used to find files of interest and move them to the user machine
    directly from the dashboard.

```{=html}
<!-- -->
```
-   FastBit indexes applied to text searches provides 50-fold speedup

```{=html}
<!-- -->
```
-   The work on using FastBit for text searches has been accepted to a
    special issue of "Annals of Information Systems" (Springer). This
    work describes a way of using FastBit to index keywords from text
    documents. In a comparison with a well-regarded MySQL text search
    functions, our indexing system was found to answer queries about 50
    times faster.

```{=html}
<!-- -->
```
-   High marks for the SDM center technologies during the CPES project
    review.

```{=html}
<!-- -->
```
-   The The SDM center participated in the review of the Center for
    Plasma Edge Simulation (CPES), since it is providing many of the
    technologies used by CPES. The technologies provided by the center
    include the Kepler workflow system, the dashboard, provenance
    tracking and recording, and SRM-based data movement. The
    workflow-dashboard system has been used to perform simulation
    monitoring as well as complex code-coupling tasks. Scott Klasky and
    Arie Shoshani presented the computer science component of CPES. The
    review was highly successful where all the reviewers gave the CPES
    project the highest possible grade of "excellent".

    ***[Publications this period]{.underline}***

Papers abstracts and pointers to the full papers are available at the
sdmcenter.lbl.gov web site, under "publications".

\[BMR+08\] S Bowers, T McPhillips, S Riddle, M Anand, B Ludäscher.
"Kepler/pPOD: Scientific Workflow and Provenance Support for Assembling
the Tree of Life". Intl. Provenance and Annotation Workshop (IPAW) 2008,

\[CA08\] Daniel Crawl, Ilkay Altintas, "A Provenance-Based Fault
Tolerance Mechanism for Scientific Workflows," International Provenance
and Annotation Workshop (IPAW), June 17, 2008.

\[CKC+08\] C S Chang, S Klasky, J Cummings, et al. Toward a
first-principles integrated simulation of tokamak edge plasmas, 2008
International Symposium on. Collaborative Technologies and Systems.
SciDAC 2008,

http://www.iop.org/EJ/article/1742-6596/125/1/012042/jpconf8_125_012042.pdf?request-id=c82c7e3e-97a7-46cf-89d8-ee8c1cbc5b54

\[CFS+08\] S. P. Callahan, J. Freire, C. E. Scheidegger, C. Silva, and
Huy T. Vo. \"Towards Provenance-Enabling ParaView\", Second
International Provenance and Annotation Workshop (IPAW 2008). URL:
http://www.sci.utah.edu/\~csilva/papers/ipaw2008a.pdf

\[CSC+08\] Chase, J., Schuchardt, K., Chin Jr., G., Daily, J., and
Schiebe, "Iterative Workflows in Numerical Simulations", T. Proceedings
of the IEEE 2008 Second International Workshop on Scientific Workflows
(SWF 2008). URL: http://subsurface.pnl.gov/docs/SWF08_Paper.pdf

\[EKA+08\] T. Ellkvist, D. Koop, E. W. Anderson, J. Freire, and C.
Silva. \"Using Provenance to Support Real-Time Collaborative Design of
Workflows\", Second International Provenance and Annotation Workshop
(IPAW 2008).

URL: http://www.sci.utah.edu/ipaw2008/papers/Paper%2019/submitted.pdf

\[KVP+08\] Scott Klasky, Mladen Vouk, Manish Parashar, Ayla Kahn,
Norbert Podhorszki, Roselyne Barreto, Deborah Silver, Steve Parker,
Collaborative Visualization Spaces for Petascale Simulations, to appear
in the 2008 International Symposium on Collaborative Technologies and
Systems (CTS 2008).

\[LKA+08\] L. Lins, D. Koop, E. W. Anderson, S. P. Callahan, E. Santos,
C. E. Scheidegger, J. Freire, and C. Silva, Examining Statistics of
Workflow Evolution Provenance: A First Study, Statistical and Scientific
Database Management (SSDBM) 2008.
http://www.cs.utah.edu/\~juliana/pub/vistrails-ssdbm2008.pdf

\[Naga08\] Mei Nagappan, An Overview of Provenance Collection in
Workflow Systems and a Privacy Policy Model to Share the Information,
April 11-22, High Performance and Applications Conference, Oak Ridge,
TN, April 11-12, 2008.
http://www.nccs.gov/user-support/training-education/workshop-archives/high-performance-computing-and-applications-conference-2008/

\[NV08\] Meiyappan Nagappan, and Mladen Vouk, A Privacy Policy Model for
Sharing of Provenance Information in a Query Based System, accepted as a
short/demo paper for publication in the proceedings of the International
Provenance and Annotation Workshop (IPAW) 2008.

\[SLA+08\] E. Santos, L. Lins, J. P. Ahrens, J. Freire, and C. Silva.
\"A First Study on Clustering Collections of Workflow Graphs\", Second
International Provenance and Annotation Workshop (IPAW 2008). URL:
http://www.sci.utah.edu/ipaw2008/papers/Paper%209/camera_ready.pdf

\[WSS08\] Kesheng Wu, Kurt Stockinger and Arie Shoshani.  Breaking Curse
of Cardinality on Bitmap Indexes. 20th International Conference on
Scientific and Statistical Database Management, (SSDBM) 2008.

\[YOC+08\] W. Yu, H.S. Oral, R.S. Canon, J.S. Vetter, R. Sankaran.
Empirical Analysis of a Large-Scale Hierarchical Storage System, The
14th European Conference on Parallel and Distributed Computing (Euro-Par
2008). 2008. URL: http://ft.ornl.gov/pubs-archive/2008-EuroPar-DDN.pdf

\[YV08\] W. Yu, J.S. Vetter. ParColl: Partitioned Collective IO on the
Cray XT. The 37th International Conference on Parallel Processing. 2008.
URL: http://ft.ornl.gov/pubs-archive/2008-IPDPS-CrayXT-IO.pdf

***[Talks and tutorials in this period]{.underline}***

-   SciDAC 2008 ADIOS tutorial.

-   SciDAC 2008 Kepler tutorial.

-   Invited talk at CCSM meeting.

-   Ilkay Altintas, "Lifecycle of Scientific Workflows and Their
    Provenance: A Usage Perspective", Invited Talk. In IEEE 2008 Second
    International Workshop on Scientific Workflows (SWF 2008) in
    conjunction with IEEE International Conference on Services
    Computing, Hawaii, USA, July 8, 2008

-   B. Ludaescher, Panelist, Provenance across workflows and databases,
    IPAW, 2008

-   C. Kamath, invited to give a talk on scientific data mining at the
    Workshop on Modern Massive Data Sets, held at Stanford, June 25-28.

***[The all-hands-meeting]{.underline}***

1\) The SDM center all-hands meeting took place on Oct 7-9, 2008. It
included reports from all institutions participating in the center, with
emphasis on three aspects:

\- Success stories from the past 7 years in the SDM center  (For this
purpose, \"success\" is a technology being/was used or about to be used
in production or otherwise by application people)\
- Vision for the future of the SDM center 

(This should include vision in your area of expertise, as well as
inter-operation between multiple technologies in the center)\
- Long term approach for the survival of SDM center technology 

(Beyond Scidac and longer).

The talks presented are posted at:
http://sdm.lbl.gov/sdmcenter/pub/2008.10.SDM.AHM.htm

Details of progress during this period are reported next.

**[Introduction]{.underline}**

Managing scientific data has been identified as one of the most
important emerging needs by the scientific community because of the
sheer volume and increasing complexity of data being collected.
Effectively generating, managing, and analyzing this information
requires a comprehensive, end-to-end approach to data management that
encompasses all of the stages from the initial data acquisition to the
final analysis of the data. Fortunately, the data management problems
encountered by most scientific domains are common enough to be addressed
through shared technology solutions. Based on the community input, we
have identified three significant requirements. First, more efficient
access to storage systems is needed. In particular, parallel file system
improvements are needed to write and read large volumes of data without
slowing a simulation, analysis, or visualization engine. These processes
are complicated by the fact that scientific data are structured
differently for specific application domains, and are stored in
specialized file formats. Second, scientists require technologies to
facilitate better understanding of their data, in particular the ability
to effectively perform complex data analysis and searches over large
data sets. Specialized feature discovery and statistical analysis
techniques are needed before the data can be understood or visualized.
To facilitate efficient access it is necessary to keep track of the
location of the datasets, effectively manage storage resources, and
efficiently select subsets of the data. Finally, generating the data,
collecting and storing the results, data post-processing, and analysis
of results is a tedious, fragmented process. Tools for automation of
this process in a robust, tractable, and recoverable fashion are
required to enhance scientific exploration.

Our approach is to employ an evolutionary development and deployment
process: from research through prototypes to deployment and
infrastructure. Accordingly, we have organized our activities in three
layers that abstract the end-to-end data flow described above. We
labeled the layers (from bottom to top):

-   Storage Efficient Access (SEA)

-   Data Mining and Analysis (DMA)

-   Scientific Process Automation (SPA)

The SEA layer is immediately on top of hardware, operating systems, file
systems, and mass storage systems, and provides parallel data access
technology, and transparent access to archival storage. The DMA layer,
which builds on the functionality of the SEA layer, consists of
indexing, feature identification, and parallel statistical analysis
technology. The SPA layer, which is on top of the DMA layer, provides
the ability to compose scientific workflows from the components in the
DMA layer as well as application specific modules.

# This report consists of the following sections, organized according to the three layers, as follows:

-   Storage Efficient Access (SEA) techniques

    -   Task 1.1: Low-Level Parallel I/O Infrastructure

    -   Task 1.2: Collaborative File Caching

    -   Task 1.3: File System Benchmarking and Application I/O Behavior

    -   Task 1.4: Application Interfaces to I/O

    -   Task 1.5: Disk Resident Extendible Array Libraries

    -   Task 1.6: Active Storage in the Parallel Filesystem

```{=html}
<!-- -->
```
-   Data Mining and Analysis (DMA) components

    -   Task 2.1 High-performance statistical computing for scientific
        applications

    -   Task 2.2: Feature Selection in Scientific Applications

    -   Task 2.3: High-dimensional indexing techniques

-   Scientific Process Automation (SPA) tools

    -   Task 3.1: Dashboard Development

    -   Task 3.2: Provenance Tracking

    -   Task 3.3: Outreach

The reports by each of the three areas, SEA, DMA, and SPA, follow.

**[1. Storage Efficient Access (SEA)]{.underline}**

***Participants:*** Rob Ross, Rajeev Thakur, Sam Lang, and Rob Latham
(ANL), Alok Choudhary, Wei-keng Liao, Arifa Nisar (NWU), Arie Shoshani
and Ekow Otoo (LBNL), Jeffrey Vetter and Weikuan Yu (ORNL), Jarek
Nieplocha and Juan Piernas Canovas (PNL)

The goal of this project is to provide significant improvements in the
parallel I/O subsystems used on today\'s machines while ensuring that
the capabilities available now will continue to be available as systems
increase in scale and technologies improve. A three-fold approach of
immediate payoff improvements, medium-term infrastructure development,
and targeted longer-term R&D is employed.

Two of our keystone components are the PVFS parallel file system and the
ROMIO MPI-IO implementation. These tools together address the
scalability requirements of upcoming parallel machines and are designed
to leverage the technology improvements in areas such as
high-performance networking. These are both widely deployed and freely
available, making them ideal tools for use in today's systems. Our work
in application I/O interfaces, embodied by our Parallel NetCDF
interface, also promises to provide short-term benefits to a number of
climate and fusion applications.

In addition to significant effort on PVFS, we recognize the importance
of other file systems in the HPC community. For this reason our efforts
include improvements to the Lustre file system, and we routinely discuss
both Lustre and GPFS during tutorials. Our efforts in performance
analysis and tuning for parallel file systems, as well as our work on
MPI-IO, routinely involve these file systems.

At the same time we continue to push for support of common,
high-performance interfaces to additional storage technologies. Our work
combining the Storage Resource Manager (SRM) with MPI-IO, in conjunction
with PVFS, will provide a single solution for the problems of
high-performance parallel storage access, integration of file system
with tertiary storage, and remote data access. Our work in Active
Storage will provide common infrastructure for moving computation closer
to storage devices, an important step in tackling the challenges of
petascale datasets.

**[Task 1.1: Low-Level Parallel I/O Infrastructure (Ross, Thakur, Lang,
Latham, Vetter, Yu)]{.underline}**

The objective of this work is to improve the state of parallel I/O
support for high-end computing (HEC) and enable the use of high
performance parallel I/O systems by application scientists. The Parallel
Virtual File System (PVFS) and ROMIO MPI-IO implementations, with
development lead by ANL, are in wide use and provide key parallel I/O
functionality. This work builds on these two components by enhancing
them in order to ensure these capabilities continue to be available as
systems continue to scale. The OPAL library, being developed at ORNL, is
a more recent development and builds in the original ROMIO library to
provide enhancements for Lustre and Cray systems.

***Progress Report***

We continued our efforts in developing the OPAL library, a feature-rich
and high-performance MPI-IO library for Lustre. In the past quarter, we
have evaluated the direct I/O support in Lustre ADIO driver for both
Cray XT and Linux clusters. Using a Linux cluster, we have observed
performance improvement with large streaming of I/O requests over
Lustre, avoiding data copies in the kernel buffer cache. We have also
observed significant benefits for Cray XT systems under CNL (Compute
Node Linux) operating system.

We investigated the performance of different network technologies for
data transfer on WAN. Using the OC192 SONET connections from
UltraScience Net, we demonstrated that a network-level bandwidth of
7.4Gbps and an MPI-level bandwidth of 758MB/sec are achievable across
two InfiniBand clusters that are 8600 miles apart. Part of results was
presented in the OpenFabrics Conference in Sonoma, California. We
currently continue to explore upper-level protocols for long distance
I/O, such as GridFTP and NFS. Both the legacy TCP/IP protocol and RDMA
on top of 10Gigabit Ethernet and InfiniBand are under investigation.

With respect to PVFS, we have begun performance testing on the 500TFlop
Blue Gene/P at the Argonne Leadership Computing Facility (ALCF).
Simultaneously we have developed and tested a set of performance
improvements for small files in the context of PVFS. This work was
detailed in a paper submitted to OSDI. Our goal is to exceed our
performance goal of 60 Gbyte/sec for block I/O workloads in the third
quarter of this year. We are hosting a collection of students this
summer, three of which are working on projects related to PVFS,
including an investigation of the use of Berkeley DB transactions in
PVFS, an evaluation of the Sun ZFS underlying storage components as a
possible replacement for local file system storage, and a study of the
FUSE system for hooking user-space file systems into operating systems
such as Linux and OSX.

We have been incorporating patches from collaborators into the ROMIO
library in preparation for the next MPICH2 release, including file
domain and strided I/O optimizations from NWU. We have also developed
additional test cases to exercise new code paths in the IBM Blue Gene/P
MPI-IO implementation resulting from their work to fix limitations due
to a 32-bit pointer on the system.

We presented material on parallel I/O optimizations at ScicomP, the IBM
Scientific Systems User Group in May, and at the INCITE workshop at ANL
in May. We have also submitted a full-day tutorial on the use of
parallel I/O systems for SC08 in conjunction with colleagues at Panasas.

***Plans for next quarter***

In the coming months, we will continue to develop more features to
improve the performance of parallel I/O on the supercomputers that are
deployed with Lustre, such as Cray XT and large-scale clusters. We will
also introduce new features into the Lustre file system for better
scalability and performance assurance under heavy load. Furthermore, we
will investigate the performance of parallel I/O between geographically
distributed computing clusters that are connected with high performance
connections.

Once performance of PVFS on the ALCF system has reached acceptable
levels, we plan to write a paper detailing performance and submit to
FAST in September. We will be participating in the CScADS workshops in
July, providing expertise in parallel I/O during one of the weeks. Rob
Ross will be presenting a talk on parallel I/O in data analysis at the
upcoming SciDAC all-hands meeting.

***Publications***

-   W. Yu, H.S. Oral, R.S. Canon, J.S. Vetter, R. Sankaran. Empirical
    Analysis of a Large-Scale Hierarchical Storage System, The 14th
    European Conference on Parallel and Distributed Computing (Euro-Par
    2008). 2008. URL:
    http://ft.ornl.gov/pubs-archive/2008-EuroPar-DDN.pdf

-   W. Yu, J.S. Vetter. ParColl: Partitioned Collective IO on the Cray
    XT. The 37th International Conference on Parallel Processing. 2008.
    URL: http://ft.ornl.gov/pubs-archive/2008-IPDPS-CrayXT-IO.pdf

    **[Task 1.2: Collaborative File Caching (Choudhary, Liao,
    Nisar)]{.underline}**

The objective of this work is to develop a layer of user-level
client-side file caching system for MPI-IO library. Moving the caching
system closer to user applications allows high-level I/O patterns to be
captured and improve data caching efficiency. We adopt two designs: one
uses an I/O thread in each application process and the other spawns a
separate group of processes as I/O delegates. In both designs, I/O
threads or delegates collaborate with each other to perform a coherent
caching.

***Progress Report***

We evaluated the design of using I/O delegates on two parallel machines:
Tungsten running Lustre and Mercury running GPFS, both at NCSA.
Performance results were collected by running the BTIO benchmark and two
application I/O kernels: FLASH and S3D. When additional 3% of compute
nodes are allocated as I/O delegates, we observed the I/O bandwidth
improvements ranging from 40% to 250%. When the number of I/O delegates
is 10% of the application nodes, we observed up to 500% improvement.
These results were summarized in a technical paper to appear in the
Super Computing conference 2008. We also added a new implementation in
this design that uses POSIX asynchronous I/O routines to flush dirty
cache data at the delegates. The flushing is only activated while the
delegates are idle so that it can increase the overlap between I/O and
other caching operations. The preliminary results showed 10 to 20 %
additional improvement.

***Plans for next quarter***

The current caching policy adopted in our design is to let the first
requesting delegate cache the requested data locally. We plan to add a
policy based on file system stripe size. We will dissect the collective
I/O implementation in ROMIO and develop a set of new collective I/O
subroutines that incorporate the caching delegation. This will replace
the communication within application processes with the communication
between application processes and I/O delegates. One objective of this
design is to eliminate entirely the allocation of collective buffers in
the application processes. Instead, the collective buffering will only
perform in the I/O delegates whose memory spaces available for caching
are much larger than the application nodes. With larger collective
buffer size, the data redistribution phase in the collective I/O
operations can be carried out with less number of messages.

**[Task 1.3: File System Benchmarking and Application I/O Behavior
(Choudhary, Liao, Nisar)]{.underline}**

The objective of this work is to study file system characteristics that
have significant impacts to the parallel I/O operations and evaluate the
relative performance of the file systems available to important SciDAC
applications on DoE compute platforms. The performance, functionality,
and scalability of MPI-IO, parallel netCDF, and HDF5 are critical for
many applications. Two groups in the SDM center, Northwestern University
and ORNL, are performing complementary efforts in this area.

***[Progress Report]{.underline}***

We continue the collaboration work on the S3D I/O kernel with Jacqueline
Chen at Sandia National Laboratories, Ramanan Sankaran and Scott Klasky
at ORNL. We have developed Fortran subroutines using MPI-IO, parallel
netCDF, and HDF5 and tested on JaguarCNL at ORNL, the interim Cray XT3
machine when Jaguar was being upgraded to XT4. Since the completion of
Jaguar's upgrade in May 12, we migrated and tested the new I/O kernel on
the new machine. In addition, we built a separated ROMIO library with a
few I/O optimizations incorporated for the Lustre file system running on
Jaguar. The new I/O kernel was tested using up to 30,720 process cores
(the total number of codes on Jaguar is 31,328). We delivered the new
S3D I/O kernel to Jacqueline Chen on May 28. Ramanan Sankaran has
performed initial tests on small numbers of cores and it passed for
checkpoint writes and restart reads using a different number of cores.
Tests with large number of cores are currently under way.

***Plans for next quarter***

We will continue the collaboration and support the new S3D I/O kernel.
We will also continue to study application I/O kernel from the Parallel
Ocean Program. Evaluation will be performed on TeraGrid machines and the
Cray XT on ORNL.

**[Task 1.4: Application Interfaces to I/O (Ross, Thakur, Latham, Liao,
Choudhary, Nisar, Vetter, Yu)]{.underline}**

The objective of this work is to improve the observed I/O throughput for
applications using parallel I/O by enhancements to or replacements for
popular application interfaces to parallel I/O resources. This task was
added in response to a perceived need for improved performance at this
layer, in part due to our previous work with the FLASH I/O benchmark.
Because of their popularity in the scientific community we have focused
on the NetCDF and HDF5 interfaces, and in particular on a parallel
interface to NetCDF files.

***Progress Report***

We started prototyping the new netCDF file format to support larger than
4 GB array size. The new format uses 64-bit integers to accommodate
array attributes. Our initial work includes changes of APIs that define
the array dimensionalities and global attributes, and we have been in
constant contact with the Unidata team that supports the serial netCDF
code base to ensure that they are comfortable with this new format,
called the "CDF-5" format. Tests are being performed using PnetCDF test
programs, and we are also experimenting with header padding and variable
alignment to improve performance on the Blue Gene/P system with GPFS. We
also continue the collaboration work with the Adaptive IO System (ADIOS)
team lead by Scott Klasky at ORNL.

As a part of our efforts to provide a versatile I/O middleware for
scientific applications, we have ported the I/O kernel of the combustion
simulation program, S3D, to the initial ADIOS middleware that is still
under development at Oak Ridge National Laboratory. We have evaluated
the performance to thousands of processes, and have shown comparable
performance to the original S3D I/O implementation.

We have been working with Jim Edwards of UCAR to clarify the use of the
PnetCDF flexible mode interface and fix some bugs found as a result of
this collaboration. We've also been helping Annette Koontz of PNNL
resolve some issues in the process of porting the GCM code to the Blue
Gene/P and other systems.

***Plans for next quarter***

We plan to develop new test programs for the new PnetCDF format based on
the I/O patterns from the ROMIO test programs. We will also continue to
collaborate with ADIOS team on developing PnetCDF interface.

**[Task 1.5: LIBDRXTA: Disk Resident Extendible Array Libraries (Otoo,
Rotem)]{.underline}**

This work focuses on implementing a library for storage management
access of Disk Resident Multidimensional Extendible Arrays for parallel
applications. The library is called pDRXTA. It is the parallel
counterpart of a Disk Resident Multidimensional Extendible Array library
referred to simply as DRXTA.

***Progress Report***

The current ongoing work is on the evaluation and comparison of its
access functions with an alternative scheme of using skip-list access
method for array chunks in HDF5. Implementation of caching for array
chunks with the BerkeleyDB cache pool module is now complete. Likewise,
implementation of skip-list access methods for chunked dense arrays with
caching using the BerkeleyDB cache pool module has been completed.

Finally, a performance comparison of DRXTA storage of dense arrays with
Skip-List index of dense array chunks was made.

***Plans for next quarter***

A recent paper in a Linux magazine claims that the skip-list is a more
concurrency-friendly data-structure for implementation in multi-core and
other shared memory architecture than data structures such as AVL-,
Red-Black --Trees and Arrays. Future work includes activities intended
to validate or refute this claim with respect to skip-list and DRXTA.
Our activities will involve:

-   Testing the applicability of SkipDB (an open-source library), for
    managing array chunks.

-   Concurrency control and multi-threaded implementations of skip-list
    and the DRXTA library using the BerkleyDB Locking Subsystem.

-   Use of the DRXTA storage of dense arrays to manage HDF5 dense array
    chunks and conducting some performance measurements.

-   Continuing work on implementing and testing of extendible array file
    for dense chunked arrays.

-   Implementation of parallel out-core multidimensional extendible
    array functions (pDRXTA) for both dense and sparse arrays.

    **[Task 1.6: Active Storage in the Parallel File System (Nieplocha,
    Canovas)]{.underline}**

    We are developing Active Storage, a promising technology for
    reducing bandwidth requirements between the storage and compute
    elements of current supercomputing systems, and leveraging the
    processing power of the storage nodes used by some modern file
    systems. To achieve both objectives, Active Storage allows certain
    processing tasks to be performed directly on the storage nodes, near
    the data they manage.

    ***Progress Report***

Active Storage is targeted at applications with I/O-intensive stages
that involve fundamentally-independent data sets. It can be used to
process, either on-line or off-line, output files from scientific
simulation runs. Some examples of tasks suitable for Active Storage
include: compression and archival of output files, statistical analysis
of the output data and storing the results in an external database,
indexing the contents of the output files, simple data transformations
such as unit conversion by multiplying by a scalar a set of numbers,
etc.

We published a paper entitled \"Efficient Management of Complex Striped
Files in Active Storage\" in Proceedings of Europar\'08.  The
paper deals with striped files and files with complex formats (e.g.,
netCDF). The experimental results on a Lustre file system not only show
that our proposal can reduce the network traffic to near zero and scale
the performance with the number of storage nodes, but also that it
provides an efficient treatment of striped files and can manage files
with complex data structures.

In addition we are working on performance model for Active Storage. The
purpose of developing an analytical model for Active Storage is to be
able evaluate its expected performance benefits before deploying user
application. The performance model we are developing includes system
parameters: network bandwidth, number of server and compute nodes, disk
bandwidth and information about the compute nodes user parallel job
would run on.

***Plans for next quarter***

In the next quarter we will continue to develop the performance model
for Active Storage and our implementation of user space Active Storage
for Lustre, PVFS, and local file systems.

**[2. Data Mining and Analysis (DMA)]{.underline}**

### 

### [Task 2.1: High-performance statistical computing for scientific applications, ORNL]{.underline}

***Contact:*** Nagiza Samatova, ORNL

The following people have contributed to the project over this quarter:

Roselyne Boretto, Paul Breimier, Nathan Green, William Hendrix, Scott
Klasky, Guru Kora, and Nagiza Samatova

This quarter, we continued our on-going work of building a scalable
analysis engine for Dashboard. Over the last quarter, we made a
transition from design phase to the implementation phase and we created
a prototypical system that is ready to be demoed at the upcoming SDM All
Hands Meeting.

## Accomplishments 

[Selection of data analysis routines for the Dashboard]{.underline}:

An experimental set of \~15 analysis routines were added to the
Dashboard web application. The initial set of routines was selected
based on the most frequently used analysis functions that are currently
used by climate and fusions communities. The set has been derived from
the analysis of climate and fusion data analysis packages, such as CDAT,
GKV, and vugyro. We also had several meetings with the SciDAC Fusion
Centers and discussed their data analysis needs and practices. The
primary focus of current data analysis routines is on:

1\. Descriptive Statistics Group

2\. Correlation Group

3\. FFT Group

4\. Time Series Group

The dashboard analysis process followed a minimal parameter passing
mechanism with pass-through system. The first version of the analysis
engine supported a typical disk based data handling process; input data
read from disk computation of data output data written to disk. This
particular process type enabled us to achieve quick prototype
life-cycle. We also explicitly wanted the dashboard backend to hook in
to the analysis system through a series of system calls, giving us the
flexibility for enhancements for future releases.

[Data type and data access support:]{.underline}

The Dashboard analysis engine has the capability to support the
following three data formats: netCDF, HDF, and ASCII.

For the initial release, netCDF was considered as a priority, so the
support for netCDF datatype was built into the system. Climate and
Fusion NetCDF datasets were used for initial testing. Transparent and
direct data access calls were made through analysis backend in
conjunction with the Dashboard front-end triggers.

[Analytical engine:]{.underline}

The Apache/PHP based Dashboard made plain system calls to the
appropriate analytical trigger routine after receiving an analysis
request from the Dashboard GUI front-end. The system call based trigger
approach was used to minimize changes to the Dashboard front-end and
backend engine.

The following process flow is realized when an analytical trigger event
occurs:

-   A request is received from the browser client for an analysis
    operation.

-   Backend PHP code identifies and locates the appropriate analysis
    routine.

-   PHP code generates the appropriate trigger string to fire the
    routine.

-   PHP code makes a system call on the generated string.

-   Analysis routines are executed in a separate process as it is a
    system call.

-   Analysis routine goes through 3 phase execution cycle
    (READ/COMPUTE/WRITE).

-   PHP code realizes the change in the status of the Analysis routine
    execution and picks up the process flow for further processing.

## Future Plans

Currently we are working with ORNL\'s Dashboard team to integrate all 15
analytical routines and getting the system out of testing phase. We hope
to get the system into the next testing phase that will involve actual
simulation data in the coming quarter.

### [Task 2.2: Feature Selection in Scientific Applications (LLNL)]{.underline}

# *Contact:* Chandrika Kamath, LLNL

## Accomplishments

Much of this quarter focused on writing proposals, the co-organization
of the ASCR \"Mathematics of the Analysis of Petascale Data\" workshop,
and contributing to the report from the workshop.

I was also invited to give a talk on scientific data mining at the
Workshop on Modern Massive Data Sets, held at Stanford, June 25-28.

A paper on \"Tracking Non-Rigid Structures in Computer Simulations\" was
accepted to the IEEE International Conference on Image Processing, to be
held in San Diego in October.

I also put together a 10-page paper on my experiences in scientific data
mining for the SciDAC 2008 conference.

I looked into why certain orbits which were clearly identifiable as
belonging to one class were being misclassified. My analysis indicated
that I need to extract features in a more robust way so that small
changes in the feature values do not result in a mis-classification. To
address this, I added a feature derived from the polynomial fitting
approach we had tried earlier. I also need to add more global features
which reflect the orbit as a whole instead of just local features.

## Plans for Next Quarter

I plan to continue refining the features for the orbits in Poincare
plots and find ways of representing more global features. I will resume
the interactions with my collaborators at PPPL.

I also expect to resume the blob tracking work.

### [Task 2.3: High-dimensional indexing techniques (LBNL)]{.underline}

# Contact: John Wu, LBNL

Contributors: Arie Shoshani

## Accomplishments

-   Developed new histograming functions in support of a project with a
    visual data analysis task jointly with the Visualization group of
    LBNL and VisIt developers. This work has resulted in a paper
    accepted for publication at SC08.

-   Working with researchers from International Computer Science
    Institute, we have developed a set of functions to perform joins on
    pairs of data tables. To support this functionality, we have also
    implemented new and more efficient sorting functions.

-   Refine earlier work on multi-level bitmap indexes with finer control
    of the number of coarse bins to use in FastBit software. This
    follows from a more detailed analysis of the average expected query
    processing costs in a paper under review.

-   Started working on a protein sequence search problem posed by Andrey
    Gorin of ORNL. Our initial approach was to treat the search problem
    as a text search problem by using bitmap indexes for text. However,
    the early indication was that this approach is less efficient than
    alternative ones based on suffix arrays or suffix trees.

-   Finished up two articles. Our work on Order-preserving Bin-based
    Clustering (OrBiC) was accepted for SSDBM 2008. OrBiC data structure
    improves the performance of binned bitmap indexes by reducing cost
    of checking whether records actually satisfy a query condition. It
    presents an alternative strategy for handling extremely high
    cardinality. Our work on using FastBit for text searches has been
    accepted to a special issue of "Annals of Information Systems"
    (Springer). This work describes a way of using FastBit to index
    keywords from text documents. In a comparison with a well-regarded
    MySQL text search functions, our indexing system was found to answer
    queries about 50 times faster.

***Future Plans***

-   Continue to support FastBit users, particularly, developers at
    Yahoo, ICSI and University of Hamburg.

-   Continue to work on the protein sequence search problem and explore
    the option of using suffix array.

-   Continuing the work on feature identification in fusion simulation
    data, the next step is to work with some visualization people to
    develop a front-end for the underlying region finding algorithms.

-   Seek applications that make use of the text search capability.

    **[3. Scientific Process Automation]{.underline}**

PNNL (Terence Critchlow, George Chin),

ORNL (Scott Klasky, Roselyne Barreto, Norbert Podhorszki)

NCSU (Mladen Vouk, Jeff Ligon, Pierre Mouallem, Mei Naggapan)

SDSC (Ilkay Altintas, Daniel Crawl)

UC Davis (Bertram Ludaescher, Ustun, Yildiz, Timothy McPhillips, Manish
Anand, Shawn Bowers)

University of Utah (Steven Parker, Claudio Silva, Ayla Khan, David Koop)

Managing scientific data has been identified as one of the most
important emerging needs by the scientific community because of the
sheer volume and increasing complexity of data being collected.
Effectively generating, managing, and analyzing this information
requires a comprehensive, end-to-end approach to data management that
encompasses all of the stages from the initial data acquisition to the
final analysis of the data. One very significant requirement is
efficient generation and movement of data, collection and storing of the
results, data post-processing, and analysis of the results, i.e.,
facilitation of the scientific process. We are developing a suite of
tools as well as an integrated system for automation of this process in
a robust, tractable, and recoverable fashion so that scientific
exploration can be enhanced. Components are: workflow technologies for
run-time management of the processes, provenance collection and
management technologies, and end-user access technologies for display
and management of both workflows, provenance, analytics and results,
i.e., dashboards.

***Major Accomplishments***

-   Completed prototype of the compilation and run-time provenance
    capture scripts and data-base feeds

-   Improved data provenance capture from Kepler, tested and implemented
    a solution that now works with relatively large workflows. A number
    of issues with Kepler provenance recorder still need to be fixed

-   Dashboard has been improved and upgrades with new visualization
    capabilities and other features

    ***Task 3.1: Dashboard Development***

ORNL (Scott Klasky, Roselyne Barreto, Norbert Podhorszki)

NCSU (Mladen Vouk, Jeff Ligon, Pierre Mouallem, Mei Naggapan)

University of Utah (Steven Parker, Claudio Silva, Ayla Khan, David Koop)

**Current Status**

-   Re-design of machine monitoring page in flash. Need to get ANL
    computers (can someone help)? We can now move windows around the
    page.

-   Calculator integrated into dashboard.

-   Integrate more Data Analysis from Samatova's team into the
    dashboard.

-   Integration of SRM-lite into dashboard.

-   Met with VisTrails designers to talk about integration with
    dashboard.

-   Further bug squashing/hardening of the dashboard.

-   We looked at a number of data server products to allow the pushing
    of data and events from the dashboard server to the browser, and
    settled on Adobe\'s open-source BlazeDS product, which has been
    implemented as a servlet-based web application.

-   Determined how to integrate BlazeDS with the dashboard\'s
    server-side architecture,

-   Identified what forms of visualization should be integrated into the
    Dashboard.

-   Created a preliminary version of the Portable dashboard, which
    currently includes the following: a) A document on how to install
    and setup the necessary components, b) Scripts to package the
    dashboard code and the mysql database, c) an interface on the
    dashboard to download the packages created in it, and d) a perl
    script that unzips those packages and properly set them on the
    client machine

    **Plans for next quarter**

```{=html}
<!-- -->
```
-   Debug with Provenance information.

-   Display codes similar to eclipse view of codes form Provenance
    tracking system.

-   More efficient data analysis routines integrated with calculator.
    Remove ascii dependence, and add interpolations for data.

-   Start looking into launching workflows from dashboard for
    analysis/post processing.

-   Complete migrating the dashboard\'s Flex client-side code SVN
    repository from Utah to ORNL.

-   Update the dashboard visualization capability, adding functionality
    that allows the composition of movies on ewok,

-   Further work on the ability to directly interact with the VisTrails
    visualizations directly on the Dashboard.

-   Improve the portable dashboard to make it more robust and automate
    more steps that are currently done manually

-   Have information on multiple shots accessible through portable
    dashboard.

-   ***Task 3.2: Provenance Tracking***

ORNL (Scott Klasky, Roselyne Barreto, Norbert Podhorszki)

PNNL (George Chin),

SDSC (Ilkay Altintas, Daniel Crawl)

UC Davis (Bertram Ludaescher, Manish Anand, Ustun Yildiz, Shawn Bowers)

University of Utah (Leena Kora)

**Current Status**

-   Harden application provenance tracking software, integrate this with
    several codes.

-   Integrate further for multiple simulations on the dashboard.

-   Continuing to refine the generic data transport and generic job
    scheduling actors. For generic transport actor, we are incorporating
    SRM lite as another support transport mechanism. For both actors, we
    are updating parameter fields and implementing expert parameter
    lists.

-   Extended the Workflow provenance schema and wrote an adapter for the
    dashboard

-   Completed the system provenance python scripts that will be used
    instead of the make command (ORNL installation). These scripts will
    capture the code and store it in a database indexed by the checksum
    of the executable. Also the environment variables and the loaded
    libraries are also stored in the database. These scripts are now
    being tested at ORNL.

-   Extend the Workflow provenance, so that it would return the shot
    number and code name in addition to the raw data file names

-   Ustun Yildiz just joined (June 23^rd^) SPA a UC Davis

-   Ad-hoc meeting among some SPA members at IPAW 2008 (Bertram sent
    summary to SPA-dev)

-   Began work on capturing evolution provenance in Kepler, specifically
    parameter value changes

    **Plans for next quarter**

```{=html}
<!-- -->
```
-   Continue work on improving Dashboard functionality and robustness.

-   Error Trapping: With Alex Sim, we are working on a solution to
    automate the process of finding where an error occurred in a SRM
    transaction. We are going to use the log files from previous SRM
    transactions. We intend to find the events and their order in a
    correct transaction. We will find the events order in similar
    transactions from the log files. Using many occurrences of these
    transactions in log files, we intend to form a Non Finite Automata
    or a pattern for every unique transaction. Then on a future
    transaction that has an error we can match the events in the log
    file to this NFA and find out where the error might have happened.

-   Add new provenance tracking features to dashboard

```{=html}
<!-- -->
```
-   Fully capture Kepler parameter changes to provenance database.

-   Demonstrate prototype for provenance-based failure recovery.

```{=html}
<!-- -->
```
-   Design integration of Kepler pPod-based provenance information
    viewer with Kepler

    ***Task 3.3: Outreach***

    **Current Status**

```{=html}
<!-- -->
```
-   Developed S3D workflow for Jackie Chen. Need to work with their team
    to further test their software.

-   Developed GTC workflow based on CPES workflow.

-   Continued to work with Tony Mezzacappa to develop the Kepler
    monitoring workflow with the Chimera code. First was to replace IO
    in code with ADIOS.

-   Ran GTC code on \~29K processors, and saw \>70TB of data written in
    2, 24 hour days. Data was produced synchronous at \~ 25GB/sec.

-   Benchmarking Chimera code for 'realistic' runs up to 8K processors.
    ADIOS shows about 100x (sometimes 1000x) performance increase over
    parallel hdf5 (with both MPI-IO and POSIX IO, and with the hints
    provided on ANL page for lustre).

-   Significant SPA participation in Kepler Stakeholders meeting (May
    13-14) and Kepler Developers meeting (May 15^)^ at UC Davis

-   Submitted and received an acceptance notice for an SC2008 tutorial
    on scientific workflows and Kepler.

    **Plans for next quarter**

```{=html}
<!-- -->
```
-   Continue to work with Energetic Particle Fusion SciDAC projects.

-   Continue to work the VACET team for integrating data analysis
    workflows with the dashboard.

-   Further discussions of workflow/ADIOS/dashboard to come in August.

-   Working to release ADIOS 1.0 in October 2008.

-   Present Kepler tutorial at SciDAC 2008

    ***New Publications and Related Presentations***

-   \"A First Study on Clustering Collections of Workflow Graphs\", E.
    Santos, L. Lins, J. P. Ahrens, J. Freire, and C. Silva. Second
    International Provenance and Annotation Workshop (IPAW 2008).

-   \"Towards Provenance-Enabling ParaView\", S. P. Callahan, J.
    Freire, C. E. Scheidegger, C. Silva, and Huy T. Vo. Second
    International Provenance and Annotation Workshop (IPAW 2008).

-   \"Using Provenance to Support Real-Time Collaborative Design of
    Workflows\", T. Ellkvist, D. Koop, E. W. Anderson, J. Freire, and C.
    Silva. Second International Provenance and Annotation Workshop (IPAW
    2008).

-   S Bowers, T McPhillips, S Riddle, M Anand, B Ludäscher. Kepler/pPOD:
    Scientific Workflow and Provenance Support for Assembling the Tree
    of Life. Intl. Provenance and Annotation Workshop (IPAW) 2008,

-   Daniel Crawl, Ilkay Altintas, A Provenance-Based Fault Tolerance
    Mechanism for Scientific Workflows, International Provenance and
    Annotation Workshop (IPAW), June 17, 2008

-   Iterative Workflows in Numerical Simulations, Chase, J., Schuchardt,
    K., Chin Jr., G., Daily, J., and Schiebe, T. Proceedings of the IEEE
    2008 Second International Workshop on Scientific Workflows (SWF
    2008).

-   

***Presentations and other activities:***

-   SciDAC 2008 ADIOS tutorial.

-   SciDAC 2008 Kepler tutorial.

-   Invited talk at CCSM meeting.

-   Ilkay Altintas, "Lifecycle of Scientific Workflows and Their
    Provenance: A Usage Perspective", Invited Talk. In IEEE 2008 Second
    International Workshop on Scientific Workflows (SWF 2008) in
    conjunction with IEEE International Conference on Services
    Computing, Hawaii, USA, July 8, 2008

-   B Ludaescher, Panelist, Provenance across workflows and databases,
    IPAW, 2008