![](media/image1.png){width="1.375in" height="0.74375in"}SDM Center
Report

January- March 2008

http://sdmcenter.lbl.gov

**[Highlights in this quarter]{.underline}**

-   1000 times speedup of particle search for the Laser Wakefield
    Particle Accelerator project attained with FastBit.

```{=html}
<!-- -->
```
-   Used FastBit to speed up the operations of searching and tracking
    particles in Laser Wakefield Particle Accelerator (LWPA) project
    (joint with Visualization group). By replacing an existing IDL based
    analysis program with a FastBit based program, we observed a three
    orders of magnitude (1000 fold!) speedup (from 300 seconds to 0.3
    seconds) in the first test run.

```{=html}
<!-- -->
```
-   Large-scale data transfers of climate simulation data from the Earth
    System Grid (ESG) portal to ORNL systems enabled.

```{=html}
<!-- -->
```
-   The user request came from a climate scientist, Marcia Branstetter,
    ORNL. This was a collaborative effort between several DOE SciDAC
    projects: SDM, ESG, and Climate Modeling. The SDM center produced a
    simple command-line interface that invoked the ESG portal
    meta-database and the DataMover tool (based on SRMs) used by the SDM
    Center.

```{=html}
<!-- -->
```
-   New high performance driver for Lustre developed.

```{=html}
<!-- -->
```
-   ORNL has developed a Luster driver for MPI-IO that enables much
    higher performance by better tuning access to avoid performance
    pitfalls with Lustre on the Cray systems. This driver has been
    integrated into MVAPICH version 1.0, which is a popular MPI
    implementation for InfiniBand clusters, and MPICH2 version 1.0.7,
    which serves as the basis of most vendor implementations. Through
    these distributions, this technology will make it into the hands of
    scientists around the world, enhancing I/O performance for a variety
    of applications using MPI-IO directly or through such high-level I/O
    libraries as HDF5 and PnetCDF.

```{=html}
<!-- -->
```
-   Workflow-Dashboard framework now used in production runs by Fusion
    scientists.

```{=html}
<!-- -->
```
-   The Dashboard has been improved and upgrades with new visualization
    capabilities and other features, including comparing images from
    multiple time-steps (shots), and display of movies composed from
    multiple images by the workflow system. We hardened the dashboard
    software to be used in production runs by Fusion scientists.

    ***[Publications this quarter]{.underline}***

Papers abstracts and pointers to the full papers are available at the
sdmcenter.lbl.gov web site, under "publications".

***Publications this quarter***

\[BMR+08\] Shawn Bowers, Timothy McPhillips, Sean Riddle, Manish Anand,
Bertram Ludaescher, Kepler/pPOD: Scientiﬁc Workﬂow and Provenance
Support for Assembling the Tree of Life, Proceedings of the
International Provenance and Annotation Workshop (IPAW) 2008.

\[CA08\] Daniel Crawl and Ilkay Altintas, A Provenance-Based Fault
Tolerance Mechanism for Scientific Workflows, Proceedings of the
International Provenance and Annotation Workshop (IPAW) 2008.

\[FS08\] Juliana Freire and Claudio T. Silva, Towards Enabling Social
Analysis of Scientific Data, CHI Social Data Analysis Workshop, 2008
http://www.cs.utah.edu/\~juliana/pub/freire-sda-chi2008.pdf

\[GBA+08\] Antoon Goderis, Christopher Brooks, Ilkay Altintas, Edward A.
Lee, Carole Goble, Heterogeneous Composition of Models of Computation,
Future Generation Computer Systems (FGCS) Journal, 2008.
http://www.eecs.berkeley.edu/Pubs/TechRpts/2007/EECS-2007-139.pdf

\[KSS08\] J. Freire, D. Koop, E. Santos, and C. Silva, Provenance for
Computational Tasks: A Survey, AIP/IEEE Computing in Science and
Engineering pp. 11-21, May/June 2008 (Vol. 10, No. 3).
http://www.cs.utah.edu/\~juliana/pub/freire-cise2008.pdf

\[KVP+08\] Scott Klasky, Mladen Vouk, Manish Parashar, Ayla Kahn,
Norbert Podhorszki, Roselyne Barreto, Deborah Silver, Steve Parker,
Collaborative Visualization Spaces for Petascale Simulations, to appear
in the 2008 International Symposium on Collaborative Technologies and
Systems (CTS 2008).

\[LKA+08\] L. Lins, D. Koop, E. W. Anderson, S. P. Callahan, E. Santos,
C. E. Scheidegger, J. Freire, and C. Silva, Examining Statistics of
Workflow Evolution Provenance: A First Study, Statistical and Scientific
Database Management (SSDBM) 2008.
http://www.cs.utah.edu/\~juliana/pub/vistrails-ssdbm2008.pdf

\[Naga08\] Mei Nagappan, An Overview of Provenance Collection in
Workflow Systems and a Privacy Policy Model to Share the Information,
April 11-22, High Performance and Applications Conference, Oak Ridge,
TN, April 11-12, 2008.
http://www.nccs.gov/user-support/training-education/workshop-archives/high-performance-computing-and-applications-conference-2008/

\[NV08\] Meiyappan Nagappan, and Mladen Vouk, A Privacy Policy Model for
Sharing of Provenance Information in a Query Based System, accepted as a
short/demo paper for publication in the proceedings of the International
Provenance and Annotation Workshop (IPAW) 2008.

\[SKF+08\] Carlos E. Scheidegger, David Koop, Juliana Freire, and
Claudio T. Silva, Querying and Re-Using Workflows with VisTrails,
Proceedings of ACM SIGMOD International Conference on Management of
Data, 2008. To appear.
http://www.cs.utah.edu/\~juliana/pub/vistrails-sigmod2008.pdf

\[ST08\] C. Silva and J. Tohline, Guest Editorial: Special Issue on
Computational Provenance, AIP/IEEE Computing in Science and Engineering
pp. 9-10, May/June 2008 (Vol. 10, No. 3).
http://csdl2.computer.org/comp/mags/cs/2008/03/mcs2008030009.pdf

\[SWW+08\] Rishi Rakesh Sinha, Marianne Winslett, Kesheng Wu, Kurt
Stockinger, Arie Shoshani: Adaptive Bitmap Indexes for Space-Constrained
Systems. Proceedings of the 24th International Conference on Data
Engineering (ICDE) 2008, pp. 1418-1420.

\[WSS08\] Kesheng Wu, Kurt Stockinger and Arie Shoshani.  Breaking Curse
of Cardinality on Bitmap Indexes. 20th International Conference on
Scientific and Statistical Database Management, (SSDBM) 2008.

\[Zinn08\] Daniel Zinn, Modeling and Optimization of Scientific
Workflows, Extending Database Technology Conference (EDBT) PhD Workshop,
Nantes, France, 2008.

Details of progress during this quarter are reported next.

**[Introduction]{.underline}**

Managing scientific data has been identified as one of the most
important emerging needs by the scientific community because of the
sheer volume and increasing complexity of data being collected.
Effectively generating, managing, and analyzing this information
requires a comprehensive, end-to-end approach to data management that
encompasses all of the stages from the initial data acquisition to the
final analysis of the data. Fortunately, the data management problems
encountered by most scientific domains are common enough to be addressed
through shared technology solutions. Based on the community input, we
have identified three significant requirements. First, more efficient
access to storage systems is needed. In particular, parallel file system
improvements are needed to write and read large volumes of data without
slowing a simulation, analysis, or visualization engine. These processes
are complicated by the fact that scientific data are structured
differently for specific application domains, and are stored in
specialized file formats. Second, scientists require technologies to
facilitate better understanding of their data, in particular the ability
to effectively perform complex data analysis and searches over large
data sets. Specialized feature discovery and statistical analysis
techniques are needed before the data can be understood or visualized.
To facilitate efficient access it is necessary to keep track of the
location of the datasets, effectively manage storage resources, and
efficiently select subsets of the data. Finally, generating the data,
collecting and storing the results, data post-processing, and analysis
of results is a tedious, fragmented process. Tools for automation of
this process in a robust, tractable, and recoverable fashion are
required to enhance scientific exploration.

Our approach is to employ an evolutionary development and deployment
process: from research through prototypes to deployment and
infrastructure. Accordingly, we have organized our activities in three
layers that abstract the end-to-end data flow described above. We
labeled the layers (from bottom to top):

-   Storage Efficient Access (SEA)

-   Data Mining and Analysis (DMA)

-   Scientific Process Automation (SPA)

The SEA layer is immediately on top of hardware, operating systems, file
systems, and mass storage systems, and provides parallel data access
technology, and transparent access to archival storage. The DMA layer,
which builds on the functionality of the SEA layer, consists of
indexing, feature identification, and parallel statistical analysis
technology. The SPA layer, which is on top of the DMA layer, provides
the ability to compose scientific workflows from the components in the
DMA layer as well as application specific modules.

# This report consists of the following sections, organized according to the three layers, as follows:

-   Storage Efficient Access (SEA) techniques

    -   Task 1.1: Low-Level Parallel I/O Infrastructure

    -   Task 1.2: Collaborative File Caching

    -   Task 1.3: File System Benchmarking and Application I/O Behavior

    -   Task 1.4: Application Interfaces to I/O

    -   Task 1.5: Disk Resident Extendible Array Libraries

    -   Task 1.6: Active Storage in the Parallel Filesystem

```{=html}
<!-- -->
```
-   Data Mining and Analysis (DMA) components

    -   Task 2.1 High-performance statistical computing for scientific
        applications

    -   Task 2.2: Feature Selection in Scientific Applications

    -   Task 2.3: High-dimensional indexing techniques

-   Scientific Process Automation (SPA) tools

    -   Task 3.1: Dashboard Development

    -   Task 3.2: Provenance Tracking

    -   Task 3.3: Outreach

The reports by each of the three areas, SEA, DMA, and SPA, follow.

**[1. Storage Efficient Access (SEA)]{.underline}**

***Participants:*** Rob Ross, Rajeev Thakur, Sam Lang, and Rob Latham
(ANL), Alok Choudhary, Wei-keng Liao, Arifa Nisar (NWU), Arie Shoshani
and Ekow Otoo (LBNL), Jeffrey Vetter and Weikuan Yu (ORNL), Jarek
Nieplocha and Juan Piernas Canovas (PNL)

**[Highlight this quarter]{.underline}**

ORNL has developed a Luster driver for MPI-IO that enables much higher
performance by better tuning access to avoid performance pitfalls with
Lustre on the Cray systems. This driver has been integrated into MVAPICH
version 1.0, which is a popular MPI implementation for InfiniBand
clusters, and MPICH2 version 1.0.7, which serves as the basis of most
vendor implementations. Through these distributions, this technology
will make it into the hands of scientists around the world, enhancing
I/O performance for a variety of applications using MPI-IO directly or
through such high-level I/O libraries as HDF5 and PnetCDF.

**[Goals]{.underline}**

The goal of this project is to provide significant improvements in the
parallel I/O subsystems used on today\'s machines while ensuring that
the capabilities available now will continue to be available as systems
increase in scale and technologies improve. A three-fold approach of
immediate payoff improvements, medium-term infrastructure development,
and targeted longer-term R&D is employed.

Two of our keystone components are the PVFS parallel file system and the
ROMIO MPI-IO implementation. These tools together address the
scalability requirements of upcomnocing parallel machines and are
designed to leverage the technology improvements in areas such as
high-performance networking. These are both widely deployed and freely
available, making them ideal tools for use in today's systems. Our work
in application I/O interfaces, embodied by our Parallel NetCDF
interface, also promises to provide short-term benefits to a number of
climate and fusion applications.

In addition to significant effort on PVFS, we recognize the importance
of other file systems in the HPC community. For this reason our efforts
include improvements to the Lustre file system, and we routinely discuss
both Lustre and GPFS during tutorials. Our efforts in performance
analysis and tuning for parallel file systems, as well as our work on
MPI-IO, routinely involve these file systems.

At the same time we continue to push for support of common,
high-performance interfaces to additional storage technologies. Our work
combining the Storage Resource Manager (SRM) with MPI-IO, in conjunction
with PVFS, will provide a single solution for the problems of
high-performance parallel storage access, integration of file system
with tertiary storage, and remote data access. Our work in Active
Storage will provide common infrastructure for moving computation closer
to storage devices, an important step in tackling the challenges of
petascale datasets.

**[Task 1.1: Low-Level Parallel I/O Infrastructure (Ross, Thakur, Lang,
Latham, Vetter, Yu)]{.underline}**

The objective of this work is to improve the state of parallel I/O
support for high-end computing (HEC) and enable the use of high
performance parallel I/O systems by application scientists. The Parallel
Virtual File System (PVFS) and ROMIO MPI-IO implementations, with
development lead by ANL, are in wide use and provide key parallel I/O
functionality. This work builds on these two components by enhancing
them in order to ensure these capabilities continue to be available as
systems continue to scale. The OPAL library, being developed at ORNL, is
a more recent development and builds in the original ROMIO library to
provide enhancements for Lustre and Cray systems.

***Progress Report***

Sharing our work in parallel I/O optimizations for users' benefits at
large has been a primary goal. During the past quarter, we have put
significant efforts to release an initial version of Lustre ADIO
implementation to the popular MPI implementations that are deployed at
supercomputing sites. Owing to the assistance from the MVAPICH team at
the Ohio State University, and that of the Argonne team, our Lustre ADIO
driver has been released as a part of both MVAPICH-1.0 and MPICH2
version 1.0.7.

In addition to merging changes into ROMIO relevant to Lustre, we have
been working with IBM to merge their Blue Gene/P optimizations into our
source tree. We implemented a lock-free ADIO drive for Blue Gene to
enable PVFS use, and we improved the implementation of stat operations
in ROMIO to minimize the overhead of file system type detection. We also
made a number of bug fixes prior to the release of MPICH2 version 1.0.7.

We continued our efforts in developing the OPAL library, a feature-rich
and high-performance MPI-IO library for Lustre. In the past quarter, we
have evaluated the direct I/O support in Lustre ADIO driver for both
Cray XT and Linux clusters. Using a Linux cluster, we have observed
performance improvement with large streaming of I/O requests over
Lustre, avoiding data copies in the kernel buffer cache. We have also
observed significant benefits for Cray XT systems under CNL (Compute
Node Linux) operating system.

With respect to PVFS, we have been focusing our development efforts on
the ALCF Blue Gene/P 100TF system, fixing problems uncovered at this new
scale of deployment. This includes testing at as many as 8192 nodes. On
the research side, we have been implementing tracing in PVFS and ROMIO
along with collaborators Al Malony (U. of Oregon) and Kwan-Liu Ma (UC
Davis) with the goal of performance visualization of the entire I/O
software stack on the Blue Gene/P system. Additionally, we continue to
collaborate with Garth Gibson's team from the PDSI in the implementation
of scalable directories in PVFS.

Finally, we presented a half-day version of our parallel I/O tutorial at
FAST 2008, a very popular file system and storage conference, to
approximately 75 attendees.

***Plans for next quarter***

In the coming months, we will continue to develop more features to
improve the performance of parallel I/O on the supercomputers that are
deployed with Lustre, such as Cray XT and large-scale clusters. We will
also demonstrate the performance benefits of OPAL to other applications
running over Jaguar and clusters. Furthermore, we plan to introduce new
features into the Lustre file system for better scalability and
performance assurance under heavy load.

We will be presenting material on parallel I/O optimizations at ScicomP,
the IBM Scientific Systems User Group in May, and at an upcoming INCITE
workshop at ANL in May.

**[Task 1.2: Collaborative File Caching (Choudhary, Liao,
Nisar)]{.underline}**

The objective of this work is to develop a layer of user-level
client-side file caching for the MPI-IO library. The design uses an I/O
thread in each MPI process and coordinates all threads to perform a
coherent file caching.

***Progress Report***

We continue working on the approach that uses a separate group of MPI
processes as I/O delegates to performance client-side file caching. The
I/O requests from application clients are forwarded to the I/O delegates
and the delegates collaborate to perform caching and I/O to the file
systems. The creation of the cache delegates and communication between
delegates and clients are implemented through MPI dynamic process
management functionality. The same caching mechanism used by the
I/O-thread based approach is adopted in this design.

Testing and development are continued on several parallel machines:
Tungsten running Lustre file system and the IBM cluster running GPFS
file system at NCSA, Jazz running PVFS file system at ANL, and Ewok
running Lustre at ORNL. We use several I/O benchmarks for performance
evaluation: the NASA BTIO benchmark, the FLASH application I/O kernel,
and S3D application I/O kernel. Performance numbers and profiling have
been collected and analyzed.

***Plans for next quarter***

We plan to develop an I/O thread on each of the I/O delegates, so that
cached pages can be flushed to file systems asynchronously at the
background. This design includes using a pthread mutex for each cache
page to protect concurrent accesses to the page. We will also develop a
scheme for flushing dirty data that are accessed less frequently.

**[Task 1.3: File System Benchmarking and Application I/O Behavior
(Choudhary, Liao, Nisar)]{.underline}**

The objective of this work is to evaluate the relative performance of
the file systems available to important SciDAC applications on DoE
compute platforms, and, in particular the sustained speed of reading and
writing large datasets. The performance, functionality, and scalability
of MPI-IO, parallel netCDF, and HDF5 are critical for many applications.
Two groups in the SDM center, NWU and ORNL, are performing complementary
efforts in this area.

***Progress Report***

We continue the collaboration work on the S3D I/O kernel with Jacqueline
Chen at Sandia National Laboratories, Ramanan Sankaran and Scott Klasky
at ORNL. We have developed Fortran subroutines using MPI-IO, parallel
netCDF, and HDF5 and tested on the Cray XT at ORNL. The performance
evaluation of using up to 11,000 process cores has been obtained, which
showed the results very closed to the system maximum bandwidth when the
number of cores is larger than 2000. As the Cray XT running catamount
operating system is upgrading to compute-node Linux (CNL), we have
started the evaluation on the new CNL environment. The initial
performance results showed worse write bandwidths than the ones obtained
on catamount.

***Plans for next quarter***

We plan to study application I/O kernel from the Parallel Ocean Program.
Evaluation will be performed on TeraGrid machines and the Cray XT on
ORNL.

**[Task 1.4: Application Interfaces to I/O (Ross, Thakur, Latham, Liao,
Choudhary, Nisar, Vetter, Yu)]{.underline}**

The objective of this work is to improve the observed I/O throughput for
applications using parallel I/O by enhancements to or replacements for
popular application interfaces to parallel I/O resources. This task was
added in response to a perceived need for improved performance at this
layer, in part due to our previous work with the FLASH I/O benchmark.
Because of their popularity in the scientific community we have focused
on the NetCDF and HDF5 interfaces, and in particular on a parallel
interface to NetCDF files.

***Progress Report***

We have started the collaboration work with the Adaptive IO System
(ADIOS) team lead by Scott Klasky at ORNL. ADIOS is an I/O system that
separates the selection and implementation of any particular I/O
routines from the scientific code offering unprecedented flexibility in
the choices for processing and storing data. ADIOS currently handles I/O
on per-process base. We have contributed the implementation of MPI-IO
collective functions for ADIOS interface for FLASH I/O benchmark. The
PnetCDF interfaces for ADIO is still under development.

As a part of our efforts to provide a versatile I/O middleware for
scientific applications, we have ported the I/O kernel of the combustion
simulation program, S3D, to the initial ADIOS middleware that is still
under development at Oak Ridge National Laboratory. We have evaluated
the performance to thousands of processes, and have shown comparable
performance to the original S3D I/O implementation.

We have been working with Jim Edwards of UCAR to clarify the use of the
PnetCDF flexible mode interface and fix some bugs found as a result of
this collaboration. We've also been helping Annette Koontz of PNNL
resolve some issues in the process of porting the GCM code to the Blue
Gene/P and other systems.

***Plans for next quarter***

We will continue to collaborate with ADIOS team on developing PnetCDF
interface and improving FLASH I/O benchmark.

**[Task 1.5: LIBDRXTA: Disk Resident Extendible Array Libraries
(Shoshani, Rotem, Otoo)]{.underline}**

This work focuses on implementing a library for storage management
access of Disk Resident Multidimensional Extendible Arrays for parallel
applications. The library is called pDRXTA. It is the parallel
counterpart of a Disk Resident Multidimensional Extendible Array library
referred to simply as DRXTA. The pDRXTA library currently is only
partially completed and is being implemented to work with PVFS and the
MPI-2 RMA methods and also as an alternative to the Disk Resident Array
(DRA) that is associated with the Global Array (GA) library. Work is
ongoing to provide equivalent APIs that are callable from the Global
Array library. There is a corresponding ongoing work to evaluate and
compare its access functions with the use of skip-list access method for
array chunks in HDF5.

***Progress Report***

We completed implementation of a cache replacement algorithm, besides
LRU, for use with the BerkeleyDB cache pool module. We also completed
implementation of skip-list access methods for chunked dense arrays.
Finally, we performed a performance comparison of DRXTA storage of dense
arrays with Skip-List index of dense array chunks.

***Plans for next quarter***

We will apply of the DRXTA storage of dense arrays to HDF5 dense array
chunks and performance measurements. We will continue work on
implementing and testing of extendible array file for dense chunked
arrays. We will investigate a multi-threaded implementation of the DRXTA
library for multi-core architectures. Finally, we will implement
parallel out-of-core multidimensional extendible array functions
(pDRXTA) for dense and sparse arrays.

**[Task 1.6: Active Storage in the Parallel File System (Nieplocha,
Canovas)]{.underline}**

We are developing Active Storage, a promising technology for reducing
bandwidth requirements between the storage and compute elements of
current supercomputing systems, and leveraging the processing power of
the storage nodes used by some modern file systems. To achieve both
objectives, Active Storage allows certain processing tasks to be
performed directly on the storage nodes, near the data they manage.

***Progress Report***

We have been developing Active Storage, a storage architecture for
reducing bandwidth requirements between the I/O and compute elements of
current supercomputing systems, and leveraging the processing power of
the storage nodes used by some modern file systems. To achieve both
objectives, Active Storage allows certain processing tasks to be
performed directly on the storage nodes, near the data they manage. In
the last quarter, our focus has been on releasing an initial version of
user-space implementation and developing a performance model for Active
Storage.

We released an initial version of the user space implementation of
Active Storage. It is available for download on the project website,
[[http://hpc.pnl.gov/projects/active-storage/]{.underline}](http://hpc.pnl.gov/projects/active-storage/)
This initial port was tested under Lustre and PVFS (with patches). To
provide a more convenient development platform for the user, support is
also available for most Unix local file systems. The user can develop
Active Storage processing code on a local desktop before deploying it in
a parallel machine.

The purpose of developing an analytical model for Active Storage is to
enable evaluation of its expected performance benefits before developing
the user application. The performance model we are developing includes
system parameters like network bandwidth, number of file system servers,
disk bandwidth and information about the compute nodes user parallel job
would run on.

***Plans for next quarter***

In the next quarter we will continue to develop the performance model
for Active Storage and our implementation of user space Active Storage
for Lustre, PVFS, and local file systems.

**[2. Data Mining and Analysis (DMA)]{.underline}**

**[Highlights in this quarter]{.underline}**

-   Used FastBit to speed up the operations of searching and tracking
    particles in Laser Wakefield Particle Accelerator (LWPA) project
    (joint with Visualization group). By replacing an existing IDL based
    analysis program with a FastBit based program, we observed a three
    orders of magnitude (1000 fold!) speedup (from 300 seconds to 0.3
    seconds) in the first test run.

```{=html}
<!-- -->
```
-   Enabled large-scale data transfers of the climate simulation data
    from the Earth System Grid (ESG) portal to ORNL systems. The user
    request came from a climate scientist, Marcia Branstetter, ORNL.
    This was a collaborative effort between several DOE SciDAC projects:
    SDM, ESG, Climate Modeling. The SDM center produced a simple
    command-line interface that invoked the use of the ESG portal
    meta-database and DataMover tool used by the SDM center.

### [Task 2.1: High-performance statistical computing for scientific applications, ORNL]{.underline}

***Contact:*** Nagiza Samatova, ORNL

The following people have contributed to the project over this quarter:

Paul Breimier, Guru Kora, Nagiza Samatova, Alex Sim, Arie Shoshani, and
Mladen Vouk

## Accomplishments 

[Enabling large-scale data transfers for climate scientists through the
DOE ESG portal]{.underline}:

With the growing amount of climate data available through the DOE Earth
Science Grid (ESG) portal, and computing power continuously increasing,
there is a burgeoning demand by climatologists to access large swaths of
data for processing. The current ESG portal provides a convenient
interface for users to find and download small subsets of data, but it
is not convenient for large-scale data transfers. While the ESG provides
a large-scale multi-file data mover that runs on the user site (called
DataMover-Lite, or DML), it was designed as a Graphical User Interface.
Many users prefer a simple command-line interface (CLI) for the entire
data movement operation, once the desired files were identified on the
ESG portal. The SDM center addressed this need by creating the ESG
Download Project (EDP). EDP provides a convenient mechanism for ESG
customers to download large amounts of data from different archives for
subsequent analysis at their own sites.

We worked with Marcia Branstetter (ORNL), the DataMover-Lite (DML) team
(Alex Sim, Arie Shoshani), one of the ESG PIs (David Bernholdt, ORNL),
and the ESG portal developers at NCAR (Luca Cinquini, et. al) to build
EDP, a custom Java CLI tool, that allows users to interact with the ESG
portal to download large numbers of files. Given a set of logical file
names identified by a user form interaction with the ESG portal, the
tool uses the portal's metadata system to identify the physical location
of files. The EDP tool divides the multi-file request into ESG
acceptable sized batches, and submits each batch request to the portal.
The portal gets the desired files for each batch into its cache, using
Storage Resource Managers (SRMs). The EDP tool then invokes DML to
download the files to the local or remote machine, provided SSH is
installed. The client runs on both Windows and Linux distributions,
allows users to configure the number of concurrent DML download
transfers, and notifies the user of completion by email. It is also
possible to find out the status of the large transfer request by
checking with DML about the status of the requests.

We would like to acknowledge and thank the cross-team contributions of
all involved, including the ESG, DOE Climate Modeling, The SRM team, and
the SDM members in this successful effort.

[Enabling the native language plug-ins to R/pR]{.underline}

Parallel computation in R is extremely useful and is an actively desired
feature request. Researchers who are using R as a day-to-day statistical
data analytics tool are seeking new ways of data analysis as size of the
data grows day by day. R foreign language support is extremely low-level
and difficult to code. Parallel computations using well known
programming APIs involve low-level programming effort and large
development life-cycles. In considerably bigger projects, active tasks
like manual memory management, thread management and synchronization is
a big challenge and one can easily get lost in tracking the efficiency
of the algorithms. Performance can suffer a lot due to memory leaks and
mismanagement.

The next release of parallel R (pR) strives to bridge this gap between
low-level R programming environment and classical parallel programming
paradigms. The new pR provides users a seamless, user-friendly API that
a typical parallel programmer can easily understand and adapt. The new
pR extensively uses optimized C++ programming constructs that are proven
to be very efficient and optimal for large-scale data analytics. C++
inherently has lots of efficient management techniques that can be
readily used. C++ "contains" C; it augments the low-level memory
management and computing efficiency of C with its own high-level
programming efficiency. pR effectively uses the better parts of C++ to
manage various parts of R foreign language interface. Some of the main
features that pR effectively uses to circumvent several drawbacks of R
are: IO stream objects that read and write character data to and from
the console and disk files, and which are extensible to other IO
devices/streams; container objects contain and manipulate collections of
data objects; generic algorithms to perform operations on the container
objects; numerical objects to represent complex numbers; localization
objects to control at run time the display of dates, currency, and other
culturally mutable signs; and exception objects to manage execution
failure of a program gracefully. Some of the things that pR addresses
are navigation issues for traversing/accessing items in a generic R
Object List, automatic clean-up of R Objects, and easier
hand-shaking/coercing (Object) with the R environment. pR effectively
brings in existing rich set of standard C++ STL based algorithms. These
algorithms have been standardized and are tuned over the years by
experienced C++ development community. As proof of principle, the new pR
has been shown utilizing external libraries like BLAS, LINPACK and
LAPACK.

## Future Plans

-   Develop the most frequently used analysis routines by various
    application domains and integrate them inside of the Dashboard --
    the project led by Scott Klasky, ORNL.

-   Extend the capability of pR for more efficient memory management and
    more comprehensive support of various data types and data
    structures.

### [Task 2.2: Feature Selection in Scientific Applications (LLNL)]{.underline}

# *Contact:* Chandrika Kamath, LLNL

## Accomplishments

I used the scripts I wrote to create a training set of feature vectors
from one plot and used it in a decision tree classifier. I then
evaluated the tree created to see if it made sense. This resulted in
several iterations, where I refined the features extracted so that the
resulting decision tree seemed to make sense. I then used the tree to
classify orbits from another plot from the same tokamak. This added some
more insights into the features and resulted in further refinements. I
then tried training the tree using data from a tokamak and classifying
the orbits in a stellarator. Surprisingly, the results with a large
number of points are quite good, even though the orbits in the tokamak
are oval and those in the stellarator are triangular.

The results when I consider fewer points are not as good, indicating
that I need to identify more robust features.

I also completed my chapter on Scientific Data Analysis for the book
being edited by Arie Shoshani and Doron Rotem. This was an overview of
the analysis process followed by sections describing work done by
different groups on mining science data.

I continue to chair the steering committee for the SIAM Data Mining
conference and am responsible for coordinating the organization of the
conference. The 2008 conference will be held Atlanta, April 24-26.

I am also involved in organizing the June workshop on Mathematics for
the Analysis of Petascale Data.

## Plans for Next Quarter

I plan to continue refining the features for the orbits in Poincare
plots to extract additional quantities which are more robust to the
diversity of plots. I will also contact the physicists at PPPL to
discuss the latest results.

I also expect to pick up on the blob tracking work to continue where my
post-doc left off 6 months ago.

### [Task 2.3: High-dimensional indexing techniques (LBNL)]{.underline}

# Contact: John Wu, LBNL

Contributor: Arie Shoshani

## Accomplishments

-   Used FastBit to speed up the operations of searching and tracking
    particles in Laser Wakefield Particle Accelerator (LWPA) project
    (joint with Visualization group). By replacing an existing IDL based
    analysis program with a FastBit based program, we observed a three
    order of magnitude speedup (from 300 seconds to 0.3 seconds) in the
    first test run.

-   Working with the Visualization group of LBNL, we are continuing to
    expand the functionality H5Part software. Recently, we worked with
    VisIt developers to provide efficient parallel histogram
    functionality for interactive exploration of terabytes of data.

-   Working with a few dedicated FastBit users, especially the users
    from University of Hamburg and ICSI (in Berkeley), we have performed
    extensive stress testing of the software, including running a very
    large number of queries through large datasets under extremely tight
    memory constraints, and running queries over data with extremely
    high attribute cardinality (without binning). In particular, the
    tests involving extremely high-cardinality attributes are highly
    instructive. It verifies the theoretically expected performance
    characteristics of FastBit compressed bitmap indexes and it also
    reveals how we might further improve the overall performance of the
    FastBit software.

-   Two articles recently accepted for publication. Our work on
    Order-preserving Bin-based Clustering (OrBiC) was accepted for
    SSDBM 2008. OrBiC data structure improves the performance of binned
    bitmap indexes by reducing the cost of checking whether records
    actually satisfy a query condition. It presents an alternative
    strategy for handling extremely high cardinality. Our work on using
    FastBit for text searches has been accepted to a special issue of
    "Annals of Information Systems" (Springer). This work describes a
    way of using FastBit to index keywords from text documents. In a
    comparison with a well-regarded MySQL text search functions, our
    indexing system was found to answer queries about 50 times faster.

***Future Plans***

-   Continue to support FastBit users, particularly, developers at
    Yahoo, ICSI and University of Hamburg.

-   Continuing the work on feature identification in fusion simulation
    data, the next step is to work with visualization people to develop
    a front-end for the underlying region finding algorithms.

-   Implement basic support for joins request for some users.

-   Seek applications that make use the text search capability.

    ***Publications***

```{=html}
<!-- -->
```
-   Rishi Rakesh Sinha, Marianne Winslett, Kesheng Wu, Kurt Stockinger,
    Arie Shoshani: Adaptive Bitmap Indexes for Space-Constrained
    Systems. Proceedings of the 24th International Conference on Data
    Engineering (ICDE) 2008, pp. 1418-1420.

-   Kesheng Wu, Kurt Stockinger and Arie Shoshani.  Breaking Curse of
    Cardinality on Bitmap Indexes. 20th International Conference on
    Scientific and Statistical Database Management, (SSDBM) 2008.

    **[3. Scientific Process Automation]{.underline}**

PNNL (Terence Critchlow, George Chin),

ORNL (Scott Klasky, Roselyne Barreto, Norbert Podhorszki)

NCSU (Mladen Vouk, Jeff Ligon, Pierre Mouallem, Mei Naggapan)

SDSC (Ilkay Altintas, Daniel Crawl)

UC Davis (Bertram Ludaescher, Timothy McPhillips, Daniel Zinn)

University of Utah (Steven Parker, Claudio Silva, Ayla Khan, David Koop)

Managing scientific data has been identified as one of the most
important emerging needs by the scientific community because of the
sheer volume and increasing complexity of data being collected.
Effectively generating, managing, and analyzing this information
requires a comprehensive, end-to-end approach to data management that
encompasses all of the stages from the initial data acquisition to the
final analysis of the data. One very significant requirement is
efficient generation and movement of data, collection and storing of the
results, data post-processing, and analysis of the results, i.e.,
facilitation of the scientific process. We are developing a suite of
tools as well as an integrated system for automation of this process in
a robust, tractable, and recoverable fashion so that scientific
exploration can be enhanced. Components are: workflow technologies for
run-time management of the processes, provenance collection and
management technologies, and end-user access technologies for display
and management of both workflows, provenance, analytics and results,
i.e., dashboards.

**[Highlights this quarter]{.underline}**

-   Completed prototype of the compilation and run-time provenance
    capture scripts and data-base feeds

-   Improved data provenance capture from Kepler, tested and implemented
    a solution that now works with relatively large workflows. A number
    of issues with Kepler provenance recorder still need to be fixed

-   Dashboard has been improved and upgrades with new visualization
    capabilities and other features, including comparing images from
    multiple time-steps (shots), and display of movies composed from
    multiple images by the workflow system. Hardened the dashboard
    software to be used in production runs by Fusion scientists.

    **[Task 3.1: Dashboard Development]{.underline}**

ORNL (Scott Klasky, Roselyne Barreto, Norbert Podhorszki)

NCSU (Mladen Vouk, Jeff Ligon, Pierre Mouallem, Mei Naggapan)

University of Utah (Steven Parker, Claudio Silva, Ayla Khan, David Koop)

**Current Status**

-   Extended the dashboard to allow for shot comparison.

-   Hardened the dashboard to allow for a full-ORNL release. Software is
    installed by ORNL system administrators, and has passed ORNL
    security team.

-   Textual display of information placed onto dashboard.

-   Annotations and web-notes now fully integrated into the database for
    the dashboard.

-   Extended the movies (through the workflow automation system) to
    allow for local and global views of movies. Now we have two movies
    per variable, which either keeps the y-axis/colormap constant
    through time, or varies to the local-min-max of each time step.

-   At users' request, improved the dashboard presentation. Change the
    order in which information is presented and how it's presented
    across browsers and screen resolutions. As the dashboard is
    developed user experience is a continuing effort for the team.

-   Created an Actor/Workflow for SRM-Lite. The actor has a few
    requirements in order to run, such as exchanging keys and setting up
    the configuration, but once that is done, it is very straightforward
    to use. Future Version would address these requirements/limitations.

-   Worked on improving the DB schema for the ORNL Dashboard, mainly the
    machine tracking and annotations databases.

-   Improve the simulation monitoring user interface and the annotations
    user interface by hardening and simplifying the user interface
    components based on bug reports and user feedback from the pilot
    dashboard.

-   Improve annotation persistent storage and management.

-   Provided support to dashboard development.

-   Worked on creating a Data Lineage system for the dashboard based on
    the data collected by the provenance recorded. It currently has some
    limitations and requires a few changes in the workflow design, in
    order to collect all the data that is needed (document on-line)

-   Currently working on packaging all the modules needed in order to
    run the dashboard locally, without the need to access the internet.

    **Plans for next quarter**

```{=html}
<!-- -->
```
-   Developing a session management system for tracking user sessions
    and for collaboration among scientists when looking at the same
    simulations.

-   Create a package for portable Dashboard

-   Demonstrate initial integration of VisIt within Dashboard

    **[Task 3.2: Provenance Tracking]{.underline}**

ORNL (Scott Klasky, Roselyne Barreto, Norbert Podhorszki)

PNNL (George Chin),

SDSC (Ilkay Altintas, Daniel Crawl)

UC Davis (Bertram Ludaescher, Timothy McPhillips, Manish Anand, Daniel
Zinn)

University of Utah (Leena Kora)

**Current Status**

-   A separate provenance recorder can now be instantiated at each level
    of the workflow hierarchy. Each recorder can be independently
    configured to ignore provenance at that level or store to a
    different location.

-   Completed base-line provenance requirements, architecture and
    prototyping of integrated system components Production prototype is
    running at ORNL. More information available on-line
    (http://groups.google.com/group/spa-dev).

-   Continued updates to provenance database schema based on use-cases.

-   Developed draft version of Kepler actor for SRM-lite

-   Implemented initial versions of the generic data transport and
    generic job scheduling actors for Kepler. Demonstrated actors to
    select Kepler users to collect additional requirements and design
    criteria. Continuing to refine actors based on user input.

    **Plans for next quarter**

```{=html}
<!-- -->
```
-   Add workflow evolution support to provenance recorder.

-   Finish implementing compile-time and run-time code provenance for
    codes at ORNL.

-   Continue work on improving Dashboard functionality and robustness.

    **[Task 3.3: Outreach]{.underline}**

    **Current Status**

```{=html}
<!-- -->
```
-   Consolidated Kepler command-line execution actors.

-   Added support for Matlab JNI actor.

-   Worked with Groundwater group to build a "smart" distributor actor
    for better job scheduling.

-   Implemented "port assertions" to detect workflow errors.

-   Submitted a tutorial on scientific workflows and Kepler to SC 2008.

-   Working on the lecture material for the accepted SciDAC 2008
    tutorial on scientific workflows and Kepler.

-   Preview release of Kepler with support for collection-oriented
    workflows ("COMAD") and embedded provenance. Target audience: AToL

-   Started to work with Jackie Chen. Starting the development of the
    Kepler monitoring workflow with the S3D code.

-   Started to work with Tony Mezzacappa to develop the Kepler
    monitoring workflow with the Chimera code.

-   Started to work with the GTS code to develop the Kepler monitoring
    workflow for this group.

-   Developed draft version of Kepler actor for SRM lite

-   Work with Marcia Branstetter

    **Plans for next quarter**

```{=html}
<!-- -->
```
-   Implement *Checkpoint* actor for recovery of workflow errors.

-   Extend interactions with Energetic Particle Fusion SciDAC projects.

-   Start integrating further with the VACET team for integrating data
    analysis workflows with the dashboard.

-   Place more analysis routines into the monitoring workflows.

-   Continue interactions with Climate researchers

    ***New Publications and Related Presentations***

-   A Provenance-Based Fault Tolerance Mechanism for Scientific
    Workflows, Daniel Crawl and Ilkay Altintas, proceedings of the
    International Provenance and Annotation Workshop (IPAW) 2008, Salt
    Lake City, UT, June 2008.

-   Kepler/pPOD: Scientiﬁc Workﬂow and Provenance Support for Assembling
    the Tree of Life, Shawn Bowers, Timothy McPhillips, Sean Riddle,
    Manish Anand, Bertram Ludaescher, IPAW 2008.

-   Heterogeneous Composition of Models of Computation, Antoon Goderis,
    Christopher Brooks, Ilkay Altintas, Edward A. Lee, Carole Goble,
    FGCS Journal.

-   Modeling and Optimization of Scientific Workflows, Daniel Zinn, EDBT
    PhD Workshop, Nantes, France, 2008

```{=html}
<!-- -->
```
-   Collaborative Visualization Spaces for Petascale Simulations, Scott
    Klasky, Mladen Vouk, Manish Parashar, Ayla Kahn, Norbert Podhorszki,
    Roselyne Barreto, Deborah Silver, Steve Parker., to appear in the
    2008 International Symposium on Collaborative Technologies and
    Systems (CTS 2008).

```{=html}
<!-- -->
```
-   A Privacy Policy Model for Sharing of Provenance Information in a
    Query Based System, Meiyappan Nagappan, and Mladen Vouk, accepted as
    a short/demo paper for publication in the proceedings of the
    International Provenance and Annotation Workshop (IPAW) June 17-18,
    2008, Salt Lake City.

-   An Overview of Provenance Collection in Workflow Systems and a
    Privacy Policy Model to Share the Information, Mei Nagappan, April
    11-22, High Performance and Applications Conference, Oak Ridge, TN,
    April 11-12, 2008.
    http://www.nccs.gov/user-support/training-education/workshop-archives/high-performance-computing-and-applications-conference-2008/

-   Examining Statistics of Workflow Evolution Provenance: A First
    Study L. Lins, D. Koop, E. W. Anderson, S. P. Callahan, E.
    Santos, C. E. Scheidegger, J. Freire, and C. Silva, Statistical and
    Scientific Database Management (SSDBM) 2008. To appear.
    http://www.cs.utah.edu/\~juliana/pub/vistrails-ssdbm2008.pdf

-   Guest Editorial: Special Issue on Computational Provenance C. Silva
    and J. Tohline AIP/IEEE Computing in Science and Engineering pp.
    9-10, May/June 2008 (Vol. 10, No. 3).
    http://csdl2.computer.org/comp/mags/cs/2008/03/mcs2008030009.pdf

-   Provenance for Computational Tasks: A Survey J. Freire, D. Koop, E.
    Santos, and C. Silva AIP/IEEE Computing in Science and Engineering
    pp. 11-21, May/June 2008 (Vol. 10, No. 3).
    http://www.cs.utah.edu/\~juliana/pub/freire-cise2008.pdf

-   Querying and Re-Using Workflows with VisTrails, Carlos E.
    Scheidegger, David Koop, Juliana Freire, and Claudio T. Silva
    Proceedings of ACM SIGMOD International Conference on Management of
    Data, 2008. To appear.
    http://www.cs.utah.edu/\~juliana/pub/vistrails-sigmod2008.pdf

-   Towards Enabling Social Analysis of Scientific Data, Juliana Freire
    and Claudio T. Silva CHI Social Data Analysis Workshop, 2008
    http://www.cs.utah.edu/\~juliana/pub/freire-sda-chi2008.pdf

***Submitted Papers (currently in review)***

-   Structured Streams: Data Services for Petascale Science
    Environments, Widener, et al., submitted CCGrid 2008.

```{=html}
<!-- -->
```
-   Iterative Workflows in Numerical Simulations, Chase, J., Schuchardt,
    K., Chin Jr., G., Daily, J., and Schiebe, T.IEEE 2008 Second
    International Workshop on Scientific Workflows (SWF 2008).
    (Submitted March 2008)

-   Experiences in the Design of Scientific Workflows for Subsurface
    Sciences, Schuchardt, K., Chin Jr., G., Chase, J., Daily, J., and
    Schiebe, T. IEEE Transactions On Automation Science and Engineering.
    (Submitted March 2008)

***Presentations and other activities:***

-   Scientific Workflow Support for AToL Data Analysis & Management,
    Bertram Ludaescher, Shawn Bowers, Timothy McPhillips, Sean Riddle,
    Manish Anand, NSF/AToL meeting, New Orleans, March 8-9, 2008