**The Scientific Data Management ISIC:**

**Lessons Learned and Future Vision**

**Arie Shoshani**

Lawrence Berkeley National Laboratory

Contributors:

Terence Critchlow, LLNL

Rob Ross, ANL

Doron Rotem, LBNL

Nagiza Samatova, ORNL

[Overview 2](#overview)

[The SDM center 2](#the-sdm-center)

[Motivation 2](#motivation)

[The SDM center Framework 3](#the-sdm-center-framework)

[The SDM center accomplishment 4](#the-sdm-center-accomplishment)

[Lessons Learned 6](#lessons-learned)

[What makes an SDM activity successful
6](#what-makes-an-sdm-activity-successful)

[Re-applying SDM technology does work
7](#re-applying-sdm-technology-does-work)

[Technology lessons 8](#technology-lessons)

[Vision: facilitating end-to-end data management
10](#vision-facilitating-end-to-end-data-management)

[Engaging application science communities
11](#engaging-application-science-communities)

[The role of CS basic research 12](#the-role-of-cs-basic-research)

[Summary 12](#summary)

# Overview

The Scientific Data Management Center (SDM) focuses on the application
of known and emerging data management technologies to scientific
applications. The Center's goals are to integrate and deploy
software-based solutions to the efficient and effective management of
large volumes of data generated by scientific applications. Our purpose
is not only to achieve efficient storage and access to the data using
specialized indexing, compression, and parallel storage and access
technology, but also to enhance the effective use of the scientist's
time by eliminating unproductive simulations, by providing specialized
data-mining techniques, by streamlining time-consuming tasks, and by
automating the scientist's workflows. Our approach is to provide an
integrated scientific data management framework where components can be
chosen by the scientists and applied to their specific domains. By
overcoming the data management bottlenecks and unnecessary
information-technology overhead through the use of this integrated
framework, scientists are freed to concentrate on their science and
achieve new scientific insights.

In this document we describe the organization of the SDM center,
summarize successful application of data management technology in
various application domains, and then proceed to describe lessons
learned from this experience. We follow with our vision for future
directions, and recommendations on ways to actively engage the
scientific community to take advantage of the SDM technology.

# The SDM center

## Motivation

Scientific exploration and discovery typically takes place in two
phases: data collection/generation and data analysis. In the data
collection/generation phase large volumes of data are generated by
simulation programs running on supercomputers or collected from
experiments' instruments. This requires efficient parallel data systems
that can keep up with the volumes of data generated. In the data
analysis phase, it is necessary to have efficient indexes and effective
analysis tools to find and focus on the information that can be
extracted from the data, and the knowledge learned from that
information. Because of the large volume of data it is also useful to
perform analysis as the data are generated. For example, a scientist
running a thousand-time-step three-dimensional simulation can benefit
from analyzing the data generated by the individual steps in order to
steer the simulation, saving unnecessary computation, and accelerating
the discovery process. This requires sophisticated workflow tools, as
well as efficient dataflow capabilities to move large volumes of data
between the analysis components.

A typical SDM scenario involves multiple steps, and the invocation of
multiple components that relay on underlying efficient I/O, indexing,
and parallel storage access.

Such a typical scenario is shown schematically in Figure 1.

![](media/image1.wmf){width="5.760236220472441in"
height="3.261023622047244in"}

Figure 1: A typical SDM scenario

For these reasons we use an integrated framework that provides a
scientific workflow capability, supports data mining and analysis tools,
and accelerates storage access and data searching. This framework
facilitates hiding the details of the underlying parallel and indexing
technology, and streamlining the assembly of modules using process
automation technologies.

## The SDM center Framework

The SDM technologies we use fall into three general categories:

-   Storage Efficient Access (SEA) techniques,

-   Data Mining and Analysis (DMA) components, and

-   Scientific Process Automation (SPA) tools.

This led us to the organization of a three-layer framework shown in
Figure 2. In this figure, the SEA layer is immediately on top of
hardware, operating systems, file systems, and mass storage systems. The
SEA layer provides parallel disk access technology (PVFS, ROMIO,
MPI-IO), parallel structured data access (parallel NetCDF), and a
Storage Resource Manager (SRM) -- a software layer to manage multifile
requests from the mass storage system (currently HPSS). On top of the
SEA layer exists the DMA layer, consisting of various indexing (bitmap
index), data analysis (PCA, ICA), and statistical analysis components
(based on the R package), as well as a parallel visualization component
(parallel VTK). This layer also provides an integration framework
(ASPECT) that allows the analysis components to interact. The SPA layer
which is shown on top of the DMA layer provides the ability to compose
workflows from the components in the DMA layer. It also contains the
technology to wrap any components as Web services, thus allowing for a
uniform method to invoke the components as services, including services
available over the Web.

![](media/image2.wmf){width="5.597916666666666in"
height="3.9347222222222222in"}

Figure 2: The Scientific Data Management Framework

From previous experience, we have realized that the concept of layering
components is practical and useful. In particular, hiding the details of
the SEA layer is a desirable feature, as it simplifies the scientist's
task. In addition, providing a flexible framework for specifying a
scientific workflow made of underlying (usually existing) components is
a powerful method for setting up repetitive tasks. This framework also
provides the ability to audit and track the scientific process, which is
essential for verifying the correctness of complex tasks and recovering
from failures. This new direction, not envisioned in the original
proposal, is the basis for the continuation of this work.

## The SDM center accomplishment

Over the last three years we have adopted, improved, and applied various
data management technologies to several scientific application areas. We
chose to concentrate on typical scenarios provided to us by scientists
from different disciplines. By working with these scenarios we not only
learned the important aspects of the data management problems from the
scientist's point of view, but also provided solutions that led to
actual results. The successful results achieved so far include:

-   More than a tenfold speedup in writing and reading NetCDF files was
    achieved by developing Parallel NetCDF software on top of the MPI-IO
    using the General Parallel File System (GPFS) from IBM. A similar
    performance level has been shown when compared to HDF5. This was
    applied to Astrophysics data (FLASH code) as well as Climate
    Modeling simulations.

-   An improved version of PVFS is now offered by cluster vendors,
    including Dell, HP, Atipa, and CRAY.

-   A new method for signal separation of observational data was
    achieved by the use of a combination of Principal Component Analysis
    (PCA) and Independent Component Analysis (PCA). This was used to
    identify accurately El Nino signals in observed data that included a
    volcano signal in a Climate application. The ICA software was
    packaged to be used with other applications. Similar techniques are
    now being applied to a Fusion application for the purpose of
    identifying the key parameters that are relevant to the presence of
    edge harmonic oscillations in a Tokomak.

-   A new specialized method for indexing mesh data using bitmaps was
    used to achieve more than a tenfold speedup in generating regions
    and tracking them over time. The key to this achievement is that
    this method works just as efficiently for selection conditions over
    multiple measures, a problem previously unsolved with any known
    indexing techniques. This bitmap-based indexing method was applied
    to Combustion applications, as well as for indexing over collisions
    (events) in High Energy Physics applications (referred to as the
    Grid Collector).

-   More than a tenfold improvement to VTK for visualizing NetCDF files
    was achieved by the development of a software layer on top of
    Parallel NetCDF. This method was applied to Astrophysics data.

-   An integrated framework, called ASPECT (Adaptable Simulation Product
    Exploration and Control Tool), was developed for simulation data
    exploration using to provide pluggable analysis tools such as PCA,
    ICA, bitmap indexing, and a suite of statistical tools based on the
    R package. This framework is being applied to a Terascale Supernova
    Astrophysics application.

-   A workflow system was adopted and enhanced for scientific
    applications including access to Web-based services and databases.
    This system is designed to streamline repetitive scientific
    workflows, such as running simulations over multiple time steps.
    This system was applied to a Biology application for the analysis of
    microarray data, a process that requires a series of component
    invocations over the Web, and has to be repeated hundreds of times.
    Automation was shown to increase productivity by as much as an order
    of magnitude, a result that would have been impossible to attain
    without automation of the scientific workflow.

# 

# Lessons Learned

## What makes an SDM activity successful

We consider an activity of the SDM center to be successful if it results
in the use of scientific data management technology by application
scientists to enhance their productivity, and to enable new science that
could not be done previously. For example, a high energy physicist who
is looking for some rare events needs to sift through terabytes of data
and millions of files to find the relevant events. Providing an
efficient index over billions of event objects based on their properties
(energy, number of particles produced, etc.) can cut down the search
from weeks to hours. Similarly, the ability of a scientist to interact
with a workflow to inspect the way a large simulation is progressing can
eliminate unnecessary computation and the waste of the scientist's time.
We have developed such technology and applied it as part of our work in
the SDM center.

What made such successful projects possible? The key is a close
collaboration with the application scientists, and the extra work
required to embed the technology in the scientists' framework. It is not
enough to have the right technology. Because the scientific problems and
the technology involved are quite complex, a close collaboration is a
must. Furthermore, because application scientists are usually busy with
their own activities, and can devote little time to such collaborations,
it is best to have joint projects where the application scientist is
actually funded to perform these joint tasks.

A case in point was a collaboration with the STAR experiment to apply a
specialized bitmap indexing technology, called FastBit. It was necessary
to embed this technology into the STAR framework before scientists could
use it. Furthermore, another technology had to be used to make sure that
the objects selected by the index are brought in from tertiary storage.
We have used Storage Resource Managers (SRMs) for this purpose, a
component developed under the Collaboratories SciDAC program. Most
importantly, there was a STAR experiment scientist funded at half time
to work with us in order to embed the components into the STAR
framework. It is this type of intensive focused activity that brought
this to fruition, and resulted in SDM components to be part of the STAR
production system. This already allowed several physicists to explore
data much faster than they could perform previously.

This experience has repeated in every domain of our activities. Only a
close collaboration with the scientists working with the Astrophysics
Flash codes, brought about their use of the parallel NetCDF technology.
Working closely with Astrophysicists and a biologist brought about the
use of our scientific workflow technology, called Kepler. Working
closely with climate modelers and fusion scientists brought about
discovery of features not understood previously. We had similar
experience in other domains, such as applying parallel technology to
ScaLAPack in the R framework.

In general, we observe that there are four stages of activities in the
center: development, adaptation, integration, and deployment. In each of
these stages, it is crucial that we have an environment that fosters
collaboration. In the development stage we need to get first hand
feedback of which problems are important to work on. In the adaptation
stage we need to collaborate to apply the technology to a particular
domain. In the integration stage, it is essential to have an application
scientist involved in putting the technology into existing frameworks
for that application's domain. In the deployment phase, only the
application scientist can make the case of the usefulness of the SDM
technology and demonstrate his/her successes as examples. Also, in this
stage, ways to simplify the use of the tools may need to be developed.

We note that the four stages mentioned above: development, adaptation,
integration, and deployment, are typically cyclical. Each stage can
present new problems that can generate new requirements to previous
stages.

## Re-applying SDM technology does work

One of the important lessons we learned is that developing generic SDM
technology (the Computer Science approach) pays off. One can always
develop specialized solutions to a specific problem, perhaps more
quickly, but such technology cannot be easily changed or re-applied to
other problems. Practically, all the software technology we have
developed or adapted for use in the SDM center is generic, and was used
to apply to multiple application domains.

Examples of such re-application of technology include:

-   Parallel netCDF -- was developed initially with an Astrophysics
    example (the FLASH code). After the advantages of this approach were
    shown to the FLASH team over using HDF5, the scientists switched to
    using parallel netCDF, by making changes to their code. Now, this
    approach is being considered for Climate simulations, as well as a
    possible option for AMR codes in APDEC

-   FastBit -- was initially developed for a High Energy Physics
    application. It was later applied to a combustion application,
    leading to the use of bitmaps for region growing and region tracking
    over time. It was then applied to triangular representation of
    visualization surfaces by ORNL to speed up remote visualization. It
    is now being used for query-based visualization in several
    application domains.

-   Scientific workflow -- was first identified as a need in a biology
    application of processing microarray data. It led to the adaptation
    of an existing tool called Ptolemy (from UC Berkeley) to develop a
    scientific workflow engine, called Kepler. Kepler was then applied
    to complex workflows in the Astrophysics domain.

-   Feature selection -- the use of analysis tools to identify the
    predominant features in a high-dimensional space was first used in
    the SDM center for a Climate Modeling problem. Similar techniques
    were used for analyzing Fusion experimental data, and are now being
    applied to Fusion simulation data.

-   Parallel R and Parallel VTK -- the parallelization of these codes
    made it possible to perform statistical analysis and visualization
    on clusters, and was applied to several application domains.

-   Storage Resource Managers -- SRMs were developed initially for
    distributed applications, and applied to High Energy Physics and
    Earth Science. Later, this technology was used in conjunction with
    FastBit for facilitating High Energy Physics analysis on a single
    site or on distributed sites. It is now being used in conjunction
    with MPI-IO for seamlessly moving data from/to mass storage systems.

The key point about these experiences is that generic technologies can
be applied successfully to multiple application domains, but there is
significant work required to adapt the technologies to the different
domains and to their specific frameworks. The re-application and
adaptation of generic technology is one of the important foundations of
the SDM center. The alternative of developing specialized solutions is
not scalable and too rigid.

## Technology lessons

The technology we use evolved over time based on experience of working
with various scientific applications. We list below the technology areas
that we believe should be emphasized in future work of the SDM center.
Some enforce our previous directions and some suggest new or expanded
technology areas.

#### Scientific workflow

When we started the work in the SDM center, the support of scientific
workflow was not part of our technology. Over time, this need became
evident, and we worked collectively to find a solution. The result was
the adaptation of an open source tool that we could adapt for the needs
of the scientific community. The advantage of this approach is that it
allows us to add new specialized components into the basic workflow
engine, and adapt it to various applications.

An important lesson learned is that scientific workflows require the
coordination of computational tasks as well as the *data flow* between
the components of the workflow. Scientific applications can be data
intensive, and the coordination of the data flow is required, especially
for long lasting workflows. Another lesson was that scientific workflows
are special in that they repeat in small granularity, such as time-steps
of a simulation. This required a specialized workflow technology.

As for technology that is yet to be developed, we have identified
tracking of the workflow progress and the estimation of the total time
to complete a workflow important and difficult requirements. In the long
run, it is essential to find ways to estimate (at least time to
completion) and finding out dynamically the progress of the workflow
process.

#### Efficient I/O is often the bottleneck

There is a lot of emphasis on computational efficiency and parallel
computing. However, large scale scientific applications are also data
intensive, and often the data movement from/to the computation engine to
the storage systems is the bottleneck.

We have a solid foundation on work in the parallel I/O area, and this
needs to be sustained and expanded in two directions. The first is the
use of hints that applications can provide on the intended use of data.
That can provide the I/O system the ability to optimize the data
organization and the data access. The second is the seamless movement of
data from mass storage systems. Today's technology and cost requirements
still dictates the use of tape storage devices. We believe that even
with better and faster storage systems, there will still be the need for
secondary and tertiary storage. Thus, we need to continue to pay
attention to this aspect.

#### General analysis tools

Our experience with applying general analysis tools to domain specific
problems is quite successful. New insights into the features of observed
or simulated data were achieved, even beyond insights of experienced
scientists in their domains.

Visualization is an important part of analysis. We believe that we need
to integrate visualization into the set of analysis tools. In
particular, we could use indexing tools to focus on the relevant
query-based data before visualization. Similarly, visualization can be
used effectively in conjunction with feature tracking tools.

#### Data movement: is not just the network bandwidth

There is the general perception that if the network bandwidth is large
enough data movement will be a solved problem. In reality, all systems
are unreliable, and even with fast networks data can be lost or
corrupted, especially when the volume of data is large. Unlike viewing a
streaming video, losing or corrupting scientific data is unacceptable.

Robust data movement requires the software that makes sure data arrives
uncorrupted, and can recover from transient failures. When performing
data movement, failures can occur not only on the network, but also
in/out of the storage systems. Thus, a software layer that monitors the
data movement and ensures their integrity is essential.

Another aspect for supporting data movement is ensuring that there is
enough space to move large volumes of data into. There are currently
many storage systems that have replicated data that were never cleaned
up. Automatic tools for space management based on lifetime management
can help this problem. We will need to use such technology in the
center.

Finally, data movement between scientific components may encounter a
mismatch of the data formats between the generated and the expected data
formats of the components. We need to adapt tools that support the
automatic transformation of data. This is an area we have only started
to address in the center.

#### Metadata management

As the volumes of experiments and associated data grows, so does the
information about them. What was the procedure that was used to generate
a dataset? When was it generated? What were the assumptions/parameters
used? Was the dataset derived from another dataset? These are some
important information items that must be recorded and made available to
the scientist. This is referred to as the metadata problem. One of the
more difficult aspects of the metadata problem is the provenance; that
is, the tracking of lineage or history of each data object.

Fundamentally, the metadata can be managed and organized as a database,
but the structure of the metadata needs to match the applications.
Therefore, one needs to use technology that is easily adaptable. That
is, given a description of the metadata structure, all the appropriate
data entry, browsing and query tools, need to be automatically
generated. This is an area of future emphasis for the center.

# Vision: facilitating end-to-end data management

How can we achieve maximal impact of the SDM center on the scientific
community? One approach is the "shrink-wrap" approach. Accordingly, each
SDM technology component should be wrapped to be self-explanatory and
readily available to be downloaded and installed. From our experience,
this approach does not work well with large scale, data intensive,
complex scientific applications. We need an alternative model.

While we have a few examples that wrapping specific components can be
useful for specific tasks (such as installing a parallel virtual file
system on a cluster, or downloading parallel-R) the problem a scientist
faces is the assembly of all the necessary components on his/her
specific hardware, and linking then into a single cohesive solution.

We believe that we need to address end-to-end requirements for
scientific scenarios. The idea is to work with application scientists on
specific scenarios, such as the data generation stage or the analysis
stage, and address all the data management problems for each scenario.
The scenario should be of general interest to the application community,
so once such a system is successfully developed, other scientists from
that community could take advantage of it, perhaps with some small level
of adaptation.

To achieve this vision, the center should be set up to use not only
technology available at the center, but also be willing to use other,
external technology if necessary. Most likely, any such external
technology may have to be adapted/modified to the needs of the scenario.
The advantage of this approach is that the center is flexible in the
technology it uses, but more importantly provides all the end-to-end
data management needs to the application scientist. We note that for a
given scenario, only the subset of the relevant components from the SDM
center framework need to be used.

In terms of supporting technology for this vision, it is necessary that
we strive to have all the SDM components callable in a standardized way
from the workflow engine. This is one of the goals of the Common
Component Architecture ISIC, and we should work with them to achieve
this goal. The supporting workflow technology should also be able to
launch tasks provided by the scientist, and permit dynamic interaction
with the workflow. We also need to have data movement technology
available at the center to support this vision.

# Engaging application science communities

We summarize below the main concepts that we believe are necessary to
engage scientific communities, not just an individual scientist at a
time.

-   Start with a close collaboration with application scientists on
    scenarios of general interest

-   Adapt the Center's technology to application domain as appropriate

-   Use external technology if necessary to support the end-to-end
    scenario

-   Work with application scientists that have used the technology to
    expose other scientists in their domain to such end-to-end
    solutions.

The key issue is how to get application scientists engaged in such
relatively long term (1-2 years) joint activities. We believe that such
joint activities should support the funding of the application scientist
as well, so that he/she becomes part of the joint project. By being
funded as part of a joint project, the scientist will consider the
success of such a project a central goal of his/her work, rather than a
peripheral activity. Our experience bears this out. As mentioned
previously, in the case of the STAR experiment, a physicist working
closely with us made it possible to integrate the SDM components into
the STAR experiment framework.

The goal is to embed SDM solutions in scientific packages or frameworks.
Examples are the framework of an experiment (such as the STAR
experiment), a mathematical package (such as APDEC), or widely-used
analysis software (such as ROOT in HENP).

We recommend that funds for supporting application scientists in joint
projects should be designated for this purpose and separate from the SDM
center's funding. They should be made available based on identified
needs, and controlled by the SDM center through an advisory board. Such
funds should also be used by the application scientist to expose his/her
community to the successful SDM end-to-end systems.

Another way to generate such funding could be from the DOE application
offices, by having the SDM center participate in application-side
proposals.

# 

# The role of CS basic research

Technology development can be viewed as performed in 4 stages:

-   Research

-   Prototype

-   Product

-   Infrastructure

We see the role of the CS basic research as providing the
research-to-prototype activity. This activity can take more risks and
experiment with novel ideas, mainly because there is no pressure to
deliver a product to waiting scientists. We consider basic research as
essential to generating new ideas. Successful research results lead to
new technology that can be funneled into SciDAC projects.

SciDAC activities should aim to take prototype-to-product. In the
process of doing so, some changes and adaptation may be required. More
importantly, SciDAC activities should provide the basic research
activities with meaningful problems that arose in making a technology a
product to be used in real systems.

The third step of taking product-to-infrastructure is an activity that
requires heavy involvement of the application scientists. It is typical
that in order to be part of an infrastructure a product must be
installed as part of the infrastructure software by default. This
typically happens when key members of the application community are
involved. This type of activity may require Postdocs or people assisting
in managing the framework to be funded.

# Summary

The key points made in this document are:

-   SDM technology can successfully be applied across multiple
    scientific applications

-   Close collaboration to establish *end-to-end* solutions is essential

-   Application scientists must be involved in incorporating the
    technology into existing frameworks and infrastructures (messengers
    of good news)

-   We recommend having a flexible funding structure to support
    application scientists on collaborative projects

    -   Level of such funding structure should be in addition to the SDM
        center funding (about 20% of level of center's funding)

    -   Funding structure to be managed by SDM center using advisory
        board

-   We recommend that the SDM Center joins application-side proposals
    when possible

```{=html}
<!-- -->
```
-   Ability to use funds where technology is needed

-   May need help from OASCR to make such connections