PACI DATA QUEST Expedition

**Quarterly report (January 1,2003 -- April 1,2003)**

### Expedition Leads: 

Ian Foster, ANL <foster@mcs.anl.gov>

Robert Brunner, NCSA <rb@ncsa.uiuc.edu>

### Coordinator

Veronika Nefedova, ANL <nefedova@mcs.anl.gov>

### Site/Subproject leads

Tom DeFanti, UIC

Miron Livny, **U.Wisconsin-Madison**

Ray Plante, NCSA

Natalia Maltsev, ANL

Nagiza Samatova, ORNL

### Partners

Bob Wilhelmson, MEAD

Dennis Gannon, Portals

Rob Gardner, U. Chicago

Paul Avery, U. Florida

Rick Cavanaugh, U. Florida

Mike Wilde, ANL

Jens Voeckler, U. Chicago

OVERVIEW

**Radio astronomy (Ray Plante, NCSA)**

The target community here is the users of the VLA/EVLA and BIMA/CARMA
radio telescopes. The BIMA Image Pipeline, now being deployed at NCSA,
automatically calibrates and images radio interferometer data as it
arrives from the BIMA telescope in northern California. The initial
version was implemented to access data from the BIMA Data Archive at
NCSA and distribute the computing between a single serial platform and
the NCSA SGI Origin system. As part of this expedition, we plan to
generalize the pipeline to use general Grid services to manage data and
access compute resources. By incorporating these generic services, we
can use the Grid with other data resources and pipelines. In particular,
we plan to establish an instance of the NRAO End-to-end system at NCSA
for processing VLA data. The distribution of data and computing between
NCSA and NRAO will make a good testbed for extension to the TeraGrid. We
can make real practical use of the distributed resources and users when
the BIMA Array combines with the Caltech OVRO Array to create the CARMA
telescope in 2005.

**Bioinformatics (Natalia Maltsev, ANL; Nagiza Samatova, ORNL)**

The first and most crucial step in genome analysis is the accurate
assignment of function to the genes. This requires the capability to
perform high throughput genetic sequence analysis based on comparative
analysis of multiple genomes, and is essential for the development of
applications in functional genomics and systems biology. However, such
factors as exponential growth of the information in the public sequence
databases, and the development of sophisticated and computationally
intensive tools and algorithms for analysis of genomes dictate an urgent
need for the development of the new approaches for high-throughput
computations to support the bioinformatics effort.

In support of this work we propose to apply existing virtual data
techniques to the labor-intensive process of gathering genomic data from
multiple sources, reformatting that data, and transforming it for
insertion into the ANL sequence database, and newly developed virtual
data techniques that extend virtual data tracking into the realm of
relation data provenance, to enable the recording and tracking of
significant portions of the processes of comparative analysis of genomes
and metabolic reconstructions from sequence data.

Algorithms Development. We will develop the following tools and
algorithms that would allow increasing precision and reliability of
high-throughput analysis of the genomes:

a.  Tools and computational infrastructure for adapting advanced
    TeraGrid based technologies for the needs of high-throughput genome
    analysis

b.  Clustering algorithms for partitioning protein space into protein
    functional families

c.  Algorithms for automated assignment of functions to the

Developed algorithms will enhance analysis of the genomes in a framework
of the public genome analysis server and will be available for the use
by the wide scientific community.

**Virtual astronomy (Robert Brunner, NCSA)**

The aim of this work is to utilize Grid-based data management for
various astrophysical data archives. This effort would first work to
integrate several public astronomical datasets, including SDSS (EDR
only), DPOSS, 2MASS, NVSSS, FIRST, and ROSAT within the NCSA domain.
Once the catalog datasets have been integrated into the Grid, we will
develop and deploy several archive services, including complex spatial
queries and data federation functionality. Different commercial and open
source data-mining toolkits will be tested on the federated datasets. We
will also integrate image datasets within the NCSA domain, requiring
several tens of Terabyte of near- and on-line storage. Finally, we will
work to extend this work to the currently planned QUEST2 survey, which,
once operational, will generate a Terabyte of new data every few months.
QUEST2 requires high-speed data transfer from SDSC (where the data
arrives from the mountain) and complex, on-demand processing for rapid
variable identification. Data management, transport, and access are
critical to the Virtual Astronomy effort, which is developing and
applying data mining technologies over the infrastructure being
developed by the NSF funded NVO project. The major tasks being tackled
by virtual astronomy can be categorized into three areas: data
transport, data management, and data mining. In addition to our
contributions to the PDQ Expedition presence at the upcoming Alliance
All Hands Meeting, three posters will be displayed that detail how
Virtual Astronomy group is beginning to deal with large astronomical
datasets.

**Mesoscale weather (Bob Wilhelmson, NCSA)**

We will work with the [MEAD
expedition](http://www.ncsa.uiuc.edu/expeditions/MEAD/) to apply PDQ
technologies to MEAD problems. In a first stage, we will apply basic
data movement and replica management technologies. In a second, we will
apply the Chimera virtual data system. Details can be found in the MEAD
expedition proposal.

**Integration of Optical Networks with Data Grid Tools (Tom DeFanti,
UIC)**

[UIC will support the execution of Alliance data-intensive applications
over high-speed optical networks, including I-WIRE (within Illinois) and
through StarLight (for international connections). Work to be done will
include: enhancements to, and integration with Data Grid tools, of UIC
Quality of Service (QoS) Adaptive Networking Toolkit ("Quanta"), which
enables application developers to utilize the extremely high-bandwidth
afforded by optical networks via support for Generalized Multi Protocol
Label/Lambda Switching and Optical Border Gateway Protocol; support for
application projects running over optical networks, via direct
partnerships with application groups. Work with Reliable Blast
UDP.]{.mark}

[UIC will also support the deployment of storage capabilities that
support overlay network services for bulk data transfer at StarLight.
These capabilities will be based on Network Storage (NeST) appliances
and store-and-forward middleware developed by the Condor Project at the
University of Wisconsin-Madison.]{.mark}

**Milestones, Deliverables and Status**

Here is the short (table) version of the report:
[**http://www-fp.mcs.anl.gov/pdq/schedule.htm**](http://www-fp.mcs.anl.gov/pdq/schedule.htm).
The long version is below.

**Accomplishments**

1.  The group has produced a number of internal documents that specify
    the basic requirements for data movement and replication.

2.  The development of new requested Data Grid capabilities has been
    finished at ANL: Virtual Data Toolkit, Reliable File Transfer, some
    GridFTP features.

3.  The important Bioninformatics computations were performed using new
    tools with an average speedup of x5.

4.  The group holds regular AG meetings to discuss the progress as a
    whole as well as in the specific technical areas.

5.  Starlight clusters are ready and enabled on the network.

> **Bioinformatics (ANL, ORNL)**

-   Further development of GADU (Genome Analysis and Database Update
    pipeline) (A.Rodrigez, D. Sulakhe)

    -   Using GADU we have analyzed the sequence data from 106 publicly
        available genomes

    -   Installed Chimera and ran examples on ANL systems Jazz and Data
        Grid Cluster

    -   Did a production run on 60 genomes using new Grid Tools
        (Chimera) and achieved a x5 speedup in computations.

-   Tools development

    -   SVMMER -- Continued working on a Support Vector Machines
        algorithm-based tool for automated functional classification of
        protein sequences. In collaboration with NCSA (Eric Jacobsson)
        developed a preliminary support for classification of
        transmembrane protein sequences.

    -   PAT (Pattern Analysis Tool) (ORNL) -- We have developed an
        automated web-based tool for comparative analysis of genomic
        sequence domains and motifs with protein surface maps,
        interfaces, and substructures. Developed a library of tools for
        analysis of surface patches in proteins and protein complexes.

    -   Prepared a demonstration of a uniform pipeline of protein
        sequence analysis tools (PhyloBlocks, BloBlah, SVMMER and PAT)
        at the NCSA All Hands Meeting, at Utah, April 30 -- May 2, 2003.

> **Radio Astronomy**

-   GridFT incorporation. GridFTP has been successfully integrated into
    the BIMA pipeline using GTK3\'s Java CoG for moving data between
    archive and compute platforms. However, the globus installation on
    the platinum cluster is very out-of-date, and we are not able to use
    transmit data there. We have temporary fix using GT2\'s
    globus-url-copy.

-   Installation of BIMA pipeline on NCSA platforms. The installation is
    largely complete; however, we continue to track down problems with
    the platinum cluster (we have work-around for Globus, including ssh
    for job submission and the file-locking problem):

    -   out-of-date Globus installation

    -   non-functional Globus job-submission

    -   NFS file-locking bug in kernel on storage nodes hangs AIPS++

    -   statd occasionally dies (appears to be fixed now).

-   Developing the long-term Data Management plan. This work is in
    progress. In collaboration with experts at Argonne, we have worked
    out best use of tools for handling mirroring.

-   Establishing of an initial mirror of VLA Archive. We have started
    this target early and are getting ready to transmit data. Due to
    limited bandwidth to the VLA archive, we will be seeding the NCSA
    mirror by shipping a terabyte disk array.

> **Virtual Astronomy**

-   In the area of data transport, we have been working with the Condor
    group at the University of Wisconsin on transferring 3 TB of DPOSS
    astronomical imaging data from the SDSC SRB to the NCSA mass storage
    system. The data transport pipeline utilizes a DAG manager to
    automate the data transfer and uses one cache system at SDSC and
    another cache system at NCSA. The pipeline currently utilizes three
    different data transport functionalities, including grid-ftp, and
    will soon incorporate reliable file transfer. Details on the
    performance of the different approaches will soon be available,
    providing an important community resource for tool evaluation.

-   Another effort has been made to move SDSS data from FNAL to NCSA
    mass storage. The first set of data moved is the public SDSS Early
    Data Release (EDR). This data consists of a large number of small
    files, totaling around 500 GB. Initially, wget was tried, and after
    several unsuccessful attempts was abandoned. In each case, around 11
    GB would be transferred before the process failed. The next approach
    tried was rsync. Rsync proved more successful than wget, but still
    had problems, probably due to the large number of small files.
    Currently, rsync has failed to transfer all of the data from FNAL to
    NCSA. We plan on trying RFT directly as the next step, both for the
    EDR data, as well as the SDSS Data Release One (DR1), which is
    around 3TB.

-   On the data management side, we have successfully replicated the
    DPOSS catalog database from Caltech to NCSA. The database utilizes
    Microsoft SQL Server, which is running on a Dell PowerEdge server
    provided by NCSA for this effort. This server is connected to a SAN
    that has 2 TB of disk dedicated to this server. Other datasets that
    will be added to this server include the NVSS, FIRST, ROSAT, 2MASS,
    and SDSS catalogs. Of these, 2MASS just released a new version,
    while SDSS is expected to release a new version this summer. This
    transfer of the DPOSS database proved extremely difficult, as the
    majority of the data transfer tools either were not supported on
    Windows 2000, had 2 GB file size limitations, or had their ports
    blocked for security reasons. In the end, a fake user was created
    and the data moved using ftp from the fake user account.

-   Finally, the last major effort has been the evaluation of different
    data mining technologies. Currently, we have worked with the
    data-to-knowledge product as well as separate tools, including
    Autoclass and FastEM. The D2K product will have a major relase soon,
    at which point we expect to extend our partnership with the D2K
    team. In the meantime, we have been exploring FastEM, and the
    visualization tool Xgobi to understand how quasars cluster in
    multi-dimensional parameter spaces. While not strictly data mining,
    we have developed a data processing pipeline to process the DPOSS
    imaging data. This pipeline is being ported to CondorG, and should
    soon begin operation processing the 3 TB of DPOSS imaging data using
    distributed condor grids. This pipeline not only ingests 3 TB of
    data, but produces another 3+ TB, which must be transported back to
    the NCSA Mass Storage system.

> **MEAD**

-   GridFTP: ANL created a very reasonable plan to extend GridFTP
    functionality, in particular, in the java client.  This has been
    implemented, and we are currently testing.  We have some issues to
    resolve, but on a whole we are on track to complete this work. 
    Minimally, the current modifications should be tested and in the
    production GridFTP libraries by the end of the next quarter.

-   We also have started an interaction with ANL GridFTP experts on
    Teragrid-scale GridFTP data transfers.  From this interaction, we
    have developed a clear understanding of the issues involved, the
    plan to resolve the issues in the new server that is being
    developed, as well as the beginnings of planning to start some
    testbed activities with the new GridFTP server on Teragrid, probably
    in Q4 CY03.  This functionality is critically important to be able
    to deal with Teragrid-scale data, on Teragrid-scale networks, and,
    will be manageable with the clients that are the target for the
    first part of the GridFTP work.  All of this will be important to
    factor in to a larger data plan for Teragrid; we view it as the
    important bedrock that we need to build the data story with, and are
    absolutely happy with the quality of the interactions and results
    from the collaboration.

-   Initial consultations with ESG on Metadata Catalog Services.

> **Integration of Optical Networks with Data Grid Tools**

-   Wisconsin has installed DiskRouter software on StarLight servers and
    has run data transfer experiments between UW Madison, Milwaukee and
    STAR TAP. Results are shown at this link (blue color is total data
    in flow (Y axis in MBPS) into STAR TAP. Green is data from Milwaukee
    and Red is data from UW Madison).
    <http://pumori.cs.wisc.edu/devise/milw_net/mil_uw-startap_4_4_sock.html>

-   Source code for Quanta\'s Reliable Blast UDP protocol has been
    provided to Bill Allcock, who is in charge of GridFTP.

-   Prototype modules have been developed to allow Quanta to directly
    control optical light paths between Calient and Glimmerglass optical
    switches.

-   A prototype IA64 port of Quanta has been completed.

> **Publications:**
>
> None in this quarter (?)