High-Speed Access for an NVO Data Grid Node

Maria A. Nieto-Santisteban, Aniruddha R. Thakar, and Alexander S.
Szalay, William O'Mullane, Tamas Budavari, George Fekete and Nolan Li,
Johns Hopkins University

We had identified the following objectives in last year's annual report
for this project:

1.  Build a parallel compute cluster using the Condor software.

2.  Add VOTable compatibility to our web services.

3.  Implement registry of our Web Services using emerging NVO registry
    standards.

4.  Investigate parallelization of data access to SQL databases using
    distributed partition views (DPVs) and other techniques.

5.  Design and build a Grid portal for our data access component.

We have met the first three of these objectives and are close to
achieving the fourth.

-   Our Condor cluster is in operation and available to the GriPhyN
    community on the IVDGL grid -- Grid3.

-   We have added VOTable compatibility to the following services on
    [http://www.voservices.net](http://www.voservices.net/):

    -   SIAP -- Simple Image Access service

    -   Cone Search -- radial search service

    -   Spectrum -- access to over 500k spectra from SDSS DR1 and 2dFGRS

    -   Filter Profile -- access to a database of nearly 100 photometric
        filters

    -   CASService -- access to the SDSS Catalog Archive Server (CAS)

    -   VO-enabled Mirage -- enhanced Mirage functionality with VOTable

    -   CasJobs -- batch query service for the SDSS CAS.

-   We have also set up a VO registry at JHU (also available on
    [www.voservices.net](http://www.voservices.net/)) that implements an
    OAI interface and harvests OAI repositories.

-   We have designed an algorithm that performs spatial partitioning of
    the SDSS catalog database and uses the partitioned data to speed up
    an astronomical galaxy cluster finding algorithm. This effort is
    described in more detail below.

# Finding BCGs with Partitioned SDSS Data

Using a zone based approach \[Gray et al.\], data get distributed among
several servers allowing parallel execution of the same application on
different partitions. We have developed a SQL version of the Finding
Galaxy Cluster algorithm implemented by Jim Annis \[Annis et al.\] with
the goal of testing and benchmarking our partitioning approach. This
algorithm identifies galaxy clusters in the data and finds the Brightest
Cluster Galaxy (BCG) in each cluster. The original version was running
on the Terabyte Access Machine (TAM), a grid cluster set up in Fermilab
composed of 10 computers. We have demonstrated that using databases
technologies and a 3-node SQL server cluster we can process the same
area of the sky 20 times faster and with higher accuracy.

## Finding Galaxy Clusters 

The algorithm includes five main steps:

1.  **Get Galaxy List**

*fieldPrep*: Extracts from the main data set the measurements of
interest.

2.  **Filter**

*brgSearch*: Calculates the unweighted BCG likelihood for each galaxy
(unweighted by galaxy count) and discards unlikely galaxies.

3.  **Check Neighbors**

*bcgSearch*: Weights BCG likelihood with the number of neighbors.

4.  **Pick Most Likely**

*bcgCoalesce*: Determines whether a galaxy is the most likely galaxy in
the neighborhood to be the center of the cluster.

5.  **Discard compromised results**

*getCatalog*: Removes suspicious results and produces and stores the
final cluster catalog.

### TAM Implementation

The implementation running on TAM approaches this problem by dividing
the sky in 0.25 deg^2^ patches and processing each of them separately on
a single CPU. This implies creating 2 files per each patch, one 0.5 x
0.5 deg^2^ TARGET to compute the likelihood of each galaxy and another 1
x 1 deg^2^ BUFFER to look for neighbors within some radius. Ideally,
BUFFER should cover 1.5 x 1.5 deg^2^ to find all neighbors within 0.5
deg and weight properly the likelihood of each galaxy to be the center
of the cluster. However, the system cannot afford such a big buffer and
only 0.25 deg is provided.

In order to determine the search radius, the algorithm uses a lookup
correction table mapping magnitudes, redshifts, and searching distances.
This table has 100 rows with redshift values between 0 and 1 in steps of
0.01. A **Candidates** file, C, is produced after this step.

In order to determine which galaxy is the center of the cluster, nodes
must interchange information about candidate neighbors. Ideally, each
node should know about all the surrounding candidate neighbors within
0.5 deg, but once again TAM can only afford considering four (N, S, E,
W) and discard the candidate neighbors on the corners (NE, NW, SE, SW)
\[Figure 1\]. Resolving a TARGET field of 0.25 deg^2^ requires about
1000 s on each of the 600 MHz PIII that compose the TAM cluster. The
time scales lineally with the number of fields being computed. TAM is
composed of 10 machines processing 0.5 x 0.5 deg^2^ fields in parallel.

  -----------------------------------------------------------------------

  Figure 1. Surrounding C files provide a searching radius close to 0.5
  deg for candidates near the border. Corners are not considered due to
  performance issues.
  -----------------------------------------------------------------------

### Parallel SQL Server

We have implemented the same algorithm (Steps 1-4) and included two main
improvements:

-   we use a finer lookup table with redshift steps of 0.001 (vs 0.01
    for TAM);

-   we provide the 0.5 deg buffer surrounding the target field that
    would be prohibitively expensive in the TAM version.

TAM would require 20--30 times longer in processing the problem with
these two improvements.

The SQL implementation can run either on a single CPU (SQL Server) or a
cluster. When running in parallel, the data distribution has been
arranged so each server is completely independent from the others. We
achieve this by duplicating some data and processing on different
servers. The duplicated computation is insignificant compared to the
whole work involved when processing big volumes of data, or
equivalently, big areas of the sky.

![](media/image1.png){width="5.544488188976378in"
height="4.927952755905512in"}

  -----------------------------------------------------------------------
  Figure 2. Data distribution among 3 SQL Servers. Total duplicated data
  = 4 x 13 deg^2^. Total duplicated work (1 object processed more than
  once) = 2 x 11 deg^2^.

  -----------------------------------------------------------------------

We have processed a target area of 66 deg^2^ \[Figure 2.\] and compared
the performance between the SQL Server cluster and TAM \[Table 1\].

+-----------------------------------+-----------------------------------+
| **SQL Server cluster**            | **TAM cluster**                   |
+-----------------------------------+-----------------------------------+
| Resolve a Target of 66 deg^2^     | Resolve a Target of 66 deg^2^     |
| using 3 SQL Servers dual 2.6 GHz. | using 10 nodes 600 MH PIII.       |
|                                   |                                   |
| For 0.5 deg buffer and z-steps of | For 0.25 deg buffer and z-steps   |
| 0.001:                            | of 0.01:                          |
|                                   |                                   |
| **T = Max (T~S~) = 2 h 15'**      | **T~0.25,\ 0.01~ = 7.3 h**        |
|                                   |                                   |
|                                   | Scaling to 0.5 deg buffer and     |
|                                   | z-steps of 0.001:                 |
|                                   |                                   |
|                                   | **T~0.5,\ 0.001~ = 7.3 h \* 25 =  |
|                                   | 7.6 days**                        |
|                                   |                                   |
|                                   | Dividing by 4 to adjust the CPU   |
|                                   | speed:                            |
|                                   |                                   |
|                                   | **T ~0.5,\ 0.001~ \~ 2 days**     |
+-----------------------------------+-----------------------------------+
| Table 1. Using a SQL Server grid  |                                   |
| we solve the same problem **20 x  |                                   |
| faster**                          |                                   |
+-----------------------------------+-----------------------------------+

# Future work

In addition to completing the remaining objective from last year's list
-- providing a grid interface to the partitioned data so that it is
accessible to grid applications -- we aim to pursue the following steps
towards developing a general purpose scalable data access facility.

-   Generalize the algorithm to deal with empty areas and boundaries
    (Step 5).

-   Set up the system to be able to register and manage tasks and
    transactions so if the process fails we don't have to reprocess the
    whole database.

-   Experiment with spatial partitioning in the RA axis. Current work
    has been done using the declination axis. However, it seems the RA
    axis would allow us to do bigger areas in parallel.

-   Incorporate the partitioning algorithm into our data import
    pipeline, sqlLoader, so that the data is partitioned as it is loaded
    into the database.

# References

\[Annis et al.\] *Applying Chimera Virtual Data Concepts to Cluster
Finding in the Sloan Sky Survey*. Annis, J., Zhao, Y., Voeckler, J.,
Wilde, M., Kent, S. and Foster, I., in SC\'2002, (2002).

\[Gray et al.\] *There Goes the Neighborhood: Relational Algebra for
Spatial Data Search* Jim Gray; Alexander S. Szalay; Aniruddha R. Thakar;
Gyorgy Fekete Fekete; William O\'Mullane; Gerd Heber; Arnold H. Rots;
Maria A. Nieto-Santisteban. MSR-TR-2004-32.