# Presentation: 010650

## Radio Telescope Data Grids

- NCSA Radio Astronomy Imaging

- Ray Plante

- PDQ Expedition Meeting  October 1-2, 2002

- Dave Mehringer

- Daniel Goscha

- Harold Ravlin

## BIMA Image Pipeline

- Data is transferred from the telescope to NCSA in real time
- Data is ingested into the BIMA Data Archive automatically
- Astronomers use Web front-end to search, browse, and retrieve data
- Raw data is automatically processed by the pipeline using AIPS++

- Web Interface

- BIMA
- Data Archive
- (current size: ~800 GB)

-        BIMA
- Image Pipeline

***AIPS++***

## BIMA Image Pipeline

- Data is transferred from the telescope to NCSA in real time
- Data is ingested into the BIMA Data Archive automatically
- Astronomers use Web front-end to search, browse, and retrieve data
- Raw data is automatically processed by the pipeline using AIPS++
*Use Grid technologies to distribute the processing *

- Web Interface

-        BIMA
- Image Pipeline

***AIPS++***

***The Grid***

- BIMA
- Data Archive
- (current size: ~800 GB)

## BIMA Image Pipeline

- Data is transferred from the telescope to NCSA in real time
- Data is ingested into the BIMA Data Archive automatically
- Astronomers use Web front-end to search, browse, and retrieve data
- Raw data is automatically processed by the pipeline using AIPS++
*Use Grid technologies to distribute the processing *
*Expedition Application Driver for PDQ, Portals, & Community Codes*

- Web Interface

-        BIMA
- Image Pipeline

***AIPS++***

***The Grid***

- BIMA
- Data Archive
- (current size: ~800 GB)

## Beyond the BIMA Image Pipeline: CARMA

- CARMA = Combined Array for Millimeter Astronomy
- A combination of the BIMA and CalTech arrays at a 3rd higher-altitude site (2003-2004)
- Data rate will increase by at least a factor of 4
- BIMA:      Raw data ~ 0.5 GB/day
-               Processed data 0.75 – 4 GB/day
- CARMA:  Raw data ~ 4 GB/day
-               Processed data  1 – 16 GB/day
- TeraGrid application: distributing data and processing between CalTech and NCSA.
- Partial mirrors at other consortium institutions
- (UC Berkeley & University of MD)
- BIMA Image Pipeline system will be used to process the data

## Grid Drivers

**Characteristic Features of the BIMA Grid application**
**Fully-Automated Processing**
- There is no user initiating action
*processing in response to new data arriving at the archive*
**Robust to machine/network outages/bottlenecks**
*when services return, system must pick up where it left off without human intervention*
**Easily monitored**
*web-based;  connecting to running (or dead) service*

***Radio Astronomy Imaging***

## Networking within the National Radio Astronomy Observatory (NRAO)

## VLA Archive & Pipeline Mirrors

- NRAO End-to-End (e2e) Project: archive & pipeline system for NRAO telescopes
- Desire archive mirrors for access by NRAO scientists
- Archive mirror at NCSA for outside access & processing on Alliance platforms (using AIPS++)
- Plan to establish mirror of the VLA pipeline processing system at NCSA

**Very Large**
**Array (VLA)**

**Greenbank**
**100 m**
**Telescope (GBT)**

**Very Long**
**Baseline**
**Array (VLBA)**

**NCSA**
**Urbana, IL**

**Array Operations Center**
**Socorro, NM**

**NRAO/UVA**

- 13 TB*

- 25 TB*

- 3 TB*

- *by 2005

- Abilene

- Frame-Relay
- Intranet

- T1

- T1

- T1

## Expedition Milestones

- 3.0
- 6.0
- 9.0
- 15.0
- 24.0

- Complete installation of basic Globus servers on pipeline platforms
- Complete initial integration of generic grid data management tools into pipeline system.  Grid tools should be used to access and move input and output data using GridFTP
- Integrate generic procedures for grid job submission and monitoring into Pipeline.
- Establish initial mirror of VLA archive using Globus replication management tools
- Demonstrate distributed data processing between NCSA and NRAO

## BIMA Pipeline Components

***Radio Astronomy Imaging***

**Archive System**

**Event Server**

** ****Ingest**
**Engine**

**Script Generator**

**Queue Manager**

**    ****Data**
**Manager**

*Signal the arrival*
*of new data*

*Determine what *
*processing can take *
*place.*

*Match data to *
*processing recipes*

*Submit and monitor*
*jobs on multiple *
*platforms*

**Serial**

**Parallel**
**    ****SM**

**   ****Grid**
**Platform**

*4-processor*
* **Linux box*

*Origin 2000*

*NCSA Linux*
*   **Clusters*

## BIMA Pipeline Components

***Radio Astronomy Imaging***

**Archive System**

**Event Server**

** ****Ingest**
**Engine**

**Script Generator**

**Queue Manager**

**    ****Data**
**Manager**

*Signal the arrival*
*of new data*

*Determine what *
*processing can take *
*place.*

*Match data to *
*processing recipes*

*Submit and monitor*
*jobs on multiple *
*platforms*

**Serial**

**Parallel**
**    ****SM**

**   ****Grid**
**Platform**

*4-processor*
* **Linux box*

*Origin 2000*

*NCSA Linux*
*   **Clusters*

**3.0 Milestone:**
**Complete installation of basic **
**Globus servers on pipeline **
**platforms**

## BIMA Pipeline Components

***Radio Astronomy Imaging***

**Archive System**

**Event Server**

** ****Ingest**
**Engine**

**Script Generator**

**Queue Manager**

**    ****Data**
**Manager**

*Signal the arrival*
*of new data*

*Determine what *
*processing can take *
*place.*

*Match data to *
*processing recipes*

*Submit and monitor*
*jobs on multiple *
*platforms*

**Serial**

**Parallel**
**    ****SM**

**   ****Grid**
**Platform**

*4-processor*
* **Linux box*

*Origin 2000*

*NCSA Linux*
*   **Clusters*

**6.0 Milestone:**
**Complete initial integration of **
**generic grid data management **
**tools into pipeline system.**

**3.0 Milestone:**
**Complete installation of basic **
**Globus servers on pipeline **
**platforms**

## BIMA Pipeline Components

***Radio Astronomy Imaging***

**Archive System**

**Event Server**

** ****Ingest**
**Engine**

**Script Generator**

**Queue Manager**

**    ****Data**
**Manager**

*Signal the arrival*
*of new data*

*Determine what *
*processing can take *
*place.*

*Match data to *
*processing recipes*

*Submit and monitor*
*jobs on multiple *
*platforms*

**Serial**

**Parallel**
**    ****SM**

**   ****Grid**
**Platform**

*4-processor*
* **Linux box*

*Origin 2000*

*NCSA Linux*
*   **Clusters*

**6.0 Milestone:**
**Complete initial integration of **
**generic grid data management **
**tools into pipeline system.**

**9.0 Milestone:**
**Integrate generic procedures for **
**grid job submission and monitoring **
**into Pipeline.**

**3.0 Milestone:**
**Complete installation of basic **
**Globus servers on pipeline **
**platforms**

## Expedition Milestones

- 3.0
- 6.0
- 9.0
- 15.0
- 24.0

- Complete installation of basic Globus servers on pipeline platforms
**install on archive on archive server & serial processing platform**
**need availability on Grid platforms**
- Complete initial integration of generic grid data management tools into pipeline system.  Grid tools should be used to access and move input and output data using GridFTP
**reimplement data manager using GridFTP (C or Java)**
- Integrate generic procedures for grid job submission and monitoring into Pipeline.
**will use Java COG kit to handle job submission**
**in collaboration with ****Portals expedition**
- Establish initial mirror of VLA archive using Globus replication management tools   **transfer to other NRAO sites**
- Demonstrate distributed data processing between NCSA and NRAO
**in collaboration with ****Community Codes expedition**
**grid job submission tool within AIPS++**

## Data Stats

- BIMA Image Pipeline
- typical per-job data traffic (per day: x3)
- Input:  0.2 – 1 GB     Output:   0.3 – 3 GB
- total archive size
- Now:  0.8 TB           end of 2003:   1.5 – 2 TB?
- VLA mirror
- currently moving tape-based archive to spinning disk
- total archive size
- Now:  0.5 TB           end of 2002:   2 TB         2005:   25 TB
- network AOC - NCSA:  T1 (frame relay) and Abilene
- distribute processing between AOC & NCSA

## Project requirements

- Ability to do simple, organized bulk data movement between NCSA storage and computing resources using GridFTP
- ability to manage replicated data between NRAO and NCSA storage systems using Globus replication management tools.
- Grid job management (submission, monitoring) should be supported by NCSA computing platforms (i.e. Linux clusters).
- Web Service-oriented implementations are highly desirable  for integration with current BIMA pipeline software and related planned work under the Portal expedition, as are Java interfaces.
- C-interfaces are useful as well, particular for support of the VLA pipeline.
**Grid job submission tool within AIPS++ (****Community Codes****)**