# Presentation: 067005

## Globus Data Services for Science

**Raj Kettimuthu**
- Argonne National Laboratory/Univ. of Chicago
**Ann Chervenak, Rob Schuler**
- USC Information Sciences Institute

## Globus Services for Data Intensive Science

- Data Movement:
- GridFTP  and Reliable File Transfer Service (RFT)
- Replica management:
- Replica Location Service (RLS) and Data Replication Service (DRS)
- New: Policy-based data placement service
- Access to databases and other data sources:
- OGSA Data Access and Integration (DAI) Service

## Talk Outline

- Examples of production data intensive science projects that use Globus services
- New features:
- GridFTP and RFT
- Replica management tools
- Data placement services
- Data access and integration services

## The LIGO Project

- Laser Interferometer Gravitational Wave Observatory
- LIGO instruments in Washington State and Louisiana
- During science runs, produce up to 2 terabytes per day
- Published along with metadata at Caltech (archival site)
- Replicated at up to 10 other LIGO sites
- LIGO scientists typically move data sets near to computational clusters at their sties
- The LIGO Data Grid

## Globus Services in the LIGO Data Grid

- Lightweight Data Replicator (LDR): data management system developed by LIGO researchers
- Globus data services:
- GridFTP: used for moving data around the Grid efficiently and securely
- Replica Location Service: catalogs deployed at all LIGO sites, keep track of locations of over 150 million files
- Data Replication Service was developed to generalize the functionality in the LDR
- Other Globus services:
- Globus security
- Starting to deploy the Globus Monitoring and Discovery Service

## Earth System Grid objectives

**To support the infrastructural needs of the national and international climate community, ESG is providing crucial technology to securely access, monitor, catalog, transport, and distribute data in today’s grid computing environment.**

- HPChardware running climate models

- ESGSites

- ESG Portal

-    Bernholdt_ESG_SC07

## ESG facts and figures

| Main ESG Portal | IPCC AR4 ESG Portal |
| --- | --- |
| 146 TB of data at four locations
1,059 datasets
958,072  files
Includes the past 6 years of joint DOE/NSF climate modeling experiments | 35 TB of data at one location
77,400 files
Generated by a modeling campaign coordinated by the Intergovernmental Panel on Climate Change
Model data from 13 countries |
| 4,910 registered users | 1,245 registered analysis projects |
| Downloads to date
30 TB
106,572  files | Downloads to date
245 TB
914,400 files
500 GB/day(average) |
|  |  |
|  | > 300 scientific papers published to date based on analysis of IPCC AR4 data |

**Worldwide ESG user base**

**IPCC Daily Downloads (through 7/2/07)**

- Slide Courtesy of Dave Bernholdt, ORNL

## ESG architecture and underlying technologies

- Climate data tools
- Metadata catalog
- NcML (metadata schema)
- OPenDAP-G (aggregation, subsetting)
- Data management
- _D_ata _M_over _L_ite
- _S_torage _R_esource _M_anager
- Globus toolkit
- _G_lobus _S_ecurity _I_nfrastructure
- GridFTP
- Monitoring and Discovery Services
- _R_eplica _L_ocation _S_ervice
- Security
- Access control
- MyProxy
- User registration

**OPeNDAP-G**

**MyProxy**

**SRM**

**DISK**
**Cache**

**ESG Web Portal**

**NCAR**
**Cache**

**RLS**

**SRM**

**RLS**

**SRM**

**RLS**

**SRM**

**RLS**

**LANL**
**Cache**

**search**
**browse**
**download**

**Web**
**Browser**

**DML**

**Data**
**User**

**publish**

**Web**
**Browser**

**Data**
**Provider**

- MSS, HPSS**: Tertiary****data storage systems**

**First Generation ESG Architecture**

**SRM**

## GridFTP Data Transfers for the Advanced Photon Source

- “One Australian user left nearly 1TB of data on our systems that we had been struggling to transfer via standard FTP for several weeks. The typical data rate using standard FTP was ~200 KB/s. Using GridFTP we are now moving data at 6 MB/s—quite a significant boost in performance!” **Brian Tieman****Advanced Photon Source**

## What’s New in Globus GridFTP and RFT

- Raj Kettimuthu
- Argonne National Laboratory and
- The University of Chicago

## What is GridFTP?

- High-performance, reliable data transfer protocol optimized for high-bandwidth wide-area networks
- Based on FTP protocol - defines extensions for high-performance operation and security
- We supply a reference implementation:
- Server
- Client tools (globus-url-copy)
- Development Libraries
- Multiple independent implementations can interoperate
- Fermi Lab and U. Virginia have home grown servers that work with ours.

## GridFTP

- Two channel protocol like FTP
- Control Channel
- Communication link (TCP) over which commands and responses flow
- Low bandwidth; encrypted and integrity protected by default
- Data Channel
- Communication link(s) over which the actual data of interest flows
- High Bandwidth; authenticated by default; encryption and integrity protection optional

## Why GridFTP?

- Performance
- Parallel TCP streams, optimal TCP buffer
- Non TCP protocol such as UDT
- Order of magnitude greater
- Cluster-to-cluster data movement
- Another order of magnitude
- Support for reliable and restartable transfers
- Multiple security options
- Anonymous, password, SSH, GSI

**Notes:**
GridFTP is intended for bulk data tranfer
GridFTP can do coordinated data transfer by using multiple computer nodes at the source and destination.

## Cluster-to-Cluster transfers

**Notes:**
GridFTP can do coordinated data transfer by using multiple computer nodes at the source and destination.

## Performance

- Mem. transfer between Urbana, IL and San Diego, CA

**Notes:**
Given resources Globus GridFTP can scale to network speeds

## Performance

- Disk transfer between Urbana, IL and San Diego, CA

## Users

- HEP community is basing its entire tiered data movement infrastructure for the LHC computing Grid on GridFTP
- Southern California Earthquake Center (SCEC), European Space Agency, Disaster Recovery Center in Japan move large volumes of data using GridFTP
- An average of more than 2 million data transfers happen with GridFTP every day

## LOSF and Pipelining

- 	   Traditional			      Pipelining
- Significant performance improvement for LOSF

- File Request 1

- File Request 2

- File Request 3

- DATA 1

- DATA 2

- DATA 3

- ACK 1

- ACK 2

- ACK 3

- File Request 1

- File Request 2

- File Request 3

- DATA 1

- DATA 2

- DATA 3

- ACK 1

- ACK 2

- ACK 3

**Notes:**
GridFTP and FTP - command response protocols
A client can sends one command and then wait for a “Finished response” before sending another
Overhead added on a per file basis
Performance is best on large files
Overhead has less impact
Performance suffers for a large data set partitioned into many small files

 PIPELINING:
 Allow many outstanding transfer requests
Send next request before previous completes
Latency is overlapped with the data transfer
When one finishes the next has already traversed the network

## GridFTP over UDT

- GridFTP uses XIO for network I/O operations
- XIO presents a POSIX-like interface to many different protocol implementations

- GSI

- TCP

- Default GridFTP

- GSI

- UDT

**Notes:**
UDT - application-level protocol that uses UDP to transfer data
Implement its own reliability and congestion control mechanisms 
Good performance on long fast networks where TCP has significant limitations
Enabled GridFTP to use UDT as an alternate transport protocol

## GridFTP over UDT

|                   
 | Argonne to NZ Throughput in Mbit/s | Argonne to LA Throughput in Mbit/s |
| --- | --- | --- |
| Iperf – 1 stream | 19.7 | 74.5 |
| Iperf – 8 streams | 40.3 | 117.0 |
| GridFTP mem TCP – 1 stream | 16.4 | 63.8 |
| GridFTP mem TCP – 8 streams | 40.2 | 112.6 |
| GridFTP disk TCP – 1 stream | 16.3 | 59.6 |
| GridFTP disk TCP – 8 streams | 37.4 | 102.4 |
| GridFTP mem UDT  | 179.3 | 396.6 |
| GridFTP disk UDT | 178.6 | 428.3 |
| UDT mem | 201.6 | 432.5 |
| UDT disk | 162.5 | 230.0 |

**Notes:**
We compared the performance of GridFTP over UDT with that of GridFTP over TCP
We transferred data between Argonne and NZ and between Argonne and LA
Compared to GridFTP using single tcp stream, gridftp using udt gives 7-10 times higher throughput
Compared to GridFTP using multiple tcp stream, gridftp using udt gives 4-5 times higher throughput

## SSH Security for GridFTP

- sshd

- Client

- GridFTP
- Server

- Port 22

- ROOT

- exec

**USER**

- ssh

- popen

- Authenticate

- Stdin/out

**Notes:**
GridFTP traditionally uses GSI for establishing secure connections
Some users do not want to deal with certificates and they prefer a SSH based security
Leverages the fact that an SSH client can remotely execute programs by forming a secure connection with SSHD 
The client (globus-url-copy) acts as an SSH client and remotely executes a Globus GridFTP server
All of the standard IO from the remote program is routed back to the client.

## Multicast / Overlay Routing

- Enable GridFTP to transfer single data set to many locations or act as an intermediate routing node_x000c_

**Notes:**
GridFTP servers act as both servers receiving data, and clients sending data.
It allow us to create a network overlay where many GridFTP servers act as routers forwarding packets along to each other until they get to the final destination where there are written to disk
Higher level service having more information about the GridFTP servers and the characteristics of the network connecting the servers, can use this feature in a intelligent way to get the data to all the destinations quicker

## GridFTP with Lotman

- GridFTP
- Server

- Client

- Lotman

**Notes:**
Developed a plugin for GridFTP to interact with Lotman, a storage management tool from Univ. of Wisc.
This plugin used to determine the availability of space before a transfer is started and thus would prevent failures due to no space after part of the data is transferred.

## Reliable File Transfer Service (RFT)

- RFT Service

- RFT Client

- SOAP Messages

- Notifications(Optional)

- GridFTP
- Server

- GridFTP
- Server

- CC

- CC

- DC

- Persistent Store

-  GridFTP client
-  WSRF complaint fault-tolerant service

**Notes:**
Queues transfer requests
Fire and Forget
Stores transfer requests and status in a persistent store
Integrated Automatic Failure Recovery
Network level failures.
System level failures etc.

## RFT - Connection Caching

- Control channel connections (and thus the data channels associated with it) are cached to reuse later (by the same user)

- RFT Service

- GridFTP
- Server

- GridFTP
- Server

- CC

- CC

- DC

## RFT - Connection Caching

- Reusing connections eliminate authentication overhead on the control and data channels
- Measured performance improvement for jobs submitted using Condor-G
- For 500 jobs - each job requiring file stageIn, stageOut and cleanup (RFT tasks)
- 30% improvement in overall performance
- No timeout due to overwhelming connection requests to GridFTP servers

## What’s new in Data Access and Integration?

- Raj Kettimuthu on behalf of OGSA-DAI team

**Notes:**
OGSA-DAI is an OMII-UK project run by EPCC and National e-Science Centre of the University of Edinburgh.
Some work in this talk has been done by NextGrid and BEinGRID.
OGSA-DAI 3.0 was released in September 2007. This talk describes some developments since then.

## What is OGSA-DAI?

- Middleware that allows data resources, such as relational or XML databases, to be accessed via web services

## What is OGSA DAI?

- OGSA-DAI executes **workflows**
- OGSA-DAI is not just for data access, also does data updates, transformations and delivery.

## OGSA DAI Workflow

## Remote resource access

- OGSA-DAI  data resource interaction
- Via a data resource plug-in
- Access a data resource managed by another OGSA-DAI server

**Notes:**
This functionality was in OGSA-DAI 2.2 and requested by the GeoGRID project for 3.0.

## Remote resource access

- Remote resource plug-in
- Basically a client to a remote OGSA-DAI server
- Runs queries via workflow submission
- Configured with URL of remote server
- Transparent to OGSA-DAI infrastructure
- Just another data resource plug-in

## OGSA-DAI 3.0 data sources

- OGSA-DAI data sources
- Resource for asynchronous data delivery
- Data source service
- Web service
- Invoke GetFully via SOAP/HTTP
- Use WS-Addressing to specify data source ID

**Expose via data source**

**DataSource**

**Client**

**getFully()**

**DataSourceService**

**...****data from workflow ...**

**Notes:**
A client can submit a workflow that streams data into a data source from the workflow.
The same, or a different, client can then get this data via a data source web service.
Asynchronous data delivery. Recipient is unaware of the workflow.

## OGSA-DAI servlet

- Data source servlet
- Invoke HTTP GET
- Use URL query string to specify data source ID

**Expose via data source**

**DataSource**

**Client**

**HTTP GET**

**DataSourceRetrievalServlet**

**...****data from workflow ...**

**Notes:**
OGSA-DAI servlet exposes data source as a servlet. 
No need for client to invoke a web service, just an HTTP GET.
This is an improved version of OGSA-DAI 2.2’s servlet.

## OGSA-DAI servlet

- Useful for service orchestration and job submission
- Taverna service-oriented workflow executor
- Taverna could submit workflow to OGSA-DAI
- OGSA-DAI returns URL
- Taverna passes URL as part of job to job submission service
- e.g. GRAM or GridSAM
- Data is pulled from the URL when the job is executed
- Advantages
- Data is only moved when needed i.e. when the job executes
- Job execution components need no OGSA-DAI-specific components

## A join activity

- Virtual Organisations for Trials and Epidemiological Studies (VOTES)
- UK Medical Research Council project
- Relational databases
- Uses OGSA-DAI
- OGSA-DAI team developed join activities

## A join activity

- This is equivalent to running:
**SELECT id, x, y FROM tableOne, tableTwo where table1.id = table2.myID;**
- Where tableOne and tableTwo are in two different databases

**Tuple merge join**

**SELECT id, x FROM tableOne ORDER by id**

**Run SQL query**

**SELECT myID, y FROM tableTwo ORDER by myID**

**joinColumn2: myID**

**joinColumn1: id**

**Run SQL query**

**Notes:**
TupleMergeJoin activity.
Inner or left outer joins of rows of relational data from two queries.
The other inputs are the column upon which the join is done.
A “tuple” is OGSA-DAI’s internal representation of a row of data.
Optional argument to select output rows to project upon.
Can also specify whether the join columns data is in asending order.

## SQL views

- Imagine we have Patient and Doctor tables
- SQL CREATE VIEW command
- Define a DrPatient view to be
- SELECT p.id, p.name, p.age, p.sex FROM Patient p, Doctor d WHERE p.DrID = d.ID;
- Client runs SELECT * FROM DrPatient;
- Shorthand for complex queries
- Data access control
- e.g. staff with only access to the DrPatient view will be unable to access a patient’s ZIP

| ID | Name | Age | Sex | ZIP | Dr ID |
| --- | --- | --- | --- | --- | --- |
| 1 | Ken | 42 | M | IL1478305 | 456 |
| 2 | Josie | 25 | F | BN1 7QP | 789 |

| ID | Name | DN |
| --- | --- | --- |
| 123 | Greene | US-Chicago-G |
| 456 | Ross | US-Chicago-R |
| 789 | Fairhead | UK-Holby-F |

**Notes:**
SQL VIEW is an SQL command that basically allows definition of a sub-procedure.
A view represents the results of an SQL query so can act as a shorthand.
e.g. the "DrPatient" VIEW acts as a short-hand for that whole query.
Views are useful since:
They are a shorthand that saves clients having to enter long queries repeatedly.
Views are just handled as another table in queries. So they can be used as a form of data access control for database administrators.
e.g. if I tell clients that they have access to a DrPatient table they'll be unaware of the actual Doctor or Patient tables and the additional fields of those that are not defined to be part of the view (e.g. Patient.ZIP)

## OGSA DAI SQL views

- Layer above the database to implement views
- Define views for databases to which you don’t have write access
- Parses query
- Maps view to SQL query over actual database
- e.g if DrPatient was defined as
- SELECT p.id, p.name, p.age, p.sex FROM Patient p, Doctor d WHERE p.DrID = d.ID AND d.dn = $DN$;
- Can replace $DN$ by client’s DN from their certificate provided using GT4 security components
- Doctors can only view their own patients
- Factor in the client’s security credentials

**Notes:**
Developed by the OGSA-DAI and BEinGRID projects.
As an OGSA-DAI deployer I may not have read access to a relational database exposed by OGSA-DAI (e.g. I may be exposing a third-party database) and so cannot create an SQL view in the database using the SQL CREATE VIEW command. 
Alternatively I may wish to expose the data in a different logical schema (table names, field names and types) from the physical schema of the database.
Implementing SQL views support in OGSA-DAI allows me to address both these.
The SQL views components of OGSA-DAI can parse SQL queries and build abstract syntax trees. These can then be manipulated e.g. replacing a table name such as DrPatient with the SQL statement across the actual database and so basically implementing the SQL views functionality outwith the database.
It also allows the query to be to transformed e.g. to map table and field names to a different schema, to perform complex calculations, or to use Grid credentials to assist in restricting access to the data.

## OGSA-DQP

- Distributed query processing
- Multiple tables on multiple databases are exposed to clients multiple tables in one “virtual database”
- Client is unaware of the multiple databases
- Databases can be exposed within one OGSA-DAI server or exposed by remote OGSA-DAI servers
- How it works
- Query is parsed
- Query plan is created
- Query plan is executed – each database has sub-queries executed on it
- Results are combined
- Good for joins and unions

## What’s new in data replication and placement services?

- Rob Schuler

## Objectives for Data Replication

- A

- A

- A

**Improve *****Durability***
- Safeguard against data loss due to disk failure

**Improve *****Availability***
- Safeguard against data inaccessibility due to network partition

**Improve *****Performance***
- Safeguard against performance bottlenecks due to resource overload

## The Globus Replica Location Service

- Distributed registry
- Records the locations of data copies
- Allows replica discovery
- RLS maintains mappings between logical identifiers and target names
- Must perform and scale well:
- support hundreds of millions of objects
- hundreds of clients
- Mature and stable component of the Globus Toolkit

- Replica Location Indexes

- Local Replica Catalogs

## New Features in RLS

- Embedded SQLite database for easier RLS deployment
- Open source relational database backends (MySQL, PostgreSQL) depend on ODBC libraries
- Compatibility problems that have made DB deployment difficult
- Embedded DB back end now allows easy installation of RLS
- Allows easier evaluation of RLS by potential users
- SQLite offers good performance and scalability on queries
- Does not support multiple simultaneous writers, so not suitable for some high performance environments

## New Features in RLS

- Pure Java client implementation
- Long-awaited
- Overcomes problems with JNI-based client, particularly on 64-bit platforms
- Improves reliability of portals that use RLS Java client
- Being used by several large applications (ESG, SCEC)
- WS-RLS interface: provides a WS-RF compatible web services interface to RLS
- Easier integration of RLS services into GT4 Web service environments

## Data Placement Services: Motivation

- Scientific applications often perform complex computational analyses that consume and produce large data sets
- Computational and storage resources distributed in the wide area
- The placement of data onto storage systems can have a significant impact on
- performance of applications
- reliability and availability of data sets
- We want to identify data placement policies that distribute data sets so that they can be
- staged into or out of computations efficiently
- replicated to improve performance and reliability

## Data Placement and Workflow Management

- Data Placement and Workflow Management

- Studied relationship between asynchronous data placement services and workflow management systems
- Workflow system can provide hints r.e. grouping of files, expected order of access, dependencies, etc.
- Contrasts with many existing workflow systems
- Explicitly stage data onto computational nodes before execution
- Some explicit data staging may still be required
- Data placement has potential to
- Significantly reduce need for on-demand data staging
- Improve workflow execution time
- Experimental evaluation demonstrates that good placement can significantly improve workflow execution performance
- “Data Placement for Scientific Applications in Distributed Environments,” Ann Chervenak, Ewa Deelman, Miron Livny,  Mei-Hui Su, Rob Schuler, Shishir Bharathi, Gaurang Mehta, Karan Vahi, in Proceedings of Grid 2007 Conference, Austin, TX, September 2007.

##         Approach: Combine Pegasus Workflow Management with Globus Data Replication Service

## Replication occurs when...

- Replica *Placement*
- I want replica X at sites A, B, and C
- I want N replicas of each file
- I want replicas near my compute clusters
- Replica *Repair*
- Due to *replica failure*: lost or corrupted
*But *it can be hard to tell the difference between permanent and temporary failure*!*

## Examples of Placement Policies

## Topology-Aware Placement

- client

- Site 1

- Site 3

- Site 2

- 1. Put Data

- 2. Replicate to 2nd Local Site

- 3. Replicate to Remote Site

- The ***Topology Aware ***policy is a type of *N-copy *policy that (in this 3-copy example) ensures that replicas are distributed within and between sites

## Publish/Subscribe Placement

- client

- Site 1

- Site 3

- Site 2

- 1.a. Publish Data “XYZ”

- 2. Query replica name service
- 	and replicate data sets

- The ***Publish/Subscribe ***policy is a *query-based *policy that identifies desired replicas based on a query and replicates them to the desired site

- client

- 1.b. Publish Data “QRS”

- client

- 1.c. Subscribe “XYZ”
- 	and “QRS”

## Reactive vs. Proactive Replication

- Reactive Replication
- When a replica failure occurs, replicate
- Difficult to tell the difference between a permanent replica failure and a temporary loss – e.g., temporary network partition
- Proactive replication
- Continually replicate files beyond the minimum required
- Avoid bursts of network traffic to repair failures; limit bandwidth for repairs
- Need  creation rate >= failure rate