# Presentation: 487319

## SRM-Lite:overcoming the firewall barrier for data movement 

**Arie Shoshani**
**Alex Sim**
**Viji Natarajan**
**Lawrence Berkeley National Laboratory**

**SDM Center All-Hands Meeting**
**November, 2007**

## Outline

**What are Resource Storage Managers (SRM)**
**Requirement of using SRM behind firewalls**
**Satisfying the Requirements**
**Architecture**
**Potential uses**

## Storage Resource Managers

**SRMs are middleware components whose function ****is to provide:**
**dynamic space allocation AND file management in spaces**
**for storage components on the local or wide-area network**
**Based on a common standard**

**SRM**
**(BeStMan)**

- client/user applications

- Examples of storage systems currently supported by SRMs

**GPFS**

## Storage Resource Managers:Main concepts

**Non-interference with local policies  **
**Advance space reservations  **
**Dynamic space management**
**Pinning file in spaces**
**Support abstract concept of a file name: Site URL (SURL)**
**Temporary assignment of file names for transfer: Transfer URL (TURL)**
**Directory Management and ACLs  **
**Multi-file requests (srmRquestToPut, srmRequestToGet, srmCopy)**
**Transfer protocol negotiation  **
**Peer to peer request support**
**Support for asynchronous multi-file requests**
**Support abort, suspend, and resume operations**
**SRM relies on other services for data movement (GridFTP, HTTPS, SCP, ...)**

## Concepts: Site URL and Transfer URL

**Provide: Site URL (SURL)**
**URL known externally – e.g. in Replica Catalogs**
**e.g. srm://ibm.cnaf.infn.it:8444/dteam/test.10193 **
**Get back: transfer URL (TURL)**
**Path can be different than SURL – SRM internal mapping**
**Protocol chosen by SRM based on request protocol preference**
**e.g. gsiftp://ibm139.cnaf.infn.it:2811//gpfs/dteam/test.10193 **
**One SURL can have many TURL**
**Files can be replicated in multiple storage components**
**Files may be in near-line and/or on-line storage**
**In light-weight SRM (a single file system on disk)**
**SURL can be the same as TURL except protocol**
**File sharing is possible**
**Same physical file, but many requests**
**Needs to be managed by SRM**

## Earth Science Grid Analysis Environment(in production for 4 years)

**Tomcat  servlet engine                       **
** **

**MCS**
**Metadata Cataloguing Services**

**RLS**
**Replica Location Services**

- SOAP

- RMI

**MyProxy**
**server**

**MCS client**

**RLS client**

**MyProxy client**

**GRAM**
**gatekeeper**

**CAS**
**Community Authorization Services**

**CAS client**

**disk**

**MSS**
**Mass Storage System**

**HPSS**
**High Performance**
**Storage System**

**disk**

**HPSS**
**High Performance**
**Storage System**

**disk**

**disk**

**DRM**
**Storage Resource **
**Management**

**HRM**
**Storage Resource **
**Management**

**HRM**
**Storage Resource **
**Management**

**HRM**
**Storage Resource **
**Management**

**gridFTP**

**gridFTP**

**gridFTP**
**server**

**gridFTP**
**server**

**gridFTP**
**server**

**gridFTP**
**server**

**openDAPg**
**server**

**gridFTP**
**Striped**
**server**

- LBNL

- LLNL

- ISI

- NCAR

- ORNL

- ANL

**DRM**
**Storage Resource **
**Management**

**Earth Science Grid Analysis Environment****(in production for 4 years)**

- >5000 users

- 160 TBs managed

- SRMs are used and inter-communicate in several sites

**SRMs**

## Robust Data Movement provided by SRMs and DataMover

_**Problem**_**: move thousands of files robustly**
**Takes many hours**
**Need error recovery**
**Mass storage systems failures**
**Network failures**
_**Solution**_**: Use Storage Resource Managers (SRMs)**
**File streaming paradigm**
**By reserving and releasing storage space automatically**
_**Problem**_**: too slow**
**Solution: **
**in GridFTP**
**Use parallel streams**
**Use large FTP windows**
**Pre-stage files from MSS**
**Use concurrent transfers**

**NCAR**

**Anywhere**

**LBNL**

**SRM-COPY**
**(thousands of files)**

**SRM-GET ****(one file at a time)**

**DataMover**

**SRM**
**(performs writes)**

**SRM**
**(performs reads)**

**GridFTP GET (pull mode)**

**stage files**

**archive files**

**Network transfer**

**Get list**
**of files**

**MSS**

- Example setup for Earth System Grid (ESG)

## File tracking shows recovery from transient failures

**Total:**
**45 GBs**

## Requirements for SRM-Lite

**Run SRM behind a firewall**
**Cannot have third party transfers (source/target is local)**
**May not be able to run GridFTP**
**Remote site may not support it**
**Some communities choose not to use GSI**
**Need support for multi-file transfer**
**Or entire directory**
**Need support for asynchronous request**
**Also support for intermediate status of request**
**Need to support concurrent file transfers**

## Satisfying the Requirements: SRM-Lite

**Run SRM behind a firewall**
**Must have a client tool (SRM-Lite)**
**May not be able to run GridFTP**
**Support high-performance SCP: Use HPN-SSS from Pittsburgh supercomputing Center**
**But, also use other transfer protocols (GridFTP, bbcp, https, ...)**
**Need support for multi-file transfer**
**Manage queues for large requests**
**Need support for asynchronous request**
**SRM-Lite returns a “request token”; token can be used for “request status”**
**Need to support concurrent file transfers**
**Use multi-threading to manage concurrent transfers**
**Monitor transfers and recover from mid-transfer interruptions**

## Process Steps
Login to ORNL using OTP
At ORNL invoke SRM-Lite
User composes XML input file, srmlite.xml for selectedfiles/directories to copy from/to another site
Or, user gives command lineoption for a selected file/directory
SRM-Lite uses srmlite.xml orcommand line inputto automatically 
Push/Pull files to/from NERSC
Use multiple threads for concurrent transfers

- Disk
- Cache

**SSH Server**

**NERSC**

**SSH**
**Channel**
**(SCP)**

**GridFTP/FTP/**
**BBCP/HTTP**
**transfers**

**Process Steps**
**Login to ORNL using OTP**
**At ORNL invoke SRM-Lite**
**User composes XML input ****file, srmlite.xml for selected****files/directories to copy from/to another site**
**Or, user gives command line****option for a selected ****file/directory**
**SRM-Lite uses srmlite.xml or****command line input****to automatically **
**Push/Pull files to/from NERSC**
**Use multiple threads for ****concurrent transfers**

- Disk
- Cache

**ORNL**

**SRM-**
**Lite**

**OTP**
**Login**

- srmlite.xml

**Local Commands**
**And**
**Protocols**

- Put example: Source: file:////my_directory/file_foo  Target: scp://host/target_dir/file_foo
- Get example: Source: GridFTP://host/target_dir/file_foo  Target: file:////my_directory/file_foo

## Scenario B: one end has a firewall,

**Scenario B: one end has a firewall,**
**The other end has ****SRM**

- Disk
- Cache

**HPSS**

**SRM**

**NERSC**

**GridFTP/FTP/**
**SCP**
**transfers**

- Disk
- Cache

**ORNL**

**SRM-**
**Lite**

**OTP**
**Login**

- srmlite.txt

**SRM**
**Request**

- Put example: Source: file:////my_directory/file_foo  Target: srm://host/target_dir/file_foo

## Process Steps
Login to Site1 using OTP
At site1 invoke SRM-Lite
SRM-Lite at site1 uses SSH to invoke SRM-Lite at site2
Use SSH channel for SCP
Same as before:
User composes XML input file, srmlite.xml for selected files/directories to copy from/to another site
Or, user gives command line option for a selected file/directory

- Disk
- Cache

**SSH Server**

**SSH**
**Channel**
**(SCP)**

**Process Steps**
**Login to Site1 using OTP**
**At site1 invoke SRM-Lite**
**SRM-Lite at site1 uses SSH to invoke SRM-Lite at site2**
**Use SSH channel for SCP**
**Same as before:**
**User composes XML ****input file, srmlite.xml for selected files/directories to copy from/to another site**
**Or, user gives command line option for a selected ****file/directory**

- Disk
- Cache

**OTP**
**Login**

- srmlite.xml

- site2

- site1

## Scenario C: SRM-Lite manages MSS access

**SSH Server**

**SSH**
**Channel**
**(SCP)**

**OTP**
**Login**

- srmlite.xml

- site2

- site1

- Disk
- Cache

**HPSS**

- Disk
- Cache

**HPSS**

## GUI for SRM-Lite

**GUI for SRM-Lite**

**Used in ESG**
**Special version for data movement ****to user workstations**

**Called DataMover-Lite**
**Versions exist for Linux, PC, Mac**

## Usage

**Combustion project**
**The Applied Partial Differential Equations Center (APDEC)**
**John Bell**
**Efficient, robust data movement from sites behind firewalls**
**At DoE and DoD sites**
**Kepler-SRM-Lite actor**
**To be used for managing multi-file transfers from sites behind firewalls**
**Launch SRM-Lite remotely through SSH**
**Initial version – help from NCSU: Pierre Mouallem**
**Two modes**
**Entire request**
**Streaming file requests**
**To be used in CPES workflows first with Norbert’s help**