# Presentation: 051928

## Storage Resource Managers:

**Storage Resource Managers:**
**Essential Components for the Grid**
**Arie Shoshani**
**Staff:**
**Alex Sim, Junmin Gu,**
**Alex Romosan, Viji Natarajan**
**Scientific Data Management Group**
**Lawrence Berkeley National Laboratory**
**http://sdm.lbl.gov/srm**

## Outline

**What are Storage Resource Managers - Motivation**
**General Analysis Scenario and the use of SRMs**
**SRM functionality**
**Real examples of working SRMs**
**Advantages of using SRMs**
**Conclusions and Future Work**

## Motivation

**Grid architecture needs to include reservation & scheduling of:**
**Compute resources**
**Storage resources**
**Network resources**
**Storage Resource Managers (SRMs) role in the data grid architecture**
_**Shared**_** storage resource allocation & scheduling**
**Especially important for **_**data intensive**_** applications**
**Often files are **_**archived**_** on a mass storage system (MSS)**
**large scientific collaborations (100’s of clients) – opportunities for **_**file sharing**_
**File **_**replication**_** and **_**caching**_** may be used**
**Need to support **_**non-blocking**_** (asynchronous) requests**

## A multi-file request to aDisk Resource Manager

## Accessing Remote Storage Resource Managers

## General Analysis Scenario

**                                           ****:****Uniform SRM Interface**

## File movement functionality: srmGet, srmPut, srmReplicate

## SRM is a Service(OGSA, CORBA, C++, Java, ...)

**SRM functionality**
**Manage **_**space**_
**Negotiate and assign space to users**
**Manage “lifetime” of spaces**
**Manage **_**files**_** on behalf of a user**
**Pin files in storage till they are released**
**Manage “lifetime” of files**
**Manage action when pins expire (depends on file types)**
**Manage file sharing**
**Policies on what should reside on a storage resource at any one time**
**Policies on what to evict when space is needed**
**Get files from remote locations when necessary**
**Purpose: to simplify client’s task**
**Manage multi-file requests**
**A brokering function: queue file requests, pre-stage when possible**
**Provide grid access to/from mass storage systems**
**HPSS (LBNL, ORNL, BNL), Enstore (Fermi), JasMINE (Jlab), Castor (CERN), MSS (NCAR),  ...**

## Types of SRMs

**Types of storage resource managers**
**Disk Resource Manager (DRM)**
**Manages one or more disk resources**
**Tape Resource Manager (TRM)**
**Manages access to a tertiary storage system (e.g. HPSS)**
**Hierarchical Resource Manager (HRM=TRM + DRM)**
**An SRM that stages files from tertiary storage into its disk cache**
**SRMs and File transfers**
**SRMs ****DO NOT**** perform file transfer**
**SRMs ****DO**** invoke file transfer service if needed ****(GridFTP, FTP, HTTP, ...)**
**SRMs ****DO**** monitor transfers and recover from failures**
**TRM: from/to MSS**
**DRM: from/to network**

## SRM works with other SRMsas well as legacy systemsby using GridFTP

**DRM**

**Berkeley**

**Berkeley**

**Chicago**

**Livermore**

**HRM**

**GridFTP**

**GridFTP**

**GridFTP**

**FTP**

**Request**
**Interpreter**

**Request**
**Manager**

**DRM**

**GridFTP**

**client**

**server**

**server**

**server**

**server**

**Logical Request**

**Data Path**

**Control path**

_**Legend:**_

## Earth Science Grid

** **

**MCS**
**Metadata Cataloguing Services**

**RLS**
**Replica Location Services**

- SOAP

- RMI

**MyProxy**
**server**

**MCS client**

**RLS client**

**MyProxy client**

**GRAM**
**gatekeeper**

**CAS**
**Community Authorization Services**

**CAS client**

**NCAR-MSS**
**Mass Storage System**

**HPSS**
**High Performance**
**Storage System**

**HPSS**
**High Performance**
**Storage System**

**DRM**
**Storage Resource **
**Management**

**HRM**
**Storage Resource **
**Management**

**HRM**
**Storage Resource **
**Management**

**HRM**
**Storage Resource **
**Management**

**gridFTP**

**gridFTP**

**gridFTP**
**server**

**gridFTP**
**server**

**gridFTP**
**server**

**gridFTP**
**server**

**openDAPg**
**server**

**gridFTP**
**Striped**
**server**

- LBNL

- LLNL

- USC-ISI

- NCAR

- ORNL

- ANL

**DRM**
**Storage Resource **
**Management**

**Earth Science Grid**

## Uniformity of Interface Compatibility of SRMs

**SRM**

**SRM**

**SRM**

**Enstore**

**JASMine**

**Client**
**USER/APPLICATIONS**

- Grid Middleware

**SRM**

**DCache**

**SRM**

**CASTOR**

**SRM**

## Screen Dump of Demo at Fermi Booth

## Where do SRMs belongin the Grid architecture?

## SRMs provide a brokering serviceby supporting multi-file requests

## DataMover: SRMs use in ESG and PPDG

** ****for Robust Muti-file replication**** **

## Web-Based File Monitoring Tool

_**Shows:**_
**Files already****   transferred****- Files during****   transfer**
** ****Files to be****   transferred**
_**Also shows for**_
_**each file:**_
**Source URL**
**Target URL**
**Transfer rate**

## File tracking helps to identify bottlenecks

**Shows that archiving is the bottleneck**

## File tracking shows recovery from transient failures

**Total:**
**45 GBs**

## File tracking shows network slowdown and recovery

**Total:**
**53 GBs**

## Types of Spaces and Files

**Space reservation services**
**Spaces and files: volatile, durable, permanent**
**Lifetime, action at end of lifetime**
**Volatile – SRM owned, files can be removed if space needed**
**Durable – files cannot be removed, but administrator notified**
**Permanent – can be removed by owner only**
**Directory services**
**Usual unix semantics**
**any type of files in directory**
**Access control services**
**Support owner/group/world permission**
**Can only be assigned by owner**
**File sharing for read-only files**
**check with source for shared file permission**
**File sharing for updatable files**
**check with “master copy” for time of last update**

## SRM Methods

_**File Movement**_
**srm(Prepare)Get:**
**srm(Prepare)Put:**
**srmReplicate:**
** **
_**Lifetime management**_
**srmReleaseFiles:**
**srmPutDone:**
**srmExtendFileLifeTime:**
_**Terminate/resume**_
**srmAbortRequest:**
**srmAbortFile**
**srmSuspendRequest:**
**srmResumeRequest:**
** **

_**Space management**_
**srmReserveSpace**
**srmReleaseSpace**
**srmUpdateSpace**
**srmCompactSpace:**
**srmGetCurrentSpace:**
** **
_**FileType management**_
**srmChangeFileType: **
_**Status/metadata**_
**srmGetRequestStatus:**
**srmGetFileStatus:**
**srmGetRequestSummary:**
**srmGetRequestID:**
**srmGetFilesMetaData:**
**srmGetSpaceMetaData:**

## Summary: advantages of using SRMs

**Synchronization between storage resources**
**Pinning file, releasing files**
**Allocating space dynamically on as “needed basis”**
**Insulate clients from storage and network system failures**
**Transient MSS failure**
**Network failures**
**Interruption of large file transfers**
**Facilitate file sharing**
**Eliminate unnecessary file transfers**
**Support “streaming model”**
**Use space allocation policies by SRMs: no reservations needed**
**Use explicit release by client for reuse of space**
**Control number of concurrent file transfers**
**From/to MSS – avoid flooding MSS and thrashing**
**From/to network – avoid flooding and packet loss**

## Ongoing and Future Work

**Ongoing work**
**Developing Standard SRM interfaces**
**Particle Physics Data Grid (PPDG) project**
**LBNL, TJNAF, FNAL **
**European Data Grid (EDG) project**
**WP2 - data management**
**WP5 – mass storage**
**Deployment**
**LBNL, BNL, ORNL, TJNAF, FNAL, CERN, (SE-England)**
**Use of SRM by other agents**
**Storage Resource Broker (SDSC) calling HRM to Stage files from HPSS**
**GridFTP invoking HRM**
**New Spec completed (SRM V2.1)**
**directory management **
**File/directory file movement**
**dynamic space management**
**Future work**
**Access authorization – community access service (CAS)**
**“****On-demand” space allocation, accounting, and charging**
**Replica management – invoke SRMs and RLS as a single service**
**Request executer (e.g. DAGMAN) to invoke SRMs**
**SRMs over NeST (Network STorage)**