# Presentation: 328310

## Storage Resource Management: 

**Storage Resource Management: **
**Providing Uniform Access to **
**Storage Systems**
**(including MSSs)**
**Arie Shoshani **
**LBNL**
**(on behalf of the SRM collaboration)**
** ****	****	****	**
**http://sdm.lbl.gov/srm-wg**

## Storage Resource Managers (SRMs)

_**Definition**_
**SRMs are middleware components **
**whose function is to provide dynamic**** **
**space allocation**** ****file management**** **
**of shared storage components on the Grid**

## History

**4 year of Storage Resource (SRM) Management activity**
**Experience with system implementations v.1.x - 2001**
**MSS: HPSS (LBNL, ORNL, BNL), Enstore (Fermi), JasMINE (Jlab), Castor (CERN), MSS (NCAR), SE (RAL) ...**
**Disk systems: DRM(LBNL), dCache(Fermi), jSRM (Jlab), ...**
**SRM v2.x spec was finalized - 2003**
**Several implementations of v2.x completed or in-progress**
**Jlab, Fermi, CERN, LBNL**
**Started GSM: GGF-BOF at GGF8 (June 2003)**
**Last SRM collaboration meeting – Sept. 2004**
**SRM v3.x spec (for GGF) being finalized - 2005**

## Uniformity of Interface Compatibility of SRMs

**SRM**

**SRM**

**SRM**

**Enstore**

**JASMine**

**Client**
**USER/APPLICATIONS**

- Grid Middleware

**SRM**

**dCache**

**SRM**

**Castor**

**SRM**

**Unix-based**
**disks**

**SRM**

**SE**

## Current Storage Resource ManagementActive Working Group

_**CERN**_**: Olof Barring, Jean-Philippe Baud, James Casey, **
**             ****Peter Kunszt**
_**Rutherford lab**_**: Jens Jensen, Owen Synge**
_**Jefferson Lab**_**: Bryan Hess, Andy Kowalski, Chip Watson**
_**Fermilab**_**: Don Petravick, Timur Perelmutov**
_**LBNL**_**: Junmin Gu , Arie Shoshani, Alex Sim, Kurt Stockinger**
_**Univa**_**: Rich Wellner**

**Current Storage Resource Management****Active Working Group**

## Basic Issues

**Suppose you want to run a job on your local machine**
**Need to allocate space**
**Need to bring all input files from a storage system**
**Need to ensure correctness of files transferred**
**Need to monitor and recover from errors**
**What if files don’t fit space? Need to manage file streaming**
**Need to remove files to make space for more files**
**Now, suppose that the machine and storage space is a shared resource**
**Need to to the above for many users**
**Need to enforce quotas**
**Need to ensure fairness of space allocation and scheduling**

## Basic Issues

**Now, suppose you want to do that on the WAN**
**Need to access a variety of storage systems**
**mostly remote systems, need at have access permission**
**Need to have special software to access mass storage systems**
**Now, suppose you want to run distributed jobs on the WAN**
**Need to allocate remote spaces**
**Need to move (stream) files to remote sites**
**Need to manage file outputs and their movement to destination site(s)**

## Types of storage resource managers

**Disk Resource Manager (DRM)**
**Manages one or more disk resources**
**Tape Resource Manager (TRM)**
**Manages access to a tertiary storage system (e.g. HPSS)**
**Hierarchical Resource Manager (HRM=TRM + DRM)**
**An SRM that stages files from tertiary storage into its disk cache**

## Peer-to-Peer Uniform Interface

## General Analysis Scenario

## Standards for Grid Storage Management

**Main concepts**
**Allocate spaces**
**Get/put files from/into spaces**
**Pin files for a lifetime**
**Release files and spaces**
**Get files into spaces from remote sites**
**Manage directory structures in spaces**
**SRMs communicate as peer-to-peer**
**Negotiate transfer protocols**
**No logical name space management (rely of GGF- GFS)**

## Where do SRMs belongin the Grid architecture?

- Compute

- Systems

- Networks

- Other
- Storage

- systems

**Storage**

**Resource**

**Manager**

- Compute

- Resource

- Management

- General Data

- Discovery

- Services

- Community

- Authorization

- Services

- Application-

- Specific Data

- Discovery Services

-  Storage

- Management

- (Brokering)

- Compute

- Scheduling

- (Brokering)

- Data Filtering or

- Transformation

- Services

- Database

- Management

- Services

- Request

- Interpretation

- and Planning

- Services

- File Transfer

- Service

- (GridFTP)

- Data

- Transport

- Services

- Monitoring/

- Auditing

- Services

- Workflow or

- Request

- Management

- Services

- Consistency Services

- (e.g., Update Subscription,

- Versioning, Master Copies)

- Data

- Federation

- Services

- C

- O

- L

- L

- E

- C

- T

- I

- V

- E

- 1

- :

- G

- E

- N

- E

- R

- A

- L

- S

- E

- R

- V

- I

- C

- E

- S

- F

- O

- R

- C

- O

- O

- R

- D

- I

- N

- A

- T

- I

- N

- G

- M

- U

- L

- T

- I

- P

- L

- E

- R

- E

- S

- O

- U

- R

- C

- E

- S

- C

- O

- L

- L

- E

- C

- T

- I

- V

- E

- 2

- :

- S

- E

- R

- V

- I

- C

- E

- S

- S

- P

- E

- C

- I

- F

- I

- C

- T

- O

- A

- P

- P

- L

- I

- C

- A

- T

- I

- O

- N

- D

- O

- M

- A

- I

- N

- O

- R

- V

- I

- R

- T

- U

- A

- L

- O

- R

- G

- .

- Resource

- Monitoring/

- Auditing

- F

- A

- B

- R

- I

- C

- C

- O

- N

- N

- E

- C

- T

- I

- V

- I

- T

- Y

- Communication

- Protocols (e.g.,

- TCP/IP stack)

- Authentication and

- Authorization

- Protocols (e.g., GSI)

- Data Filtering or

- Transformation

- Services

- C

- O

- L

- L

- E

- C

- T

- I

- V

- E

- This figure based on the
- Grid Architecture paper
- by Globus Team

- Mass Storage
- System
- (HPSS)

## SRMs supports data movement betweenstorage systems

- Compute

- Systems

- Networks

- Other
- Storage

- systems

- Storage

- Resource

- Manager

- Compute

- Resource

- Management

- Community

- Authorization

- Services

- Application-

- Specific Data

- Discovery Services

- Data

- Compute

- Scheduling

- (Brokering)

- Data Filtering or

- Transformation

- Services

- Database

- Management

- Services

- Request

- Interpretation

- and Planning

- Services

- File Transfer

- Service

- (GridFTP)

- Data

- Transport

- Services

- Monitoring/

- Auditing

- Services

- Workflow or

- Request

- Management

- Services

- Consistency Services

- (e.g., Update Subscription,

- Versioning, Master Copies)

- Data

- Federation

- Services

- R

- E

- S

- O

- U

- R

- C

- E

- :

- S

- H

- A

- R

- I

- N

- G

- S

- I

- N

- G

- L

- E

- R

- E

- S

- O

- U

- R

- C

- E

- S

- C

- O

- L

- L

- E

- C

- T

- I

- V

- E

- 1

- :

- G

- E

- N

- E

- R

- A

- L

- S

- E

- R

- V

- I

- C

- E

- S

- F

- O

- R

- C

- O

- O

- R

- D

- I

- N

- A

- T

- I

- N

- G

- M

- U

- L

- T

- I

- P

- L

- E

- R

- E

- S

- O

- U

- R

- C

- E

- S

- C

- O

- L

- L

- E

- C

- T

- I

- V

- E

- 2

- :

- S

- E

- R

- V

- I

- C

- E

- S

- S

- P

- E

- C

- I

- F

- I

- C

- T

- O

- A

- P

- P

- L

- I

- C

- A

- T

- I

- O

- N

- D

- O

- M

- A

- I

- N

- O

- R

- V

- I

- R

- T

- U

- A

- L

- O

- R

- G

- .

- Resource

- Monitoring/

- Auditing

- F

- A

- B

- R

- I

- C

- C

- O

- N

- N

- E

- C

- T

- I

- V

- I

- T

- Y

- Communication

- Protocols (e.g.,

- TCP/IP stack)

- Authentication and

- Authorization

- Protocols (e.g., GSI)

- C

- O

- L

- L

- E

- C

- T

- I

- V

- E

- This figure based on the
- Grid Architecture paper
- by Globus Team

- Mass Storage
- System
- (HPSS)

- General Data

- Discovery

- Services

- Data Filtering or

- Transformation

- Services

- Movement

- Storage

## SRM Functional Concepts

**Manage Spaces dynamically**
**Reservation, lifetime**
**Negotiation**
**Manage files in spaces**
**Request to put files in spaces**
**Request to get files from spaces**
**Lifetime, pining of files, release of files**
**No logical name space management (done by replica location services)**
**Access remote sites for files**
**Bring files from other sites and SRMs as requested**
**Use existing transport services (GridFTP, https, ...)**
**Transfer protocol negotiation**
**Manage multi-file requests**
**Manage request queues**
**Manage caches**
**Manage garbage collection**
**Directory Management**
**Uxix semantics: srmLs, srmMkdir, srmMv, srmRm, srmRmdir**

## Concepts: Types of Files

**Volatile: temporary files with a lifetime guarantee**
**Files are “pinned” and “released”**
**Files can be removed by SRM when released or when lifetime expires**
**Permanent**
**No lifetime**
**Files can only be removed by creator (owner)**
**Durable: files with a lifetime that CANNOT be removed by SRM**
**Files are “pinned” and “released”**
**Files can only be removed by creator (owner)**
**If lifetime expires – invoke administrative action (e.g. notify owner, archive and release)**

## Concepts: Types of Spaces

**Types**
**Volatile**
**Space can be reclaimed by SRM when lifetime expires**
**durable **
**Space can be reclaimed by SRM only if it does NOT contain files**
**Can choose to archive files and release space**
**Permanent**
**Space can only be released by owner or administrator**
**Assignment of files to spaces**
**Files can only be assigned to spaces of the same type**
**Spaces can be reserved**
**No limit on number of spaces**
**Space reference handle is returned to client**
**Total space of each type are subject to SRM and/or VO policies**
**Default spaces**
**Files can be put into SRM spaces without explicit reservation**
**Defaults are not visible to client**
**Compacting space**
**Release all unused space – space that has no files or files whose lifetime expired**

## Concepts: Directory Management

**Usual unix semantics**
**srmLs, srmMkdir, srmMv, srmRm, srmRmdir**
**A single directory for all file type**
**No directories for each type**
**File assignment to types is virtual**
**File can be placed in SRM-managed directories by maitaining mapping to client’s directory**
**Access control services**
**Support owner/group/world permission**
**Can only be assigned by owner**
**When file requested by user, SRM should check permission with source site**

## Examples of Directory Structures(user defined)

**D1**

**D3**

**D2**

**D4**

**F2 (P)**

**F4 (P)**

**F5 (D)**

**F1 (D)**

**F3 (V)**

**D1**

**D3**

**D2**

**D4**

**F1 (V)**

**F2 (V)**

**F3 (V)**

**F4 (D)**

**F5 (D)**

**F6 (D)**

**F7 (P)**

**F8 (P)**

**(1) Mixed file types**

**(2) By file type**

** **_**Supported function**_**: ChangeFileType**
** **_**Advantage of (1)**_**: no need to move files when file types are changed **

## Concepts: Space Reservations

**Negotiation**
**Client asks for space: C-guaranteed, MaxDesired**
**SRM return: S-guaranteed <= C-guaranteed, ****                      best effort <= MaxDesired**
**Type of space**
**Can be specified**
**Subject to limits per client (SRM or VO policies)**
**Default: volatile**
**Lifetime**
**Negotiated: C-lifetime requested**
**SRM return: S-lifetime <= C-lifetime**
**Reference handle**
**SRM returns space reference handle**
**User can provide: srmSpaceTokenDescription to recover handles**

## Concepts: Transfer Protocol Negotiation

**Negotiation**
**Client provides an ordered list**
**SRM return: highest possible protocol it supports **
**Example**
**Protocols list: bbftp, gridftp, ftp**
**SRM returns: gridftp**
**Advantages**
**Easy to introduce new protocols**
**User controls which protocol to use**
**Default – SRM policy choice**
**How it is returned?**
**The protocol of the Transfer URL (TURL)**
**Example: bbftp://dm.slac.edu/temp/run11/File678.txt**

## Concepts: Multi-file requests

**Can srmRequestToGet multiple files**
**Required: Files URLs**
**Optional: space file type, space handle, Protocol list**
**Optional: total retry time**
**Provide: Site URL (SURL)**
**URL known externally – e.g. in Rep Catalogs**
**e.g. srm://sleepy.lbl.gov:4000/tmp/foo-123**
**Get back: transfer URL (TURL)**
**Path can be different that in SURL – SRM internal mapping**
**Protocol chosen by SRM**
**e.g. gridftp://dm.lbl.gov:4000/home /level1/foo-123**
**Managing request queue**
**Allocate space according to policy, system load, etc.**
**Bring in as many files as possible**
**Provide information on each file brought in or pinned**
**Bring additional files as soon as files are released**
**Support file streaming**

## SRM Methods

_**File Movement**_
**srmPrepareToGet**
**srmPrepareToPut**
**srmCopy**
** **
_**Lifetime management**_
**srmReleaseFiles**
**srmPutDone**
**srmExtendFileLifeTime**
_**Terminate/resume**_
**srmAbortRequest**
**srmAbortFile**
**srmSuspendRequest**
**srmResumeRequest**
** **

_**Space management**_
**srmReserveSpace**
**srmReleaseSpace**
**srmUpdateSpace**
**srmCompactSpace**
** **
_**FileType management**_
**srmChangeFileType **
_**Status/metadata**_
**srmGetRequestStatus**
**srmGetFileStatus**
**srmGetRequestSummary**
**srmGetRequestID**
**srmGetFilesMetaData**
**srmGetSpaceMetaData**

## SRM v3.x: Basic vs. Advanced Features

**SRM v3.x: Basic vs. Advanced Features**

**File movement**
**PrepareToGet**
**PrepareToPut**
**Copy**
**Request capabilities**
**Multi-file Streaming**
**Trans. Prot. Negotiation**
**File lifetime negotiation**
**File types**
**Volatile**
**Permanent**
**durable**

**    ****BASIC             ADVANCED**

**yes**

**yes**

**yes**

**yes**

**no**

**yes**

**yes**

**yes**

**yes**

**yes**

**no**

**yes**

**yes**

**yes**

**yes (for MSS)**

**yes**

**no**

**yes**

## Features in Basic vs. Advanced SRM

**    ****BASIC             ADVANCED**

**Space reservations**
**Space-time negotiation**
**Space types**
**Remote access**
**gridFTP**
**Other SRMs**
**User-specified Directory**
**Volatile**
**Permanent**
**Durable**
**Terminate/suspend**
**Abort file**
**Abort request**
**Suspend/resume request**

**no**

**yes**

**no**

**yes**

**no**

**yes**

**no**

**yes**

**no**

**yes**

**yes**

**yes**

**no**

**yes**

**yes**

**yes**

**yes**

**yes**

**no**

**yes**

## Use of SRMs forRobust directory-to-directoryfile replication

**Use Case**

## Massive Robust File Replication

**Multi-File Replication – why is it a problem?**
_**Tedious task**_** – many files, repetitious**
_**Lengthy task**_** – long time, can take hours, even days**
_**Error prone**_** – need to monitor transfers**
_**Error recovery**_** – need to restart file transfers**
_**Stage and archive from MSS**_** – limited concurrency, down time, transient failures**
_**Use of FTP**_** –  no large windows / multiple streams**
_**Security**_** – both for local MSS and the network**
_**Firewalls**_** – transfer from/to MSS must be internal to the site**
_**Specialized MSS**_** – HPSS at NERSC, ORNL, ..., **
_**Legacy MSS**_** – MSS at NCAR **

## Main Idea

**Leverage off Storage Resource Managers (SRMs) Technology**
**Supported by SRM middleware project**
**Leverage from experience with other SciDAC projects – PPDG**
**What do you get?**
**SRMs queue multi-file requests**
**SRMs allocate space and release space automatically**
**SRMs request files from remote SRMs**
**Recover from network failures**
**SRMs invoke GridFTP – use large windows & parallel streams**

## DataMover: HRMs use in ESG for

**Robust Muti-file replication**** **

**Anywhere**

**DataMover**

**HRM**
**(performs writes)**

**LBNL/**
**ORNL**

**HRM**
**(performs reads)**

**BNL**

## Slide 29

## Web-Based File Monitoring Tool

_**Shows:**_
**Files already****   transferred****- Files during****   transfer**
** ****Files to be****   transferred**
_**Also shows for**_
_**each file:**_
**Source URL**
**Target URL**
**Transfer rate**

## File tracking helps to identify bottlenecks

**Shows that archiving is the bottleneck**

## File tracking shows recovery from transient failures

**Total:**
**45 GBs**

## File tracking shows network slowdown and recovery

**Total:**
**53 GBs**

## Multi-file Transfer plot from BNL to LBNL (27/02/04)

- 1 = Request ACCEPTED 2 = File SpaceReserved 3 = Grid FTPStart 4 = Grid FTPEnd 5 = HPSS MIGRATION_REQUEST 6 = HPSS ARCHIVE_START 7 = HPSS ARCHIVED 8 = File Released

## Multi-file Transfer plot from BNL to LBNL (10/02/04)

- 1 = Request ACCEPTED 2 = File SpaceReserved 3 = Grid FTPStart 4 = Grid FTPEnd 5 = HPSS MIGRATION_REQUEST 6 = HPSS ARCHIVE_START 7 = HPSS ARCHIVED 8 = File Released
- 9 = File SpaceClaimed 10 = HPSS Archivig_Error

## Summary

**Storage Resource Management – essential for Grid**
**SRM is a functional definition**
**Adaptable to different frameworks (WS, OGSA, WSRF, ...)**
**Multiple implementations interoperate**
**Permit special purpose implementations for unique products**
**Permits interchanging one SRM product by another**
**SRM implementations exist and some in production use**
**Particle Physics Data Grid**
**Earth System Grid**
**More coming ...**
**Cumulative experience in GGF-WG**
**Specifications SRM v3.0 complete**

## Extra Slides

## Space Reservation Functional Spec

**srmReserveSpace**
**	****In: ****	****TUserID****	****	****	****userID,**
**	****	****TSpaceType****	****	****	**_**typeOfSpace, **_
**	****	****String****	****	****	****	****userSpaceTokenDescription,**
**	****	****TSizeInBytes****	****	****	****sizeOfTotalSpaceDesired**_**,**_** **
**	****	****TSizeInBytes****	****	****	****sizeOfGuaranteedSpaceDesired,**
**	****	****TLifeTimeInSeconds****	****lifetimeOfSpaceToReserve,**
**	****	****TStorageSystemInfo****	****	****storageSystemInfo**
**  ****Out:  TSpaceType****	****	****	****typeOfReservedSpace, **
**	****	****TSizeInBytes****	****	****	****sizeOfTotalReservedSpace,**
**	****	****TSizeInBytes****	****	****	****sizeOfGuaranteedReservedSpace,**
**	****	****TLifeTimeInSeconds****	****lifetimeOfReservedSpace,   **
**	****	****TSpaceToken, ****	****	****referenceHandleOfReservedSpace*****,***
**	****	****TReturnStatus****	****	**_**returnStatus**_

## “Request-to-Get” Files Functional Spec

**srmPrepareToGet**
**	****In:****	****TUserID****	****	****	****userID,**
**	****	****TGetFileRequest[ ]****	**_**arrayOfFileRequest,**_** **
**	****	****string[]****	****	****	****arrayOfTransferProtocols,****	**
**	****	****string****	****	****	****userRequestDescription,****	**
**	****	****TStorageSystemInfo****	****storageSystemInfo, **
**	****	****TLifeTimeInSeconds****	****TotalRetryTime**
**Out:****	****TRequestToken****	****	**_**requestToken,**_
**	****	****TReturnStatus****	****	**_**returnStatus,**_** **
**	****	****TGetRequestFileStatus[ ]****	****arrayOfFileStatus**

## “TGetFileRequest” typedef Functional Spec

**typedef****	****struct {TSURLInfo****	****	**_**fromSURLInfo,**_** **
**	****	****	****TLifeTimeInSeconds****	****	****lifetime, // pin time**
**	****	****	****TFileStorageType****	****	****fileStorageType,**
**	****	****	****TSpaceToken****	****	****	****spaceToken,****	**
**	****	****	****TDirOption****	****	****	****	****dirOption **
**	****	****	****} TGetFileRequest**

## Detailed sequence of actions

**For each file being replicated**

**Anywhere**

**srmCopy** {(sourceURL=hpss.lbnl.gov/xyz/file_x,
-                  targetURL =mss.ncar.gov/uvw/file_y)}

**Get list**
**of files from**
**directory**

**Request **
**files**