# Presentation: 266952

## Storage Resource Management

**Storage Resource Management**
**(SRM)**
**For Grid Applications**
**A SciDAC supported**
**middleware component**
**Arie Shoshani**
**Computing Sciences Directorate**
**Lawrence Berkeley National Laboratory**
**http://sdm.lbl.gov/srm**

## Participants

- PI:  Arie Shoshani
- LBNL – 2 FTEs:
- Arie Shoshani, PI
- Alex Sim, co-PI
- Junmin Gu
- Andreas Mueller
- Fermilab – 1⁄2 FTE:
- Don Petravick, Co-PI
- Rich Wellner

## Motivation

**Grid architecture emphasized in the past**
**Security**
**Compute resource coordination & scheduling**
**Network resource coordination & scheduling (QOS)**
**SRMs role in the data grid architecture**
**Storage resource coordination & scheduling**
**Types of storage resource managers**
**Disk Resource Manager (DRM)**
**Tape Resource Manager (TRM)**
**Hierarchical Resource Manager (TRM + DRM)**

## Where Do SRMs Fit in Grid Architecture?

- property-file
-  index

- logical
- query

- pinning & file
- transfer requests

- network

- ...

- logical files

- site-specific
-  files

**Client’s site**

- ...

- site-specific
-  files requests

## Challenges (1)

**Managing storage resources in an unreliable distributed large heterogeneous system**
**Long lasting data intensive transactions**
**Can’t afford to restart jobs**
**Can’t afford to loose data, especially from experiments**
**Type of failures**
**Storage system failures **
**Mass Storage System (MSS)**
**Disk system**
**Server failures**
**Network failures**

## Challenges (2)

**Heterogeneity**
**Operating systems (well understood)**
**MSS - HPSS, Castor, Enstore, ...**
**Disk systems – system attached, network attached, parallel**
**Optimization issues**
**avoid extra file transfers - What to keep in each disk caches over time **
**How to maximize sharing for multiple users**
**Global optimization**
**Multi-Tier storage system optimization**

## Specific Problems

**Managing resource space allocation**
**What if there is no space?**
**Managing pinning of files**
**What if files can be removed in the middle of a transfer**
**Space reservations**
**What if multiple files are needed concurrently**
**File streaming**
**For processing a large set of files**
**Pin-lock**
**What if you pinned files, and system deadlocks**
**User priorities**
**Access control – who can read/write a file**

## HRMs in PPDG (high level view)

- HRM-COPY

- HRM-GET

**Replica **
**Coordinator**

**HRM**
**(performs writes)**

**HRM**
**(performs reads)**

-  Monitors files written   into BNL’s HPSS
-  Selects files to replicate
-  Issues request_to_put  for file (or many files)

- LBNL

- BNL

- GridFTP GET (pull mode)

## Details of Interactions

## Measurements

- FILE_REQUEST_FAILED

- Notified_Client

- Migration_Finished

- Migration_Requested

- Transfered_to_PDSF_from_BNL

- Staging_finished_at_BNL

- Staging_started_at BNL

- Staging_requested_at_BNL

- File replication request start

## SC 2001 Demo Setup

**DRM**

**Berkeley**

**Berkeley**

**Chicago**

**Livermore**

**HRM**

**GridFTP**

**GridFTP**

**GridFTP**

**FTP**

**BIT-MAP**
**Index**

**Request**
**Manager**

**File Transfer**
**Monitoring**

**DRM**

**GridFTP**

**Denver**

**client**

**server**

**server**

**server**

**server**

**Logical Request**

**Data Path**

**Control path**

_**Legend:**_

## Monitoring File Transfer

## Accomplishment

**Developed HRMs and DRMs using the same uniform protocols**
**Deployed in PPDG**
**Developed Command Line interface to HRM**
** ****Wrote a joint design specification in cordination with EDG, Jlab, and Fermi (to be presented at GGF)**
**Wrote a paper for MSS conference**
**Future: develop a standard protocol**
**Future: deploy HRM in ORNL & NERSC for ESG II project**

## Slide 14

## Slide 15

## Slide 16