# Presentation: 512244

## Scientific Data Management Group LBNL

**Scientific Data Management Group LBNL**

_**SRM related demos SC 2002**__** **_
**Demos **
**Robust File Replication of Massive Datasets on the Grid**
**GridFTP-HPSS access Provided through HRM**
**HRM enables NCAR’s MSS to be accessed on the Grid**
**Uniform Grid Access to Different Mass Storage Systems**
**Projects**
_**Earth Science Grid**_**: Use of HRMs in ESG Demo, new HRM-NCAR included**
_**Particle Physics Data Grid**_**: File replication for STAR project – Demo**

## Demo: Robust File Replication of Massive Datasets on the Grid

**http://sdm.lbl.gov/srm**

_**Demo**_**: Robust File Replication of Massive Datasets on the Grid**

**This demo illustrates:**
_**Replicating Thousands of Files **__***Robustly***__** between **__**Two remote HPSS installations**_
**  ****Using a single command for a request to replicate all files**
**  ****Using an intermediate middleware storage management****   (called Hierarchical Storage Manager - HRM)**
**  ****Using a client HRM command-line module**
**  ****Without any modifications to the MSS**
**  ****Synchronizing pre-staging, file transfer, and archiving**
**  ****Robustness: recovers from MSS and network transient failures**

## SRMs in Action : PPDG

**HRM-COPY**
**(thousands of files)**

**HRM-GET ****(one file at a time)**

**HRM-Client**
**Command-line Interface**

**HRM**
**(performs writes)**

**HRM**
**(performs reads)**

**LBNL**

**BNL**

**GridFTP GET (pull mode)**

**Anywhere**

**stage files**

**archive files**

**Network transfer**

## Web-Based File Monitoring Tool

_**Shows:**_
**Files already****   transferred****- Files during****   transfer**
** ****Files to be****   transferred**
_**Also shows for**_
_**each file:**_
**Source URL**
**Target URL**
**Transfer rate**

## Recent Measurements 

**of large multi-file replication**

**Shows that the network is the bottleneck**

## Demo: GridFTP-HPSSAccess Provided through HRM

**http://sdm.lbl.gov/srm**

_**Demo**_**: GridFTP-HPSS****Access Provided through HRM**

**               ****This demo illustrates the capability of:**
_**Providing GridFTP access to Mass Storage Systems (MSSs) **_
**  ****Using an intermediate middleware storage management****   (called Hierarchical Storage Manager - HRM)**
**  ****Without any modifications to the MSS**
**  ****Managing queues of multiple requests to the MSS**
**  ****Minimizing tape mounts**
**  ****Recovers from MSS transient failures**

## Where does this service belongin the Grid architecture?

- Compute

- Systems

- Networks

- Other
- Storage

- systems

- Hierarchical

- Resource

- Manager

- Compute

- Resource

- Management

- General Data

- Discovery

- Services

- Community

- Authorization

- Services

- Application-

- Specific Data

- Discovery Services

-  Storage

- Management

- (Brokering)

- Compute

- Scheduling

- (Brokering)

- Data Filtering or

- Transformation

- Services

- Database

- Management

- Services

- Request

- Interpretation

- and Planning

- Services

- File Transfer

- Service

- (GridFTP)

- Data

- Transport

- Services

- Monitoring/

- Auditing

- Services

- Workflow or

- Request

- Management

- Services

- Consistency Services

- (e.g., Update Subscription,

- Versioning, Master Copies)

- Data

- Federation

- Services

- R

- E

- S

- O

- U

- R

- C

- E

- :

- S

- H

- A

- R

- I

- N

- G

- S

- I

- N

- G

- L

- E

- R

- E

- S

- O

- U

- R

- C

- E

- S

- C

- O

- L

- L

- E

- C

- T

- I

- V

- E

- 1

- :

- G

- E

- N

- E

- R

- A

- L

- S

- E

- R

- V

- I

- C

- E

- S

- F

- O

- R

- C

- O

- O

- R

- D

- I

- N

- A

- T

- I

- N

- G

- M

- U

- L

- T

- I

- P

- L

- E

- R

- E

- S

- O

- U

- R

- C

- E

- S

- C

- O

- L

- L

- E

- C

- T

- I

- V

- E

- 2

- :

- S

- E

- R

- V

- I

- C

- E

- S

- S

- P

- E

- C

- I

- F

- I

- C

- T

- O

- A

- P

- P

- L

- I

- C

- A

- T

- I

- O

- N

- D

- O

- M

- A

- I

- N

- O

- R

- V

- I

- R

- T

- U

- A

- L

- O

- R

- G

- .

- Resource

- Monitoring/

- Auditing

- F

- A

- B

- R

- I

- C

- C

- O

- N

- N

- E

- C

- T

- I

- V

- I

- T

- Y

- Communication

- Protocols (e.g.,

- TCP/IP stack)

- Authentication and

- Authorization

- Protocols (e.g., GSI)

- Data Filtering or

- Transformation

- Services

- C

- O

- L

- L

- E

- C

- T

- I

- V

- E

- This figure based on the
- Grid Architecture paper
- by Globus Team

- Mass Storage
- System
- (HPSS)

## HRMs and GridFTP

**HRM**

**GridFTP**

**SRM-API**

**GridFTP-API**

**Client**

**HRM**

**GridFTP move**

**SRM-API**

**GridFTP-API**

**Client**

_**Using HRM protocol**_

_**New: GridFTP-HPSS**_
_**through HRM**_

**GridFTP entry**

## GridFTP-HRM-Layerimplementation detail

**HRM**

**GridFTP-API**

**Client**

**GridFTP entry**
**GridFTP move**
**GridFTP exit**

**FTP-**
**HRM**
**Layer**

- Shared
- memory

- Corba

**1a**

**1b**

**2a**

**2b**

**3a**

**3b**

**1a: stor/retv**
**1b: hrm_get/hrm_put**

**2b: call_back**
**2a: unblock semaphore**

**3a: success_code**
**3b: hrm_release**

## Demo: HRM enables NCAR’s MSS to be accessed on the Grid

**LBNL**
**http://sdm.lbl.gov/srm**

_**Demo**_**: HRM enables NCAR’s MSS to be accessed on the Grid**

**This demo illustrates:**
_**The application of Storage Resource Management (SRM) technology to make NCAR’s MSS Grid-enabled**_
**NCAR’s MSS is a unique system developed in house**
**  ****Making HRM-HPSS grid-enabled is a **_**big task**_
**  ****HRM is a specialized SRM designed for Mass Storage****   Systems (MSS).  HRM-HPSS already exists**
**  ****Adapting HRM-HPSS to work with HRM-HPSS is a **_**small task**_
***  *****Demo illustrates success and ease of this approach**

## How Was It Done?only MSS-specific module modified

**Disk Resource**
**Manager (DRM)**

**Tape Resource**
**Manager (TRM)**

**HPSS-specific**
**Access Module**

**HRM-HPSS**

**Disk Resource**
**Manager (DRM)**

**Tape Resource**
**Manager (TRM)**

**NCAR-specific**
**Access Module**

**HRM-HPSS**

- NCAR-MSS

**Specialize for **
**NCAR-MSS**

## Demo details: File Replication from NCAR MSS  

**to NERSC HPSS controlled at NCAR **

## SRMs in Action : ESG

## Demo: Uniform Grid Accessto Different Mass Storage Systems

_**JLAB**_**: Bryan Hess, Andy Kowalski**
_**FNAL**_**: Don Petravick, Timur Perelmutov, Rich Wellner**
_**LBNL**_**: Junmin Gu, Vijaya Natarayan,** **Ekow Otoo, **
**	**** ****Alex Romosan,** **Alex Sim, Arie Shoshani**

_**Demo**_**: Uniform Grid Access****to Different Mass Storage Systems**

**This demo illustrates: **_**Uniform Grid Access to:**_
_**three MSSs: HPSS, Enstore, JASMine **_
**  ****Using the Storage Resource Manager (SRM) interface**
**  ****SRM modules were developed independently for the 3 MSSs**
**  ****Shows the power of standard SRM functionality**

## High Level View of Demo Setup

**SRM**

**SRM**

**SRM**

**Enstore**

**JASMine**

- Client
- (USER/APPLICATIONS)

- uniform interface

## Where do SRMs belongin the Grid architecture?

- Compute

- Systems

- Networks

- Other
- Storage

- systems

**Storage**

**Resource**

**Manager**

- Compute

- Resource

- Management

- General Data

- Discovery

- Services

- Community

- Authorization

- Services

- Application-

- Specific Data

- Discovery Services

-  Storage

- Management

- (Brokering)

- Compute

- Scheduling

- (Brokering)

- Data Filtering or

- Transformation

- Services

- Database

- Management

- Services

- Request

- Interpretation

- and Planning

- Services

- File Transfer

- Service

- (GridFTP)

- Data

- Transport

- Services

- Monitoring/

- Auditing

- Services

- Workflow or

- Request

- Management

- Services

- Consistency Services

- (e.g., Update Subscription,

- Versioning, Master Copies)

- Data

- Federation

- Services

- R

- E

- S

- O

- U

- R

- C

- E

- :

- S

- H

- A

- R

- I

- N

- G

- S

- I

- N

- G

- L

- E

- R

- E

- S

- O

- U

- R

- C

- E

- S

- C

- O

- L

- L

- E

- C

- T

- I

- V

- E

- 1

- :

- G

- E

- N

- E

- R

- A

- L

- S

- E

- R

- V

- I

- C

- E

- S

- F

- O

- R

- C

- O

- O

- R

- D

- I

- N

- A

- T

- I

- N

- G

- M

- U

- L

- T

- I

- P

- L

- E

- R

- E

- S

- O

- U

- R

- C

- E

- S

- C

- O

- L

- L

- E

- C

- T

- I

- V

- E

- 2

- :

- S

- E

- R

- V

- I

- C

- E

- S

- S

- P

- E

- C

- I

- F

- I

- C

- T

- O

- A

- P

- P

- L

- I

- C

- A

- T

- I

- O

- N

- D

- O

- M

- A

- I

- N

- O

- R

- V

- I

- R

- T

- U

- A

- L

- O

- R

- G

- .

- Resource

- Monitoring/

- Auditing

- F

- A

- B

- R

- I

- C

- C

- O

- N

- N

- E

- C

- T

- I

- V

- I

- T

- Y

- Communication

- Protocols (e.g.,

- TCP/IP stack)

- Authentication and

- Authorization

- Protocols (e.g., GSI)

- Data Filtering or

- Transformation

- Services

- C

- O

- L

- L

- E

- C

- T

- I

- V

- E

- This figure based on the
- Grid Architecture paper
- by Globus Team

- Mass Storage
- System
- (HPSS)

## Same uniform interface to all types of storage systems
Disk Storage System
Mass Storage System
For Mass Storage Systems it provides:
Queuing and pre-staging
Queued multi-file “get” requests (avoid flooding MSS)
Pre-staging of files (concurrent with transfer)
Queued archiving of “put” requests (avoid flooding MSS)
Robustness and efficiency
Recover in case of transient failures from MSS
Reorders pre-staging requests to minimize tape mounts
Recovery from failed GridFTP transfers
Re-issues requests in case of failure

**Same uniform interface to all types of storage systems**
**Disk Storage System**
**Mass Storage System**
**For Mass Storage Systems it provides:**
**Queuing and pre-staging**
**Queued multi-file “get” requests (avoid flooding MSS)**
**Pre-staging of files (concurrent with transfer)**
**Queued archiving of “put” requests (avoid flooding MSS)**
**Robustness and efficiency**
**Recover in case of transient failures from MSS**
**Reorders pre-staging requests to minimize tape mounts**
**Recovery from failed GridFTP transfers**
**Re-issues requests in case of failure**