# Presentation: 177017

## Supporting Ad-hoc Data Explorationfor Large Scientific Databases

_**LBNL:**_
**Arie Shoshani **
**Ekow Otoo**
**Alex Sim**
**Kesheng John Wu**
_**ORNL:**_
**Randy Burris**
**Dan Million**
**All Hands Meeting**
**March 26-27, 2002**

## P3: Efficient Access from Large Datasets 

**grid**

**storage**

**Adaptive file caching**

**file**

**Request**
**Interpreter**

**dataset**

**Data mining**

**Distributed**

**Large**

**Storage**
**Resource**
**Management**

**HPSS/**
**Disk **

## Typical Scientific Exploration Process

**Generate large amounts of raw data**
**large simulations**
**collect from experiments**
**Post-processing of data**
**analyze data (find particles produced, tracks)**
**generate summary data**
**e.g. momentum, no. of pions, transverse energy**
**Number of properties is large (50-100)**
**Analyze data**
**use summary data as guide**
**extract subsets from the large dataset**
**Need to access events based on partial****properties specification (range queries)**
**e.g. ((0.1 < AVpT < 0.2) ^ (10 < Np < 20)) v (N > 6000)**
**apply analysis code**

## Problem Statement

**Large number of objects reside in files on a distributed Data Grid**
**10****8**** – 10****9**** objects**
**O.5 – 5 million files**
**15,000 – 150,000 tapes**
**Distributed system can be across continents**
**100’s of sites**
**Some of the data is **_**replicated**_** based on demand or ****pre-assigned replication**
**Request expressed as **_**logical request**_** by user**
**Systems and network may **_**fail**_
_**Problem:**_** given a logical request,****                 get relevant data to local system ****                 without human intervention**

## The big picture

*Request interpreter*

*Logical request*

*“**physical” objects*

*Storage Resource Managers*

*HPSS/shared disk*

*File access management*** *

*Request Manager*

*Logical objects*

**(73.39 < zdc2Energy < 94.94 **
**AND -24.99 < qxb < -7.25)**

**{****set192_01.STAR,..., set287_07.STAR****}**

**gsiftp://dg0n1.mcs.anl.gov/homes/**
**asim/gsiftp/ set192_01.STAR**
**hrm://DRMServerAlone@srm.lbl.gov:4000/**
**home/dm/srm/data1/gsiftp/ set287_07.STAR**

**Sites:**
**dg0n1.mcs.anl.gov**
**srm.lbl.gov:4000**

** Grid Enabled Access*

## SC 2001 Demo

**DRM**

**Berkeley**

**Berkeley**

**Chicago**

**Livermore**

**HRM**

**GridFTP**

**GridFTP**

**GridFTP**

**FTP**

**BIT-MAP**
**Index**

**Request**
**Executor**

**File Transfer**
**Monitoring**

**DRM**

**GridFTP**

**Denver**

**client**

**server**

**server**

**server**

**server**

**Logical Request**

**Data Path**

**Control path**

_**Legend:**_

## Middleware Components

**1) BitMap index**
**Size of data to be indexed: ****10****8 ****objects x 500 attributes x 4 bytes = 200 GB**
**2) Request Executer**
**Uses Replica Catalog**
**Monitors transfer progress**
**3) Storage Resource Managers (SRMs)**
**Disk Resource Manager (DRM)**
**Hierarchical Resource Manager (HRM)**
**4) File Transfer Visualization tool (FTV)**
**View by file size and fraction of file transferred**
**View by % of files transferred**

## Monitoring File Transfer

## Earth Science Data Grid (ESG II)

**Discovery**
**Apps**

**Analysis**
**Apps**

**Request Management Services**

**Replica**
**Services**

**Vis**
**Services**

**Analysis**
**Services**

**Data Services**

**On-line**
**data**

**Archival**
**data**

**Publication**

**Portals**

**Servers**

**Middleware**

**General and Use**
**Metadata**
**catalog**

**Ancillary**
**catalog**

**Security**
**(Authen+Author)**
**Services**

**Earth Science Data Grid (ESG II)**
**Architecture**

**Dataset**
**Metadata**
**Services**

**Discovery**
**Metadata**
**Services**

## Storage Resource Management

**Storage Resource Management**
**A Collaboratory**
**middleware project**
**Arie Shoshani**
**Alex Sim**
**Junmin Gu**
**Computing Sciences Directorate**
**Lawrence Berkeley National Laboratory**
**http://sdm.lbl.gov/srm**

## Motivation

**Grid architecture emphasized in the past**
**Security**
**Compute resource coordination & scheduling**
**Network resource coordination & scheduling (QOS)**
**SRMs role in the data grid architecture**
**Storage resource coordination & scheduling**
**Types of storage resource managers**
**Disk Resource Manager (DRM)**
**Tape Resource Manager (TRM)**
**Hierarchical Resource Manager (TRM + DRM)**

## Where Do SRMs Fit in Grid Architecture?

- property-file
-  index

- logical
- query

- pinning & file
- transfer requests

- network

- ...

- logical files

- site-specific
-  files

**Client’s site**

- ...

- site-specific
-  files requests

## Challenges (1)

**Managing storage resources in an unreliable distributed large heterogeneous system**
**Long lasting data intensive transactions**
**Can’t afford to restart jobs**
**Can’t afford to loose data, especially from experiments**
**Type of failures**
**Storage system failures **
**Mass Storage System (MSS)**
**Disk system**
**Server failures**
**Network failures**

## Challenges (2)

**Heterogeneity**
**Operating systems (well understood)**
**MSS - HPSS, Castor, Enstore, ...**
**Disk systems – system attached, network attached, parallel**
**Optimization issues**
**avoid extra file transfers - What to keep in each disk caches over time **
**How to maximize sharing for multiple users**
**Global optimization**
**Multi-Tier storage system optimization**

## Specific Problems

**Managing resource space allocation**
**What if there is no space?**
**Managing pinning of files**
**What if files can be removed in the middle of a transfer**
**Space reservations**
**What if multiple files are needed concurrently**
**File streaming**
**For processing a large set of files**
**Pin-lock**
**What if you pinned files, and system deadlocks**
**User priorities**
**Access control – who can read/write a file**

## HRMs in PPDG (high level view)

- HRM-COPY

- HRM-GET

**Replica **
**Coordinator**

**HRM**
**(performs writes)**

**HRM**
**(performs reads)**

-  Monitors files written   into BNL’s HPSS
-  Selects files to replicate
-  Issues request_to_put  for file (or many files)

- LBNL

- BNL

- GridFTP GET (pull mode)

## Measurements

- FILE_REQUEST_FAILED

- Notified_Client

- Migration_Finished

- Migration_Requested

- Transfered_to_PDSF_from_BNL

- Staging_finished_at_BNL

- Staging_started_at BNL

- Staging_requested_at_BNL

- File replication request start

## The Other Talks

*Request interpreter*

*Logical request*

*Qualified objects*

*Storage Resource Managers*

*HPSS/shared disk*

** Grid Enabled Access*

*File access management *** *

*Request Manager*

*Selected Logical objects*

**Bitmap Indexing**
**(John Wu)**

**Shared Disk File Caching**
**(Ekow Otoo)**

**Optimizing Shared Access **
**to Tertiary Storage**
**(Randy Burris)**

## P3 Tasks

**Deployment of compressed BitMap index (COMBIX) for HEP and Combustion applications (millions-billions objects)**
**Logical range query to find qualified files - HENP**
**Logical range query to find “flame fronts” – Combustion**
**Developing optimal disk caching policies**
**Using simulation and real test with DRM**
**Testing a new caching policy method based on “hazard rates”**
**Deploying of HRM at ORNL and BNL for use with Climate and HENP applications**
**To support data movement of files to NERSC for climate simulation production data**
**To support event subset access for HENP simulations  **
**Developing efficient access to HPSS (ORNL)**
**Parallel streams**
**Partial file reads**