# Presentation: 184000

## Data Placement on the 

**Data Placement ****on th****e**** **

## Customer orders:Place y = F(x) at L!Grid delivers. 

## Planning, scheduling, 

**Planning, scheduling, **
**execution,**
**error recovery, **
**monitoring ... **

**Logical Request**

**Physical Resources**

## Key Challenges

- Trustworthy and Robust services
- Effective Communication between consumer and provider of services
- Tell me what to do
- Tell what you did and what happened
- Tell me what is going on
- Reasonable (planning, scheduling, recovery) policies (on client and servise sides)

## Define your Objective(s)!!!

- Minimize turn around time (avg, std, max ...)
- Maximize throughput (all customers, time interval, ... )
- Minimize FTE costs
- Maximize resource utilization

## A simple plan for y=F(x) -> L

- Allocate size(x)+size(y) at SE(i)
- Move x from SE(j) to SE(i)
- Place F on CE(k)
- Compute F(x) at CE(k)
- Move y to L
- Release allocated space

- Storage Element (SE); Compute Element (CE)

## What we have here is a simple six-nodesDirected Acyclic Graph(DAG)

- Logical vs Physical

## Some of the Services Needed:

- Collection and retrieval of information  – what is where, who can do what, ...
- Planning of a physical Directed Acyclic Graph (DAG)
- All jobs at DAG submission time (eager)
- A job at a time when job is ready to go (lazy) or when resource is available (very lazy)
- Any combination of the above ...
- Execution of a DAG – A DAG Manager dispatches a node (job) when the conditions it waits for are satisfied.
- Scheduling of Data Placement (DaP) and Processing jobs
- Allocation of storage space – Give me a “lot” of X MB for Y days at location Z and call it L
- Transfer files – Move file F (or part of) from A to B and place it in L
- Run job J on machine M
- Undo/release allocations, placement, job submissions ...

## The Layers of Condor

**Resource**

**Local Resource Manager**

**Owner Agent**

**Matchmaker **

**Customer Agent**

**Application Agent**

**Application**

**Remote Execution Agent**

**Submit**
**(client)**

**Execute**
**(service)**

## Grid

**Fabric (****processing, storage, communication****)**

**Grid**

## PPDG-MOP Demonstration at SC-01

**PPDG-MOP Demonstration at SC-01**

## How Does MOP Work?

- From the perspective of the CMS production system (IMPALA), MOP is almost like a local batch system.  Instead of submitting jobs to PBS or Condor, the system can submit them to MOP.
- For each physics job that IMPALA submits to MOP, MOP creates a DAG containing sub-jobs necessary to run that job on the Grid...

## MOP Job Stages

- Stages

**MOP Job Stages**

- Stage-in – get the program and its data to a remote site
- Run – run the job at the remote site
- Stage-back – get the program logs back from the remote site
- Publish – advertise the results so they will be sent to sites that want it
- Cleanup – clean up remote site

## MOP Job Stages

- MOP combines the five-stage DAG for each IMPALA job into one giant DAG, and submits it to DAGMan.

- Combined DAG

## Slide 15

## Client

**Client**

**Application**

**DAGMan**

**Condor-G**

**DaPSched**

**  ****NeST**

**RFT**

**Planning**

**GateKeeper**

## NeST

- Develop a portable self-contained storage “appliance”
- Lot management
- User Management
- File Management
- File transfer support
- POSIX support

## Multi-protocol support

- Interoperability between admin domains
- NeST currently speaks
- Grid FTP and FTP
- HTTP
- NFS (beta)
- Chirp
- Designed for integration of new protocols

## Design structure

- Storage Mgr

- Chirp

- FTP

- Grid ftp

- NFS

- Common protocol layer

- HTTP

- Physical network layer

- Physical storage layer

**Design structure**

## Storage Management

- NeST makes storage space schedulable and reservable
- NeST allows creation of dynamic users
- Use OS inter user “protection” capabilities
- NeST offers lots
- Reservations of time and space
- Maps files to lots
- Uses OS quota enforcement

## Data Placement (DaP) Jobs

- Define, manage and schedule DaPs like any other “CPU” job.
- Interface with providers of storage services (NeST).
- Interface with providers of file transfer and caching services (GridFTP, SRB, RFT, DRM).
- Logging monitoring and error recovery

##  Monitoring

- Exploit Condor technology to offer monitoring (and administration) services:
- Deployment and management of agents
- Runing “probes”
- ClassAds and matchmaking
- Information gathering

## Hawkeye Architecture

**Hawkeye Architecture**

**Master**

**Hawkeye**
**Agent**

**Probe****i**

**Master**

**Hawkeye**
**Collector**

**Probe****i**

- LDIF

- Config

**Web Interface**

- GRIS

## A simple plan for y=F(x)->L

- Allocate size(x)+size(y) at SE(i)
- Move x from SE(j) to SE(i)
- Place F on CE(k)
- Compute F(x) at CE(k)
- Move y to L
- Release allocated space

## Logical!!! 

**Do not be**
**picky, be**
**Logical!!!**** **