# Presentation: 325331

## Just-in-time Staging of Large Input Data for Supercomputing Jobs

- Just-in-time Staging of Large Input Data for Supercomputing Jobs

- Henry Monti, Ali R. Butt

- Sudharshan S. Vazhkudai

## HPC Center Data Stage-in Problem

- Data stage-in entails moving all necessary input files for a job to a center’s local storage
- Requires significant commitment of center resources while waiting for the job to run
- Storage failures are common, and users may be required to restage data
- Delaying input data causes costly job rescheduling
- Staging data too early is undesirable
- From a center standpoint:
- Wastes scratch space that could be used for other jobs
- From a user job standpoint:
- Potential job rescheduling due to storage system failure
******Coinciding Input Data Stage-in time with job execution time improves HPC center serviceability**

## Current Methods to Stage-in Data

- No standardized method
- Utilizes common point-to-point transfer tools:
- GridFTP, HSI, scp, ...
- Limitations
- Entail early stage-in to ensure data availability
- Do not leverage orthogonal bandwidth
- Oblivious to deadlines or job start times
- Not an optimal approach for HPC data stage-in

## Our Contribution: A Just-in-time Data Stage-in Service

- Coincides data stage-in with job start time
- Attempts to reduce overall scratch space usage
- Uses an army of intermediate storage locations
- Allows for quick stage-in during storage failures
- Integrates with real-world tools
- Portable Batch System (PBS)
- BitTorrent
- Provides a fault-tolerant way to stage data while efficiently utilizing scratch space

**Notes:**
To address these problems we propose

## Storage system failures may require data to be re-staged!

- Transfer complete much earlier than job startup time

- Job Rescheduled!

**Notes:**
_x0003_omplete transfer much earlier than job startup time
Resend arrow
Turutle
Job reschuledued

##  Provides better opportunities for accomplishing JIT Staging

-  Provides better opportunities for accomplishing JIT Staging

- Time between stage-in and job start minimal

**Notes:**
Fanout
User choosing ul as necessary
Solve many problem of point-to-point

Hare
Time between stage-in and jobstrart

## Challenges Faced in Our Approach

- Obtaining accurate job start times
- Adapting to dynamic network behavior
- Ensuring data reliability and availability
- Managing deadlines during Stage-in
- Utilizing intermediate nodes
- Providing incentives to participate

**Notes:**
Number

## Obtaining Accurate Job Start Times 

- Accurate estimates of job start time needed
- Prevent job delay with Just-in-time staging
- Batch Queue Prediction (BQP)
- Provides statistical upper bound on job wait
- Predicts probability of job starting by the deadline
- Obtain predictions of job start time from BQP
- Stage-in data using this deadline

**Notes:**
completes design

## Adapting Data Distribution To Dynamic Network Behavior

- Available bandwidth can change
- Distribute data randomly - may not be effective
- Utilize network monitoring
- Network Weather Service (NWS)
- Provides bandwidth Measurement
- Predicts future bandwidth
- Choose dynamically changing data paths
- Select enough nodes to satisfy a given deadline
- Monitor and update the selected nodes

- 10 Mb/s

- 5 Mb/s

- 4 Mb/s

- 1 Mb/s

- ?

**Notes:**
Question mark, 4 after

## Protecting Data from Intermediate Storage Location Failure

- Problem: Node failure may cause data loss
- Solution:
- Use data replication
- Achieved through multiple data flow paths
- Employ Erasure coding
- Can be done by the user or at the intermediates

## Managing Deadlines during Stage-in

- Use NWS to measure available bandwidths
- Use Direct if it can meet a deadline
- Otherwise, perform decentralized stage-in
- If end host fails or cannot meet deadline
- Utilize decentralized stage-in approach

*T**Stage** <= T**JobStartup*

**Notes:**
Maybe add BPQ

## Intermediate Node Discovery

- User specifies known and trusted nodes
- Utilize P2P Overlay
- Nodes advertise their availability to others
- Receiving nodes *discovers  *the advertiser
- Discovered nodes utilized as necessary

**Notes:**
Solution found node, arrange them them in a peer to peer:

## P2P Data Storage and Dissemination 

- P2P-based storage
- Enables robust storage of data on loosely coupled distributed participants: CFS, PAST, OceanStore, ...
- P2P-based multicast
- Enables application-level one to many communication
- Example: BitTorrent
- Uses a scatter-gather protocol to distribute files
- Leverages Seeds - peers that store entire files
- Employs a tracker to maintain lists of peers
- Uses a “torrent file” containing metadata for data retrieval

**Notes:**
Add slide find nodes user specified disovery

## Incentives to Participate inStage-in Process

- Modern HPC jobs are often collaborative
- “Virtual Organizations” - set of geographically distributed users from different sites
- Jobs in TeraGrid usually from such organizations
- Resource bartering among participants to facilitate each others stage-in over time
- Nodes specified and trusted by the user

**Notes:**
Orthogonly problem user

## Integrating Stage-in with PBS

- Provide new PBS directives
- Specifies destination, intermediate nodes, and deadline
*	***#PBS -N myjob**
**	****#PBS -l nodes=128, walltime=12:00**
**	****mpirun -np 128 ~/MyComputation**
**	****#Stagein file://SubmissionSite:/home/user/input1**
**	****#InterNode node1.Site1:49665:50GB**
**	****...**
**	****#InterNode nodeN.SiteN:49665:30GB**
**	****#Deadline 1/14/2007:12:00**

## Adapting BitTorrent Functionality to Data Stage-in 

- Tailor BitTorrent to meet the needs of our stage-in
- Restrict the amount of result-data sent to a peer
- Peers with less storage than the input size can be utilized
- Incorporate global information into peer selection
- Use NWS bandwidth measurements
- Use knowledge of node capacity from PBS scripts
- Choose the appropriate nodes with storage capacity
- Recipients are not necessarily end-hosts
- They may simply pass data onward

## Evaluation: Experimental Setup

**Evaluation: Experimental Setup**

- Objectives
- Compare with direct transfer, and BitTorrent
- Validate our method as an alternative to other stage-in methods
- PlanetLab test bed
- 6 PlanetLab nodes:
- 	center + end user + 4 intermediate nodes
- Experiments:Compare the proposed method with
- Point-to-point transfer (scp)
- Standard BitTorrent
- Observe the effect of bandwidth changes

**Notes:**
Proof of concept initial evaulation

## Results: Data Transfer Times with Respect to Direct Transfer

**Results: Data Transfer Times with Respect to Direct Transfer**

- Times are in seconds

| File Size | 100 MB | 240 MB | 500 MB | 2.1 GB |
| --- | --- | --- | --- | --- |
| Direct | 172 | 351 | 794 | 3082 |
| Client Offload | 139 | 258 | 559 | 2164 |
| Pull | 43 | 106 | 193 | 822 |

- A JIT stage-in is capable of significantly improving transfer times

## Results: Data Transfer Times with Respect to Standard BitTorrent

**Results: Data Transfer Times with Respect to Standard BitTorrent**

- Times are in seconds
- Transferring 2.1 GB file

| Phase | BitTorrent | Our Method |
| --- | --- | --- |
| Send to all intermediate nodes 
(Client Offload) | 2653 | 2164 |
| Submission site download (Pull) | 960 | 822 |

- Monitoring based stage-in is capable of outperforming standard BitTorrent

**Notes:**
Increase file size
Monitoring based offloading is cap...

## Conclusion

- A fresh look at Data Stage-in
- Decentralized approach
- Monitoring-based adaptation
- Considers deadlines and job start times
- Integrated with real-world tools
- Outperforms direct transfer by **73.3% **in our experiments

## Future Work

- Measuring scratch space savings
- Measuring potential job delays
- Testing other stage-in scenarios
- Contact
- Virginia Tech.
- Distributed Systems and Storage Lab._[http://research.](http://research.cs.vt.edu/dssl/)__[cs](http://research.cs.vt.edu/dssl/)__[.](http://research.cs.vt.edu/dssl/)__[vt](http://research.cs.vt.edu/dssl/)__[.](http://research.cs.vt.edu/dssl/)__[edu/dssl/](http://research.cs.vt.edu/dssl/)_
- {hmonti, butta}@cs.vt.edu
- ORNL
- _[http://www.](http://www.csm.ornl.gov/~vazhkuda/Storage.html)__[csm](http://www.csm.ornl.gov/~vazhkuda/Storage.html)__[.](http://www.csm.ornl.gov/~vazhkuda/Storage.html)__[ornl](http://www.csm.ornl.gov/~vazhkuda/Storage.html)__[.](http://www.csm.ornl.gov/~vazhkuda/Storage.html)__[gov/~vazhkuda/Storage](http://www.csm.ornl.gov/~vazhkuda/Storage.html)__[.html](http://www.csm.ornl.gov/~vazhkuda/Storage.html)_
- vazhkudaiss@ornl.gov

## Evaluation Objectives 

- Compare with direct transfer, and BitTorrent
- Observe how system reacts to failures and bandwidth fluctuations:
- How SLAs are enforced?
- How fault tolerance is achieved?
- Validate our method as a viable alternative to other stage-in methods

**Notes:**
Pro

## Overlay Networks

**Overlay Networks**

- P2P networks are self-organizing overlay networks without central control

- ISP3

- ISP1

- ISP2

- Site 1

- Site 4

- Site 3

- Site 2

- N

- N

- N

- N

- N

- N

- N

## Structured P2P Overlays

- Overlays with imposed structure
- Each node has a unique random **nodeId**
- Each message has a key
- The **nodeId** and key reside in the same name space
- Routing: Takes a message with a key and sends it to a unique node
- Implements Distributed Hash Table (DHT) abstraction
- DHT abstraction is preserved in the presence of node failure/departure
- Many implementations available, e.g. Pastry, Tapestry, Chord, CAN ...

**Notes:**
Simple slide about explanation of p2p overlays
Messages unique mapped to node