# Presentation: 305876

## Network metrics usage for optimization of the Grid

- DataGrid Project Work Package 7
- Written by Tiziana Ferrari
- Presented by Richard Hughes-Jones

## Why Grid Optimization?

- performance of Grid middleware and applications can be enhanced through the use of information about the status and behaviour of network connections
***Network-based Optimiser***
- the Grid building block that provids an estimate of the transmission quality between two or more Grid nodes
- can be used by any Grid service and aplication
- relies on *raw* network performance metrics regularly monitored by the *network monitor system *and* *available in the Grid information Service

## The network-based Grid optimization service

## Application scenarios of the Network-based Grid Optimizer  (NGO)

***Resource Brokerage**** *:
- given a set of candidate Computing Elements (CE) that are equivalently good for the execution of a given job, and a list of candidate Storage Elements (SE), where a given input file requested by the job is stored, the  **NGO** can identify the most suitable CE and SE according to a *combined* set of network criteria:
- packet loss minimization
- Round Trip Time  minimization
- throughput maximization

## Application scenarios of the Network-based Grid Optimizer  (NGO) - cont

***Data management:***
- Estimation of the time needed to transfer a file of known size from a source SE to a destination SE
- Identification of the most appropriate SE from which a given file replica can be retrieved
- given a set of input files or file fragments located in different SEs, identification of an individual SE in the set where such files or file fragments can be all transferred to, in the most efficient way

## Application scenarios of the Network-based Grid Optimizer  (NGO) - cont

***A******daptive remote file access***
- If the file/files needed by a given application are only known at run time, CEs and SEs cannot be selected by the Resource Broker statically prior to execution.
-       In this case the NGO allows the application to optimise the access to remote files at run time. The optimisation is based on the dynamic selection of the most appropriate Storage Element set that the application is using as the file access pattern changes.

## Network Cost Functions

- The Network- based Grid Optimizer is based on a set of *Network Cost Functions:*
- The network cost function is a *compound* network performance metric that provides a *high-level* view of connectivity between two or more  Grid nodes
- The network cost function is based on raw performance metrics (RTT, packet loss probability, available bandwidth, link utilization etc)
- The NGO applies different network cost functions according to the application scenario

## Cost functions: examples

- Example 1:
**File_size (bits) * Throughput (SourceSE, DestinationSE)**
- Example 2: *Closeness **C**i,j** **(p**l**i,j** ,r **i,j**,th **i,j** )** *
*Given*:
- 	**r****i,j**:    Round Trip Time between CEi and SEj
*	***R****max**: max s,t  *r**s,t*  for all CEs and SEt in a given set
*	***pl****i,j*** *:  packet loss probability on the network path connecting
- 	         CEi and     SEj’
*    ***th ****i,j*** *:  throughput of data transfer between CEi and SEj
-     **TH****max**: max s,t  *th**s,t* 
**C****i,j**** ****(p****l****i,j**** ,r ****i,j**** ****,th ****i,j**** ) ****=1***  **	*if CEi*  *is local to* *SEj
**C****i,j**** ****(p****l****i,j**** ,r ****i,j**** ****,th ****i,j**** ) ****=0** 	if SE unreachable or if  *p**l**i,j** **> P  *where P
- 				is a configurable threshold
**C****i,j**** ****(p****l****i,j**** ,r ****i,j**** ****,th ****i,j**** ) =   ****(****th ****i,j**** / ****TH****max****)**** **** * ******** **** (****  r ****i,j**** ****/**** ****R ****rmax****)***                             *
-    				where* 0 <****  **< 1** *otherwise

##  Closeness Ci,j (pli,j ,r i,j,th i,j )

- Fig 1: alpha = 1/2

- Fig 2: alpha = 1/10

## Implementation

- Two APIs currently implemented in the DataGrid project for the following two application cases:
- Resource Brokerage: selection of the best CE and SE for each input file needed by a given job
- File transfer between a source SE and a destination SE

## Conclusions and future work

- Use of high-level metrics for estimation of connectivity between two or more nodes has proved to be very important in the DataGrid project for the optimization of  Grid middleware and applications based on network transactions
- Future work:
- Study of new optimization scenarios
- Analysis of new cost functions
- analysis of Grid Information Service performance when used for the solution of complex optimization problems