# Presentation: 692676

## The Ka toolsandOSCAR

- Simon Derr, INRIA
- Simon.Derr@imag.fr

## Goals of this presentation

- Integrate some ideas of Ka in OSCAR
- Establish a collaboration between INRIA and OSCAR

## Who are we ?

- INRIA : institut national de recherche en informatique et automatismes
- French public institute that does research in computer science
- the APACHE project
- 	City of Grenoble
- Fundings from MS, BULL for previous works
- Fundings from the French Govt for a “cluster oriented Linux distribution” in association with Mandrake.

## ID-Apache

**Objectives :  ****Distributed **** computing**
- Cluster  of multiprocessors (CLUMP) for CPU intensive applications
- Performance, “easy access”, scalability, heterogeneity and resilience
**Research directions**
- Parallel programming model
- Scheduling and load balancing
- Management tools
- Parallel algorithms
**Validation**
- A parallel programming environment Athapascan
- For real applications
- On significant parallel platforms (few hundreds to thousands)

**Notes:**
The target of our work is mainly what can be referred to as ‘CLUMPS’[...]
The main goals we are looking for on this platform are : performance, an easy access, scalability (to be able to work on large numbers of machines), heterogeneity (to have different kind of hardware working together), and resilience (to work on a group of machines, that the machines can leave or that new machines can join at any time)

The main research directions are: parallel programming model (make // progs more easily and efficiently that the classical explicit message passing techniques), scheduling and load balancing, management tools for // machines, and also // algorithms.

The validation comes from : an implementation of a // prog. Env. : Athapascan, that is being used to write real world applications, on run on significant platforms, such ash this one.

## Interest in clusters of PC’s

- One-year old cluster of 225 uniprocessors PIII
- 100 mbit fast ethernet
- Process of buying a more powerful machine
- Around 128 dual-processor nodes
- High performance network

## Ka tools

- Scalable tools
- Designed to fulfill the needs we had on our 225-node fast-ethernet cluster
- Ka-deploy
- OS** **installations
- Ka-run
- launching of parallel programs, run commands on the cluster
- files distribution
- And also...
- Monitoring
- Distributed NFS

**Ka tools**

**sderr:**
- On arrive dans ce qui me concerne

**Notes:**
I’ll talk a bit more deeply about what we have called... THE KA(kay)-tools
Scalable tools for management/use of big clusters 
Most problems can be summed up by two Main problems: 1/running commands 2/file broadcast – quick/efifcient
General idea : many machines  use Multicast. But our experimentation showed that we can achieve better performances on the kind of network we have on the icluster by using ‘trees’, both for 1/ and 2/
Dummy example : the binomial tree : for a given operation, do it in 2^n nodes with only n steps
Questions : which tree for what ?
And the trees gave us the possibilty to write scalable and efficient tools using only very standard techniques : unmodified rsh daemon, standard TCP connections,..

## Idea behind Ka

- 2 goals
- Contact many nodes from one node (contact = run a remote command)
- Send big amounts of data to many nodes from one node
- On our ‘slow’ switched fast-ethernet network
- Problem : source node bottleneck
- One common solution : trees

## Using trees to run a command

- Objective : quickly contact many nodes (contact = rsh)
- Contacting many nodes from a single host produces a lot of network traffic and cpu work
- Idea: contact a few nodes and then delegate some of the work to the nodes that have already been contacted == use a tree
- 	ex: binomial

## Using trees to run a command

- Implementation : *rshp*

## Comparison with C3

- Running commands with C3 *cexec*
- All nodes contacted by a single node
- Network  traffic
- A process forked() for each destination node -> high cpu load on the source node
- Running commands with rshp-enabled *cexec*
- Each node contacts only a few other nodes
- No per node fork() (when rsh -not ssh- is used)
- Tree brings scalability

## Comparison with C3

- Time to run the *uname* command on 130 machines of our cluster:
- Time with *cexec*: 0:02.07 elapsed 85%CPU
- Time with *rshp-enabled cexec* : 0:01.50 elapsed 8%CPU
- Using a binomial tree
- Future : Non-blocking connect() calls to improve speed

## Using trees to send data

- Objective : high bandwidth
- Idea : create a structure of TCP connections that will be used to send the data to all the machines

**sderr:**
- Ce transparent la est un peu lourd
- Vivement le dessin

- On a SWITCHED ethernet-like network:
- One node receiving data and repeating them to N other nodes
- Bandwidth = network bandwidth / N

## Using trees to send data

- Objective : high bandwidth
- Idea : create a structure of TCP connections that will be used to send the data to all the machines
- On a switched network where the bandwidth limitation is *per host *and where there is no limitation on the total traffic on the network. *where* the bottlenecks are the hosts and not the switchs...
- Consider a node on a fast ethernet network (full duplex). If it receives data on one side, and sends it to *k *nodes on the other, it can do so at the speed of 100/k mbit/s (in real world, ~about 10/k MB/s)
- 						+pitit schema de 1 noeud

**sderr:**
- Ce transparent la est un peu lourd
- Vivement le dessin

## Using trees to send data

- Binary tree on a fast ethernet network : ~ 5 MB/s

- Chain tree on a fast ethernet network : ~ 10 MB/s
-  BUT tree creation takes longer (very deep tree)

## File transfer

**Notes:**
Small files -> binary tree
Big files -> chain

## Topologies and their use

- Chain : best  when start-up is small, eg file transfert

- Binomial : best when synchronous, eg  rsh

- Binary : better for
- subsequent comm.,
- eg monitoring

- Sequential : best when asynchronous until ....eg rsh  async.

**sderr:**
- Ce transparent la doit disparaitre
- En fait il doit etre vampirise : piquer les schemas et les mettre ailleurs

**Notes:**
Depending on what one wants to do, the tree to use is not always the same.
Examples: chain -> big files
	      sequential (flat tree): command on few nodes / asynchronous rsh
	     binary : not-so-big files

## Comparison with C3

- Sending files with C3 *cpush*
- Use of rsync : efficient for modified files
- Sending new files *(blind mode):*
- Network bottleneck on the sending node
- Transfer time linear / number of nodes
- Sending files with rshp-enabled cpush
- rshp duplicates stdin : sending a file is merely :
- cat filein | rshp *options *dd of=fileout
- Transfer time *almost* independent / number of nodes

## Comparison with C3

- Time to send a 30MB file to 20 nodes:
- Time with *cpush*: 1:12.67 elapsed 99%CPU
- Time with *rshp-enabled cpush* : 0:05.88 elapsed 21%CPU

## Possible integration with C3

- Current C3 code handles inter-cluster stuff, reads the cluster description files, parses the command line, ...
- Rshp only handles and accelerates intra-cluster communications for *cexec*, and intra-cluster data transmission in *cpush*’s *blind* mode.
- For now only if C3_RSH is ‘rsh’
- Next version of *rshp* should be able to use ssh

## Ka-deploy

- Scalable operating system installation (*almost*)
- Node duplication
- PXE-capable cluster nodes network-boot and use a TCP chain-tree to efficiently transfer OS files
- Works on Linux, for Linux and Windows

**sderr:**

**Notes:**
Now I’m gonna present you the tool we have developed for operating system installation on our cluster
Goal: scalable OS install
How it works : we have a very homogenous system -> machine cloning (for know)
Much data - chain tree
Network boot, diskless Linux system, ...
-------------- ka-deploy DEMO --------------

## Ka-deploy

- Speed : installation of a 1-2 GB system on 200 machines can take less than 15 minutes
- Very little flexibity
- Machines must be homogenous
- Very painful to set up

## Ka-deploy and LUI

- Same environment : PXE boot, etc...
- Different goals:
- LUI is headed towards flexibility, and ease of use
- Ka-deploy is headed towards speed and scalability
- Maybe the diffusion scheme used by ka-deploy can be added in LUI
- But with SIS ??

##      NFS server for clusters

***Interconnect (udp)***

-  **Slaves**

**Master**

**Request**

**Distributed **
**files**

**The cluster is the file system**

** ****NFS client unchanged**
**  ****files placement**
** ****parallel access**
** ****scalability  ??**
** ****optimized read**
** ****write??**

-      NFS server for clusters

**sderr:**
- Demander a Pierre pour le ‘adaptive’

**Notes:**
As an example of what has been done on this cluster, a member of the project, Pierre Lombard, has developped a distributed NFS server.
-much unused disk space on the cluster
....
Questions remain: scalability,...

## Conclusion

- Very interested in a collaboration
- Some manpower, and one (soon 2) clusters for testing
- Visitors are welcome
- Maybe even host a future meeting
- Other research directions:
- Peer to peer machine cloning
- Intranet clusters
- Web : icluster.imag.fr, ka-tools.sourceforge.net