# Presentation: 805463

## PVM and MPI 

- PVM and MPI
- What Else is Needed
- For Cluster Computing?

**Al Geist**
**Oak Ridge National Laboratory**
- www.csm.ornl.gov/~geist

- DAPSYS/EuroPVM-MPI
- Balatonfured, Hungary
- September 11, 2000

##             EuroPVM-MPI

**Dedicated to the hottest developments of PVM and MPI **

- PVM and MPI are the most used tools for parallel programming
- The hottest trend driving PVM and MPI today is PC clusters running Linux and/or Windows
- This talk will look at gaps in what PVM and MPI provide for Cluster Computing.
- What role the GRID may play and
- What is happening to fill the gaps...

## New release this summer – PVM 3.4.3  includes:
Optimized msgbox routines
More scalable, more robust
New Beowulf-linux port
Allows clusters to be behind firewalls yet work together
Smart virtual machine startup
Automatically determines the reason for “Can’t start pvmd”
Works with Windows2000
Installshield version available
Improved Win32 communication performance

**Optimized msgbox routines**
- More scalable, more robust
**New Beowulf-linux port**
- Allows clusters to be behind firewalls yet work together
**Smart virtual machine startup**
- Automatically determines the reason for “Can’t start pvmd”
**Works with Windows2000**
- Installshield version available
- Improved Win32 communication performance

**PVM Latest News: **

**New Third party PVM Software: **
**PythonPVM 0.9, JavaPVM, **
**PVM port using SCI**** interface**

## Ten Years of Cluster Computing

- 1989

- 94

- 90

- 96

- 97

- 99

- 2000

- PVM-1

- MPI-1

- MPI-2

- I-MPI

- PVM-2

- PVM-3

- PVM-3.4

- Harness

- Wide-area GRID experiments

**Ten Years of Cluster Computing**
**Building a Cluster Computing Environment for 21****st**** Century **

- Networks
- of Workstations

- PC Clusters

## TOP500 Trends – Next 10 years

**0.1**

**1**

**10**

**100**

**1000**

**10000**

**100000**

**1000000**

**Jun-93**

**Nov-94**

**Jun-96**

**Nov-97**

**Jun-99**

**Nov-00**

**Jun-02**

**Nov-03**

**Jun-05**

**Nov-06**

**Jun-08**

**Nov-09**

**Performance [GFlop/s]**

**N=1**

**N=500**

**N=10**

**1 TFlop/s**

**1 PFlop/s**

**2005 Entry at 1TFlop/s         2010 Peak at 1PFlop/s **

**TOP500 Trends – Next 10 years**
**Even the largest machines are clusters **

- http://www.netlib.org/benchmark/top500.html

## Trend in Affordable PC clusters 

- PC Clusters are cost effective from a hardware perspective
- Many Universities and companies can afford 16 to 100 nodes.
- System administration is an overlooked cost:
-     people to maintain cluster
-     software written for each cluster
-     higher failure rates for COTS
**Presently there is lack of tools for managing large clusters**

- www.csm.ornl.gov/torc

**Trend in Affordable PC clusters **

## Cluster Computing Tools 

**C3 Command line cluster toolset**

**Cluster Computing Tools **

-  cl_pushimage() – push system image across cluster
-  cl_shutdown() – shutdown specified nodes
-  cl_push() – push files/directories across cluster
-  cl_rm() – remove files from multiple nodes
-  cl_get() – gather cluster files to one location
-  cl_ps() – returns results of multi-node ps
-  cl_kill() – kill application across entire cluster
-  cl_exec() – execution of any command across specified nodes

_**Functions**_

**C3 is a command line based toolset for system administration and user level operations on a single cluster. C3 functions may also be called in a program. C3 is multithreaded and each function executes in parallel.**

- only executable
- by sysadmin

**Software available   www.csm.ornl.gov/torc**

## VTK ported to Linux Clusters
Visualization toolkit making use of C3 cluster package
AVS Express ported to PC Cluster
Expensive but standard package asked for by apps. 
Requires AVS Site license to eliminate the cost of individual node licenese
Cumulvs plug-in for AVS Express
Combines the interactive visualization and computational steering of Cumulvs with the visualization tools of AVS.

- Visualization toolkit making use of C3 cluster package
**AVS Express ported to PC Cluster**
- Expensive but standard package asked for by apps.
- Requires AVS Site license to eliminate the cost of individual node licenese
**Cumulvs plug-in for AVS Express**
- Combines the interactive visualization and computational steering of Cumulvs with the visualization tools of AVS.

**Visualization using Clusters **

**Lowering the Cost of High Performance Graphics**

## Cluster Computing Tools 

**M3C tool suite**

**Cluster Computing Tools **

-  Reserve nodes within or across clusters
-  Submit Job to queue system
-  Monitor nodes of cluster – also adding a PAPI interface
-  Install software on selected nodes
-  Reboot, shutdown, add user, etc. to selected cluster nodes.
-  Display properties of nodes

_**Growing list of plug-in modules**_

**Suite of user interface tools for system administration and simultaneous monitoring of multiple PC clusters.**
**Written in Java – web based remote access.**

- www.csm.ornl.gov/torc

## OSCAR

**OSCAR**

-  Bring uniformity to cluster creation and use
-  Make clusters more broadly acceptable
-  Foster commercial versions of the cluster software

_**Goals**_

**OSCAR is a collection of the best known software for building, programming, and using clusters. The collection effort is lead by a national consortium which includes: IBM, SGI, ****Intel, ORNL, NCSA, MCS Software. Other vendors invited.**

**www.csm.ornl.gov/oscar**

**National Consortium for Cluster software**

**For more details see Stephen Scott**
- (talk Monday 12:00 DAPSYS track)

## ORNL M C Tool

**Architecture designed to work both within and across organizations **
** **

**3**

- URL    CGI

- C3 Scripts/monitors

- URL    CGI

- Custom Scripts

- URL    CGI

- Third-party Scripts

- GUI

- proxy

- URL

- front-end

- back-end

- Java applet
- Based GUI

- cluster 1

- cluster 2

- cluster 1

- cluster 1

- Interface thru
- XML files

- M3C proxy allows one sysadmin
- to monitor and update multiple
- clusters

- M3C GUI allows
- user to submit
- and monitor jobs

- ORNL

- UTK

- SDSC

## Information services
Uniform naming, locating, and allocating distirbuted resources
Data management and access
Single log-on security

- Information services
- Uniform naming, locating, and allocating distirbuted resources
- Data management and access
- Single log-on security

**GRID “Ubiquitous” Computing**** **

**GRID Forum is helping define higher level services**

**MPI and PVM are often seen as**
**lower level capabilities that**
**GRID frameworks support.**

- Globus
- NetSolve
- Condor

- Legion
- Neos
- SinRG

## Cumulvs – collaborative computational steering 

**Cumulvs ****– collaborative computational steering**** **

**Recent Highlights**

- Release of new version
- Development of CAVE viewer
- Works w/ PNL global arrays
**Made CCA compliant**

**Cumulvs was the initial reason we started Harness.**

**Collective port**

- A DOE effort to provide a standard for interoperability of high performance components developed by many different groups in different languages or frameworks.

**Common Component Architecture**

- http://z.ca.sandia.gov/~cca-forum/port-spec

## HARNESS 

**Exploring New Capabilities in Heterogeneous Distributed Computing**

- www.epm.ornl.gov/harness

** ****Parallel Plug-in environment  **
- Extend the concept of a plug-in to the parallel computing world.
- Dynamic with no restrictions on functions.
** ****Distributed peer-to-peer control  **
- No single point of failure unlike typical client/server models.
** ****Multiple distributed virtual machines merge/split   **
- Provide a means for short-term sharing of resources and collaboration between teams.

- Building on our experience and success with PVM
- create a fundamentally new heterogeneous virtual machine
- based on three research concepts:

**Goal**

## HARNESS Virtual Machine

- Host D

- Host C

- Host B

- Host A

- Virtual
- Machine

- Operation within VM uses
- Distributed Control

**HARNESS Virtual Machine**
**Scalable Distributed control and Component based Daemon **

## HARNESS Latest News 

**Provide a practical environment and illustrate extensibility **

** ****Harness Core **(beta release ready)(see talk Monday 16:30 Track 1)** **
- Task library and Harness Daemon software
- Provides API to load, unload plug-ins and distributed control.
** ****PVM Plug-in **(stalled for summer now back on track)
- Provides PVM API veneer to support exiting PVM applications.
** ****Fault tolerant MPI plug-in **(see talk Monday 16:50 Track 1)**  **
- Provides MPI API for 30 most used functions. Semantics adjusted to allow recovery from corrupted communicator.
** ****VIA communication plug-in **(looking at multi-interface transfer)
- To illustrate how different low level communication plug-ins
- can be used within Harness. And to provide high performance

## Parallel Plug-in Research

**Parallel Plug-in Research**
**For Heterogeneous Distributed Virtual Machine**

- Tough Research problems include:
- Heterogeneity (has delayed ‘C’ H-core development.)
- Synchronization - Dynamic installation
- Interoperation
-  between same plug-in on different tasks
-  between task plug-in and daemon plug-in
-  between daemon plug-ins
- Partial success

**One research goal is to understand and implement **
**a dynamic parallel plug-in environment.**

- provides a method for many users to extend Harness
- in much the same way that third party serial plug-ins
- extend Netscape, Photoshop, and Linux.

## Fault Tolerant MPI 

**Motivation   **

- As application and machine sizes grow
- the **MTBF is less than the application run time**.
- MPI standard is based on a static model so any decrease in tasks
- leads to **corrupted communicator **(MPI_COMM_WORLD).
- Develop MPI plugin that **takes advantage of Harness **
**robustness**  to allow a range of recovery alternatives
- to an MPI application. Not just another MPI implementation.
- FT-MPI follows the syntax of MPI standard
- Communication performance on par with MPICH
- Presently uses PVM3.4.3 fault recovery until Harness is ready)

## MPI_COMM_CREATE ( comm, group, newcomm)
MPI_COMM_SPLIT ( comm, color, key, newcomm)
Creates a new communicator that contains all surviving processes 
Allows MPI_COMM_WORLD to be specified as both input and output communicator.

**Fault Tolerant MPI**** **
**Recovery requires MPI semantic changes  **

- Key step to MPI recovery is creating a
- communicator that app can use to continue.
- Accomplished by modifying the semantics
- of two MPI functions.

- MPI_COMM_CREATE ( comm, group, newcomm)
- MPI_COMM_SPLIT ( comm, color, key, newcomm)
- Creates a new communicator that contains all surviving processes
- Allows MPI_COMM_WORLD to be specified as both input and output communicator.

## No single point (or set of points) of failure for Harness. It survives as long as one member still lives.

All members know the state of the virtual machine, and their knowledge is kept consistent w.r.t. the order of changes of state. (Important parallel programming requirement!)

No member  is more important than any other (at any instant) i.e. here isn’t a pass-around “control token”

- All members know the state of the virtual machine, and their knowledge is kept consistent w.r.t. the order of changes of state. (Important parallel programming requirement!)
- No member  is more important than any other (at any instant) i.e. here isn’t a pass-around “control token”

**Symmetric Peer-to-Peer Distributed Control**
**Characteristics**

## Distributed Control

- Harness kernels on each host
- have arbitrary priority assigned to them
- (new kernels are always given the lowest priority)

**Virtual machine**

- A  task on this host
- requests a new host be added

- VM state
- held by
- each kernel

**2.**

- Each adds request to a
- list of pending changes

**1.**

- Send host/T#/data
- to neighbor in ring

**3.**

**Distributed Control**
**Harness Two Phase Arbitration**

## Harness Distributed Control

- Supports
- multiple
- simultaneous
- updates

**Harness Distributed Control**
**Control is Scalable, Asynchronous, and Parallel**

- add
- host

- Fast
- host delete
- or recovery
- from fault

- Parallel recovery
- from multiple
- host failures

- Supports
- fast host
- adding

- Scalable Design

- 1<=S<=P

## Follow the links from my Web site

www.csm.ornl.gov/~geist

**www.csm.ornl.gov/~geist**

**For more Information**
**Also - Copy of these slides **