# Presentation: 132013

## Development of Compatible Component Software(or CUMULVS, CCA, Harness, PVM, MPIand other assorted acronyms... )

- Development of Compatible Component Software(or CUMULVS, CCA, Harness, PVM, MPIand other assorted acronyms... )

- James “Jeeembo” Kohl
- Computer Science and Mathematics Division
- Oak Ridge National Laboratory

- Tuesday, July 24, 2001

**Research supported by the Mathematics, Information and Computational Sciences Office, Office of Advanced Scientific Computing Research, U. S. Department of Energy,**
**under contract No. DE-AC05-00OR22725 with UT-Battelle, LLC.**

**?**

## What a Great Idea!

- Let’s Componentize Software!
- Experts Write Many Specialized Components
-  Pick the Right Component, Then Plug & Chug...
- Software Re-Use, Interoperability
-  Define “Ports” for Component Interfaces
-  Connect Up Ports to Assemble Applications
- What Can Possibly Go Wrong?
-  Go Wrong... Go Wrong...  Go Wrong...
-             (echoes from 20 years ago...  still resonating... )

- ORNL

## Reality of Software Development

- Vendors Customize Operating Systems
-  Every Hardware Architecture is Different
- Version 1.1 May Not Be Compatible with 1.0
-  Software Developers Don’t Always Look “Back”
-  True of Both O.S. and Tools...
- My Dad Can Beat Up Your Dad!
-  Always Plenty of “Standards” to Choose From
-  Caught Wearing the Same Costume to the Ball
-  Force Interoperability After the Fact...  huhuhuhuh...

- ORNL

## “War” Story:  PVM and MPI

- PVM ~ Research Project, Started ‘89:
-  Message-Passing, Task & Resource Management, Heterogeneity, MPPs, Fault-Tolerance...
- MPI ~ Message-Passing “Standard”, ‘93:
-  Comprehensive Point-to-Point, Collective, Context, Communicators, Topologies, Profiling
- MPI_Connect / “PVMPI”, ’95...
- MPI-2 ~ When 250 Routines Isn’t Enough, ’95:
-  One-Sided, Spawn, I/O, Language Interop...
- IMPI ~ Interoperability Among MPIs, ’97...

- ORNL

## PVM versus MPI Reality...

- Different Design Goals:
-  Research Platform versus Ultimate Performance
-  PVM ~ Dynamics and Flexibility, Fault Recovery...
-  MPI ~ Fast, Static Model; No Daemons... (yet)
- Different Applications Have Different Needs
-  Either PVM or MPI could be the “Right Choice”
- Some “Convergence” Over Time...
-  PVM has added Context...
-  MPI has added Spawn...
- Coexistence...

- ORNL

## PVM versus MPI ~ Ramifications

- There is No ONE Message-Passing “Standard” that Pleases Everyone...
- Distributed Computing Tools Must Therefore:
-  Support BOTH Systems, Or...
-  Choose ONE and Abandon Users of the Other...
- Is This So Terrible?
-  Users “Mostly” Satisfied, One Way or Another...
-  Compatibility and Interoperability?  Um, NOPE.

- ORNL

## Collaborative Infrastructure for Interacting with Scientific Simulations On-The-Fly:
 Run-Time Visualization by Multiple Viewers
 Dynamic Attachment
 Coordinated Computational Steering
 Model  &  Algorithm
 Heterogeneous Checkpointing / Fault Tolerance
 Automatic Fault Recovery and Task Migration
 Coupled Models...

-  Run-Time Visualization by Multiple Viewers
-  Dynamic Attachment
-  Coordinated Computational Steering
-  Model  &  Algorithm
-  Heterogeneous Checkpointing / Fault Tolerance
-  Automatic Fault Recovery and Task Migration
-  Coupled Models...

- (_C_ollaborative,  _U_ser _M_igration,  _U_ser _L_ibrary  for  _V_isualization  and  _S_teering)

- www.csm.ornl.gov/cs/cumulvs.html

## CUMULVScoordinates the consistent collection and disseminationof information to / from parallel tasks to multiple viewers

- distributed parallel application or simulation
- supports most target platforms (PVM / MPI, Unix / NT, etc.)

**Unix Host A**

**NT Host B**

**Unix Host C**

**remote person**
**using virtual reality**
**interface**

**remote person**
**using AVS**

**local  person**
**using custom**
**GUI**

**exists in three pieces:**
**task part, viewer part,**
**and separate fault **
**recovery daemon**

## CUMULVS “War” Stories(of which there are many... )

- CUMULVS and MPI
-  Nexus / Mpich-G
-  MPI-1
-  MPI-2
- CUMULVS and Global Arrays
- CUMULVS and CCA

- ORNL

## CUMULVS  &  MPI

- CUMULVS Originally Written Over PVM
-  Dynamic Attachment ~ Independent Spawns
-  Application Discovery ~ Key-Value / Mbox
-  Fault Tolerance / Recovery ~ Notify, Host Add
-  Works Great!  (for applications written in PVM )
- What About MPI Applications?  Hmmm...
-  MPI Tasks Also Register as PVM Tasks
-  Application Uses MPI / CUMULVS Uses PVM
-  Yechhhh...  Functional, But Not User Friendly

- ORNL

## Solution #1:  CUMULVS & Nexus

- MPI Doesn’t Support:
-  Independent Spawn, Application Discovery, Fault Tolerance (with some upcoming exceptions...)
- Mpich-G (Globus) Built on Nexus
-  Nexus Supports Some Dynamics and Fault Notify
-  Port CUMULVS to Nexus
-  Application and “Viewer” Use Mpich-G!  
- What a Great Idea!  (Uh-Oh...)

- ORNL

## There’s Some Bad News,		and Some Worse News...

- Nexus is Message-Handler Based...
-  COMPLETELY!  No Point-to-Point Messaging!
-  O.K., so Convert CUMULVS to ALL Message-Handler Based Messaging...
-  Not so Bad, Good for Fault Tolerant State Protocols
-  PVM 3.4 Supports a Form of Message Handlers, too...
-  Lotsa Work, But Well Justified.  We Did It.

- 

- 

- BUTT, Nexus is DEAD?!  Bummer...  :-o
-  Globus Team Decided to Replace Nexus
-  Simple Messaging System in its Place...  D-Oh!

- ORNL

## Back to Square “One” (MPI-1)

- O.K., What If CUMULVS Wasn’t So “Cool”?  
-  Can We Still “Attach” to an MPI-1 Application?
- MPI-1 Has No “Spawn”...
-  Something Has To Be There All Along...
-  Start Extra CUMULVS Proxy Task in MPIRUN?
-  What About MPI_COMM_WORLD?
-  How to Exclude the Proxy Task from Collective Calls?
-  Force Applications to Use Alt World Communicator?!
- Requires Good Ole TCP/IP for Viewer Conns
-  Bogus, But Doable... (work in progress)

- ORNL

## Square “Two” (MPI-2)

- MPI-2 Has SPAWN!  :-D
-  Now We Can “Attach” Dynamically!
-  What a Great Idea!  (Uh-oh... )

- Hmmm...  Just Like “Get Smart” Cyanide Pill:
-  “How Do We Get Them to Take It?”  
-  Need Some Way to Externally “Trigger” Collective Spawn in MPI-2 Application...
-  File-Based Flag?  Another Simple TCP/IP Hook?
-  Good “Student” Project...  Heh, heh, heh...

- ORNL

## Another Tool Interoperability Study:  CUMULVS  &  Global Arrays

- Proof of Concept ~ Jarek @ PNNL
- Production Interface ~ David @ ORNL
- Idea:  Wrap Global Arrays for CUMULVS
-  Eases CUMULVS Instrumentation Hassle
-  CUMULVS Library Extensions for GA
-  “One-Call” Field Definitions for CUMULVS
- SUCCESS!  This Idea “Just Worked”... 
-  Minor Glitches:  Row Major / Column Major, 	    Processor Topology Munging...

- ORNL

## CUMULVS  &  CCA

- Symbiotic Relationship:
-  CUMULVS Drives “MxN” Development...
-  CCA MxN Interface “Stretches” CUMULVS
-  (Merging with Other Tools ~ PAWS, Meta-Chaos)
- SC Demos ~ CUMULVS “Eyes” Component
-  SC99:  Wrapped CUMULVS as One Component
-  SC00:  Started Pulling Apart CUMULVS
-  “Data Holders” Interfaced to CUMULVS (Like GA)
-  New “Eyes” Component Just Handled Comm
-  SC01:  First Shot at “Real” MxN Interface...

- ORNL

## CUMULVS  &  MxN

- MxN Parallel Data Redistribution Interface
-  Specification “0.5” Hot Off the Press... 
- Integrates CUMULVS, PAWS, Meta-Chaos
-  Several Approaches/Variations ~ One Interface
-  “One-Shot” Transfers, No Persistent Synchronization
-  “Periodic” Transfers, Ongoing Data Frames, a la Viz
-  Solidify & Formalize Internals ~ Comm Sched
- No One Tool Fully Implements MxN!
-  Everyone has to Stretch Their Tools...
-  Multi-Stage Development & Deployment

- ORNL

## MxN Technology Sources

- CUMULVS (ORNL):

- PAWS (LANL):

- Meta-Chaos (U Maryland):

- PADRE (LANL):

- Global Arrays
- (PNNL)

- PVM

- MPI

- KeLP
- (UCSD)

- PETSc
- (ANL)

- Other Data
-         Dist Tools...

- ORNL

## MxN Technology Integration

- Parallel Data
- Map & Comm

- Data Decomp
- Specification

- LOCAL DATA

- CUMULVS, PAWS, Meta-Chaos...		plus PADRE...

- CCA MxN
- Specification

- CCA MxN
- Component

- CCA Data
- Object

- New & Existing
- Tools, including:
- Global Arrays (PNNL),
- KeLP (UCSD), PETSc (ANL)...

- ORNL

## CUMULVS  &  CCA Data

- CCA Generalized Data Interface
-  Interface for Structured / Unstructured Meshes
-  Use to Describe Existing / Develop New Data
- Existing “MxN” Tools:
-  Will Require Modification for New Mesh Types
-  Nobody (in MxN) Handles Unstructured Very Well Yet
- CUMULVS “Particle” Interface...
-  Benefit From “Standardized” Interface
-  Now We Have a “Chance” at Handling Unstructured!
- CCA Data Should Ease Data Instrumentation...

- ORNL

## Harness*: Parallel Plug-in Virtual Machine Research 

- Harness*: Parallel Plug-in Virtual Machine Research

- www.csm.ornl.gov/harness

** ****Parallel Plug-in Environment  **
- Extend the concept of a plug-in to the parallel computing world.
** ****Distributed Peer-to-Peer Control  **
- No single point of failure unlike typical client/server models.
** ****Multiple Distributed Virtual Machines Merge/Split   **
- Provide a means for short-term sharing of resources and collaboration between teams.

**Building on our experience and success with PVM...**
** ****Create a fundamentally new heterogeneous virtual machine**
** ****Based on three research concepts:**

- * Collaborative Project ~ ORNL, UTK and Emory.

## Harness Motivated By Needs From Simulation Science

-  Develop applications by plugging together
***Component Models!***
-  Customize/tune virtual environment for application’s needs and for performance on existing resources.
-  Support long-running simulations despite maintenance, faults, and migration (dynamically evolving VM).
-  Adapt virtual machine to faults and dynamic
- scheduling in large clusters (DASE).
-  Provide framework for collaborative simulations (in spirit of CUMULVS).

- ORNL

## Subtle Warning:  Harness

- Harness Concept is Simple, Yet Powerful
-  Harness is Effectively a Component Framework
-  (What a Great Idea! )

- BUTT, the Devil is in the Details:
-  Huge Number of  “Base” Components Needed
-  The “Noose” of  Flexibility Hangs High...
-  Dynamic Pluggability
-  Can *Everything* be a “Plug-In”?  System Modules?
-  How Much Can Be Made Truly “Reusable”?

- ORNL

## True Component Compatibility?

- Will Component ***A*** Ever Be Re-Used?
-  Designed for Use with Component ***B***...
-  Component ***C*** has Something Different in Mind...
- True Re-Use Requires “Standards”:
-  Specific Method Invocation Interfaces
-  Data Description Interfaces
-  Semantic Standards?  Yikes!  (Run Away! )
- Each Application Domain Must Have Them!
-  If We’re Lucky, Some General Interfaces Will 	     Propagate...  Go SciDAC Go!  :-D

- ORNL

## Summary

- Obstacles to Overcome
- 			in “Componentland”
-  Multiple “Overlapping” Solutions Already Exist
-  PVM & MPI...  CUMULVS & PAWS...
- Sometimes Co-Exist...  Sometimes Merge Functionality...
-  Not All Software Systems are “Compatible”
-  CUMULVS & MPI...  Specific Components...
- Need Application Domain-Specific Interface “Standards”...
-  Nothing Lasts Forever ~ Platform Selection...
-  “Nexus is Dead”...  C++ vs. Java vs. Python...?
- CCA ~ On Right Track; Work Cut Out For Us!

- ORNL

## Unanswered Killer Question...

- How Can Rob Still Look So Young & Sexy When He’s Been In The Game For So Long?

- ORNL