# Presentation: 002447

## Process Manager Update – May 6

- The Process Manager component  (PM)
- The Process Manager implementation (MPD2)
- Issues generated for other components by process management

## The Process Manager Component

- Added limits to interface definition
- Example on next page
- Not implemented yet in terms of parsing and passing on to MPD
- Dynamic jobs (MPI_Comm_spawn)
- Current interface allows the process manager to be given a list of nodes and a number of processes to start, independently.
- Process manager implementation can then use unused nodes to start spawned processs (or not)
- MPI_UNIVERSE_SIZE allows MPI job to get hint about how many processes can usefully be spawned

## Limits Specification Example

- <create-process-group
-            totalprocs='2'>
-       <process-spec    range='0'    cwd='/home/rbutler/mpd2'  exec='infloop'>
-         <arg idx='1' value="hello"> </arg>
-         <arg idx='2' value="from 0"> </arg>
-         <limit type='cpu' value="2"/>
-       </process-spec>
-       <process-spec  range='1'   cwd='/home/rbutler/mpd2'     exec='infloop'>
-         <arg idx='1' value="hello"> </arg>
-         <arg idx='2' value="from 1"> </arg>
-         <env name='foo' value="bar"> </env>
-         <limit type='cpu' value="3"/>
-       </process-spec>
-       <host-spec>
-            magpie
-       </host-spec>
- </create-process-group>

## The Process Manager Implementation

- Improvements to MPD resulting from production use on Chiba
- Mostly in recovering from errors and crashes by applications
- Support for limits (those supported in setrlimit)
- Improvements in configuring and building along with MPICH2
- Support for MPI_Comm_spawn through PMI interface to MPICH2 application
- Interactive debugging via mpigdb

## Coercing gdb Into Functioning as a Primitive Parallel Debugger

- Key is control of stdin, stdout, stderr by MPD, through mpigdb
- Replaces mpiexec or mpirun on interactive command line
- Usable through SSS process manager component
- Stdout, stderr collected in tree, labeled by rank, and merged for scalability
-   (0-9) (gdb)  p x
-   (0-2):  $1 = 3.4
-   (3):     $1 = 3.8
-   (4-9):  $1 = 4.1
- Stdin can be broadcast to all or to a subset of processes
- z 3     (to send input to process 3 only)
- Same for interrupts
- Can run under debugger control, interrupt and query hung processes, parallel attach to running parallel job

## Coercing gdb Into a Parallel Debugger

- donner% mpigdb -n 5 cpi
- 0-4:  (gdb) b 30
- 0-4:  Breakpoint 1 at 0x804930c: file /home/lusk/mpich2/examples/cpi.c, line 30.
- 0-4:  (gdb) r
- 0-4:  Starting program: /sandbox/lusk/mpich2-mpd-mar25/examples/cpi
- 0-4:  Breakpoint 1, main (argc=1, argv=0xbffff224)
- 0-4:      at /home/lusk/mpich2/examples/cpi.c:30
- 0-4:  30            n = 10000;                  /* default # of rectangles */
- 0-4:  (gdb) n
- 0-4:  31            if (myid == 0)
- 0-4:  (gdb) p n
- 0-4:  $1 = 10000
- 0-4:  (gdb) p myid
- 0:  $2 = 0
- 2:  $2 = 2
- 1:  $2 = 1
- 3:  $2 = 3
- 4:  $2 = 4
- 0-4:  (gdb)

## mpigdb (cont)

- 0-4:  (gdb) n
- 0:  32          startwtime = MPI_Wtime();
- 1-4:  34            MPI_Bcast(&n, 1, MPI_INT, 0, MPI_COMM_WORLD);
- 0-4:  (gdb) z 0
- (gdb) n
- 0:  34      MPI_Bcast(&n, 1, MPI_INT, 0, MPI_COMM_WORLD);
- 0:  (gdb) z 0-4
- 0-4:  (gdb) n
- 0-4:  36            h   = 1.0 / (double) n;
- 0-4:  (gdb) n
- ..............
- 0-4:  42                sum += f(x);
- 0-4:  (gdb) p sum
- 0:  $4 = 23.999941940268087
- 2:  $4 = 23.999928500379845
- 1:  $4 = 23.999935460320089
- 3:  $4 = 23.999921060448177
- 4:  $4 = 23.99991314052599
- 0-4:  (gdb) q

## Issues Generated For Other Components 

- Job steps
- Option 1:  QM handles  (preferred)
- Process manager starts process groups directly
- Need public definition of user interface to QM
- Option 2:  PM implementation handles PBS-like scripts from QM
- A bit weird:  mpirun in a PBS script is trapped by extra layer (MPISH) because the “real” mpirun is a call to MPD itself
- In use on Chiba
- QM interface for requesting allocation of some number of nodes but starting up on different number of nodes, particularly for option 1.
- QM interface for requesting dynamic rebuilds
- Limits in QM interface?

## Tale of Two Queue Manager Implementations

- PM

- QM2

- QM1

- qsub1

- qsub2

- Same XML syntax;
- different content

- Different XML syntax

- totalprocs=1, exec=myscript

- totalprocs=64, exec=cpi

- mycript contains:  mpirun –np 64 cpi

- Underlying process manager
- (MPD)

- QM Interface (XML)

- QM Interface (XML)

- (mpirun intercepted)

- (mpirun is interactive interface
- to underlying process manager)