# Presentation: 243215

## An Early Experience on 

**An Early Experience on **
**Job Checkpoint/Restart **
**- Working with SGI Irix OS and **
**the Portable Batch System (PBS) **

**Sherry Chang**

**schang@nas.nasa.gov**
**Scientific Consultant **
**NASA Advanced Supercomputing Division**
**NASA Ames Research Center**
** ****Moffett Field, CA**

## Helping Our Users 

**Helping Our Users **

-  Scientific Consultants at NAS help users to run their jobs
-    successfully with little or no impact on other users

-  This presentation originated from helping with a user’s case

## Outline 

**Outline **

-  Why, How, and Who ?

-   Introduction to SGI’s cpr

-  Examples : a Gaussian job and an MPI job

-  Future testing and wish list

-  Success and Failures

-  Four Methods for checkpoint/restart

## Why Do Checkpointing?

**Why Do Checkpointing?**

-  Halt and restart resource-intensive codes that
-   take a long time to run

-  Replace hardware, maintenance

-  Improve a system's load balancing and scheduling

- - Prevent job loss if system fails

##  User code has its own checkpoint capability 

-  User code has its own checkpoint capability

**How to Do Checkpoint/Restart ?**

- Example:
- The Cray Unicos OS - chkpnt and restart
- The SGI Irix OS - cpr implemented in 6.n releases

- Example:
- many CFD codes
- certain gaussian jobs

-  OS has built-in checkpoint/restart utility

- Batch systems NQE, LSF, PBS support checkpoint/restart

## Who Can Checkpoint and Restart ?

- •

-  Only the checkpoint owner or superuser is permitted

-    to perform a checkpoint.

- •

-  If the processes have multiple owners, only the superuser

-   is permitted to checkpoint them.

- •

-  Only the checkpoint owner or superuser can restart

- checkpointed

-  process(

- es

- ) from a statefile.

- •

-  If the superuser performed a checkpoint, only the superuser

-    can restart it.

**Who Can Checkpoint and Restart ?**

- •

-  owner of process(

- es

- )

- •

- superuser

**Rules:**

## Sample Gaussian Job

- % **nproc=2**
- % **chk=o2.chk**
- #p **CCSD****/6-31g* ****OPT**
- O2 Geometry Optimization
- 0 1
- O
- O 1 r
- r 1.500

**Sample Gaussian Job**

**Gaussian Script : o2.com**

** **

**Sequence of calculations (links) **
**of this gaussian job **

- Link      1  initialization

- Link  101  read title and molecule specification

- Link  114  EF numerical optimization

- Link  202

- reorientate

- coordinates, etc.

- Link  301  generates basis set info

- Link  302  calculate overlap,  kinetic,  and potential integral

- Link  303  calculate

- multipole

-  integral

- Link  401  form initial MO guess

- Link  502

- iteratively

-  solve SCF

**Link  801  initialize transformation of 2-e integrals**

**Link  804  integral transformation**

**Link  913  calculate post-SCF energies and gradient terms**

- Link  601  population and related analysis

## Sample MPI Job

**Sample MPI Job**

** **

- % mpirun **-miser** -np 3 ./pi > pi.out &

-  Program pi.f : calculate the value of 

-  Use -miser or -cpr option to allow checkpoint/restart

- % mpirun -np 3 ./pi > pi.out &

## SGI's CPR Commands

- The cpr command provides a command-line interface for

**SGI's CPR Commands**

- •

**checkpoint**

- •

**find information**

-  of an existing checkpoint statefile

- •

**restart**

- •

**delete**

-  checkpoint statefile

-  % cpr -c statefile -p id[:type],[id[:type]...] [ -fgku ]

-  % cpr -i statefile

-  % cpr [ -j ] -r statefile

-  % cpr -D statefile

## Checkpoint: cpr -c statefile -p id:type

- •

**SGP**

-     for IRIX sproc shared group; see sproc(2)

**Checkpoint: cpr -c statefile -p id:type**

- -p  specifies the process or set of processes to checkpoint

- Processes may have any type in the following list:

- •

**PID**

-     for Unix process and POSIX pthread ID (

**default**

- )

-             (

- use 'ps' command to find PID

- )

- •

**GID**

-     for Unix process group ID

- •

**SID**

-      for Unix process session ID; see termio(7)

- •

**ASH**

-    for IRIX Array Session ID; see array_services(5)

-                   (

- use 'array ps' command to find ASH ID

- )

- •

**HID**

-     for process hierarchy (tree) rooted at that PID

## Using CPR Interactively

- [1] **19432**      (parent process ID)
- % ps
- PID    TTY    TIME CMD
- 19431 ttyq2   0:01    l101.exe
**19432** ttyq2   0:00    g98
- 19435 ttyq2   0:02    l101.exe
** ****Do the first checkpoint**
- % cpr -c chk1 -p **19432:HID** -k
- Checkpointing id 19432 (type HID) to **directory chk1**
**Checkpoint done**

**Using CPR Interactively**

** ****Start a Gaussian Job**

- child process

- child process

- parent process

**Caveat:** Multiple-processor Gaussian jobs do not automatically
- clear its 'shared memory segments' when the job is checkpointed.

## A Caveat with Multiple-processor 

- Multiple-processor gaussian jobs do not automatically

- clear its 'shared memory segments' when the job is aborted.

- - memory not freed, cause problems for other jobs

- -

- cpr

-  -r may fail if older version of OS is used

**A Caveat with Multiple-processor **

**Gaussian**

** ****Job**

- %

**ipcs**

** ****-a**

- IPC status from /

- dev

- /

- kmem

-  as of Tue Nov 28 13:24:36 2000

- Shared Memory:

- T         ID     KEY        MODE       OWNER    GROUP  CREATOR   CGROUP   NATTCH       SEGSZ       CPID       LPID    ATIME    DTIME    CTIME

- m          0 0x53637445 --

- rw

- -r--r--     root     root     root     root       1       48124        845        845 17:04:57 no-entry 17:04:57

- m         73 0x0000116d --

- rw

- -------   schang    g1179   schang    g1179       0    50331664      19340      19340 13:03:56 no-entry 13:03:56

- m        144 0x000072ea --

- rw

- -------   schang    g1179   schang    g1179       0    50331664      19431      19431 13:20:51 13:21:09 13:17:35

- %

** **

**ipcrm**

** ****-m 73**

- %

** **

**ipcrm**

** ****-m 144**

## Using CPR Interactively - continued

- % **cpr -r chk1** &
- [1] 19458
- % Restarting processes from directory chk1
**Process restarted successfully**.
**[1]    Done                 cpr -r chk1**
- % ps
-  PID TTY     TIME CMD
- 19431 ttyq2   0:27 l114.exe
**19432** ttyq2   0:00 g98

**Using CPR Interactively - continued**

** ****Restart **

## Failure - 1

**Failure - 1**

-  **checkpoint stalled using cpr interactively**

- % mpirun -miser -np 3 ./pi > pi.out &
- [1] 3527372
- % cpr -c chk1 -p 3527372:HID -k (&)
- [2] 3539292
- (no progress at all)

- - for mpi jobs

- Production systems : checkpoint stalled most of times
- Test-bed systems     : successful

## Using CPR within PBS

- # PBS -l ncpus=2
- # PBS -l mem=50mw
- # PBS -l walltime=1:00:00
- setenv g98root /usr/local/pkg
- setenv $g98root/g98/bsd/g98.login
- cd $PBS_O_WORKDIR
**g98 o2.com o2.out &**
**sleep 20**
**cpr -c chk1 -p `ps -u schang | grep g98 | awk ’{print $1}’`:HID -k**

** ****PBS script for first checkpoint**

** **

**Using CPR within PBS**

** ****PBS script for subsequent cpr**

- # PBS -l ncpus=2
- # PBS -l mem=50mw
- # PBS -l walltime=1:00:00
- setenv g98root /usr/local/pkg
- setenv $g98root/g98/bsd/g98.login
- cd $PBS_O_WORKDIR
**cpr -r chk1 &**
**sleep 60**
**cpr -c chk2 -p 3971074:HID -k**

- Find the PID of the parent process

** **Alternative : start job and do first
-   checkpoint interactively

** **Caveat : Restart will fail if PBS
-    stdout/stderr not present

## A Caveat with PBS

- If a job is started within PBS instead of interactively, the PBS

- standard output/error files associated with the first PBS script

- have to be present in /PBS/spool in order for subsequent

- cpr

- to succeed.

**A Caveat with PBS**

- •

-  Solutions

-   - simply use 'touch' to recreate the needed

- stdout

- and

-  stderr

-  files in /PBS/spool,

-      even though they have no content.

-      %

- cd

-  /PBS/spool

-      %

**touch 131.t3.**

**nas**

**..ER**

-   - An alternative to avoid this trouble is to run your job and do the first

-     checkpoint through an interactive session

- •

-  The PBS standard output and error files are copied over to $PBS_O_WORKDIR

-   and removed from /PBS/spool when PBS job is completed.

- •

-  Subsequent

- cpr

-  will fail if such files are not present in /PBS/spool

-   CPR Error:

-  ckpt

- _

- fstat

- : open /PBS/spool/131.t3.

- nas

- ..ER (No such file or directory)

## }

**Failure - 2**

-  **restart failed using cpr in PBS script**

- - both for mpi and gaussian jobs

- - checkpoint/restart successful for a few cycles,
-   restart failed in a later cycle

- cpr -c chk1 -p:xxxx:HID -k &
- cpr -r chk1
- cpr -c chk2 -p:xxxx:HID -k &
- cpr -r chk2
- ....
- cpr -c chkn -p:xxxx:HID -k &
- cpr -r chkn                                              **}**

- CPR Error: **Failed to place mld 0** (Invalid argument)
- CPR Error: Unexpected status EOF
- CPR Error: Cleaning up the failed restart

**Error Messages:**

- successful

- failed

## }

**Failure - 3**

-  **restart failed from a checkpoint state which was once **
**  ****successfully restarted **

- - both for mpi and gaussian jobs

- cpr -c chk1 -p:xxxx:HID -k &
- cpr -r chk1
- cpr -c **chk2** -p:xxxx:HID -k &
- cpr -r chk2
- ....
- cpr -c chkn -p:xxxx:HID -k &
- cpr -r chkn                                               **}**

- successful

- failed

- cpr -r **chk2**

**failed**

**}**

**Error Message :**
- same as previous page

## Using qhold and qrls of PBS

- 1121.evelyn.nas.nasa.gov
- % **qhold** **1121**
- % **ls** **-l** **/PBS/checkpoint**
- drwxr-xr-x    3 root     root          28 May  1 12:20   **1121.evelyn.CK**
- % **qstat -a**
- Job ID                   S
- 1121.evelyn          **H**
- % **qrls** **1121**
- Job ID                  S
- 1121.evelyn         **R**

** **

**Using qhold and qrls of PBS**

**PBS script: o2.script**

- % **qhold**
- % **qrls**

## Failure - 4 (hopper)

**Failure - 4 (hopper)**

- Turing %** qsub mpi.pbs**
**8128.fermi.nas.nasa.gov**

## Failure on hopper :

**Failure on hopper :**

**qhold**

**/**

**qrls **

**- continued**

- pbs

- _mom;

**File exists (17) in **

**create_**

**cpuset**

**,**

**failed to create**

** ****cpuset**

** ****8128.**

**fer**

- pbs

- _mom;

- Svr

- ;

- pbs

- _mom;File exists (17) in

**mach_restart**

- ,

-                                         Cannot assign

-  cpuset

-  to 8128.

- fermi

- .

- nas

- .

- nasa

- .

- gov

- pbs

- _mom;

- Svr

- ;

- pbs

- _mom;Error 0 (0) in

**mom_restart_job**

- , 8128.

- fermi

- .

- nas

- .

- nasa

- .

- gov

- :

-                                         task 1 failed from file /PBS/checkpoint/8128.

- fermi

- ..CK/0000000001

- pbs

- _mom;Job;8128.

- fermi

- .

- nas

- .

- nasa

- .

- gov

- ;Restart failed, error 0

- pbs

- _mom;Job;8128.

- fermi

- .

- nas

- .

- nasa

- .

- gov

- ;kill_job

**Error Message from /PBS/mom_logs**

**Ed Hook:**

-  What might cause the function

-  'createCpuset

- ()'

- (from

-  libcpuset

- .so) to error out, with

-  'errno'

- set to 'EEXIST' (=17) ??

**Bron**

** ****Nelson:**

-  ... looking at the kernel sources, this error is only

- returned when the kernel believes that there is already an existing

- cpuset

-  with the requested name.

## Failure -5 (t3)

**Failure -5 (t3)**

- t3%** qsub mpi.pbs**
**148.t3.nas.nasa.gov**

**33 sec**

**72 sec**

**Job ran for ~40 seconds and then got killed**

## Failure on t3 :

**Failure on t3 :**

**qhold**

**/**

**qrls **

**- continued**

**Job failed due to PBS time-tracking error**

**  **

** ****Walltime**

** ****used ( **

**t1 **

**+ **

**t2 **

**)**

**= time job completed -**

** **

**   **

**adjusted wall start time **

**< allowed **

**walltime **

**?**

## Failure on t3 :

**Failure on t3 :**

**qhold**

**/**

**qrls **

**- continued**

**Message from /PBS/bin/**

**tracejob**

**qsub**

**qhold**

**qrls**

**PBS Mom should have adjusted wall start ~34 sec earlier than 11:19:01. **

**It mistakenly adjusted wall start 150 sec later than 11:19:01**

**Bug in **

**pbs**

**_mom :**

## Using qsub -c of PBS for 

**Using qsub -c of PBS for **
**Automatic Checkpointing Periodically**

- # PBS -l ncpus=2
- # PBS -l mem=50mw
- # PBS -l walltime=1:00:00
**# PBS -c c=3**
- setenv g98root /usr/local/pkg
- setenv $g98root/g98/bsd/g98.login
- cd $PBS_O_WORKDIR
**g98 o2.com o2.out**

- % **qsub** **o2.script**
- or
- % **qsub -c c=3 o2.script**
- 1120.evelyn.nas.nasa.gov

- PBS script : o2.script

**If PBS mom or system crashes :**

**PBS should automatically restart a job that has a checkpoint **
**directory associated with it after the system is back**

## Summary of cpr, qhold/qrls and qsub-c

- •

**qhold**

**/**

**qrls**

- •

**qsub**

** ****-c**

** **

- system crash : prevent loss of valuable results

- - by job owner

- PBS should automatically restart the job that has a checkpoint directory

- •

**cpr**

** ****within PBS**

- Job needs very long

- walltime

- which exceeds the limit of any queue

- prevent loss of valuable results if system crashes

- - by job owner

- •

** ****cpr**

** ****interactively**

- For interactive jobs, prevent loss of valuable results if system crashes

- - typically by job owner

**Summary of cpr, qhold/qrls and qsub-c**

## Future Testing and Wish List

**Future Testing and Wish List**

** ****System-wide checkpoint/restart**

**Make sure checkpoint/restart is **_**reliable**_** **
**in a real production environment**

** ****A wide variety of user applications - OpenMP, pvm, mpi**

** ****Large parallel jobs**

**Ultimate Goal :**

**Future Testing :**

** ****Efficiency**

** ****System crash simulation**

## IRIX/CPR - A Popular Topic 

**IRIX/CPR - A Popular Topic **

**Recent email-exchanges on this topic, sgi-tech@cug.org**

- Barry Sharp - Boeing

- Paul White - CSC

- Miroslaw Kupczyk - Poznan Supercomputing and Network Center

- Torgny Faxen** **- National Supercomputing Center, Sweden** **

-  NQE, LSF

-  Irix OS

-  MPI, OpenMP, Gaussian - no pvm yet

-  Irix vs Unicos

_**would like to see more exchange on the details of the success and failure cases**_

**SGI- supportfolio bug report provides limited information**

## Acknowledgement 

**Acknowledgement **

** **CUG

-  Lorraine Freeman - sysadmin

-  Ed Hook - SciCon, PBS expert

-  Bron Nelson - SGI on-site analyst

-  Chuck Niggley - SciCon Group Lead

-  NASA Advanced Supercomputing Division