# Presentation: 260826

## Debugging Programs with (and without) TotalView

***Mark Durst***
- NERSC User Services
-  **MJDurst@nersc.gov**

## Overview

- A simple (dumb) example
- Some basics about the interface
- How to...
- Common problems/complaints

## A Simple Debug Job

## Program text

- C	PROGRAM ex1: Example to show use of Totalview Debugger
- C
-       Program ex1
-       implicit none
-       include "mpif.h"
-       integer i_status(MPI_STATUS_SIZE)
-       integer im1,me,i,nproc,ierr
-       integer count
-       real x1(100)
- C******************************************************************
- C	Initialize MPI Library Routines
- C******************************************************************
-       call mpi_init(ierr)
-       call mpi_comm_rank(mpi_comm_world,me,ierr)
-       call mpi_comm_size(mpi_comm_world,nproc,ierr)
-       im1 = 100
- C******************************************************************
- C	Processor 0 (zero) executes this code
- C******************************************************************
-       if(me.eq.0) then
-          do i = 1, im1
-             x1(i) = rand()
-          enddo
- 	 write(*,*) 'All these values should be the same:'
-          write(*,*)'Processor Number :',me,' Before send  =',x1(3)
-          do i = 1, nproc - 1

- C******* Send x1 to all other processors **************************
- C******* We made a mistake here. i in the second call position
- C******* should be im1
-          call mpi_send(x1,i,mpi_real,i,i,
-      1                 mpi_comm_world,ierr)
-          enddo
-       endif
- C******************************************************************
- C	Other processors (not zero) execute this code
- C******************************************************************
-       if(me.ne.0) then
-          do i = 1, im1
-             x1(i) = 0
-          enddo
- C****** Receive x1 from processor 0
-          call mpi_recv(x1,im1,mpi_real,0,me,
-      1                 mpi_comm_world,i_status,ierr)
-       endif
- C******************************************************************
-       write(*,*)'x1(3) on Processor No. :',me,' After  recv  =',x1(3)
-       call mpi_finalize(ierr)
-       end

## Debug compilation

- % mpxlf -o ex1 -g ex1.f
- ** ex1   === End of Compilation 1 ===
- 1501-510  Compilation successful for file ex1.f.
-  -g enables debugging
- To run normally:
- % poe ./ex1 –nodes 2 –procs 4 -stdoutmode \ ordered
- Serial jobs run without poe
- Parallel jobs will sometimes invoke poe on their own

## Output

- All these values should be the same:
-  Processor Number : 0  Before send  = 0.6456298828
-  x1(3) on Processor No. : 0  After  recv  = 0.6456298828
-  x1(3) on Processor No. : 1  After  recv  = 0.0000000000E+00
-  x1(3) on Processor No. : 2  After  recv  = 0.0000000000E+00
-  x1(3) on Processor No. : 3  After  recv  = 0.6456298828
- Typically screwy output for malfunctioning parallel program

## Running TotalView

- module load totalview
-  module initadd totalview puts it in one’s startup files (modulo some pain)
- Some conversion issues for users of older versions
- See _[TotalView](http://hpcf.nersc.gov/software/ibm/totalview/#tv1)__[ Web page](http://hpcf.nersc.gov/software/ibm/totalview/#tv1)_
-  totalview poe –a ./ex1 –nodes 2 \ –procs 4 -stdoutmode ordered
-  totalview’s sole argument is “poe”
- –a says all remaining arguments are for the program that totalview is calling
- Debugging serial programs: just “totalview pgm”

## This window comes up.  TotalView is “Attached” to poe on the interactive node...

## ...immediately followed by this window.  No source; poe not compiled with –g.  Press “Go” to start poe...

## And answer “Yes” to this (unless you want to run to a crash)...

## ...to arrive at these windows.

## Your terminal has meanwhile received a huge mass of text.

- > totalview poe -a ./ex1 nodes 2 procs 4 -stdoutmode orderedAIX TotalView 6.2.0-0Copyright 1999-2003 by Etnus, LLC. ALL RIGHTS RESERVED.Copyright 1999 by Etnus, Inc.Copyright 1996-1998 by Dolphin Interconnect Solutions, Inc.Copyright 1989-1996 by BBN Inc.Reading symbols for process 1, executing "poe"Library /usr/lpp/ppe.poe/bin/poe, with 2 asects, was linked at 0x10000000, and initially loaded at 0xff00000040000000Reading and digesting 3115 symbols...doneLibrary /usr/lib/libcpr.a(shr.o), with 2 asects, was linked at 0xfffffffffffffedc, and initially loaded at 0xff00000040319100Reading and digesting 98 symbols...doneLibrary /usr/lib/libpthreads.a(shr_xpg5.o), with 2 asects, was linked at 0xfffffffffffffe80, and initially loaded at 0xff00000040319d00Reading and digesting 3005 symbols...doneLibrary /usr/lib/libpthreads.a(shr_comm.o), with 2 asects, was linked at 0xfffffffffffffedc, and initially loaded at 0xff0000004034a400Reading and digesting 311 symbols...doneLibrary /usr/lib/libcrypt.a(shr.o), with 2 asects, was linked at 0xfffffffffffffedc, and initially loaded at 0xff000000403b0100Reading and digesting 86 symbols...doneLibrary /usr/lib/libc.a(shr.o), with 2 asects, was linked at 0xfffffffffffffea0, and initially loaded at 0xff000000403b0c00Reading and digesting 27334 symbols...doneLibrary /usr/lib/nls/loc/en_US, with 2 asects, was linked at 0xfffffffffffffed8, and initially loaded at 0xff00000040739b00WARNING: Entrypoint address 0xff000000407de86b in library '/usr/lib/nls/loc/en_US' appears invalidReading and digesting 31 symbols...doneLibrary /usr/lib/libiconv.a(shr4.o), with 2 asects, was linked at 0xfffffffffffffec0, and initially loaded at 0xff0000004073ed00Reading and digesting 589 symbols...doneLibrary /usr/lib/libi18n.a(shr.o), with 2 asects, was linked at 0xfffffffffffffec0, and initially loaded at 0xff0000004075d400Reading and digesting 884 symbols...doneLibrary /usr/lib/libvaliduser.a(shr.o), with 2 asects, was linked at 0xfffffffffffffec0, and initially loaded at 0xff00000040768900Reading and digesting 1275 symbols...doneLibrary /usr/lib/libC.a(shr3.o), with 2 asects, was linked at 0x00000000, and initially loaded at 0xff0000004078ab00Reading and digesting 67 symbols...doneLibrary /usr/lib/libC.a(ansi_32.o), with 2 asects, was linked at 0x00000000, and initially loaded at 0xff0000004078b800Reading and digesting 17955 symbols...doneLibrary /usr/lib/libc128.a(shr.o), with 2 asects, was linked at 0xfffffffffffffedc, and initially loaded at 0xff000000408b6100Reading and digesting 287 symbols...doneLibrary /usr/lib/libC.a(shr2.o), with 2 asects, was linked at 0x00000000, and initially loaded at 0xff000000408ba500Reading and digesting 91 symbols...doneLibrary /usr/lib/libC.a(shrcore.o), with 2 asects, was linked at 0x00000000, and initially loaded at 0xff000000408bb400Reading and digesting 2775 symbols...doneLibrary /usr/lib/libC.a(ansicore_32.o), with 2 asects, was linked at 0x00000000, and initially loaded at 0xff000000408e1400Reading and digesting 2447 symbols...doneLibrary /usr/lib/libC.a(shr.o), with 2 asects, was linked at 0x00000000, and initially loaded at 0xff00000040901f00Reading and digesting 2549 symbols...doneLibrary /usr/lib/libllapi.a(shr.o), with 2 asects, was linked at 0xfffffffffffffea0, and initially loaded at 0xff00000040924800Reading and digesting 34170 symbols...doneLibrary /usr/lib/llapi_shr.o, with 2 asects, was linked at 0x10000000, and initially loaded at 0xff00000040e57800WARNING: Entrypoint address 0xff00000030e57800 in library '/usr/lib/llapi_shr.o' appears invalid -------------------------------------------------------------- User: mjdurst                    Repo: mpccc Job Name: <none specified>       Group: mpccc Class Of Service: interactive    Job Class: interactive Job Accepted: Thu Dec  4 11:00:48 2003-------------------------------------------------------------- llsubmit: Processed command file through Submit Filter: "/usr/common/nsg/etc/subfilter".Launching TotalView Debugger Server with command:rsh s00909.nersc.gov -n "/usr/common/nsg/toolworks/totalview.6.2.0-0/rs6000/bin/tvdsvr -working_directory /u1/mjdurst/TotViewTut/Ex1/. -callback s00609.nersc.gov:16382 -set_pw 5a2562c0:2596e8ce -verbosity info". . .

## We place a breakpoint at line 52 by clicking on the line number, and then press Go.

## Now we’ve stopped at line 52.  Right-clicking x1 lets us “Dive” to display it (lower left).  We can also switch processes...

## ...to see all four processes’ versions of the x1 array.

## Error Found

- Critical code: call mpi_send(x1, i, mpi_real,i,i, mpi_comm_world,ierr)
- 2nd argument to mpi_send is number of elements to send
- i is the processor number; im1 is the array size

## Corrected Program Output

- > poe ./ex2 -nodes 2 -procs 4 -stdoutmode ordered
- --------------------------------------------------------------
-  User: mjdurst                    Repo: mpccc
-  Job Name: <none specified>       Group: mpccc
-  Class Of Service: interactive    Job Class: interactive
-  Job Accepted: Thu Dec  4 18:34:10 2003
- --------------------------------------------------------------
- llsubmit: Processed command file through Submit Filter: "/usr/common/nsg/etc/subfilter".
-  All these values should be the same:
-  Processor Number : 0  Before send  = 0.6456298828
-  x1(3) on Processor No. : 0  After  recv  = 0.6456298828
-  x1(3) on Processor No. : 1  After  recv  = 0.6456298828
-  x1(3) on Processor No. : 2  After  recv  = 0.6456298828
-  x1(3) on Processor No. : 3  After  recv  = 0.6456298828

## Interface Basics – Root Window

- Root window lists processes (attached or not)
- Shows status of each process/thread
- Single letter codes:
- Running
- Breakpoint
- Hold
- Idle
- Sleeping
- Thread Stopped
- etc.

## Interface Basics – Process Window

- Shows most process-specific info
- Program variables appear in Stack Frame
- Note color-coded process summary line

## Interface Basics – Variable Window

- After diving from process window
- Note matching color code
- (only tie to process)
- Colored bar includes dive/undive arrows

## Stepping through a program

- Next executes a line completely
- Step moves into complex lines (function calls, etc.)
- StepI: machine instruction level
- Out finishes a function

## Setting Breakpoints

- Click on source line
- Click again to clear
- To stop in function:
- View → Lookup Function and click somewhere
- Action Points → At Location; type function name in prompt box

## Setting Watchpoints

- Tools → Watchpoint from variable window
- Stops when value changes

## Other Action Point options

- Either breakpoints or watchpoints can be conditional ( “if (I > 30) $stop”)
- Can stop every *n*th time (“$count 10”)
- Can execute code fragments (“eval point”)
- Subsets of C, Fortran
- Special-purpose additions as above
- Can disable
- Breakpoints can be saved on exit
- File → Preferences → Action Points

## Examining variables or arrays

- Right-click → Dive on variable’s name
- Can leave up while continuing to run
- Bold values can be altered during debugging
- Can redimension window view

## Slices and Filters

## Summary Statistics

- Tools → Statistics from variable window
- Cannot slice/filter from here
- Requires manual update

## Visualize

- Creates simple line (1-d) or surface (2-d) graph
- Can be invoked as action at breakpoint ($visualize())

## TotalView Miscellany

-  **totalview *****filename corefile*** to debug post-mortem
- Can attach to running executable (“Unattached” button of root window, select, Dive)
- File → Search Path... lets you connect to source files not in same directory

## What can go wrong

- Users report:
- Totalview crashes (optimized code, version mismatches)
- Symbolic information incorrect
- Data values incorrect or nonsense
- Problems with multi-threaded/OpenMP code
- Problems with complex objects
- Large derived types
- C++ (polymorphism, STL)

## References

- _[NERSC](http://hpcf.nersc.gov/software/ibm/totalview)__[ TotalView](http://hpcf.nersc.gov/software/ibm/totalview)__[ page](http://hpcf.nersc.gov/software/ibm/totalview)_
- Easy to read and use.  I cribbed from the examples.
- _[LLNL](http://www.llnl.gov/computing/tutorials/totalview/)__[ TotalView](http://www.llnl.gov/computing/tutorials/totalview/)__[ Tutorial](http://www.llnl.gov/computing/tutorials/totalview/)_
- Two parts; second extends into very advanced material.
- Etnus _[Tutorial ](http://www.etnus.com/Products/TotalView/started/getting_started.html)_and _[User Guide](http://www.etnus.com/Support/docs/online_doc/user_guide/index.html)_
- The tutorial isn’t really, but provides nice basic how-to material
- Previous NERSC training (10/17/02, 6/4/02)
- Majdi Baddourah’s terse slides cover more advanced material.

## Debugging Without TotalView

- Other debuggers and debugging tools
- Clean up code, compilation, etc.
- Understand failures
- Reduce optimizations
- Use debugging options
- Avoid pitfalls
- Simplify problem

## Other debuggers and debugging tools

- xldb
- dbx/pdbx
- ZeroFault
- Assure
- Mainly for OpenMP
- Only if already using (frozen) KAP toolset

## Clean up: good practice

- Comments, spacing, indentation
- Use make
- Ensure common compiler flags, etc.
- Version control (RCS/CVS)
- Changes more obvious (if you comment)
- Allows easy backtracking
- RCS not complex:
- mkdir RCS
- ci source.f
- co –l source.f

## Clean up: options, etc.

-  IMPLICIT NONE
-  -u (-qundef) for large legacy codes
-  -langlvl to tease out Fortran differences
-   -langlvl=77std, 90std, 90pure, etc.
- Default is full Fortran 95 plus *all* extensions
- MPI: check subroutine argument sequences carefully (particularly C vs. Fortran)

## Understand failures

- Keep core files while debugging
- Environment variable MP_COREDIR
- Use info from batch logs
- In batch scripts:
- Labeled I/O:
- 	export MP_LABELIO=yes
- /usr/common/usg/bin/phost (prior to parallel program) maps machine names to POE tasks

## Reduce optimizations

- Must *specify* optimization (default none)
- Optimization and –g not incompatible
- If optimizing, consider –qstrict
- Examine –qfloat options
- Option groups: –O*n*, –qhot, –qipa, –qarch, –qtune, –qcache, –Q

## Use debugging options

-  –g necessary for debugging tools
- Check memory references: –qcheck (-C), -qextcheck, -lhmd
- Uninitialized variables: –qinitauto
- Floating point exceptions: –qflttrap
- Assumed persistent variables: –qsave
- Variable size: –qautodbl
- Details in IBM compiler manuals on-line

## Avoid pitfalls

- Don’t save *too* many core files
- Only need one if all processes crash similarly, or if a problem task can be identified
- MP_COREDIR=/dev/null for most processes
- Limitations of running in 32-bit mode
- Poor defaults for stack size (32 MB), data region size (128MB)
- Alter by linking with –bmaxstack and –bmaxdata, or
- Use –q64, _r compilers, and 64-bit libraries
- Beware hidden multithreading (random numbers, some libraries)

## Simplify your problem

- Debugging hundreds of processes can be cumbersome and error-prone
- To help debug MPI problems, keep communication and eliminate computation
- Reductions in scale and complexity are invaluable in eliminating possibilities
- PRINT statements are not an admission of failure

## Summary

- TotalView is a valuable tool for dynamic debugging
- Compiler options and other good practices can be useful as well
- Debugging problems that occur only at scale is still an art

*This page intentionally left blank.*

*This page intentionally left blank.*