# Presentation: 680719

## An Insiders Guide: Janus Optimization and Troubleshooting

- Ben Cole, Intel Computational Scientist at SNL
- Pat Fay, Intel Computational Scientist at LANL
- Greg Henry, Intel Computational Scientist in OR
- http://www.sandia.gov/ASCI/Red/usage/optimize.html
- http://www.sandia.gov/ASCI/Red/usage/optimize.ps

## Attaching to a running job with debug -a

- What to do when large production jobs stop making progress and you’d like to figure out why (or gather enough information to beat on your code team, at least.)
- Attach the debugger to the ‘running’ job.
- debug -a <PID of the yod command for this run>
- “spacebar” through the initial messages (once they come up).
- Issue these commands:
- more off
- log <filename>
- All output now logged to file

## Suggested debug commands to run:
process
recvqueue
someday: sendqueue
where 
Real Sandia Apps where this has been done on over 1024 nodes:
CTH
Pronto
Zapotec
Alegra

- process
- recvqueue
- someday: sendqueue
- where
- Real Sandia Apps where this has been done on over 1024 nodes:
- CTH
- Pronto
- Zapotec
- Alegra

**Other Useful Debug Commands After Attaching**

## (When debugging your application)
on a large number of nodes, “where” can cause the debugger itself to memory fault.

App. Itself is not impacted.

Log file is preserved

workaround: use the context command to get this information in chunks (256, 512) of nodes.

- on a large number of nodes, “where” can cause the debugger itself to memory fault.
- App. Itself is not impacted.
- Log file is preserved
- workaround: use the context command to get this information in chunks (256, 512) of nodes.

**Debug’s ‘where’ Command Bug**

## Debug: Using Watchpoints

- Set a watchpoint on a variable. Debug will interrupt your program when variable is read/written (with an optional conditional statement).
- Useful for memory problems (say you know x(100) is getting trashed but you don’t know how).
- Example: checking errno. I want to find where in my program free() is failing. Free() sets errno = EINVAL when it finds a problem. Free() can fail due to memory overruns/underruns, invalid pointers.
- Other useful variables for watchpoints:
- profile_total_bytes_used: set by malloc/free for current total heap usage. Use profiling to find out max heap used and then use ‘stop -w’ to break when profile_total_bytes_used is near to the max heap used
- libdbmalloc.a sets malloc_errno when it finds an error.

## Debug : Using Watchpoints Example

- Example: Find free() failures with watchpoints
- janus ~/tst 148 > cat ck_errno.c
- #include <stdio.h>
- #include <errno.h>
- #include <nx.h>
- #include <malloc.h>
- #include <stdlib.h>
- #include <sys/types.h>
- int *my_errno = &errno;
- void ck_errno(void) { /*now my_errno is in scope*/
-    if(mynode()== -1 && *my_errno == 0)
-    { printf("This will never happen\n");}
- }
- int main(int argc, char **argv) {
-      int *int_ptr;
-      ck_errno();
-      int_ptr = (int *)malloc(100*sizeof(int));
-      int_ptr = int_ptr + 1;
-      /* int_ptr != base of the malloc'd area so free fails*/
-      free(int_ptr);
-      printf("above free() should fail.\n");
-      printf("errno= %d\n",errno);
-      return 0;
- }

## Debug: Compiling and Running the Watchpoint Example

- Compiling
- janus ~/tst 148 > pgcc -cougar -g -O0 ck_errno.c -o tstc
- Debugging
- janus ~/tst 149 > debug -sz 1 tstc
- *** Debug (Parallel Debugger), Release 2.4
- *** reading symbol table for /Net/usr/home/pfay/tst/tstc...
- *** initializing Debug for parallel application...
- *** load complete
- (0) > stop ck_errno
- (0) > run
- (0) > where   #**now errno and my_errno are in scope**
- ***** (0) *****
- ck_errno(void) [ck_errno.c #10]
- main(int, char**) [ck_errno.c #16]
- cstart() [unknown 0x00026243]
- __start() [unknown 0x00020120]
- (0) > stop -w errno if *my_errno > 2 #**only stop if errno > 2**
- (0) > cont    #**now we get an interrupt when the free() fails**
- (0) > where
- ***** (0) *****
- __free() [unknown 0x000464bc]
- free() [unknown 0x000460c6]
- main(int, char**) [ck_errno.c #20]
- cstart() [unknown 0x00026243]
- __start() [unknown 0x00020120]
- (0) > print errno
- ***** (0) *****
-  ** `unknown`__free`errno **
- errno = 0x00000016