The user added write statements so each PE would create its own formatted file for debugging output. The program still crashed and the debugging files were filled with binary gibberish. The new debug files were readable. This fixed the problems with the debug files, and let him look at debugging info right up to the crash. The debugging output pointed to a write outside array bounds.
Compilers are debugging tools, too.
MPI information sources. Snir, Otto, Huss-Lederman, Walker and Dongarra. The Complete Reference, volume 2. Gropp et al. Gropp, Lusk, Skjellum.