ENG-4158, ENG-4164, ENG-4172: Just avoid doing any snapshot work
at all on a node which has any rejoining sites.  All snapshot fragments
get returned the rejoining status, and no site on a rejoining node ever
touches any of the snapshot machinery until all sites are fully
rejoined.
Add more machinery to dump the state of the system when the MPI
deadlocks.
Fixes for status dump based on code review.
ENG-4172: Add a comment for some confusing horrible hackery.
Merge pull request #329 from VoltDB/eng4172

Eng4172 - more work on coordinating snapshot and partially rejoined nodes.