attempt to fix #316

This only fixes the specific state race discussed in the ticket.  Seems to solve the problems I saw on archer.
This commit also fixes the MPI tests on archer