hpc/libcircle — pull request #10 by adammoody (2013-03-07 03:12:37+00:00)
Title: rank=-1 bug on rank 0, return terminate/msg flags in wait_on_probe

I'm also debugging a hang on our system. I have a deadlock case where a process gets stuck in the Issend while sending work to a rank that is not responding, because it has already received the termination message. Of course it shouldn't have gotten the termination message if there is still work to do. I think this was happening because the token_partner for rank 0 is computed as (rank - 1) % size --> -1, which happens to be MPI_PROC_NULL. Thus, it immediately can receive the token (a receive from PROC_NULL always succeeds) and so rank 0 thinks that signal travelled around the whole ring. A simple change to (rank - 1 + size) % size seemed to do the trick, but I'm still testing to be sure.

While debugging this, I came across another possible bug in that size returns either TERMINATION, the output from MPI_Get_count, or 0. The problem is that MPI_Get_count can sometimes return MPI_UNDEFINED, which can be any arbitrary negative value (which might happen to equal the value used for TERMINATION). Thus, I've changed the prototype on this function to set a flag for termination and another flag to indicate a message is ready for receiving. In the case that a message is ready, it also returns the status from the Iprobe. Then the size can be computed external to the wait_for_probe() function.

Also, in my tracing, I could see cases where processes sometimes transfer all of their work to the requestor process. This is somewhat inefficient since after communication, the net effect is just to swap the role of the two processes. I'll modify this, so that the process with work always keeps at least one element.