@Hudon :  why would we ever ignore the response with poller.poll() ?  Wouldn't it be the other way around?  Meaning that if you use poller.poll(10000) you ignore the message if it comes after 10 seconds, but in the former case you wait forever.

@artemip   I don't think that automatically detecting hosts are are down is going to be a huge use case for this project since we're only working with a few computers, and the user will still need to initially set up and start the daemon on these hosts (which can be scripted, but that's up to the user.)

I do think it's a good idea to automatically detect if a host is we can do this easily.  I haven't had time to investigate why exactly the response doesn't come back in 10 seconds, but since it can take that long, I suspect that the root cause is us doing one of these calls and waiting on a ensemble to be constructed or something,  If that's the case the simulation will always fail in this way for at least some arbitrarily large ensemble/input/etc.  I just checked the ping between ctngpu machines and it is like 200 microseconds (not milliseconds) so I don't think that's the problem.

If I get some time to root cause this, then perhaps we can add an altered method of determining if a host is down.