Deal cleanly with bidirectional timeouts.

Signed-off-by: Matt Benjamin <matt@linuxbox.com>