Ensure select() doesn't timeout too early.

To quote the Linux man page "select() may update the timeout argument to
indicate how much time was left."  Once the timeout argument had been
updated with a tiny value, the select() would timeout almost
immediately.