Sleep right up to the timeout instead of the first half if we find ourselves polling in _usb_read