Only get extra work in fill_queue if we don't have any unqueued work in the list.
Small timeouts on select() instead of instant timeout increase reliability of socket reads and writes.