Even under heavy load the server now properly purge dead workers.

When we started to receive and process messages in less than 1 millisecond
we would fail to purge dead workers. The reason for this is that we asked
the polling monitor to return in milliseconds the duration from the last
time we polled. And when this was sub 1 millisecond we would get 0 back.