Don't assert that wakeup > now, skip the sleep if we shouldn't.

Ticket: #5759