Ensure we have a minimal setTimeout when we have a target time