Lower the delay when no jobs are available

* This can lower latency for the low activity case

Change-Id: Ic83ac417725ca59ba1d9aca3afbf998008f833f3