Only wake the worker if we have not already indicated it to wake