remove delayed job and thus the worker