Give listeners X time to finish their job, otherise kill them

This is to prevent rare bugs causing workers to get stuck to block
the queue.