Avoid working on queue if it has been unrefed (issue #96)