Only cleanup when queue should be empty