When moving a task to the slow queue, only move it to slow queue if it existed in the fast queue.