Fix error in logic.

Was repeatedly getting the first items in the set/queue, since we
were never dropping those we've already consumed.