Keep track of the first thread per queue.

Makes it faster, but still worse than before.  Arg.
Use a set to keep track of active priorities.