Now am using a set to store urls to use instead of a queue.

Should give better (random) distribution over topics.