cgiffard/node-simplecrawler — pull request #20 by Cordazar (2013-02-06 17:06:41+00:00)
Title: Modified if statement in queue.exists to prevent duplicates falling through.

While using this node.js module in our project we noticed that the crawling never ended by itself. Instead it continued crawling the same web site until the node process got stuck and couldn't continue.

After some investigation we discovered that the queue.exists method did not work as expected as it always added an url regardless of whether it had already crawled it before which meant a never ending crawl.
We changed how the if statement checked if there was an earlier key with the same url. We haven't tested how this effects the speed of the lookup.