nexcess/magento-turpentine — pull request #548 by jeroenvermeulen (2014-05-27 12:22:14+00:00)
Title: Added configuration option "Enable Crawling All URLs"

The queue of URLs to crawl is store in the Magento Cache.
Because of a cache flush or a fresh installation the queue can be empty.
In this case the crawler used to add all possible URLs to the queue to crawl.
On stores with a large amount of products this is not usable, however it would be very nice to use the Crawler to warm certain URLs after a single product is updated. That is why I added the config option "Enable Crawling All URLs". If you enable it (default) the crawler will act like it did before my change. If you disable it, the crawler will not add all URLs to the queue when it discovers an empty queue.

I also made the options for the Crawler disappear when the crawler is disabled.