scrapy/slybot — pull request #5 by kalessin (2012-04-18 00:42:46+00:00)
Title: separate code that makes the url filter, and allow to specify not to follow links

This fix is needed for scrapinghub in order to mark spiders as not follow links, a feature which has been broken for a while. Not sure if this flag should be included in slybot code, or is considered a scrapinghub specific feature, but in second alternative the separation of code that builds the url filter must be conserved in order to override.