Try to effectivelly capture the errors raised while processing response in downloader middlewares. Whenever the error occurs, start processing the exceptions.
Added strip_utm_tags as a submethod of url canonicalization.
Output dropped items in WARNING mode. It makes more sense. Simply don't drop good items.
Output more details when error occurs.
Replaced yields in CrawlSpider with array return.
yields in spiders doesn't work well with exceptions.
Added "modelargs" property to DjangoItem.
More explicit error message in HttpErrorMiddleware.
Added "redirects" property to request.
Added possibility to choose errback method in crawl spider's rules.
Created get_mw_setting() to specify settings per mw/spider/request.
Applied get_mw_setting() to HttpCacheMiddleware.
Use SCRAPY_CFG environment variable for a location of .cfg file.