scrapy/scrapy — pull request #206 by dangra (2012-12-14 14:58:31+00:00)
Title: AutoThrottle and Downloader enhancements

Recently the limitation of one concurrent request for slots with download delays configured was removed. Now slots obey delays between requests but can send more than one request in case previous are still downloading. A hard concurrency limit per slot still exists, and it is 8 by default.

The changes in this pull request improves AutoThrottle extension by reusing concurrency handling from downloader instead of doing it in the extension. This extension alters download delays and increased slot concurrency in some cases, previously the poor support in downloader for delays+concurrency also caused unwanted burst of requests and much longer feedback times in loop control.

The changes also includes a new request meta keyword named `download_slot` to ease looking up for downloader slots once requests are downloaded, *but also can be used to instruct downloader what slot it must use for the request before downloading*

And as a good complement change, now downloader slot is assigned post middlewares easing slot selection directly from a downloader middleware. So, no need to override downloader anymore to select or control slots.