scrapy/scrapy — pull request #946 by tpeng (2014-11-12 11:32:06+00:00)
Title: avoid download large response

introduce DOWNLOAD_MAXSIZE and DOWNLOAD_WARNSIZE in settings and
download_maxsize/download_warnsize in spider/request meta, so
downloader stop downloading as soon as the received data exceed the
limit. also check the twsisted response's length in advance to stop
downloading as early as possible.

NB, this PR redone the changes in https://github.com/scrapy/scrapy/pull/336