scrapy/scrapy — pull request #336 by tpeng (2013-07-08 05:00:17+00:00)
Title: allow spider to limit response size to make the broad crawl more robust

default download size limit is 32mb, but the limit can also set to per request via request.meta['download_sizelimit']