scrapy/scrapy — pull request #323 by tpeng (2013-06-19 16:21:01+00:00)
Title: allow spider to limit response size to make the broad crawl more robust

implement `get_response_size_limit` in spider to limit the size per request. e.g.

def get_response_size_limit(self, request):
if self.is_safe_url(request.url):
return 16 * 1024 * 1024
return 1 * 1024 * 1024