scrapy/scrapy — Issue #52: Real support for returning iterators on parse() method
Author: warvariuc
Created: 2011-11-18 12:56:45+00:00
State: open

New features and settings:

> ...
> Real support for returning iterators on start_requests() method. The iterator is now consumed during the crawl when the spider is getting idle (r2704)
> ...

This works! Thanks!

But the issue with iterators on parse() methods still exists:

class AmazonSpider(BaseSpider):
name = 'amazon'
allowed_domains = ['amazon.com']
start_urls = ['http://www.amazon.com/dp/B005890G8Y/']

def parse(self, response):
print 'parse'
for i in xrange(100000000):
url = 'http://www.amazon.com/dp/%i/' % i
print i,
yield self.make_requests_from_url(url)

This causes memory consumption to grow very fast.

Also (i guess), because of this when scrapyd in 'scrapy server' is interrupted one of my 'scrapy crawl' subprocesses cannot stop for a long time:

2011-11-18 14:14:32+0200 [carbusiness_it] DEBUG: Requesting search: brand 102, model 978, years 2007-2007
2011-11-18 14:14:32+0200 [scrapy] INFO: Received SIGINT, shutting down gracefully. Send again to force unclean shutdown
2011-11-18 14:14:32+0200 [carbusiness_it] DEBUG: Requesting search: brand 622, model 3404, years 2002-2002
2011-11-18 14:14:32+0200 [carbusiness_it] DEBUG: Requesting search: brand 108, model 1451, years 2003-2003
2011-11-18 14:14:32+0200 [carbusiness_it] DEBUG: Requesting search: brand 82, model 2293, years 2007-2007
2011-11-18 14:14:32+0200 [carbusiness_it] DEBUG: Requesting search: brand 42, model 805, years 2011-2011
2011-11-18 14:14:32+0200 [carbusiness_it] DEBUG: Requesting search: brand 25, model 3345, years 2011-2011
2011-11-18 14:14:32+0200 [carbusiness_it] DEBUG: Requesting search: brand 33, model 665, years 2011-2011
2011-11-18 14:14:32+0200 [carbusiness_it] DEBUG: Requesting search: brand 130, model 2139, years 2009-2009
2011-11-18 14:14:32+0200 [carbusiness_it] DEBUG: Requesting search: brand 92, model 1975, years 2009-2009
2011-11-18 14:14:32+0200 [carbusiness_it] DEBUG: Requesting search: brand 134, model 3453, years 2010-2010
2011-11-18 14:14:32+0200 [carbusiness_it] DEBUG: Requesting search: brand 21, model 375, years 2011-2011
2011-11-18 14:14:32+0200 [carbusiness_it] DEBUG: Requesting search: brand 8, model 720, years 2011-2011
2011-11-18 14:14:32+0200 [carbusiness_it] DEBUG: Requesting search: brand 95, model 644, years 2009-2009
2011-11-18 14:14:32+0200 [carbusiness_it] DEBUG: Requesting search: brand 72, model 2117, years 2007-2007
2011-11-18 14:14:32+0200 [carbusiness_it] DEBUG: Requesting search: brand 32, model 636, years 2011-2011
2011-11-18 14:14:32+0200 [carbusiness_it] DEBUG: Requesting search: brand 70, model 654, years 2011-2011
2011-11-18 14:14:32+0200 [carbusiness_it] DEBUG: Requesting search: brand 106, model 2843, years 2011-2011
2011-11-18 14:14:32+0200 [carbusiness_it] DEBUG: Found item link: http://www.carbusiness.it/527781/auto-nuova/MICROCAR_MC2.ashx
2011-11-18 14:14:32+0200 [carbusiness_it] DEBUG: Redirecting (302) to from
2011-11-18 14:14:32+0200 [carbusiness_it] DEBUG: Crawled (200) (referer: http://www.carbusiness.it/jp/jp.aspx?action=load_modelli&id_marca=82)
2011-11-18 14:14:32+0200 [carbusiness_it] DEBUG: Crawled (200) (referer: http://www.carbusiness.it/jp/jp.aspx?action=load_modelli&id_marca=25)
2011-11-18 14:14:32+0200 [carbusiness_it] INFO: Closing spider (shutdown)
2011-11-18 14:14:32+0200 [carbusiness_it] DEBUG: Requesting search: brand 131, model 2152, years 2010-2010
2011-11-18 14:14:33+0200 [carbusiness_it] DEBUG: Requesting search: brand 79, model 2164, years 2007-2007
2011-11-18 14:14:33+0200 [carbusiness_it] DEBUG: Requesting search: brand 102, model 2464, years 2007-2007
2011-11-18 14:14:33+0200 [carbusiness_it] DEBUG: Requesting search: brand 622, model 3403, years 2002-2002
2011-11-18 14:14:33+0200 [carbusiness_it] DEBUG: Requesting search: brand 108, model 2853, years 2002-2002
2011-11-18 14:14:33+0200 [carbusiness_it] DEBUG: Requesting search: brand 82, model 266, years 2007-2007
2011-11-18 14:14:33+0200 [carbusiness_it] DEBUG: Requesting search: brand 42, model 2432, years 2011-2011
2011-11-18 14:14:33+0200 [carbusiness_it] DEBUG: Requesting search: brand 25, model 413, years 2011-2011
2011-11-18 14:14:33+0200 [carbusiness_it] DEBUG: Requesting search: brand 33, model 658, years 2011-2011
2011-11-18 14:14:33+0200 [carbusiness_it] DEBUG: Requesting search: brand 130, model 2137, years 2009-2009
2011-11-18 14:14:33+0200 [carbusiness_it] DEBUG: Requesting search: brand 92, model 1977, years 2008-2008
2011-11-18 14:14:33+0200 [carbusiness_it] DEBUG: Requesting search: brand 134, model 3452, years 2010-2010
2011-11-18 14:14:33+0200 [carbusiness_it] DEBUG: Requesting search: brand 21, model 383, years 2011-2011
2011-11-18 14:14:33+0200 [carbusiness_it] DEBUG: Requesting search: brand 8, model 2404, years 2011-2011
2011-11-18 14:14:33+0200 [carbusiness_it] DEBUG: Requesting search: brand 95, model 3214, years 2009-2009
2011-11-18 14:14:33+0200 [carbusiness_it] DEBUG: Requesting search: brand 72, model 3477, years 2007-2007
2011-11-18 14:14:33+0200 [carbusiness_it] DEBUG: Requesting search: brand 32, model 2388, years 2011-2011

Because in a parse method there is really big loop:

def parse2(self, response):
'''Request search my brands, models and year.'''

hxs = HtmlXPathSelector(response)
models = hxs.select("//option/@value").extract()[1:]
random.shuffle(models)

brandId = response.meta['brand_id']
self.log('parse2, brandId=%s, models=%s' % (brandId, models))

searchPageResponse = response.meta['prev_response']
hxs = HtmlXPathSelector(searchPageResponse)
years = hxs.select("//*[@id='AnnoDa']/option/@value").extract()[1:]
years = list(map(int, years))
years.sort(reverse= True) # be sure its sorted desc

for i in range(len(years) - 1): #
yearTo = years[i]
yearFrom = years[i + 1]
if yearFrom == yearTo - 1:
yearFrom = yearTo # search is inclusive. so do not search two consecutive years

for modelId in models:
formdata = {'ddlMarca': brandId, 'ddlModello': modelId,
'AnnoDa': yearFrom, 'Annoa': yearTo, 'ddlRisultatiPerPagina': 50}
formRequest = FormRequest.from_response(searchPageResponse, 'ctl00', formdata= formdata, # we specify which submit button to click
callback= self.parseBrand, clickdata= {'name': 'btnRicerca'}, priority= -i)
self.log('Requesting search: brand %s, model %s, years %d-%d' %
(brandId, modelId, yearFrom, yearTo), log.DEBUG)
yield formRequest