scrapy/scrapely — Issue #11: Slow Extraction Times
Author: javadi82
Created: 2011-12-09 16:44:53+00:00
State: open

It's currently taking me around 2s to run the extraction on a single page.

'''
def extract(url, page, scraper):
page = unicode(page, errors = 'ignore')
html_page = HtmlPage(url, body=page, encoding = 'utf-8')

ex = InstanceBasedLearningExtractor(scraper.templates)
records = ex.extract(html_page)[0]
return records[0]
'''

Following is the output of the lineprofiler:
'''
Line # Hits Time Per Hit % Time Line Contents
==============================================================

53 def extract(url, page, scraper):
54 """Returns a dictionary containing the extraction output
55 """
56 10 2923 292.3 0.1 page = unicode(page, errors = 'ignore')
57 10 704147 70414.7 17.8 html_page = HtmlPage(url, body=page, encoding = 'utf-8')
58
59 10 2604545 260454.5 65.9 ex = InstanceBasedLearningExtractor(scraper.templates)
60 10 640413 64041.3 16.2 records = ex.extract(html_page)[0]
61 10 141 14.1 0.0 return records[0]
'''

Am I doing something wrong ? The extraction code is similar to that found in tool.py and __init__.py
But, I get faster extraction times when I run scrapely from the command line than using the code above.

Please advice.