scrapy/scrapy — pull request #79 by seriyps (2012-01-15 00:05:21+00:00)
Title: ~10x speed-up for libxml2 XPathSelector

Fix of the following questions
https://groups.google.com/forum/#!topic/scrapy-users/UJLrXMhpXyo http://habrahabr.ru/blogs/python/134918/#comment_4486742 (RU)

So, following test

```python
# -*- coding: utf-8 -*-
from scrapy.selector import HtmlXPathSelector
import urllib
from lxml.html import fromstring

data = urllib.urlopen('http://tubesexclips.com/').read()

def test_hxs():
hxs = HtmlXPathSelector(text=data)
scrapy_results = set()
for elem in hxs.select('//div[@class="added-download"]/a'):
href, text = elem.select('@href').extract()[0], elem.select('text()').extract()[0]
scrapy_results.add((href, text))
return scrapy_results

def test_lxml():
tree = fromstring(data)
lxml_results = set()
for elem in tree.xpath('//div[@class="added-download"]/a'):
href, text = elem.xpath('@href')[0], elem.xpath('text()')[0]
lxml_results.add((href, text))
return lxml_results

def timetest():
import timeit
for call in "test_hxs", "test_lxml":
t = timeit.Timer(call + "()", "from __main__ import " + call)
print call, t.timeit(10)

if __name__ == "__main__":
timetest()
```
shows

test_hxs 10.2274649143
test_lxml 0.294875144958

before, and after

test_hxs 0.291007995605
test_lxml 0.304569005966

This is because `if parent:` will call `__nonzero__()` which call heavy `extract()`