scrapy/scrapy — Issue #76: Plugable Link Extractor backends
Author: pablohoffman
Created: 2012-01-05 16:41:32+00:00
State: open

We want to had pluggable link extractor backends, maybe having a `LINKEXTRACTOR_CLASS` setting.

Some backends that come to mind: pure-regex, scrapely, libxml2, lxml, sgml

The sgml backend is not working very well, as there have been some issues reported about it:

* http://stackoverflow.com/questions/6443485/scrapy-parsing-issue-with-malformed-br-tags
* https://groups.google.com/d/topic/scrapy-users/E_-cbj7-Wyk/discussion

The scrapely backend workrks quite well and it's pure-python, so it's a good choice. But we'd have to add the dependency to scrapely.