scrapy/scrapely — pull request #48 by tpeng (2014-01-24 07:29:42+00:00)
Title: allow use html tag attributes in similar_region

before this change, similar_region only use prefix/suffix token ( the tag name, e.g ul, div, p, etc) sequence to calculate a score, the intuition is the similar region tend to have same tag sequence. but sometimes these token are not discriminative enough. on the other hand, nowadays HTML tend to use same attributes for similar elements, so this change improve the similar_region by counting these information too:

* the prefix sequence was compared to find the start point of where to start.
* only when prefix matches, the class attributes are compared.

using other information could be possible too, e.g. the html tag data fragment, this could be helpful to mach the *label* text content.