scrapinghub/webpager — pull request #1 by tpeng (2014-01-17 06:20:30+00:00)
Title: use edit distance feature

webpager sometimes give false positive for some links. e.g. on http://www.zoeken.nu/115755/bedrijf/rabobank-drechtsteden.html. it labeled following links as pagination:
`http://www.zoeken.nu/zoek/j.html` , `http://www.zoeken.nu/zoek/t.html`, `http://www.zoeken.nu/zoek/z.html`
it might because these link has single text which has bigger weight in the logistic classifier...
but apparently they're not. this change add a edit distance between the base url (the url of the page to classify) and link's our url as one feature, this should be able to catch the idea those urls tend to be similar.

also some minor improvements.