scrapy/scrapely — Issue #17: Support for passing HTML, not just URLs
Author: dragondave
Created: 2012-02-06 16:54:07+00:00
State: open

http://groups.google.com/group/scraperwiki/browse_thread/thread/d750d093ca5220bf
... was posted, wanting to use Mechanize to download HTML [since the data was behind a login] and Scrapely to parse it.

As far as I can see, Scrapely doesn't support that.

I've made https://scraperwiki.com/scrapers/scrapely-hack/ to try to work around that.

The core change is in `Scraper._get_page` where:

if html:
body=html.decode(encoding)
else:

is added before

body = urllib.urlopen(url).read().decode(encoding)

, an optional 'html' parameter is added to `Scraper.scrape`, `.train` and `_get_page` [and passed to `_get_page`], and the 'url' parameter is made optional.