venturemedias/discount_mafia — pull request #1 by getnashty (2014-09-06 16:45:25+00:00)
Title: Setting up a test scraper, run with 'rake scraper:all'

#This is essentially just framing up a test.. might as well fork this branch to play around.

Right now every time you run rake scraper:all, a new text file will appear in lib/assets/test_scrapes, so you can see the results.

I figure we might have different kinds of rake tasks around this. But anyway, our goal from here, will be to make this scraper accept a "set of rules" that'll be custom to whatever site it/we select.

So lets both propose a solution - we should discuss before either of us codes that part though, as there will be some serious opinions.

The best way to think about it, would be to grab some sites we expect to scrape (like 5-10), and figure out what things we'd scrape exactly.. If we're scraping similar things from site to site its easy, if we're scraping wildly different things, it'll obviously make it more complicated. Commit any proposal as an .md document, to a docs directory - will be a better way to proceed over email, and will serve as official docs for future use.