niryariv/opentaba-server — pull request #63 by florpor (2014-01-23 14:26:00+00:00)
Title: New scraper

Scraper revised for the new site, please review the code.
It is already working on https://dashboard.heroku.com/apps/opentaba-dev-server/resources
you can also check out the mongo contents from there.

Also - I think if this is published it is time to drop the current data. Running this revised scraper got me just below 15000 plans and only one is blacklisted, while the current mongo has over 150000 plans and a ton are blacklisted. I assume a lot of the extra plans are ones that appeared on many gushim and are duplicated a lot throughout our data...
If anyone just wants to test the new data, just point your local opentaba-client to https://opentaba-dev-server.herokuapp.com/ as the API url.

And there's a small change to opentaba-client that accompanies this one (the plan details url). Will post a link in the first comment.