joshua-hull/Reddit-Image-Scraper — pull request #13 by aggrolite (2013-01-26 08:24:52+00:00)
Title: Refactor and support pages

Sorry, I changed a lot of code. Hopefully my changes clean up my previous contributions to your project. Though it is 12AM and I might look at my code tomorrow and hate it.

Please read my code, ask questions, and let me know if there is anything better I can do.

Notable changes:

-crawler has a static limit set to 1,000 AND we are now fetching the next page! yay! This means that we scrape up to 1,000 images for each subreddit. *However*, each subreddit I tested goes no farther that half of that. For some reason the 'after' param in the JSON is null after so many pages. I will have to look at this more closely, but I suspect we could just do HTML scraping in the future instead of JSON...but I am probably just overlooking something.

-I have refactored most of the code I wrote a long time ago. I tried to simplify it a bit and make it a little less confusing. Hopefully I succeeded, but there is always room for improvement