So I'm scraping Google Finance's company news page for historical company news articles.

It's been loads of fun (partial sarcasm), the process hasn't been as bad as I thought. Using readability's parsing API to extract data about the news article that google links to. Right now I'm using grequests to make a bunch of requests to readability.com asynchronously* but that's giving me a lot of connection errors unless I throttle it back pretty hard. I suspect this is because Readability doesn't like us DOSing them.

I plan to stick all of the article data in an sqlite3 database with some simple ORM.

The google news scraper only scrapes the first page of results, so we need to work on calling the whole proceedure multiple times with different GET parameters to specify which news articles to get. The whole process is abstracted from the actual stock symbol, so it should work on different companies.