openeventdata/scraper — pull request #10 by ahalterman (2014-02-06 22:14:33+00:00)
Title: Customized scraping for some sources

**UPI** wasn't scraping correctly, but now it is (cleaned up links so they could be followed). I also have it delete a few sentences of UPI information at the end of each story.

**BBC**- removed a few sentences from each article about not getting the best viewing experience because our browser doesn't render CSS correctly (ha).

Added **Haaretz**, but took it back out. The website requires clicking through an intermediary page between RSS and story and requires a subscription.

**All other sources** checked out fine, except...

**Two remaining issues with Google News**:
1. Google news stories have a different URL from the story reported on the original page, so using URLs to check for duplicates in MongoDB doesn't work in this case. (e.g. "nytimes.com/africa/some-story" vs. "news.google.com/aslkdjflsakdf/nytimes.com/africa/some-story")
2. In a few cases, comments on articles in Google news got scraped, too. (Specifically, stories from Time.com)