reddit/reddit — pull request #921 by newhouseb (2013-11-26 20:25:26+00:00)
Title: Added gzip support to the scraper

Helpful because Amazon S3 doesn't support switching between gzipped and non-gzipped content so often times resources are uploaded directly gzipped with gzipped headers. Because all modern browsers support gzip, S3 always serves gzip and it's not generally a problem. Urllib, however, doesn't support gzip so gzipped resource currently fail to decode when scraped.