gwu-libraries/social-feed-manager — pull request #268 by cazzerson (2014-10-03 19:07:24+00:00)
Title: Improve performance of fetch_urls

This PR includes:

* Restricting the version of requests to avoid an infinite redirect loop bug.
* Catching more specific exceptions and printing the URLs and error messages.
* Optimizing refetching.
* Using `django-queryset-iterator` to process large TwitterUserItem QuerySets in batches.
* Using streaming HTTP requests to avoid pulling down the response body.
* Starting with the Twitter API-provided `expanded_url`, where possible.

For me, on a big `fetch_urls` job, these changes reduce resident memory usage from 1.5-2GB to 40-100MB. This will hopefully remove the need to routinely filter by dates.