sunlightlabs/scrapelib — pull request #27 by twneale (2014-09-21 03:55:35+00:00)
Title: Fixes bug with non-deterministic order of GET params in urls

Sorry for the dupe--just moving this one into its own branch so it's not combined with the gzip PR.

"""
Adds a normalize_url function to cache.CachingSession that rewrites the url with GET params ordered alphabetically, preventing cache misses due to non-deterministic server-side ordering of params in the query string.

While spidering a site that contained urls with non-deterministic order of get params, probably generated from a dictionary-like structure, I noticed the unpredictable ordering was causing cache misses because scrapelib computes cache keys based on the url, params included. Sorting alphabetically fixed it in this case.
"""