sunlightlabs/scrapelib — pull request #25 by twneale (2014-09-08 19:53:54+00:00)
Title: Fixes bug with non-deterministic order of GET params in urls

Adds a normalize_url function to cache.CachingSession that rewrites the url with GET params ordered alphabetically, preventing cache misses due to non-deterministic server-side ordering of params in the query string.

While spidering a site that contained urls with non-deterministic order of get params, probably generated from a dictionary-like structure, I noticed the unpredictable ordering was causing cache misses because scrapelib computes cache keys based on the url, params included. Sorting alphabetically fixed it in this case.