dbalatero/typhoeus — Issue #119: Not unzipping gzipped pages when Max Concurrency is low
Author: bkimble
Created: 2011-10-12 18:38:02+00:00
State: open

I was trying to get stats on how much bandwidth we were pulling down from an API (that returns json) per day. I began recording Content-Length as returned from the server. As I watched it run I noticed some of the pages were exceptionally large. Most of the pages uncompressed are at the most 100k so I expected at LEAST an 80% compression on the text. Upon further investigation, I noticed that for some of the requests, the server I connected to was not returning Content-Encoding: gzip so it was coming back to us uncompressed.

Typhoeus doesn't send the "Accept-Encoding: deflate, gzip" header by default so I added that to the headers so we were explicit. That's when things got weird. All of the remote requests properly had "Content-Encoding: gzip " in the headers, but Typhoeus/Curl was not decompressing the content unless max_concurrency was set close to or higher than the total number of requests about to fire.

I've created a reproduction script here: https://gist.github.com/1282088 .. if you run it you will see the 2nd request come back garbled, but by setting max_concurrency to 2 or 3 it will decompress properly.

Does Hydra lose the CURL settings when it queues up requests after the max concurrency has been hit? Thanks in advance.