scrapy/scrapy — pull request #469 by redapple (2013-11-21 18:10:42+00:00)
Title: Remove "x-gzip" from Requests' "Accept-Encoding" header

Having "x-gzip" as first encoding in "Accept-Encoding" header seems to cause encoding issues with some (IIS?) web servers (response not compressed, valid but not compressed)

"x-gzip" is not included in "Accept-Encoding" headers sent by latest Chrome or Firefox releases

After discussing with @dangra and @pablohoffman (https://github.com/scrapy/scrapy/commit/81fbe8c9a4b3940cd1cdf27ff0848a7ec5360512#commitcomment-4617865) the consensus is to remove it in requests still accepting it in responses

Moving "x-gzip" to the end of the accepted encoding list also works (with the server exhibiting the issue)