Repository: scrapy/scrapy

Issue:
IOError, 'Not a gzipped file'
while trying to access sitemap from robots.txt , Scrapy fails with **IOError, 'Not a gzipped file'** error

not sure if this issue is related to following issue(s)
https://github.com/scrapy/scrapy/issues/193 -> closed issue
https://github.com/scrapy/scrapy/pull/660 -> merged pull request to address issue 193
https://github.com/scrapy/scrapy/issues/951 -> open issue

> line where code fails in gzip.py at line # 197
> 
> ``` python
> def _read_gzip_header(self):
>         magic = self.fileobj.read(2)
>         if magic != '\037\213':
>             raise IOError, 'Not a gzipped file'
> ```
# Response Header

```
Content-Encoding: gzip
Accept-Ranges: bytes
X-Amz-Request-Id: BFFF010DDE6268DA
Vary: Accept-Encoding
Server: AmazonS3
Last-Modified: Wed, 15 Jun 2016 19:02:20 GMT
Etag: "300bb71d6897cb2a22bba0bd07978c84"
Cache-Control: no-transform
Date: Sun, 19 Jun 2016 10:54:53 GMT
Content-Type: binary/octet-stream
```

Error Log:

``` log
 Traceback (most recent call last):
  File "c:\venv\scrapy1.0\lib\site-packages\scrapy\utils\defer.py", line 102, in iter_errback
    yield next(it)
  File "c:\venv\scrapy1.0\lib\site-packages\scrapy\spidermiddlewares\offsite.py", line 29, in process_spider_output
    for x in result:
  File "c:\venv\scrapy1.0\lib\site-packages\scrapy\spidermiddlewares\referer.py", line 22, in <genexpr>
    return (_set_referer(r) for r in result or ())
  File "c:\venv\scrapy1.0\lib\site-packages\scrapy\spidermiddlewares\urllength.py", line 37, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "c:\venv\scrapy1.0\lib\site-packages\scrapy\spidermiddlewares\depth.py", line 58, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "D:\projects\sitemap_spider\sitemap_spider\spiders\mainspider.py", line 31, in _parse_sitemap
    body = self._get_sitemap_body(response)
  File "c:\venv\scrapy1.0\lib\site-packages\scrapy\spiders\sitemap.py", line 67, in _get_sitemap_body
    return gunzip(response.body)
  File "c:\venv\scrapy1.0\lib\site-packages\scrapy\utils\gz.py", line 37, in gunzip
    chunk = read1(f, 8196)
  File "c:\venv\scrapy1.0\lib\site-packages\scrapy\utils\gz.py", line 21, in read1
    return gzf.read(size)
  File "c:\python27\Lib\gzip.py", line 268, in read
    self._read(readsize)
  File "c:\python27\Lib\gzip.py", line 303, in _read
    self._read_gzip_header()
  File "c:\python27\Lib\gzip.py", line 197, in _read_gzip_header
    raise IOError, 'Not a gzipped file'
```

i did download file manually and was able to extract the content so it is not like file is corrupted 

as an example sitemap url : you can follow amazon robots.txt



Discussion:
as a work around to this problem 
if one is to set **`COMPRESSION_ENABLED = False`** in settings.py file, it will work as expected

Thanks @DharmeshPandav  for reminding us of this issue.
It's the same problem as @juraseg commented on in https://github.com/scrapy/scrapy/issues/951#issuecomment-180249747
For http://www.amazon.fr/sitemaps.f3053414d236e84.SitemapIndex_0.xml.gz for example you get

```
{b'Accept-Ranges': [b'bytes'],
 b'Content-Encoding': [b'gzip'],
 b'Content-Type': [b'binary/octet-stream'],
 b'Date': [b'Mon, 20 Jun 2016 10:50:16 GMT'],
 b'Etag': [b'"48f5a0d2cfff8d96700c49b5799c6d35"'],
 b'Last-Modified': [b'Wed, 15 Jun 2016 19:29:30 GMT'],
 b'Server': [b'AmazonS3'],
 b'Vary': [b'Accept-Encoding'],
 b'X-Amz-Request-Id': [b'674AEA9C05B9A833']}
```

from the web server, with `HttpCompressionMiddleware` decompressing the response based on Content-Encoding being "gzip".

When reading IANA's page on [`application/octet-stream`](https://www.iana.org/assignments/media-types/application/octet-stream)

> The recommended action for an implementation that receives an
> "application/octet-stream" entity is to simply offer to put the data
> in a file, with any Content-Transfer-Encoding undone, or perhaps to
> use it as input to a user-specified process.

I'd go for considering `Content-Type: binary/octet-stream` the same way, taking precedence over `Content-Encoding` (for the same reasons as in https://github.com/scrapy/scrapy/issues/193#issuecomment-12845302), i.e. not try to decompress it at `HttpCompressionMiddleware` level as a gzipped HTML/XML/JSON... response, but like a (compressed) file for another layer to interpret (here, `SitemapSpider`)


Resolving patch:
diff --git a/scrapy/utils/gz.py b/scrapy/utils/gz.py
--- a/scrapy/utils/gz.py
+++ b/scrapy/utils/gz.py
@@ -50,9 +50,12 @@ def gunzip(data):
                 raise
     return output
 
-_is_gzipped_re = re.compile(br'^application/(x-)?gzip\b', re.I)
+_is_gzipped = re.compile(br'^application/(x-)?gzip\b', re.I).search
+_is_octetstream = re.compile(br'^(application|binary)/octet-stream\b', re.I).search
 
 def is_gzipped(response):
     """Return True if the response is gzipped, or False otherwise"""
     ctype = response.headers.get('Content-Type', b'')
-    return _is_gzipped_re.search(ctype) is not None
+    cenc = response.headers.get('Content-Encoding', b'').lower()
+    return (_is_gzipped(ctype) or
+            (_is_octetstream(ctype) and cenc in (b'gzip', b'x-gzip')))


Test patch:
diff --git a/tests/test_downloadermiddleware_httpcompression.py b/tests/test_downloadermiddleware_httpcompression.py
--- a/tests/test_downloadermiddleware_httpcompression.py
+++ b/tests/test_downloadermiddleware_httpcompression.py
@@ -145,6 +145,26 @@ def test_process_response_gzipped_contenttype(self):
         self.assertEqual(response.headers['Content-Encoding'], b'gzip')
         self.assertEqual(response.headers['Content-Type'], b'application/gzip')
 
+    def test_process_response_gzip_app_octetstream_contenttype(self):
+        response = self._getresponse('gzip')
+        response.headers['Content-Type'] = 'application/octet-stream'
+        request = response.request
+
+        newresponse = self.mw.process_response(request, response, self.spider)
+        self.assertIs(newresponse, response)
+        self.assertEqual(response.headers['Content-Encoding'], b'gzip')
+        self.assertEqual(response.headers['Content-Type'], b'application/octet-stream')
+
+    def test_process_response_gzip_binary_octetstream_contenttype(self):
+        response = self._getresponse('x-gzip')
+        response.headers['Content-Type'] = 'binary/octet-stream'
+        request = response.request
+
+        newresponse = self.mw.process_response(request, response, self.spider)
+        self.assertIs(newresponse, response)
+        self.assertEqual(response.headers['Content-Encoding'], b'gzip')
+        self.assertEqual(response.headers['Content-Type'], b'binary/octet-stream')
+
     def test_process_response_head_request_no_decode_required(self):
         response = self._getresponse('gzip')
         response.headers['Content-Type'] = 'application/gzip'