Repository: scrapy/scrapy

Issue:
gzip-Content-Encoded sitemap.xml.gz is not parsed
From https://stackoverflow.com/questions/40499288/python-scrapy-sitemapspider-callbacks-not-being-called

Sitemap spider for http://www.newegg.com/Siteindex_USA.xml fails to parse sub-sitemaps files.

Excerpt from sitemap file:

```
<?xml version="1.0" encoding="utf-8"?>
<sitemapindex xmlns="http://www.google.com/schemas/sitemap/0.9">
<sitemap>
<loc>http://www.newegg.com/Sitemap/USA/newegg_sitemap_store01.xml.gz</loc>
<lastmod>2016-11-06</lastmod>
</sitemap>
<sitemap>
<loc>http://www.newegg.com/Sitemap/USA/newegg_sitemap_product01.xml.gz</loc>
<lastmod>2016-11-06</lastmod>
</sitemap>
...
```

Example spider:

```
import scrapy


class NeweggSpider(scrapy.spiders.SitemapSpider):
    name = "newegg"
    allowed_domains = ["newegg.com"]
    sitemap_urls = ['http://www.newegg.com/Siteindex_USA.xml']

    def parse(self, response):
        self.logger.info('parsing %r' % response.url)
```

With scrapy 1.2, you get the following exceptions:

```
$ scrapy runspider spider.py 
2016-11-09 15:53:10 [scrapy] INFO: Scrapy 1.2.1 started (bot: scrapybot)
(...)
2016-11-09 15:53:10 [scrapy] INFO: Spider opened
2016-11-09 15:53:10 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2016-11-09 15:53:10 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-11-09 15:53:11 [scrapy] DEBUG: Crawled (200) <GET http://www.newegg.com/Siteindex_USA.xml> (referer: None)
2016-11-09 15:53:11 [scrapy] DEBUG: Crawled (200) <GET http://www.newegg.com/Sitemap/USA/newegg_sitemap_store01.xml.gz> (referer: http://www.newegg.com/Siteindex_USA.xml)
2016-11-09 15:53:11 [scrapy] ERROR: Spider error processing <GET http://www.newegg.com/Sitemap/USA/newegg_sitemap_store01.xml.gz> (referer: http://www.newegg.com/Siteindex_USA.xml)
Traceback (most recent call last):
  File "/home/paul/.virtualenvs/scrapy12/local/lib/python2.7/site-packages/scrapy/utils/defer.py", line 102, in iter_errback
    yield next(it)
  File "/home/paul/.virtualenvs/scrapy12/local/lib/python2.7/site-packages/scrapy/spidermiddlewares/offsite.py", line 29, in process_spider_output
    for x in result:
  File "/home/paul/.virtualenvs/scrapy12/local/lib/python2.7/site-packages/scrapy/spidermiddlewares/referer.py", line 22, in <genexpr>
    return (_set_referer(r) for r in result or ())
  File "/home/paul/.virtualenvs/scrapy12/local/lib/python2.7/site-packages/scrapy/spidermiddlewares/urllength.py", line 37, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "/home/paul/.virtualenvs/scrapy12/local/lib/python2.7/site-packages/scrapy/spidermiddlewares/depth.py", line 58, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "/home/paul/.virtualenvs/scrapy12/local/lib/python2.7/site-packages/scrapy/spiders/sitemap.py", line 44, in _parse_sitemap
    s = Sitemap(body)
  File "/home/paul/.virtualenvs/scrapy12/local/lib/python2.7/site-packages/scrapy/utils/sitemap.py", line 17, in __init__
    rt = self._root.tag
AttributeError: 'NoneType' object has no attribute 'tag'

```

The root cause seem to be that the server sends `.xml.gz` files, and gzip-encodes them:

```
GET /Sitemap/USA/newegg_sitemap_product29.xml.gz HTTP/1.1
Accept-Language: en
Accept-Encoding: gzip,deflate
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
User-Agent: Scrapy/1.2.1 (+http://scrapy.org)
Host: www.newegg.com
Referer: http://www.newegg.com/Siteindex_USA.xml

HTTP/1.1 200 OK
Content-Type: application/x-gzip
Last-Modified: Tue, 08 Nov 2016 18:44:56 GMT
Accept-Ranges: bytes
ETag: "471fb033f039d21:0"
Server: NEWEGG
x-server-id: 106
X-Served-By: 40015
X-Ver: 08161601
x-newegg-flow: MISS
X-newegg-index: 0
Accept-Ranges: bytes
X-Frame-Options: SAMEORIGIN
Vary: Accept-Encoding
Content-Encoding: gzip
Expires: Wed, 09 Nov 2016 14:15:46 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Wed, 09 Nov 2016 14:15:46 GMT
Transfer-Encoding:  chunked
Connection: keep-alive
Connection: Transfer-Encoding
```

So scrapy should first gunzip the body to get that "raw" `.xml.gz` file, and then gunzip again to get a valid XML sitemap file.

But [`HttpCompressionMiddleware` does not touch the response](https://github.com/scrapy/scrapy/blob/129421c7e31b89b9b0f9c5f7d8ae59e47df36091/scrapy/downloadermiddlewares/httpcompression.py#L28) since `is_gzipped()` returns `True`, leaving it for subsequent layers to interpret.
Error load chunked xls-file
I try get xls-file from http://www.wilo.ru/no_cache/glavnaja-stranica/library/dokumentacija/prais-list/?cid=108257&did=20858&sechash=27bdca8a
But scrapy load only first part:

```
$ scrapy fetch 'http://www.wilo.ru/no_cache/glavnaja-stranica/library/dokumentacija/prais-list/?cid=108257&did=20858&sechash=27bdca8a' > ddd.xls
2016-08-02 15:04:24 [scrapy] INFO: Scrapy 1.1.1 started (bot: scrapybot)
2016-08-02 15:04:24 [scrapy] INFO: Overridden settings: {}
2016-08-02 15:04:24 [scrapy] INFO: Enabled extensions:
['scrapy.extensions.logstats.LogStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.corestats.CoreStats']
2016-08-02 15:04:24 [scrapy] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.chunked.ChunkedTransferMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2016-08-02 15:04:24 [scrapy] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2016-08-02 15:04:24 [scrapy] INFO: Enabled item pipelines:
[]
2016-08-02 15:04:24 [scrapy] INFO: Spider opened
2016-08-02 15:04:24 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2016-08-02 15:04:24 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-08-02 15:04:27 [scrapy] DEBUG: Crawled (200) <GET http://www.wilo.ru/no_cache/glavnaja-stranica/library/dokumentacija/prais-list/?cid=108257&did=20858&sechash=27bdca8a> (referer: None)
2016-08-02 15:04:27 [scrapy] INFO: Closing spider (finished)
2016-08-02 15:04:27 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 307,
 'downloader/request_count': 1,
 'downloader/request_method_count/GET': 1,
 'downloader/response_bytes': 928590,
 'downloader/response_count': 1,
 'downloader/response_status_count/200': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2016, 8, 2, 8, 4, 27, 849093),
 'log_count/DEBUG': 2,
 'log_count/INFO': 7,
 'response_received_count': 1,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2016, 8, 2, 8, 4, 24, 687076)}
2016-08-02 15:04:27 [scrapy] INFO: Spider closed (finished)
$ ll ddd.xls 
-rw-rw-r-- 1 tonal tonal 928218 2016-08-02 15:04 ddd.xls
```

If get from curl, all file load:

```
$ curl -v 'http://www.wilo.ru/no_cache/glavnaja-stranica/library/dokumentacija/prais-list/?cid=108257&did=20858&sechash=27bdca8a' > dddd.xls
* Hostname was NOT found in DNS cache
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 46.30.61.245...
* Connected to www.wilo.ru (46.30.61.245) port 80 (#0)
> GET /no_cache/glavnaja-stranica/library/dokumentacija/prais-list/?cid=108257&did=20858&sechash=27bdca8a HTTP/1.1
> User-Agent: curl/7.35.0
> Host: www.wilo.ru
> Accept: */*
> 
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0< HTTP/1.1 200 OK
< Date: Tue, 02 Aug 2016 08:08:28 GMT
* Server Apache is not blacklisted
< Server: Apache
< X-Powered-By: PHP/5.3.19
< Pragma: private
< Cache-control: private, must-revalidate
< Content-disposition: attachment; filename="Price_04_2016.xls"
< Set-Cookie: fe_typo_user=4b09a3460fc97ad39a657da3e69bd734; path=/
< Transfer-Encoding: chunked
< Content-Type: application/octet-stream
< 
{ [data not shown]
100 5825k    0 5825k    0     0   204k      0 --:--:--  0:00:28 --:--:-- 66331
* Connection #0 to host www.wilo.ru left intact
$ ll dddd.xls 
-rw-rw-r-- 1 tonal tonal 5965312 2016-08-02 15:08 dddd.xls
```



Discussion:
I'm really tempted to rethink [the assumptions around `is_gzipped`](https://github.com/scrapy/scrapy/issues/193#issuecomment-12845302) switching decoding at  `HttpCompressionMiddleware` level ON or OFF.
I believe we can:
- always gunzip at `HttpCompressionMiddleware` level whenever there's `Content-Encoding: gzip`,
- not trust file extension at SitemapSpider level, but instead check for leading `0x1f 0x8b` bytes in the response to gunzip further or not

@juraseg [said it before](https://github.com/scrapy/scrapy/issues/951#issuecomment-180249747) and I believe he's correct. I would still leave Sitemap cope with gzipped bodies (I mean potentially gunzipping the response if needed, for double-gzipped responses)

@tonal , sorry for the late feedback.
So it appears that the response if gzip encoded.
If you decode the response's body, you get the same amount of data:

```
$ scrapy shell 'http://www.wilo.ru/no_cache/glavnaja-stranica/library/dokumentacija/prais-list/?cid=108257&did=20858&sechash=27bdca8a'
2016-09-01 13:10:27 [scrapy] INFO: Scrapy 1.1.2 started (bot: scrapybot)
(...)
2016-09-01 13:10:33 [scrapy] DEBUG: Crawled (200) <GET http://www.wilo.ru/no_cache/glavnaja-stranica/library/dokumentacija/prais-list/?cid=108257&did=20858&sechash=27bdca8a> (referer: None)
(...)
In [1]: from pprint import pprint

In [2]: pprint(response.headers)
{b'Cache-Control': [b'private, must-revalidate'],
 b'Content-Disposition': [b'attachment; filename="Price_04_2016.xls"'],
 b'Content-Encoding': [b'gzip'],
 b'Content-Type': [b'application/octet-stream'],
 b'Date': [b'Thu, 01 Sep 2016 11:10:30 GMT'],
 b'Pragma': [b'private'],
 b'Server': [b'Apache'],
 b'Set-Cookie': [b'fe_typo_user=d22d059af603b025f35a2a6adc3ad67f; path=/'],
 b'Vary': [b'Accept-Encoding'],
 b'X-Powered-By': [b'PHP/5.3.19']}

In [3]: import zlib

In [4]: d = zlib.decompress(response.body, 16+zlib.MAX_WBITS)

In [5]: len(d)
Out[5]: 5965312

In [6]: len(response.body)
Out[6]: 928217
```

When I write `response.body` to a file (here "test.xls.gz") and compare with what you get with `scrapy fetch` ("ddd.xls"), you have 1 byte difference, a LF character:

```
$ hexdump -C ddd.xls | tail
000e2950  5f b5 f4 df ff ed ff da  5f 55 ef 1d fe 69 f9 4f  |_......._U...i.O|
000e2960  83 ff 19 47 cb ff fd 8e  ff d7 99 11 fe 09 f0 ff  |...G............|
000e2970  e8 ff bf 71 fc ef 36 50  34 59 8d ff 39 35 f0 3f  |...q..6P4Y..95.?|
000e2980  15 ff e9 fc 4f 6b 6b fd  ff b9 f6 e2 66 a1 c9 ff  |....Okk.....f...|
000e2990  8c e9 bf 3d 9d fe 69 ee  7b 9b ff 37 8d 26 d3 ff  |...=..i.{..7.&..|
000e29a0  bf 8e 07 e4 7f b7 ff c8  9e 73 ff d5 5e fe 3f fb  |.........s..^.?.|
000e29b0  2f a4 87 86 ff 33 86 c6  ff 73 07 fe 3f 1b 4f b6  |/....3...s..?.O.|
000e29c0  ff 2f ae 3f f6 ff 53 ff  ff 07 00 00 00 ff ff 03  |./.?..S.........|
000e29d0  00 79 b4 60 41 00 06 5b  00 0a                    |.y.`A..[..|
000e29da
$ hexdump -C test.xls.gz | tail
000e2950  5f b5 f4 df ff ed ff da  5f 55 ef 1d fe 69 f9 4f  |_......._U...i.O|
000e2960  83 ff 19 47 cb ff fd 8e  ff d7 99 11 fe 09 f0 ff  |...G............|
000e2970  e8 ff bf 71 fc ef 36 50  34 59 8d ff 39 35 f0 3f  |...q..6P4Y..95.?|
000e2980  15 ff e9 fc 4f 6b 6b fd  ff b9 f6 e2 66 a1 c9 ff  |....Okk.....f...|
000e2990  8c e9 bf 3d 9d fe 69 ee  7b 9b ff 37 8d 26 d3 ff  |...=..i.{..7.&..|
000e29a0  bf 8e 07 e4 7f b7 ff c8  9e 73 ff d5 5e fe 3f fb  |.........s..^.?.|
000e29b0  2f a4 87 86 ff 33 86 c6  ff 73 07 fe 3f 1b 4f b6  |/....3...s..?.O.|
000e29c0  ff 2f ae 3f f6 ff 53 ff  ff 07 00 00 00 ff ff 03  |./.?..S.........|
000e29d0  00 79 b4 60 41 00 06 5b  00                       |.y.`A..[.|
000e29d9
```

The file from `response.body` can be decompressed. The file from `scrapy fetch` cannot be decompressed.

Scrapy does not decompress responses with `Content-Encoding: gzip`, it is left for the consuming callback to do.
Here, for `scrapy fetch` it may be worth decrompression before writting to the output.
Adding this extra `\x0A` at the end also looks like a bug for octet stream responses.

The extra `\n` byte at the end comes from [this line](https://github.com/scrapy/scrapy/blob/ebef6d7c6dd8922210db8a4a44f48fe27ee0cd16/scrapy/commands/fetch.py#L46).

@redapple
Why Scrapy does not decompress responses with `Content-Encoding: gzip`?
I use `scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`

> This middleware allows compressed (gzip, deflate) traffic to be sent/received from web sites

@tonal , I believe the reason is because of the Content-Type value.
See https://github.com/scrapy/scrapy/issues/193 for a discussion on how browsers handle `application/gzip` responses.
Here, the content-type is `application/octet-stream`, so HttpCompressionMiddleware leaves it as it is.

@tonal , this is the latest change https://github.com/scrapy/scrapy/pull/2065 about `binary/octet-stream` and `application/octet-stream
`

In this case though, in Chrome, the file is written uncompressed to disk.

The current implementation is debatable.

If in [httpcompression.py](https://github.com/scrapy/scrapy/blob/master/scrapy/downloadermiddlewares/httpcompression.py#L28) change condition from

``` python
 if content_encoding and not is_gzipped(response):
```

to

``` python
 if content_encoding and is_gzipped(response):
```

body uncompressed ok.
Bat add `\n` to end

```
$scrapy fetch 'http://www.wilo.ru/no_cache/glavnaja-stranica/library/dokumentacija/prais-list/?cid=108257&did=20858&sechash=27bdca8a' > ddd.xls
2016-09-05 12:23:58,224 [scrapy] INFO: Scrapy 1.1.2 started (bot: amon)
2016-09-05 12:23:58,225 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'amon.spiders', 'DOWNLOAD_TIMEOUT': 60, 'LOG_FORMATTER': 'amon.logformatter.LogFormatter', 'HTTPCACHE_EXPIRATION_SECS': 86400, 'CONCURRENT_REQUESTS': 100, 'LOG_LEVEL': 'INFO', 'DOWNLOAD_WARNSIZE': 67108864, 'HTTPCACHE_IGNORE_HTTP_CODES': [400, 500, 502, 503, 504], 'SPIDER_MODULES': ['amon.spiders'], 'HTTPCACHE_ENABLED': True, 'BOT_NAME': 'amon', 'CLOSESPIDER_TIMEOUT': 86400, 'LOG_DATEFORMAT': None, 'USER_AGENT': u'Mozilla/5.0 (Windows NT 6.0; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0', 'RETRY_ENABLED': False, 'HTTPCACHE_DIR': './cache', 'FEED_FORMAT': 'null', 'HTTPCACHE_POLICY': 'scrapy.extensions.httpcache.RFC2616Policy', 'FEED_EXPORT_FIELDS': ('re_num', 'idgood', 'num', 'code', 'title', 'price', 'artikul', 'valuta', 'url_id', 'url_rsp', 'is_auto_valuta')}
2016-09-05 12:23:58,298 [scrapy] INFO: Enabled extensions:
['scrapy.extensions.closespider.CloseSpider',
 'scrapy.extensions.logstats.LogStats',
 'scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'amon.mongo_ext.MongoExtension']
2016-09-05 12:23:58,338 [scrapy] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'amon.downloadermiddleware.js_refresh.JSRefreshWithCookie',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'amon.downloadermiddleware.blocked.AntiBanMiddleware',
 'amon.downloadermiddleware.blocked.UrlBlockedInBase',
 'amon.downloadermiddleware.blocked.BrokenSiteMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.chunked.ChunkedTransferMiddleware',
 'amon.downloadermiddleware.forget_header.ForgetHeader',
 'amon.downloadermiddleware.forget_header.StripAuthorize',
 'amon.downloadermiddleware.blocked.DNSErrorMiddleware',
 'amon.downloadermiddleware.blocked.ResponseNeverReceivedErrorMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats',
 'amon.downloadermiddleware.httpcache.HttpCache']
2016-09-05 12:23:58,341 [scrapy] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2016-09-05 12:23:58,346 [scrapy] INFO: Enabled item pipelines:
['amon.pipelines.Amon4DbPipeline', 'amon.pipelines.MySQLStorePipeline']
2016-09-05 12:23:58,346 [scrapy] INFO: Spider opened
...
$ ll ddd.xls 
-rw-rw-r-- 1 tonal tonal 5965313 2016-09-05 12:24 ddd.xls
```


Resolving patch:
diff --git a/scrapy/downloadermiddlewares/httpcompression.py b/scrapy/downloadermiddlewares/httpcompression.py
--- a/scrapy/downloadermiddlewares/httpcompression.py
+++ b/scrapy/downloadermiddlewares/httpcompression.py
@@ -1,6 +1,6 @@
 import zlib
 
-from scrapy.utils.gz import gunzip, is_gzipped
+from scrapy.utils.gz import gunzip
 from scrapy.http import Response, TextResponse
 from scrapy.responsetypes import responsetypes
 from scrapy.exceptions import NotConfigured
@@ -34,7 +34,7 @@ def process_response(self, request, response, spider):
             return response
         if isinstance(response, Response):
             content_encoding = response.headers.getlist('Content-Encoding')
-            if content_encoding and not is_gzipped(response):
+            if content_encoding:
                 encoding = content_encoding.pop()
                 decoded_body = self._decode(response.body, encoding.lower())
                 respcls = responsetypes.from_args(headers=response.headers, \
diff --git a/scrapy/spiders/sitemap.py b/scrapy/spiders/sitemap.py
--- a/scrapy/spiders/sitemap.py
+++ b/scrapy/spiders/sitemap.py
@@ -5,7 +5,8 @@
 from scrapy.spiders import Spider
 from scrapy.http import Request, XmlResponse
 from scrapy.utils.sitemap import Sitemap, sitemap_urls_from_robots
-from scrapy.utils.gz import gunzip, is_gzipped
+from scrapy.utils.gz import gunzip, gzip_magic_number
+
 
 logger = logging.getLogger(__name__)
 
@@ -59,12 +60,19 @@ def _get_sitemap_body(self, response):
         """
         if isinstance(response, XmlResponse):
             return response.body
-        elif is_gzipped(response):
+        elif gzip_magic_number(response):
             return gunzip(response.body)
-        elif response.url.endswith('.xml'):
+        # actual gzipped sitemap files are decompressed above ;
+        # if we are here (response body is not gzipped)
+        # and have a response for .xml.gz,
+        # it usually means that it was already gunzipped
+        # by HttpCompression middleware,
+        # the HTTP response being sent with "Content-Encoding: gzip"
+        # without actually being a .xml.gz file in the first place,
+        # merely XML gzip-compressed on the fly,
+        # in other word, here, we have plain XML
+        elif response.url.endswith('.xml') or response.url.endswith('.xml.gz'):
             return response.body
-        elif response.url.endswith('.xml.gz'):
-            return gunzip(response.body)
 
 
 def regex(x):
diff --git a/scrapy/utils/gz.py b/scrapy/utils/gz.py
--- a/scrapy/utils/gz.py
+++ b/scrapy/utils/gz.py
@@ -59,3 +59,7 @@ def is_gzipped(response):
     cenc = response.headers.get('Content-Encoding', b'').lower()
     return (_is_gzipped(ctype) or
             (_is_octetstream(ctype) and cenc in (b'gzip', b'x-gzip')))
+
+
+def gzip_magic_number(response):
+    return response.body[:3] == b'\x1f\x8b\x08'


Test patch:
diff --git a/tests/test_downloadermiddleware_httpcompression.py b/tests/test_downloadermiddleware_httpcompression.py
--- a/tests/test_downloadermiddleware_httpcompression.py
+++ b/tests/test_downloadermiddleware_httpcompression.py
@@ -8,6 +8,7 @@
 from scrapy.downloadermiddlewares.httpcompression import HttpCompressionMiddleware, \
     ACCEPTED_ENCODINGS
 from scrapy.responsetypes import responsetypes
+from scrapy.utils.gz import gunzip
 from tests import tests_datadir
 from w3lib.encoding import resolve_encoding
 
@@ -173,9 +174,9 @@ def test_process_response_gzipped_contenttype(self):
         request = response.request
 
         newresponse = self.mw.process_response(request, response, self.spider)
-        self.assertIs(newresponse, response)
-        self.assertEqual(response.headers['Content-Encoding'], b'gzip')
-        self.assertEqual(response.headers['Content-Type'], b'application/gzip')
+        self.assertIsNot(newresponse, response)
+        self.assertTrue(newresponse.body.startswith(b'<!DOCTYPE'))
+        self.assertNotIn('Content-Encoding', newresponse.headers)
 
     def test_process_response_gzip_app_octetstream_contenttype(self):
         response = self._getresponse('gzip')
@@ -183,9 +184,9 @@ def test_process_response_gzip_app_octetstream_contenttype(self):
         request = response.request
 
         newresponse = self.mw.process_response(request, response, self.spider)
-        self.assertIs(newresponse, response)
-        self.assertEqual(response.headers['Content-Encoding'], b'gzip')
-        self.assertEqual(response.headers['Content-Type'], b'application/octet-stream')
+        self.assertIsNot(newresponse, response)
+        self.assertTrue(newresponse.body.startswith(b'<!DOCTYPE'))
+        self.assertNotIn('Content-Encoding', newresponse.headers)
 
     def test_process_response_gzip_binary_octetstream_contenttype(self):
         response = self._getresponse('x-gzip')
@@ -193,9 +194,51 @@ def test_process_response_gzip_binary_octetstream_contenttype(self):
         request = response.request
 
         newresponse = self.mw.process_response(request, response, self.spider)
-        self.assertIs(newresponse, response)
-        self.assertEqual(response.headers['Content-Encoding'], b'gzip')
-        self.assertEqual(response.headers['Content-Type'], b'binary/octet-stream')
+        self.assertIsNot(newresponse, response)
+        self.assertTrue(newresponse.body.startswith(b'<!DOCTYPE'))
+        self.assertNotIn('Content-Encoding', newresponse.headers)
+
+    def test_process_response_gzipped_gzip_file(self):
+        """Test that a gzip Content-Encoded .gz file is gunzipped
+        only once by the middleware, leaving gunzipping of the file
+        to upper layers.
+        """
+        headers = {
+            'Content-Type': 'application/gzip',
+            'Content-Encoding': 'gzip',
+        }
+        # build a gzipped file (here, a sitemap)
+        f = BytesIO()
+        plainbody = b"""<?xml version="1.0" encoding="UTF-8"?>
+<urlset xmlns="http://www.google.com/schemas/sitemap/0.84">
+  <url>
+    <loc>http://www.example.com/</loc>
+    <lastmod>2009-08-16</lastmod>
+    <changefreq>daily</changefreq>
+    <priority>1</priority>
+  </url>
+  <url>
+    <loc>http://www.example.com/Special-Offers.html</loc>
+    <lastmod>2009-08-16</lastmod>
+    <changefreq>weekly</changefreq>
+    <priority>0.8</priority>
+  </url>
+</urlset>"""
+        gz_file = GzipFile(fileobj=f, mode='wb')
+        gz_file.write(plainbody)
+        gz_file.close()
+
+        # build a gzipped response body containing this gzipped file
+        r = BytesIO()
+        gz_resp = GzipFile(fileobj=r, mode='wb')
+        gz_resp.write(f.getvalue())
+        gz_resp.close()
+
+        response = Response("http;//www.example.com/", headers=headers, body=r.getvalue())
+        request = Request("http://www.example.com/")
+
+        newresponse = self.mw.process_response(request, response, self.spider)
+        self.assertEqual(gunzip(newresponse.body), plainbody)
 
     def test_process_response_head_request_no_decode_required(self):
         response = self._getresponse('gzip')
diff --git a/tests/test_spider.py b/tests/test_spider.py
--- a/tests/test_spider.py
+++ b/tests/test_spider.py
@@ -328,6 +328,10 @@ def test_get_sitemap_body_xml_url_compressed(self):
         r = Response(url="http://www.example.com/sitemap.xml.gz", body=self.GZBODY)
         self.assertSitemapBody(r, self.BODY)
 
+        # .xml.gz but body decoded by HttpCompression middleware already
+        r = Response(url="http://www.example.com/sitemap.xml.gz", body=self.BODY)
+        self.assertSitemapBody(r, self.BODY)
+
     def test_get_sitemap_urls_from_robotstxt(self):
         robots = b"""# Sitemap files
 Sitemap: http://example.com/sitemap.xml