Repository: scrapy/scrapy

Issue:
Multiple-compressed responses support
HTTP response body can be compressed multiple times. In such cases `Content-Encoding` header contains a list of comma-separated encodings. Some servers instead can send multiple `Content-Encoding` headers.
This fix allows HttpCompressionMiddleware handle that.
See also: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Encoding
Fix #5143
Fixes #5143 


Discussion:
Nice, I did not know that.

Could you please include a test?
Please note that the existing tests failed too.
## [Codecov](https://app.codecov.io/gh/scrapy/scrapy/pull/5143?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) Report
> Merging [#5143](https://app.codecov.io/gh/scrapy/scrapy/pull/5143?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) (12e1bb8) into [master](https://app.codecov.io/gh/scrapy/scrapy/commit/c5b1ee810167266fcd259f263dbfc0fe0204761a?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) (c5b1ee8) will **decrease** coverage by `45.64%`.
> The diff coverage is `41.18%`.

> :exclamation: Current head 12e1bb8 differs from pull request most recent head d950ff0. Consider uploading reports for the commit d950ff0 to get more accurate results

```diff
@@             Coverage Diff             @@
##           master    #5143       +/-   ##
===========================================
- Coverage   88.18%   42.54%   -45.64%     
===========================================
  Files         162      162               
  Lines       10485    11289      +804     
  Branches     1517     1835      +318     
===========================================
- Hits         9246     4803     -4443     
- Misses        965     6125     +5160     
- Partials      274      361       +87     
```

| [Impacted Files](https://app.codecov.io/gh/scrapy/scrapy/pull/5143?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) | Coverage Δ | |
|---|---|---|
| [scrapy/\_\_main\_\_.py](https://app.codecov.io/gh/scrapy/scrapy/pull/5143?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L19fbWFpbl9fLnB5) | `0.00% <0.00%> (-33.34%)` | :arrow_down: |
| [scrapy/cmdline.py](https://app.codecov.io/gh/scrapy/scrapy/pull/5143?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2NtZGxpbmUucHk=) | `0.00% <0.00%> (-67.21%)` | :arrow_down: |
| [scrapy/commands/bench.py](https://app.codecov.io/gh/scrapy/scrapy/pull/5143?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2NvbW1hbmRzL2JlbmNoLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [scrapy/commands/check.py](https://app.codecov.io/gh/scrapy/scrapy/pull/5143?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2NvbW1hbmRzL2NoZWNrLnB5) | `0.00% <0.00%> (-71.02%)` | :arrow_down: |
| [scrapy/commands/crawl.py](https://app.codecov.io/gh/scrapy/scrapy/pull/5143?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2NvbW1hbmRzL2NyYXdsLnB5) | `0.00% <0.00%> (-60.00%)` | :arrow_down: |
| [scrapy/commands/edit.py](https://app.codecov.io/gh/scrapy/scrapy/pull/5143?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2NvbW1hbmRzL2VkaXQucHk=) | `0.00% <0.00%> (-44.45%)` | :arrow_down: |
| [scrapy/commands/genspider.py](https://app.codecov.io/gh/scrapy/scrapy/pull/5143?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2NvbW1hbmRzL2dlbnNwaWRlci5weQ==) | `0.00% <0.00%> (-87.00%)` | :arrow_down: |
| [scrapy/commands/list.py](https://app.codecov.io/gh/scrapy/scrapy/pull/5143?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2NvbW1hbmRzL2xpc3QucHk=) | `0.00% <0.00%> (-77.78%)` | :arrow_down: |
| [scrapy/commands/runspider.py](https://app.codecov.io/gh/scrapy/scrapy/pull/5143?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2NvbW1hbmRzL3J1bnNwaWRlci5weQ==) | `0.00% <0.00%> (-89.14%)` | :arrow_down: |
| [scrapy/commands/settings.py](https://app.codecov.io/gh/scrapy/scrapy/pull/5143?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2NvbW1hbmRzL3NldHRpbmdzLnB5) | `0.00% <0.00%> (-71.88%)` | :arrow_down: |
| ... and [148 more](https://app.codecov.io/gh/scrapy/scrapy/pull/5143?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) | |

## [Codecov](https://app.codecov.io/gh/scrapy/scrapy/pull/5964?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) Report
> Merging [#5964](https://app.codecov.io/gh/scrapy/scrapy/pull/5964?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) (e42556a) into [master](https://app.codecov.io/gh/scrapy/scrapy/commit/c5b1ee810167266fcd259f263dbfc0fe0204761a?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) (c5b1ee8) will **increase** coverage by `1.01%`.
> The diff coverage is `87.67%`.

> :exclamation: Current head e42556a differs from pull request most recent head 3a865d2. Consider uploading reports for the commit 3a865d2 to get more accurate results

```diff
@@            Coverage Diff             @@
##           master    #5964      +/-   ##
==========================================
+ Coverage   88.18%   89.19%   +1.01%     
==========================================
  Files         162      162              
  Lines       10485    11290     +805     
  Branches     1517     1835     +318     
==========================================
+ Hits         9246    10070     +824     
+ Misses        965      928      -37     
- Partials      274      292      +18     
```

| [Impacted Files](https://app.codecov.io/gh/scrapy/scrapy/pull/5964?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) | Coverage Δ | |
|---|---|---|
| [scrapy/\_\_main\_\_.py](https://app.codecov.io/gh/scrapy/scrapy/pull/5964?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L19fbWFpbl9fLnB5) | `0.00% <0.00%> (-33.34%)` | :arrow_down: |
| [scrapy/commands/crawl.py](https://app.codecov.io/gh/scrapy/scrapy/pull/5964?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2NvbW1hbmRzL2NyYXdsLnB5) | `60.00% <0.00%> (ø)` | |
| [scrapy/downloadermiddlewares/decompression.py](https://app.codecov.io/gh/scrapy/scrapy/pull/5964?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2Rvd25sb2FkZXJtaWRkbGV3YXJlcy9kZWNvbXByZXNzaW9uLnB5) | `100.00% <ø> (ø)` | |
| [scrapy/downloadermiddlewares/defaultheaders.py](https://app.codecov.io/gh/scrapy/scrapy/pull/5964?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2Rvd25sb2FkZXJtaWRkbGV3YXJlcy9kZWZhdWx0aGVhZGVycy5weQ==) | `100.00% <ø> (ø)` | |
| [scrapy/downloadermiddlewares/downloadtimeout.py](https://app.codecov.io/gh/scrapy/scrapy/pull/5964?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2Rvd25sb2FkZXJtaWRkbGV3YXJlcy9kb3dubG9hZHRpbWVvdXQucHk=) | `92.85% <ø> (ø)` | |
| [scrapy/downloadermiddlewares/httpauth.py](https://app.codecov.io/gh/scrapy/scrapy/pull/5964?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2Rvd25sb2FkZXJtaWRkbGV3YXJlcy9odHRwYXV0aC5weQ==) | `100.00% <ø> (ø)` | |
| [scrapy/downloadermiddlewares/httpcache.py](https://app.codecov.io/gh/scrapy/scrapy/pull/5964?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2Rvd25sb2FkZXJtaWRkbGV3YXJlcy9odHRwY2FjaGUucHk=) | `93.97% <ø> (ø)` | |
| [scrapy/downloadermiddlewares/httpcompression.py](https://app.codecov.io/gh/scrapy/scrapy/pull/5964?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2Rvd25sb2FkZXJtaWRkbGV3YXJlcy9odHRwY29tcHJlc3Npb24ucHk=) | `92.95% <ø> (+0.31%)` | :arrow_up: |
| [scrapy/downloadermiddlewares/httpproxy.py](https://app.codecov.io/gh/scrapy/scrapy/pull/5964?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2Rvd25sb2FkZXJtaWRkbGV3YXJlcy9odHRwcHJveHkucHk=) | `96.61% <ø> (-3.39%)` | :arrow_down: |
| [scrapy/downloadermiddlewares/redirect.py](https://app.codecov.io/gh/scrapy/scrapy/pull/5964?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2Rvd25sb2FkZXJtaWRkbGV3YXJlcy9yZWRpcmVjdC5weQ==) | `97.36% <ø> (+0.35%)` | :arrow_up: |
| ... and [148 more](https://app.codecov.io/gh/scrapy/scrapy/pull/5964?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) | |


Resolving patch:
diff --git a/scrapy/downloadermiddlewares/httpcompression.py b/scrapy/downloadermiddlewares/httpcompression.py
--- a/scrapy/downloadermiddlewares/httpcompression.py
+++ b/scrapy/downloadermiddlewares/httpcompression.py
@@ -1,6 +1,7 @@
 from __future__ import annotations
 
 import warnings
+from itertools import chain
 from logging import getLogger
 from typing import TYPE_CHECKING, List, Optional, Union
 
@@ -102,18 +103,18 @@ def process_response(
         if isinstance(response, Response):
             content_encoding = response.headers.getlist("Content-Encoding")
             if content_encoding:
-                encoding = content_encoding.pop()
                 max_size = request.meta.get("download_maxsize", self._max_size)
                 warn_size = request.meta.get("download_warnsize", self._warn_size)
                 try:
-                    decoded_body = self._decode(
-                        response.body, encoding.lower(), max_size
+                    decoded_body, content_encoding = self._handle_encoding(
+                        response.body, content_encoding, max_size
                     )
                 except _DecompressionMaxSizeExceeded:
                     raise IgnoreRequest(
                         f"Ignored response {response} because its body "
-                        f"({len(response.body)} B) exceeded DOWNLOAD_MAXSIZE "
-                        f"({max_size} B) during decompression."
+                        f"({len(response.body)} B compressed) exceeded "
+                        f"DOWNLOAD_MAXSIZE ({max_size} B) during "
+                        f"decompression."
                     )
                 if len(response.body) < warn_size <= len(decoded_body):
                     logger.warning(
@@ -121,6 +122,7 @@ def process_response(
                         f"({len(decoded_body)} B) is larger than the "
                         f"download warning size ({warn_size} B)."
                     )
+                response.headers["Content-Encoding"] = content_encoding
                 if self.stats:
                     self.stats.inc_value(
                         "httpcompression/response_bytes",
@@ -144,6 +146,28 @@ def process_response(
 
         return response
 
+    def _handle_encoding(self, body, content_encoding, max_size):
+        to_decode, to_keep = self._split_encodings(content_encoding)
+        for encoding in to_decode:
+            body = self._decode(body, encoding, max_size)
+        return body, to_keep
+
+    def _split_encodings(self, content_encoding):
+        to_keep = [
+            encoding.strip().lower()
+            for encoding in chain.from_iterable(
+                encodings.split(b",") for encodings in content_encoding
+            )
+        ]
+        to_decode = []
+        while to_keep:
+            encoding = to_keep.pop()
+            if encoding not in ACCEPTED_ENCODINGS:
+                to_keep.append(encoding)
+                return to_decode, to_keep
+            to_decode.append(encoding)
+        return to_decode, to_keep
+
     def _decode(self, body: bytes, encoding: bytes, max_size: int) -> bytes:
         if encoding == b"gzip" or encoding == b"x-gzip":
             return gunzip(body, max_size=max_size)


Test patch:
diff --git a/tests/sample_data/compressed/html-gzip-deflate-gzip.bin b/tests/sample_data/compressed/html-gzip-deflate-gzip.bin
new file mode 100644
Binary files /dev/null and b/tests/sample_data/compressed/html-gzip-deflate-gzip.bin differ
diff --git a/tests/sample_data/compressed/html-gzip-deflate.bin b/tests/sample_data/compressed/html-gzip-deflate.bin
new file mode 100644
Binary files /dev/null and b/tests/sample_data/compressed/html-gzip-deflate.bin differ
diff --git a/tests/test_downloadermiddleware_httpcompression.py b/tests/test_downloadermiddleware_httpcompression.py
--- a/tests/test_downloadermiddleware_httpcompression.py
+++ b/tests/test_downloadermiddleware_httpcompression.py
@@ -27,6 +27,8 @@
     "x-gzip": ("html-gzip.bin", "gzip"),
     "rawdeflate": ("html-rawdeflate.bin", "deflate"),
     "zlibdeflate": ("html-zlibdeflate.bin", "deflate"),
+    "gzip-deflate": ("html-gzip-deflate.bin", "gzip, deflate"),
+    "gzip-deflate-gzip": ("html-gzip-deflate-gzip.bin", "gzip, deflate, gzip"),
     "br": ("html-br.bin", "br"),
     # $ zstd raw.html --content-size -o html-zstd-static-content-size.bin
     "zstd-static-content-size": ("html-zstd-static-content-size.bin", "zstd"),
@@ -205,6 +207,62 @@ def test_multipleencodings(self):
         assert newresponse is not response
         self.assertEqual(newresponse.headers.getlist("Content-Encoding"), [b"uuencode"])
 
+    def test_multi_compression_single_header(self):
+        response = self._getresponse("gzip-deflate")
+        request = response.request
+        newresponse = self.mw.process_response(request, response, self.spider)
+        assert newresponse is not response
+        assert "Content-Encoding" not in newresponse.headers
+        assert newresponse.body.startswith(b"<!DOCTYPE")
+
+    def test_multi_compression_single_header_invalid_compression(self):
+        response = self._getresponse("gzip-deflate")
+        response.headers["Content-Encoding"] = [b"gzip, foo, deflate"]
+        request = response.request
+        newresponse = self.mw.process_response(request, response, self.spider)
+        assert newresponse is not response
+        self.assertEqual(
+            newresponse.headers.getlist("Content-Encoding"), [b"gzip", b"foo"]
+        )
+
+    def test_multi_compression_multiple_header(self):
+        response = self._getresponse("gzip-deflate")
+        response.headers["Content-Encoding"] = ["gzip", "deflate"]
+        request = response.request
+        newresponse = self.mw.process_response(request, response, self.spider)
+        assert newresponse is not response
+        assert "Content-Encoding" not in newresponse.headers
+        assert newresponse.body.startswith(b"<!DOCTYPE")
+
+    def test_multi_compression_multiple_header_invalid_compression(self):
+        response = self._getresponse("gzip-deflate")
+        response.headers["Content-Encoding"] = ["gzip", "foo", "deflate"]
+        request = response.request
+        newresponse = self.mw.process_response(request, response, self.spider)
+        assert newresponse is not response
+        self.assertEqual(
+            newresponse.headers.getlist("Content-Encoding"), [b"gzip", b"foo"]
+        )
+
+    def test_multi_compression_single_and_multiple_header(self):
+        response = self._getresponse("gzip-deflate-gzip")
+        response.headers["Content-Encoding"] = ["gzip", "deflate, gzip"]
+        request = response.request
+        newresponse = self.mw.process_response(request, response, self.spider)
+        assert newresponse is not response
+        assert "Content-Encoding" not in newresponse.headers
+        assert newresponse.body.startswith(b"<!DOCTYPE")
+
+    def test_multi_compression_single_and_multiple_header_invalid_compression(self):
+        response = self._getresponse("gzip-deflate")
+        response.headers["Content-Encoding"] = ["gzip", "foo,deflate"]
+        request = response.request
+        newresponse = self.mw.process_response(request, response, self.spider)
+        assert newresponse is not response
+        self.assertEqual(
+            newresponse.headers.getlist("Content-Encoding"), [b"gzip", b"foo"]
+        )
+
     def test_process_response_encoding_inside_body(self):
         headers = {
             "Content-Type": "text/html",