Repository: mitmproxy/mitmproxy

Issue:
GZIP request body encoding falling back to raw
#### Problem Description
GZIP requests are not decoded correctly.
This only occurs in the request and not in the response. In the response I see the body as uncompromised gzip but not in the request. 

I created a own content view to test the gzip decompression:

```python
from typing import Optional
from mitmproxy import contentviews, flow
from mitmproxy import http
import gzip


class ViewGZIPCase(contentviews.View):
    name = "GZIP"

    def __call__(self, data, **metadata) -> contentviews.TViewResult:
        return "gzip", contentviews.format_text(gzip.decompress(data))

    def render_priority(
            self,
            data: bytes,
            *,
            content_type: Optional[str] = None,
            flow: Optional[flow.Flow] = None,
            http_message: Optional[http.Message] = None,
            **unknown_metadata,
    ) -> float:

        if http_message.headers["content-encoding"] == "GZIP":
            return 1

        else:
            return 0


view = ViewGZIPCase()


def load(l):
    contentviews.add(view)


def done():
    contentviews.remove(view)
```

##### request

with selected self-made view
<img width="1637" alt="Screenshot 2021-08-05 at 16 42 57" src="https://user-images.githubusercontent.com/1338331/128370647-68c5af49-dfca-4862-a3ff-b3413c4226fd.png">

without any special view
<img width="1637" alt="Screenshot 2021-08-05 at 16 42 42" src="https://user-images.githubusercontent.com/1338331/128370652-e2001fd6-0909-491a-8c77-19cfa226b7d6.png">


##### response

Mitmproxy standalone decompression and json parsing in the response but why not in the request?
<img width="1637" alt="Screenshot 2021-08-05 at 16 47 55" src="https://user-images.githubusercontent.com/1338331/128370639-75287a4f-5c54-47e9-a126-3ff95a1dc90f.png">


#### Steps to reproduce the behavior:
1. start Mitmproxy 
2. wait for requests ( iPhone 12 as client is connected)
3. open the request

#### System Information
Mitmproxy: 7.0.2
Python:    3.9.6
OpenSSL:   OpenSSL 1.1.1k  25 Mar 2021
Platform:  macOS-11.5.1-x86_64-i386-64bit



GZIP request body encoding falling back to raw
#### Problem Description
GZIP requests are not decoded correctly.
This only occurs in the request and not in the response. In the response I see the body as uncompromised gzip but not in the request. 

I created a own content view to test the gzip decompression:

```python
from typing import Optional
from mitmproxy import contentviews, flow
from mitmproxy import http
import gzip


class ViewGZIPCase(contentviews.View):
    name = "GZIP"

    def __call__(self, data, **metadata) -> contentviews.TViewResult:
        return "gzip", contentviews.format_text(gzip.decompress(data))

    def render_priority(
            self,
            data: bytes,
            *,
            content_type: Optional[str] = None,
            flow: Optional[flow.Flow] = None,
            http_message: Optional[http.Message] = None,
            **unknown_metadata,
    ) -> float:

        if http_message.headers["content-encoding"] == "GZIP":
            return 1

        else:
            return 0


view = ViewGZIPCase()


def load(l):
    contentviews.add(view)


def done():
    contentviews.remove(view)
```

##### request

with selected self-made view
<img width="1637" alt="Screenshot 2021-08-05 at 16 42 57" src="https://user-images.githubusercontent.com/1338331/128370647-68c5af49-dfca-4862-a3ff-b3413c4226fd.png">

without any special view
<img width="1637" alt="Screenshot 2021-08-05 at 16 42 42" src="https://user-images.githubusercontent.com/1338331/128370652-e2001fd6-0909-491a-8c77-19cfa226b7d6.png">


##### response

Mitmproxy standalone decompression and json parsing in the response but why not in the request?
<img width="1637" alt="Screenshot 2021-08-05 at 16 47 55" src="https://user-images.githubusercontent.com/1338331/128370639-75287a4f-5c54-47e9-a126-3ff95a1dc90f.png">


#### Steps to reproduce the behavior:
1. start Mitmproxy 
2. wait for requests ( iPhone 12 as client is connected)
3. open the request

#### System Information
Mitmproxy: 7.0.2
Python:    3.9.6
OpenSSL:   OpenSSL 1.1.1k  25 Mar 2021
Platform:  macOS-11.5.1-x86_64-i386-64bit





Discussion:
Thanks for the super clear report! Here's what we do for gzip:
https://github.com/mitmproxy/mitmproxy/blob/7efefb716eb004369a264491f0ca558e32337c8a/mitmproxy/net/encoding.py#L148

Would you mind plugging your own gzip decompression in there to see if that helps? We're really just using Python's standard library here. Alternatively, a saved flow would be really helpful.
I'm just throwing out there that the `GZIP` is upper case in the request. Not sure if we handle that? Didn't look into it.
Good catch, @Prinzhorn! Content-Encoding is case-insensitive (https://datatracker.ietf.org/doc/html/rfc7231#section-3.1.2.1), so this is a mitmproxy bug. :)
Maybe [this line](https://github.com/mitmproxy/mitmproxy/blob/0ca458fd6475ee48728147f3b529467a75e912a4/mitmproxy/net/encoding.py#L70) needs to read

```py
if isinstance(encoding, str) and encoding.lower() in ("gzip", "deflate", "br", "zstd"):
```
instead of 
```py
if encoding in ("gzip", "deflate", "br", "zstd"):
```
Thank you all. I'll try the whole thing and test it. 
Many thanks to @Mattwmaster58  for the pull request. 
Thanks for the super clear report! Here's what we do for gzip:
https://github.com/mitmproxy/mitmproxy/blob/7efefb716eb004369a264491f0ca558e32337c8a/mitmproxy/net/encoding.py#L148

Would you mind plugging your own gzip decompression in there to see if that helps? We're really just using Python's standard library here. Alternatively, a saved flow would be really helpful.
I'm just throwing out there that the `GZIP` is upper case in the request. Not sure if we handle that? Didn't look into it.
Good catch, @Prinzhorn! Content-Encoding is case-insensitive (https://datatracker.ietf.org/doc/html/rfc7231#section-3.1.2.1), so this is a mitmproxy bug. :)
Maybe [this line](https://github.com/mitmproxy/mitmproxy/blob/0ca458fd6475ee48728147f3b529467a75e912a4/mitmproxy/net/encoding.py#L70) needs to read

```py
if isinstance(encoding, str) and encoding.lower() in ("gzip", "deflate", "br", "zstd"):
```
instead of 
```py
if encoding in ("gzip", "deflate", "br", "zstd"):
```
Thank you all. I'll try the whole thing and test it. 
Many thanks to @Mattwmaster58  for the pull request. 

Resolving patch:
diff --git a/mitmproxy/net/encoding.py b/mitmproxy/net/encoding.py
--- a/mitmproxy/net/encoding.py
+++ b/mitmproxy/net/encoding.py
@@ -52,6 +52,7 @@ def decode(
     """
     if encoded is None:
         return None
+    encoding = encoding.lower()
 
     global _cache
     cached = (
@@ -67,7 +68,7 @@ def decode(
             decoded = custom_decode[encoding](encoded)
         except KeyError:
             decoded = codecs.decode(encoded, encoding, errors)  # type: ignore
-        if encoding in ("gzip", "deflate", "br", "zstd"):
+        if encoding in ("gzip", "deflate", "deflateraw", "br", "zstd"):
             _cache = CachedDecode(encoded, encoding, errors, decoded)
         return decoded
     except TypeError:
@@ -108,6 +109,7 @@ def encode(decoded: Union[None, str, bytes], encoding, errors='strict') -> Union
     """
     if decoded is None:
         return None
+    encoding = encoding.lower()
 
     global _cache
     cached = (
@@ -123,7 +125,7 @@ def encode(decoded: Union[None, str, bytes], encoding, errors='strict') -> Union
             encoded = custom_encode[encoding](decoded)
         except KeyError:
             encoded = codecs.encode(decoded, encoding, errors)  # type: ignore
-        if encoding in ("gzip", "deflate", "br", "zstd"):
+        if encoding in ("gzip", "deflate", "deflateraw", "br", "zstd"):
             _cache = CachedDecode(encoded, encoding, errors, decoded)
         return encoded
     except TypeError:
@@ -216,7 +218,7 @@ def encode_deflate(content: bytes) -> bytes:
     "identity": identity,
     "gzip": decode_gzip,
     "deflate": decode_deflate,
-    "deflateRaw": decode_deflate,
+    "deflateraw": decode_deflate,
     "br": decode_brotli,
     "zstd": decode_zstd,
 }
@@ -225,7 +227,7 @@ def encode_deflate(content: bytes) -> bytes:
     "identity": identity,
     "gzip": encode_gzip,
     "deflate": encode_deflate,
-    "deflateRaw": encode_deflate,
+    "deflateraw": encode_deflate,
     "br": encode_brotli,
     "zstd": encode_zstd,
 }


Test patch:
diff --git a/test/mitmproxy/net/test_encoding.py b/test/mitmproxy/net/test_encoding.py
--- a/test/mitmproxy/net/test_encoding.py
+++ b/test/mitmproxy/net/test_encoding.py
@@ -17,6 +17,7 @@ def test_identity(encoder):
 
 @pytest.mark.parametrize("encoder", [
     'gzip',
+    'GZIP',
     'br',
     'deflate',
     'zstd',