Repository: mitmproxy/mitmproxy

Issue:
Crash when opening specific flows
##### Steps to reproduce the problem:

1. Record a flow with `mitmproxy -T --host` from a specific application
2. Open one of the flows (I can mail you an example)
3. Whole program closes with the following error message

##### Any other comments? What have you tried so far?

```
Traceback (most recent call last):                                                                   
  File "/usr/lib/python3.5/site-packages/mitmproxy/contentviews.py", line 657, in get_content_view
    ret = viewmode(data, **metadata)
  File "/usr/lib/python3.5/site-packages/mitmproxy/contentviews.py", line 138, in __call__
    return content_types_map[ct][0](data, **metadata)
  File "/usr/lib/python3.5/site-packages/mitmproxy/contentviews.py", line 228, in __call__
    pj = pretty_json(data)
  File "/usr/lib/python3.5/site-packages/mitmproxy/contentviews.py", line 55, in pretty_json
    p = json.loads(s.decode('utf-8'))
AttributeError: 'str' object has no attribute 'decode'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/mitmproxy/console/master.py", line 482, in run
    self.loop.run()
  File "/usr/lib/python3.5/site-packages/urwid/main_loop.py", line 278, in run
    self._run()
  File "/usr/lib/python3.5/site-packages/urwid/main_loop.py", line 376, in _run
    self.event_loop.run()
  File "/usr/lib/python3.5/site-packages/urwid/main_loop.py", line 682, in run
    self._loop()
  File "/usr/lib/python3.5/site-packages/urwid/main_loop.py", line 719, in _loop
    self._watch_files[fd]()
  File "/usr/lib/python3.5/site-packages/urwid/raw_display.py", line 393, in <lambda>
    event_loop, callback, self.get_available_raw_input())
  File "/usr/lib/python3.5/site-packages/urwid/raw_display.py", line 493, in parse_input
    callback(processed, processed_codes)
  File "/usr/lib/python3.5/site-packages/urwid/main_loop.py", line 403, in _update
    self.process_input(keys)
  File "/usr/lib/python3.5/site-packages/urwid/main_loop.py", line 503, in process_input
    k = self._topmost_widget.keypress(self.screen_size, k)
  File "/usr/lib/python3.5/site-packages/mitmproxy/console/window.py", line 86, in keypress
    k = super(self.__class__, self).keypress(size, k)
  File "/usr/lib/python3.5/site-packages/urwid/container.py", line 1128, in keypress
    return self.body.keypress( (maxcol, remaining), key )
  File "/usr/lib/python3.5/site-packages/mitmproxy/console/flowlist.py", line 385, in keypress
    return urwid.ListBox.keypress(self, size, key)
  File "/usr/lib/python3.5/site-packages/urwid/listbox.py", line 985, in keypress
    key = focus_widget.keypress((maxcol,),key)
  File "/usr/lib/python3.5/site-packages/mitmproxy/console/flowlist.py", line 238, in keypress
    self.master.view_flow(self.flow)
  File "/usr/lib/python3.5/site-packages/mitmproxy/console/master.py", line 576, in view_flow
    flowview.FlowView(self, self.state, flow, tab_offset),
  File "/usr/lib/python3.5/site-packages/mitmproxy/console/flowview.py", line 143, in __init__
    tab_offset
  File "/usr/lib/python3.5/site-packages/mitmproxy/console/tabs.py", line 31, in __init__
    self.show()
  File "/usr/lib/python3.5/site-packages/mitmproxy/console/tabs.py", line 69, in show
    body = self.tabs[self.tab_offset][1](),
  File "/usr/lib/python3.5/site-packages/mitmproxy/console/flowview.py", line 166, in view_request
    return self.conn_text(self.flow.request)
  File "/usr/lib/python3.5/site-packages/mitmproxy/console/flowview.py", line 260, in conn_text
    msg, body = self.content_view(viewmode, conn)
  File "/usr/lib/python3.5/site-packages/mitmproxy/console/flowview.py", line 203, in content_view
    flow_modify_cache_invalidation
  File "/usr/lib/python3.5/site-packages/mitmproxy/utils.py", line 32, in get
    ret = gen(*args)
  File "/usr/lib/python3.5/site-packages/mitmproxy/console/flowview.py", line 200, in <lambda>
    lambda *args: self._get_content_view(message, *args),
  File "/usr/lib/python3.5/site-packages/mitmproxy/console/flowview.py", line 208, in _get_content_view
    viewmode, message
  File "/usr/lib/python3.5/site-packages/mitmproxy/contentviews.py", line 635, in get_message_content_view
    viewmode, content, headers=headers, query=query
  File "/usr/lib/python3.5/site-packages/mitmproxy/contentviews.py", line 665, in get_content_view
    _, content = get("Raw")(data, **metadata)
  File "/usr/lib/python3.5/site-packages/mitmproxy/contentviews.py", line 156, in __call__
    return "Raw", format_text(strutils.bytes_to_escaped_str(data, True))
  File "/usr/lib/python3.5/site-packages/netlib/strutils.py", line 88, in bytes_to_escaped_str
    raise ValueError("data must be bytes, but is {}".format(data.__class__.__name__))
ValueError: data must be bytes, but is str

mitmproxy has crashed!
Please lodge a bug report at:
        https://github.com/mitmproxy/mitmproxy
Shutting down...
```

Only some of the Request/Response-pairs causes the crash. It also crashes when loading a saved version of the flow.

##### System information

Mitmproxy version: 0.18.2
Python version: 3.5.2
Platform: Linux-4.7.2-1-ARCH-x86_64-with-arch
SSL version: OpenSSL 1.0.2h  3 May 2016
Linux distro: arch

<!--
    Cut and paste the output of "mitmdump --sysinfo".

    If you're using an older version if mitmproxy, please specify the version
    and OS.
-->

Crash when opening specific flows
##### Steps to reproduce the problem:

1. Record a flow with `mitmproxy -T --host` from a specific application
2. Open one of the flows (I can mail you an example)
3. Whole program closes with the following error message

##### Any other comments? What have you tried so far?

```
Traceback (most recent call last):                                                                   
  File "/usr/lib/python3.5/site-packages/mitmproxy/contentviews.py", line 657, in get_content_view
    ret = viewmode(data, **metadata)
  File "/usr/lib/python3.5/site-packages/mitmproxy/contentviews.py", line 138, in __call__
    return content_types_map[ct][0](data, **metadata)
  File "/usr/lib/python3.5/site-packages/mitmproxy/contentviews.py", line 228, in __call__
    pj = pretty_json(data)
  File "/usr/lib/python3.5/site-packages/mitmproxy/contentviews.py", line 55, in pretty_json
    p = json.loads(s.decode('utf-8'))
AttributeError: 'str' object has no attribute 'decode'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/mitmproxy/console/master.py", line 482, in run
    self.loop.run()
  File "/usr/lib/python3.5/site-packages/urwid/main_loop.py", line 278, in run
    self._run()
  File "/usr/lib/python3.5/site-packages/urwid/main_loop.py", line 376, in _run
    self.event_loop.run()
  File "/usr/lib/python3.5/site-packages/urwid/main_loop.py", line 682, in run
    self._loop()
  File "/usr/lib/python3.5/site-packages/urwid/main_loop.py", line 719, in _loop
    self._watch_files[fd]()
  File "/usr/lib/python3.5/site-packages/urwid/raw_display.py", line 393, in <lambda>
    event_loop, callback, self.get_available_raw_input())
  File "/usr/lib/python3.5/site-packages/urwid/raw_display.py", line 493, in parse_input
    callback(processed, processed_codes)
  File "/usr/lib/python3.5/site-packages/urwid/main_loop.py", line 403, in _update
    self.process_input(keys)
  File "/usr/lib/python3.5/site-packages/urwid/main_loop.py", line 503, in process_input
    k = self._topmost_widget.keypress(self.screen_size, k)
  File "/usr/lib/python3.5/site-packages/mitmproxy/console/window.py", line 86, in keypress
    k = super(self.__class__, self).keypress(size, k)
  File "/usr/lib/python3.5/site-packages/urwid/container.py", line 1128, in keypress
    return self.body.keypress( (maxcol, remaining), key )
  File "/usr/lib/python3.5/site-packages/mitmproxy/console/flowlist.py", line 385, in keypress
    return urwid.ListBox.keypress(self, size, key)
  File "/usr/lib/python3.5/site-packages/urwid/listbox.py", line 985, in keypress
    key = focus_widget.keypress((maxcol,),key)
  File "/usr/lib/python3.5/site-packages/mitmproxy/console/flowlist.py", line 238, in keypress
    self.master.view_flow(self.flow)
  File "/usr/lib/python3.5/site-packages/mitmproxy/console/master.py", line 576, in view_flow
    flowview.FlowView(self, self.state, flow, tab_offset),
  File "/usr/lib/python3.5/site-packages/mitmproxy/console/flowview.py", line 143, in __init__
    tab_offset
  File "/usr/lib/python3.5/site-packages/mitmproxy/console/tabs.py", line 31, in __init__
    self.show()
  File "/usr/lib/python3.5/site-packages/mitmproxy/console/tabs.py", line 69, in show
    body = self.tabs[self.tab_offset][1](),
  File "/usr/lib/python3.5/site-packages/mitmproxy/console/flowview.py", line 166, in view_request
    return self.conn_text(self.flow.request)
  File "/usr/lib/python3.5/site-packages/mitmproxy/console/flowview.py", line 260, in conn_text
    msg, body = self.content_view(viewmode, conn)
  File "/usr/lib/python3.5/site-packages/mitmproxy/console/flowview.py", line 203, in content_view
    flow_modify_cache_invalidation
  File "/usr/lib/python3.5/site-packages/mitmproxy/utils.py", line 32, in get
    ret = gen(*args)
  File "/usr/lib/python3.5/site-packages/mitmproxy/console/flowview.py", line 200, in <lambda>
    lambda *args: self._get_content_view(message, *args),
  File "/usr/lib/python3.5/site-packages/mitmproxy/console/flowview.py", line 208, in _get_content_view
    viewmode, message
  File "/usr/lib/python3.5/site-packages/mitmproxy/contentviews.py", line 635, in get_message_content_view
    viewmode, content, headers=headers, query=query
  File "/usr/lib/python3.5/site-packages/mitmproxy/contentviews.py", line 665, in get_content_view
    _, content = get("Raw")(data, **metadata)
  File "/usr/lib/python3.5/site-packages/mitmproxy/contentviews.py", line 156, in __call__
    return "Raw", format_text(strutils.bytes_to_escaped_str(data, True))
  File "/usr/lib/python3.5/site-packages/netlib/strutils.py", line 88, in bytes_to_escaped_str
    raise ValueError("data must be bytes, but is {}".format(data.__class__.__name__))
ValueError: data must be bytes, but is str

mitmproxy has crashed!
Please lodge a bug report at:
        https://github.com/mitmproxy/mitmproxy
Shutting down...
```

Only some of the Request/Response-pairs causes the crash. It also crashes when loading a saved version of the flow.

##### System information

Mitmproxy version: 0.18.2
Python version: 3.5.2
Platform: Linux-4.7.2-1-ARCH-x86_64-with-arch
SSL version: OpenSSL 1.0.2h  3 May 2016
Linux distro: arch

<!--
    Cut and paste the output of "mitmdump --sysinfo".

    If you're using an older version if mitmproxy, please specify the version
    and OS.
-->



Discussion:
For some reason it seems like the data of the request was stored in a string instead of a bytestring, which made the pretty printing functions crash. Adding the following to the top of get_content_view is a workaround (but not a fix):

``` 
--- /usr/lib/python3.5/site-packages/mitmproxy/contentviews.py.old      2016-12-19 14:09:36.816351044 +0100
+++ /usr/lib/python3.5/site-packages/mitmproxy/contentviews.py.new      2016-12-19 14:10:34.841016211 +0100
@@ -654,6 +654,8 @@
             In contrast to calling the views directly, text is always safe-to-print unicode.
     """
     try:
+        if isinstance(data, str):
+            data = data.encode('utf-8')
         ret = viewmode(data, **metadata)
         if ret is None:
             ret = "Couldn't parse: falling back to Raw", get("Raw")(data, **metadata)[1]
```
Could it be possible that you recorded this with Python2/3 but opened it
with the other Python version?

On Mon, 19 Dec 2016, 14:12 Andreas Källberg, <notifications@github.com>
wrote:

> For some reason it seems like the data of the request was stored in a
> string instead of a bytestring, which made the pretty printing functions
> crash. Adding the following to the top of get_content_view is a workaround
> (but not a fix):
>
> --- /usr/lib/python3.5/site-packages/mitmproxy/contentviews.py.old      2016-12-19 14:09:36.816351044 +0100
> +++ /usr/lib/python3.5/site-packages/mitmproxy/contentviews.py.new      2016-12-19 14:10:34.841016211 +0100
> @@ -654,6 +654,8 @@
>              In contrast to calling the views directly, text is always safe-to-print unicode.
>      """
>      try:
> +        if isinstance(data, str):
> +            data = data.encode('utf-8')
>          ret = viewmode(data, **metadata)
>          if ret is None:
>              ret = "Couldn't parse: falling back to Raw", get("Raw")(data, **metadata)[1]
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/mitmproxy/mitmproxy/issues/1877#issuecomment-267960735>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AA-NPn_BTEa9EOuiz7_5IBsywcj8MdFVks5rJoKegaJpZM4LQfdz>
> .
>

Nope. I got the error even without saving and loading the flows. Just recording (with mitmproxy) and opening (in the same process) while recording is sufficient.
For some reason it seems like the data of the request was stored in a string instead of a bytestring, which made the pretty printing functions crash. Adding the following to the top of get_content_view is a workaround (but not a fix):

``` 
--- /usr/lib/python3.5/site-packages/mitmproxy/contentviews.py.old      2016-12-19 14:09:36.816351044 +0100
+++ /usr/lib/python3.5/site-packages/mitmproxy/contentviews.py.new      2016-12-19 14:10:34.841016211 +0100
@@ -654,6 +654,8 @@
             In contrast to calling the views directly, text is always safe-to-print unicode.
     """
     try:
+        if isinstance(data, str):
+            data = data.encode('utf-8')
         ret = viewmode(data, **metadata)
         if ret is None:
             ret = "Couldn't parse: falling back to Raw", get("Raw")(data, **metadata)[1]
```
Could it be possible that you recorded this with Python2/3 but opened it
with the other Python version?

On Mon, 19 Dec 2016, 14:12 Andreas Källberg, <notifications@github.com>
wrote:

> For some reason it seems like the data of the request was stored in a
> string instead of a bytestring, which made the pretty printing functions
> crash. Adding the following to the top of get_content_view is a workaround
> (but not a fix):
>
> --- /usr/lib/python3.5/site-packages/mitmproxy/contentviews.py.old      2016-12-19 14:09:36.816351044 +0100
> +++ /usr/lib/python3.5/site-packages/mitmproxy/contentviews.py.new      2016-12-19 14:10:34.841016211 +0100
> @@ -654,6 +654,8 @@
>              In contrast to calling the views directly, text is always safe-to-print unicode.
>      """
>      try:
> +        if isinstance(data, str):
> +            data = data.encode('utf-8')
>          ret = viewmode(data, **metadata)
>          if ret is None:
>              ret = "Couldn't parse: falling back to Raw", get("Raw")(data, **metadata)[1]
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/mitmproxy/mitmproxy/issues/1877#issuecomment-267960735>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AA-NPn_BTEa9EOuiz7_5IBsywcj8MdFVks5rJoKegaJpZM4LQfdz>
> .
>

Nope. I got the error even without saving and loading the flows. Just recording (with mitmproxy) and opening (in the same process) while recording is sufficient.

Resolving patch:
diff --git a/mitmproxy/net/http/message.py b/mitmproxy/net/http/message.py
--- a/mitmproxy/net/http/message.py
+++ b/mitmproxy/net/http/message.py
@@ -103,7 +103,11 @@ def get_content(self, strict: bool=True) -> bytes:
         ce = self.headers.get("content-encoding")
         if ce:
             try:
-                return encoding.decode(self.raw_content, ce)
+                content = encoding.decode(self.raw_content, ce)
+                # A client may illegally specify a byte -> str encoding here (e.g. utf8)
+                if isinstance(content, str):
+                    raise ValueError("Invalid Content-Encoding: {}".format(ce))
+                return content
             except ValueError:
                 if strict:
                     raise


Test patch:
diff --git a/test/mitmproxy/net/http/test_message.py b/test/mitmproxy/net/http/test_message.py
--- a/test/mitmproxy/net/http/test_message.py
+++ b/test/mitmproxy/net/http/test_message.py
@@ -141,6 +141,15 @@ def test_unknown_ce(self):
         assert r.headers["content-encoding"]
         assert r.get_content(strict=False) == b"foo"
 
+    def test_utf8_as_ce(self):
+        r = tutils.tresp()
+        r.headers["content-encoding"] = "utf8"
+        r.raw_content = b"foo"
+        with tutils.raises(ValueError):
+            assert r.content
+        assert r.headers["content-encoding"]
+        assert r.get_content(strict=False) == b"foo"
+
     def test_cannot_decode(self):
         r = tutils.tresp()
         r.encode("gzip")