Repository: mitmproxy/mitmproxy

Issue:
SNI in ClientConnection that caused json.dumps broken
The problem seems occurred after #1664 .
It seems that the Connection.get_servername() from OpenSSL return bytes.
And that cause json dumps failed.
I would like pick up this problem, and help display sni  and cipher (from #582).
Is that just used `str(self.connection.get_servername())` to cast it into str is OK?
##### Steps to reproduce the problem:
1. mitmweb with latest master version
2. curl -x https://example.com
3. mitmweb display error stacktrace

```
Traceback (most recent call last):
  File "/Users/han-pc/.pyenv/versions/3.5.2/envs/mitm/lib/python3.5/site-packages/tornado/ioloop.py", line 1041, in _run
    return self.callback()
  File "/Users/han-pc/.pyenv/versions/3.5.2/envs/mitm/lib/python3.5/site-packages/mitmproxy/tools/web/master.py", line 172, in <lambda>
    tornado.ioloop.PeriodicCallback(lambda: self.tick(timeout=0), 5).start()
  File "/Users/han-pc/.pyenv/versions/3.5.2/envs/mitm/lib/python3.5/site-packages/mitmproxy/master.py", line 105, in tick
    handle_func(obj)
  File "/Users/han-pc/.pyenv/versions/3.5.2/envs/mitm/lib/python3.5/site-packages/mitmproxy/controller.py", line 70, in wrapper
    master.addons(f.__name__, message)
  File "/Users/han-pc/.pyenv/versions/3.5.2/envs/mitm/lib/python3.5/site-packages/mitmproxy/addonmanager.py", line 90, in __call__
    self.invoke(i, name, *args, **kwargs)
  File "/Users/han-pc/.pyenv/versions/3.5.2/envs/mitm/lib/python3.5/site-packages/mitmproxy/addonmanager.py", line 85, in invoke
    func(*args, **kwargs)
  File "/Users/han-pc/.pyenv/versions/3.5.2/envs/mitm/lib/python3.5/site-packages/mitmproxy/addons/state.py", line 285, in request
    self.add_flow(f)
  File "/Users/han-pc/.pyenv/versions/3.5.2/envs/mitm/lib/python3.5/site-packages/mitmproxy/addons/state.py", line 205, in add_flow
    self.flows._add(f)
  File "/Users/han-pc/.pyenv/versions/3.5.2/envs/mitm/lib/python3.5/site-packages/mitmproxy/addons/state.py", line 112, in _add
    view._add(f)
  File "/Users/han-pc/.pyenv/versions/3.5.2/envs/mitm/lib/python3.5/site-packages/mitmproxy/tools/web/master.py", line 33, in _add
    data=app.convert_flow_to_json_dict(f)
  File "/Users/han-pc/.pyenv/versions/3.5.2/envs/mitm/lib/python3.5/site-packages/mitmproxy/tools/web/app.py", line 174, in broadcast
    message = json.dumps(kwargs, ensure_ascii=False)
  File "/Users/han-pc/.pyenv/versions/3.5.2/lib/python3.5/json/__init__.py", line 237, in dumps
    **kw).encode(obj)
  File "/Users/han-pc/.pyenv/versions/3.5.2/lib/python3.5/json/encoder.py", line 198, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/Users/han-pc/.pyenv/versions/3.5.2/lib/python3.5/json/encoder.py", line 256, in iterencode
    return _iterencode(o, 0)
  File "/Users/han-pc/.pyenv/versions/3.5.2/lib/python3.5/json/encoder.py", line 179, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: b'example.com' is not JSON serializable
```

SNI in ClientConnection that caused json.dumps broken
The problem seems occurred after #1664 .
It seems that the Connection.get_servername() from OpenSSL return bytes.
And that cause json dumps failed.
I would like pick up this problem, and help display sni  and cipher (from #582).
Is that just used `str(self.connection.get_servername())` to cast it into str is OK?
##### Steps to reproduce the problem:
1. mitmweb with latest master version
2. curl -x https://example.com
3. mitmweb display error stacktrace

```
Traceback (most recent call last):
  File "/Users/han-pc/.pyenv/versions/3.5.2/envs/mitm/lib/python3.5/site-packages/tornado/ioloop.py", line 1041, in _run
    return self.callback()
  File "/Users/han-pc/.pyenv/versions/3.5.2/envs/mitm/lib/python3.5/site-packages/mitmproxy/tools/web/master.py", line 172, in <lambda>
    tornado.ioloop.PeriodicCallback(lambda: self.tick(timeout=0), 5).start()
  File "/Users/han-pc/.pyenv/versions/3.5.2/envs/mitm/lib/python3.5/site-packages/mitmproxy/master.py", line 105, in tick
    handle_func(obj)
  File "/Users/han-pc/.pyenv/versions/3.5.2/envs/mitm/lib/python3.5/site-packages/mitmproxy/controller.py", line 70, in wrapper
    master.addons(f.__name__, message)
  File "/Users/han-pc/.pyenv/versions/3.5.2/envs/mitm/lib/python3.5/site-packages/mitmproxy/addonmanager.py", line 90, in __call__
    self.invoke(i, name, *args, **kwargs)
  File "/Users/han-pc/.pyenv/versions/3.5.2/envs/mitm/lib/python3.5/site-packages/mitmproxy/addonmanager.py", line 85, in invoke
    func(*args, **kwargs)
  File "/Users/han-pc/.pyenv/versions/3.5.2/envs/mitm/lib/python3.5/site-packages/mitmproxy/addons/state.py", line 285, in request
    self.add_flow(f)
  File "/Users/han-pc/.pyenv/versions/3.5.2/envs/mitm/lib/python3.5/site-packages/mitmproxy/addons/state.py", line 205, in add_flow
    self.flows._add(f)
  File "/Users/han-pc/.pyenv/versions/3.5.2/envs/mitm/lib/python3.5/site-packages/mitmproxy/addons/state.py", line 112, in _add
    view._add(f)
  File "/Users/han-pc/.pyenv/versions/3.5.2/envs/mitm/lib/python3.5/site-packages/mitmproxy/tools/web/master.py", line 33, in _add
    data=app.convert_flow_to_json_dict(f)
  File "/Users/han-pc/.pyenv/versions/3.5.2/envs/mitm/lib/python3.5/site-packages/mitmproxy/tools/web/app.py", line 174, in broadcast
    message = json.dumps(kwargs, ensure_ascii=False)
  File "/Users/han-pc/.pyenv/versions/3.5.2/lib/python3.5/json/__init__.py", line 237, in dumps
    **kw).encode(obj)
  File "/Users/han-pc/.pyenv/versions/3.5.2/lib/python3.5/json/encoder.py", line 198, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/Users/han-pc/.pyenv/versions/3.5.2/lib/python3.5/json/encoder.py", line 256, in iterencode
    return _iterencode(o, 0)
  File "/Users/han-pc/.pyenv/versions/3.5.2/lib/python3.5/json/encoder.py", line 179, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: b'example.com' is not JSON serializable
```



Discussion:
If the `get_servername` is expected to be a domain name, I think using

```
self.connection.get_servername().decode('idna')
```

should resolve this issue.

Hi guys - thanks for catching that!
What @mike820324 said - IDNA encoding should be the right way! :smiley:

If the `get_servername` is expected to be a domain name, I think using

```
self.connection.get_servername().decode('idna')
```

should resolve this issue.

Hi guys - thanks for catching that!
What @mike820324 said - IDNA encoding should be the right way! :smiley:


Resolving patch:
diff --git a/mitmproxy/connections.py b/mitmproxy/connections.py
--- a/mitmproxy/connections.py
+++ b/mitmproxy/connections.py
@@ -104,7 +104,11 @@ def make_dummy(cls, address):
     def convert_to_ssl(self, *args, **kwargs):
         super().convert_to_ssl(*args, **kwargs)
         self.timestamp_ssl_setup = time.time()
-        self.sni = self.connection.get_servername()
+        sni = self.connection.get_servername()
+        if sni:
+            self.sni = sni.decode("idna")
+        else:
+            self.sni = None
         self.cipher_name = self.connection.get_cipher_name()
         self.tls_version = self.connection.get_protocol_version_name()
 


Test patch: