Repository: mitmproxy/mitmproxy

Issue:
[web] Failed to dump flows into json when visiting https website.
##### Steps to reproduce the problem:

1. start mitmweb and set the correct proxy configuration in the browser.
2. visit [github](https://github.com), or any other website with https
3. mitmweb stuck and throw an exception:
```python
ERROR:tornado.application:Exception in callback <function WebMaster.run.<locals>.<lambda> at 0x7f8871ebb378>
Traceback (most recent call last):
  File "/home/matthew/Hack/mitmproxy/venv3.5/lib/python3.5/site-packages/tornado/ioloop.py", line 1041, in _run
    return self.callback()
  File "/home/matthew/Hack/mitmproxy/mitmproxy/tools/web/master.py", line 109, in <lambda>
    tornado.ioloop.PeriodicCallback(lambda: self.tick(timeout=0), 5).start()
  File "/home/matthew/Hack/mitmproxy/mitmproxy/master.py", line 109, in tick
    handle_func(obj)
  File "/home/matthew/Hack/mitmproxy/mitmproxy/controller.py", line 70, in wrapper
    master.addons(f.__name__, message)
  File "/home/matthew/Hack/mitmproxy/mitmproxy/addonmanager.py", line 90, in __call__
    self.invoke(i, name, *args, **kwargs)
  File "/home/matthew/Hack/mitmproxy/mitmproxy/addonmanager.py", line 85, in invoke
    func(*args, **kwargs)
  File "/home/matthew/Hack/mitmproxy/mitmproxy/addons/view.py", line 327, in request
    self.add(f)
  File "/home/matthew/Hack/mitmproxy/mitmproxy/addons/view.py", line 255, in add
    self.sig_view_add.send(self, flow=f)
  File "/home/matthew/Hack/mitmproxy/venv3.5/lib/python3.5/site-packages/blinker/base.py", line 267, in send
    for receiver in self.receivers_for(sender)]
  File "/home/matthew/Hack/mitmproxy/venv3.5/lib/python3.5/site-packages/blinker/base.py", line 267, in <listcomp>
    for receiver in self.receivers_for(sender)]
  File "/home/matthew/Hack/mitmproxy/mitmproxy/tools/web/master.py", line 58, in _sig_view_add
    data=app.flow_to_json(flow)
  File "/home/matthew/Hack/mitmproxy/mitmproxy/tools/web/app.py", line 197, in broadcast
    message = json.dumps(kwargs, ensure_ascii=False).encode("utf8", "surrogateescape")
  File "/usr/lib/python3.5/json/__init__.py", line 237, in dumps
    **kw).encode(obj)
  File "/usr/lib/python3.5/json/encoder.py", line 198, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.5/json/encoder.py", line 256, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python3.5/json/encoder.py", line 179, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: b'-----BEGIN CERTIFICATE-----\nMIIC5jCCAc6gAwIBAgIGDYj0HL5MMA0GCSqGSIb3DQEBCwUAMCgxEjAQBgNVBAMM\nCW1pdG1wcm94eTESMBAGA1UECgwJbWl0bXByb3h5MB4XDTE3MDIyNTA5MDM0M1oX\nDTIwMDIyNzA5MDM0M1owFTETMBEGA1UEAwwKZ2l0aHViLmNvbTCCASIwDQYJKoZI\nhvcNAQEBBQADggEPADCCAQoCggEBAMTLqdlVNA4h2xzkX5XhLO1wtqZX0X0JpsXC\nHUO+KE3Pf2IBHWFAzeB3SVuaTSIa55UvRUDgZm+gYpl/qswf3MpPB8rkosLtwSJt\ns7ziAYF0JlrwYW+ZBaH/baQZ4JmgpY3qFzrkNhXwrVW+Wg3uO47w/9GaIuUNVv5t\nElfbCDBO0wvWt9tgEuaFNLVwOnibN4LEioQw/xnnUZu4JU6u+16rWasARxU7vlGs\no+CB6wgoK62W4VnSK7aQv6PMAOR49tyzhLXO6LKHQtZA4DG34zXWTYfXhuTC7rnA\nQ6haZ9qyVyeYclIXpJkmf10q2eJTjQbj8ff4Cj3LYlVmBtC2qbsCAwEAAaMpMCcw\nJQYDVR0RBB4wHIIKZ2l0aHViLmNvbYIOd3d3LmdpdGh1Yi5jb20wDQYJKoZIhvcN\nAQELBQADggEBABRJcH+lDB6ec343S+tNYDtr+wWgSiGw7WggKcUpMawBuqY61K4L\nLoxous98ie5XFfLbZI2rW/sIbMEuhjjamEMNmt83ZmZxo/YzMTXO/HlmHZYm+Vjw\nTdhGxe5cGTxjCwXhygRHX+IupDjanniwmh2jfg/0SlW7S4YE/MQJ1mcbGyzppwkg\n4hZ6sEcGe+RC7Sn1tJWlVpA3V8a6udZE8ejlaZV0/PYbJUWyRxAl00PlvRG2sPu5\nEJM7Xbd0TxtqVX7oagImBhqlhf0CyJfRMq0DU34j0oeUqtV/0FaapMumOODcnloI\nJeldz1QeX2hHksE1hYeVjZNFNKQLtzvEpgg=\n-----END CERTIFICATE-----\n' is not JSON serializable

```

##### Any other comments? What have you tried so far?
`Flow.client_conn.mitmcert`is in the type of `bytes`, so `json.dumps()` could not handle it and throw an exception saying that it is not JSON serializable.
I noticed that in `flow_to_json()` function, there is a comment say: 
> Remove flow message content and cert to save transmission space.

And we indeed remove the `server_conn.cert` from the returning dict, but left `client_conn.mitmcert`.
I have tried to add one more line of code to remove `client_conn.mitmcert` from the returning dict and it solve this exception. 
However, I am not sure whether it is appropriate to remove this item. Or should we convert it into a string and keep it in the returning dict?

##### System information

Mitmproxy version: 3.0.0 (2.0.0dev0028-0x0fdf2c0) 
Python version: 3.5.2
Platform: Linux-4.4.0-63-generic-x86_64-with-Ubuntu-16.04-xenial
SSL version: OpenSSL 1.0.2g  1 Mar 2016
Linux distro: Ubuntu 16.04 xenial



Discussion:

Resolving patch:
diff --git a/mitmproxy/tools/web/app.py b/mitmproxy/tools/web/app.py
--- a/mitmproxy/tools/web/app.py
+++ b/mitmproxy/tools/web/app.py
@@ -85,6 +85,7 @@ def flow_to_json(flow: mitmproxy.flow.Flow) -> dict:
                 "is_replay": flow.response.is_replay,
             }
     f.get("server_conn", {}).pop("cert", None)
+    f.get("client_conn", {}).pop("mitmcert", None)
 
     return f
 


Test patch: