Repository: mitmproxy/mitmproxy

Issue:
Properly check for UTF8 console
##### Steps to reproduce the problem:

1. `mitmproxy` and `mitmproxy --save-stream-file +/tmp/mitmdump`
2. select some event, enter
3. scroll down

```python
Traceback (most recent call last):
  File "/usr/local/Cellar/mitmproxy/3.0.0/libexec/lib/python3.6/site-packages/mitmproxy/tools/console/master.py", line 216, in run
    self.loop.run()
  File "/usr/local/Cellar/mitmproxy/3.0.0/libexec/lib/python3.6/site-packages/urwid/main_loop.py", line 286, in run
    self._run()
  File "/usr/local/Cellar/mitmproxy/3.0.0/libexec/lib/python3.6/site-packages/urwid/main_loop.py", line 384, in _run
    self.event_loop.run()
  File "/usr/local/Cellar/mitmproxy/3.0.0/libexec/lib/python3.6/site-packages/urwid/main_loop.py", line 788, in run
    self._loop()
  File "/usr/local/Cellar/mitmproxy/3.0.0/libexec/lib/python3.6/site-packages/urwid/main_loop.py", line 816, in _loop
    self._entering_idle()
  File "/usr/local/Cellar/mitmproxy/3.0.0/libexec/lib/python3.6/site-packages/urwid/main_loop.py", line 777, in _entering_idle
    callback()
  File "/usr/local/Cellar/mitmproxy/3.0.0/libexec/lib/python3.6/site-packages/urwid/main_loop.py", line 572, in entering_idle
    self.draw_screen()
  File "/usr/local/Cellar/mitmproxy/3.0.0/libexec/lib/python3.6/site-packages/urwid/main_loop.py", line 586, in draw_screen
    canvas = self._topmost_widget.render(self.screen_size, focus=True)
  File "/usr/local/Cellar/mitmproxy/3.0.0/libexec/lib/python3.6/site-packages/urwid/widget.py", line 144, in cached_render
    canv = fn(self, size, focus=focus)
  File "/usr/local/Cellar/mitmproxy/3.0.0/libexec/lib/python3.6/site-packages/urwid/container.py", line 1086, in render
    focus and self.focus_part == 'body')
  File "/usr/local/Cellar/mitmproxy/3.0.0/libexec/lib/python3.6/site-packages/urwid/widget.py", line 144, in cached_render
    canv = fn(self, size, focus=focus)
  File "/usr/local/Cellar/mitmproxy/3.0.0/libexec/lib/python3.6/site-packages/urwid/decoration.py", line 226, in render
    canv = self._original_widget.render(size, focus=focus)
  File "/usr/local/Cellar/mitmproxy/3.0.0/libexec/lib/python3.6/site-packages/urwid/widget.py", line 144, in cached_render
    canv = fn(self, size, focus=focus)
  File "/usr/local/Cellar/mitmproxy/3.0.0/libexec/lib/python3.6/site-packages/urwid/container.py", line 1086, in render
    focus and self.focus_part == 'body')
  File "/usr/local/Cellar/mitmproxy/3.0.0/libexec/lib/python3.6/site-packages/urwid/widget.py", line 144, in cached_render
    canv = fn(self, size, focus=focus)
  File "/usr/local/Cellar/mitmproxy/3.0.0/libexec/lib/python3.6/site-packages/urwid/container.py", line 1086, in render
    focus and self.focus_part == 'body')
  File "/usr/local/Cellar/mitmproxy/3.0.0/libexec/lib/python3.6/site-packages/urwid/widget.py", line 144, in cached_render
    canv = fn(self, size, focus=focus)
  File "/usr/local/Cellar/mitmproxy/3.0.0/libexec/lib/python3.6/site-packages/urwid/widget.py", line 1765, in render
    canv = get_delegate(self).render(size, focus=focus)
  File "/usr/local/Cellar/mitmproxy/3.0.0/libexec/lib/python3.6/site-packages/urwid/widget.py", line 144, in cached_render
    canv = fn(self, size, focus=focus)
  File "/usr/local/Cellar/mitmproxy/3.0.0/libexec/lib/python3.6/site-packages/urwid/container.py", line 1086, in render
    focus and self.focus_part == 'body')
  File "/usr/local/Cellar/mitmproxy/3.0.0/libexec/lib/python3.6/site-packages/urwid/widget.py", line 144, in cached_render
    canv = fn(self, size, focus=focus)
  File "/usr/local/Cellar/mitmproxy/3.0.0/libexec/lib/python3.6/site-packages/urwid/listbox.py", line 489, in render
    focus_canvas = focus_widget.render((maxcol,), focus=focus)
  File "/usr/local/Cellar/mitmproxy/3.0.0/libexec/lib/python3.6/site-packages/urwid/widget.py", line 144, in cached_render
    canv = fn(self, size, focus=focus)
  File "/usr/local/Cellar/mitmproxy/3.0.0/libexec/lib/python3.6/site-packages/urwid/widget.py", line 1004, in render
    return apply_text_layout(text, attr, trans, maxcol)
  File "/usr/local/Cellar/mitmproxy/3.0.0/libexec/lib/python3.6/site-packages/urwid/canvas.py", line 1315, in apply_text_layout
    return TextCanvas(t, a, c, maxcol=maxcol)
  File "/usr/local/Cellar/mitmproxy/3.0.0/libexec/lib/python3.6/site-packages/urwid/canvas.py", line 358, in __init__
    raise CanvasError("Canvas text is wider than the maxcol specified \n%r\n%r\n%r"%(maxcol,widths,text))
urwid.canvas.CanvasError: Canvas text is wider than the maxcol specified
153
[153, 153, 154, 125]
[b'[REDACTED NAME]:        ....$.......?.E=?[LONG REDACTED GLIBBERISH]"]
```
![pasted_image_22_02_18_18_36](https://user-images.githubusercontent.com/2737108/36554418-801662a6-17ff-11e8-88af-6c82a95c3d37.png)

##### System information

<!-- Paste the output of "mitmproxy --version" here. -->
```shell
$ mitmproxy --version
Mitmproxy: 3.0.0
Python:    3.6.4
OpenSSL:   OpenSSL 1.0.2n  7 Dec 2017
Platform:  Darwin-13.4.0-x86_64-i386-64bit
```
(via Brew, installed earlier today, so should be the latest version)


From the second run with dumping to file,
this seems to be the binary form-data where it crashed.
The request is to an API's upload function.
```
2018-02-23 08:00:00
--Boundary+1DAA87079F6BACB5
Content-Disposition: form-data; name="[REDACTED NAME]"; filename="file"
Content-Type: application/octet-stream

^A^@^@^@[REDACTED GLIBBERISH]
```
As I am not certain about the contents of the recorded data, so I won't post the dump public.
However, I have keep it for now.

Also might be another occurrence of #1796.



Discussion:
Basically it seems to render the raw binary data in too long lines.

On a previous crash I got those values:
```python
maxcol = 153
widths = [152, 153, 134, 153, 132, 154, 138, 98, 133, 154, 154, 153, 156, 153, 81, 121, 43, 154, 153, 154, 153, 153, 121, 153, 154, 153, 153, 153, 136, 102, 81, 149, 154, 154, 153, 131, 55, 153, 153, 153, 153, 154, 154, 155, 153, 114, 153, 11, 153, 83, 154, 72, 100, 109, 153, 156, 152, 143, 110, 110, 146, 117, 153, 64, 153, 145, 82, 131, 154, 153, 155, 153, 154, 155, 153, 133, 148, 83, 155, 153, 97, 154, 153, 153, 144, 102, 134, 117, 153, 133, 153, 133, 101, 154, 153, 130, 132, 154, 143, 108, 92, 91, 154, 153, 94, 153, 153, 105, 153, 87, 154, 105, 137, 88, 79, 124, 94, 138, 153, 154, 89, 132, 21, 154, 153, 153, 104, 133, 153, 154, 94, 100, 77, 153, 76, 137, 153, 153, 154, 153, 154, 154, 154, 153, 153, 118, 96, 148, 151, 154, 153, 110, 154, 35, 153, 154, 153, 153, 112, 151, 1, 153, 58, 125, 153, 153, 153, 154, 154, 153, 153, 41, 133, 86, 140, 153, 153, 154, 109, 125, 123, 153, 143, 62, 153, 154, 153, 17, 153, 85, 95, 133, 73, 122, 113, 102, 134, 156, 154, 153, 153, 155, 153, 93, 96, 131, 154, 154, 153, 153, 122, 132, 117, 144, 155, 154, 153, 32, 135, 156, 22, 153, 153, 122, 154, 125, 148, 6, 154, 153, 130, 140, 153, 155, 121, 108, 153, 154, 154, 55, 155, 154, 70, 154, 153, 153, 153, 153, 155, 34, 153, 153, 153, 155, 154, 34, 154, 153, 155, 155, 154, 152, 155, 153, 153, 13, 141, 28, 153, 117, 154, 153, 154, 119, 68, 153, 119, 86, 91, 153, 154, 153, 153, 154, 153, 28, 139, 154, 154, 27, 128, 153, 153, 153, 153, 154, 81, 103, 132, 134, 153, 84, 117, 66, 122, 154, 102, 152, 154, 140]
```
And the `text` was `[b'file:                 PK............?w4/\\ ` and _pages_ of glibberish and `']`


Yeah, some lines are 1 or 2 characters to long:
```python
print("[" + (", ".join(["+"+str(w-maxcol) if w > maxcol else "ok" for w in widths])) + "]")
[
 ok, ok, ok, ok, ok, +1, ok, ok, ok, +1, +1, ok, +3, ok, ok, ok, ok, +1, ok, +1,
 ok, ok, ok, ok, +1, ok, ok, ok, ok, ok, ok, ok, +1, +1, ok, ok, ok, ok, ok, ok,
 ok, +1, +1, +2, ok, ok, ok, ok, ok, ok, +1, ok, ok, ok, ok, +3, ok, ok, ok, ok,
 ok, ok, ok, ok, ok, ok, ok, ok, +1, ok, +2, ok, +1, +2, ok, ok, ok, ok, +2, ok,
 ok, +1, ok, ok, ok, ok, ok, ok, ok, ok, ok, ok, ok, +1, ok, ok, ok, +1, ok, ok,
 ok, ok, +1, ok, ok, ok, ok, ok, ok, ok, +1, ok, ok, ok, ok, ok, ok, ok, ok, +1,
 ok, ok, ok, +1, ok, ok, ok, ok, ok, +1, ok, ok, ok, ok, ok, ok, ok, ok, +1, ok,
 +1, +1, +1, ok, ok, ok, ok, ok, ok, +1, ok, ok, +1, ok, ok, +1, ok, ok, ok, ok,
 ok, ok, ok, ok, ok, ok, ok, +1, +1, ok, ok, ok, ok, ok, ok, ok, ok, +1, ok, ok,
 ok, ok, ok, ok, ok, +1, ok, ok, ok, ok, ok, ok, ok, ok, ok, ok, ok, +3, +1, ok,
 ok, +2, ok, ok, ok, ok, +1, +1, ok, ok, ok, ok, ok, ok, +2, +1, ok, ok, ok, +3,
 ok, ok, ok, ok, +1, ok, ok, ok, +1, ok, ok, ok, ok, +2, ok, ok, ok, +1, +1, ok,
 +2, +1, ok, +1, ok, ok, ok, ok, +2, ok, ok, ok, ok, +2, +1, ok, +1, ok, +2, +2,
 +1, ok, +2, ok, ok, ok, ok, ok, ok, ok, +1, ok, +1, ok, ok, ok, ok, ok, ok, ok,
 +1, ok, ok, +1, ok, ok, ok, +1, +1, ok, ok, ok, ok, ok, ok, +1, ok, ok, ok, ok,
 ok, ok, ok, ok, ok, +1, ok, ok, +1, ok
]
```
Hi, I could not reproduce the issue on my system.But considering the recent fixes regarding the UI, i would advice you to update to the latest release candidate of mitmproxy.
Same problem here when uploading an image to my server while mitmproxy is running
bug still exist on recent version

```python
Traceback (most recent call last):
  File "/home/q/envs/mitmproxy_image/lib/python3.7/site-packages/mitmproxy/master.py", line 86, in run_loop
    loop()
  File "/home/q/envs/mitmproxy_image/lib/python3.7/site-packages/urwid/main_loop.py", line 286, in run
    self._run()
  File "/home/q/envs/mitmproxy_image/lib/python3.7/site-packages/urwid/main_loop.py", line 384, in _run
    self.event_loop.run()
  File "/home/q/envs/mitmproxy_image/lib/python3.7/site-packages/urwid/main_loop.py", line 1484, in run
    reraise(*exc_info)
  File "/home/q/envs/mitmproxy_image/lib/python3.7/site-packages/urwid/compat.py", line 58, in reraise
    raise value
  File "/usr/lib/python3.7/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/home/q/envs/mitmproxy_image/lib/python3.7/site-packages/urwid/main_loop.py", line 1443, in faux_idle_callback
    callback()
  File "/home/q/envs/mitmproxy_image/lib/python3.7/site-packages/urwid/main_loop.py", line 572, in entering_idle
    self.draw_screen()
  File "/home/q/envs/mitmproxy_image/lib/python3.7/site-packages/urwid/main_loop.py", line 586, in draw_screen
    canvas = self._topmost_widget.render(self.screen_size, focus=True)
  File "/home/q/envs/mitmproxy_image/lib/python3.7/site-packages/urwid/widget.py", line 144, in cached_render
    canv = fn(self, size, focus=focus)
  File "/home/q/envs/mitmproxy_image/lib/python3.7/site-packages/urwid/container.py", line 1086, in render
    focus and self.focus_part == 'body')
  File "/home/q/envs/mitmproxy_image/lib/python3.7/site-packages/urwid/widget.py", line 144, in cached_render
    canv = fn(self, size, focus=focus)
  File "/home/q/envs/mitmproxy_image/lib/python3.7/site-packages/urwid/decoration.py", line 226, in render
    canv = self._original_widget.render(size, focus=focus)
  File "/home/q/envs/mitmproxy_image/lib/python3.7/site-packages/urwid/widget.py", line 144, in cached_render
    canv = fn(self, size, focus=focus)
  File "/home/q/envs/mitmproxy_image/lib/python3.7/site-packages/urwid/container.py", line 1086, in render
    focus and self.focus_part == 'body')
  File "/home/q/envs/mitmproxy_image/lib/python3.7/site-packages/urwid/widget.py", line 144, in cached_render
    canv = fn(self, size, focus=focus)
  File "/home/q/envs/mitmproxy_image/lib/python3.7/site-packages/urwid/listbox.py", line 489, in render
    focus_canvas = focus_widget.render((maxcol,), focus=focus)
  File "/home/q/envs/mitmproxy_image/lib/python3.7/site-packages/urwid/widget.py", line 144, in cached_render
    canv = fn(self, size, focus=focus)
  File "/home/q/envs/mitmproxy_image/lib/python3.7/site-packages/urwid/widget.py", line 1765, in render
    canv = get_delegate(self).render(size, focus=focus)
  File "/home/q/envs/mitmproxy_image/lib/python3.7/site-packages/urwid/widget.py", line 144, in cached_render
    canv = fn(self, size, focus=focus)
  File "/home/q/envs/mitmproxy_image/lib/python3.7/site-packages/urwid/container.py", line 1523, in render
    canv = w.render((maxcol,), focus=focus and item_focus)
  File "/home/q/envs/mitmproxy_image/lib/python3.7/site-packages/urwid/widget.py", line 144, in cached_render
    canv = fn(self, size, focus=focus)
  File "/home/q/envs/mitmproxy_image/lib/python3.7/site-packages/urwid/container.py", line 2087, in render
    focus = focus and self.focus_position == i)
  File "/home/q/envs/mitmproxy_image/lib/python3.7/site-packages/urwid/widget.py", line 144, in cached_render
    canv = fn(self, size, focus=focus)
  File "/home/q/envs/mitmproxy_image/lib/python3.7/site-packages/mitmproxy/tools/console/common.py", line 173, in render
    return urwid.TextCanvas([c_text.encode()], [c_attr], maxcol=width)
  File "/home/q/envs/mitmproxy_image/lib/python3.7/site-packages/urwid/canvas.py", line 358, in __init__
    raise CanvasError("Canvas text is wider than the maxcol specified \n%r\n%r\n%r"%(maxcol,widths,text))
urwid.canvas.CanvasError: Canvas text is wider than the maxcol specified
23
[25]
[b'           \xe5\xb7\xa8\xe4\xb9\xb3.pic-b.com']

mitmproxy has crashed!
```

```console
$ mitmproxy --version
Mitmproxy: 5.0.0
Python:    3.7.3
OpenSSL:   OpenSSL 1.1.0j  20 Nov 2018
Platform:  Linux-5.0.0-37-generic-x86_64-with-Ubuntu-19.04-disco
```
Yeah I am also having the same issue

```
➜ mitmproxy --version
Mitmproxy: 5.0.0
Python:    3.8.0
OpenSSL:   OpenSSL 1.1.0j  20 Nov 2018
Platform:  Linux-4.19.88-1-MANJARO-x86_64-with-glibc2.2.5
```
It would be of great help if anyone has steps to reliably and easily reproduce this.
I have two terminal

in one terminal I run `mitmproxy` and in another terminal I run the following commands:

```
➜ export http_proxy=http://127.0.0.1:8080
➜ curl google.com
```
after running `curl` command when I check the `mitmproxy` terminal window I have the same traceback as posted above.
I had the same error.  mitmproxy-5.0.1-linux

I had to set "LANG=en_US.UTF-8"  to be able to run mitmproxy, but the environment already had "LC_ALL=en_US.ISO-8859-1" (and LC_CTYPE).  

mitmproxy would work if # of putty window columns was <= 99, but crash if I expanded the window >= 100.  ("tput cols"), but only after a new flow  (curl google.com) with "application/*" as the type (maybe others, but not "text/html" since that seems not to be colored in the same way)

When I unset "LC_ALL" & LC_CTYPE or set them to "en_US.UTF-8" it did not crash when window cols >= 100.


Thanks @daspri! This sounds super promising, but I can't replicate it. Here's my setup:

Console 1:
```
env LC_ALL=en_US.ISO-8859-1 LC_CTYPE=en_US.ISO-8859-1 LANG=en_US.UTF-8 mitmproxy
```

Console 2:
```
curl -x localhost:8080 -k https://google.com
```

Environment:
```
$ mitmproxy --version
Mitmproxy: 6.0.0.dev (+19, commit 7b638f1)
Python:    3.8.0
OpenSSL:   OpenSSL 1.1.1  11 Sep 2018
Platform:  Linux-4.4.0-18362-Microsoft-x86_64-with-glibc2.27

$ printenv
EDITOR=nano
GIT_HOME=/mnt/c/Users/user/git
GOPATH=/home/user/git/go
GPG_TTY=/dev/tty1
HOME=/home/user
HOSTTYPE=x86_64
LANG=C.UTF-8
LOGNAME=user
NAME=xps
PATH=[clip]
PWD=/mnt/c/Users/user
SHELL=/usr/bin/fish
SHLVL=1
SSH_AGENT_PID=520
SSH_AUTH_SOCK=/tmp/ssh-9r6AgD/agent.519
TERM=xterm-256color
TOX_WORK_DIR=.tox-linux
USER=user
VIRTUAL_ENV=/home/user/venv
WSLENV=
WSL_DISTRO_NAME=Ubuntu
_OLD_FISH_PROMPT_OVERRIDE=/home/user/venv
```

No matter how I resize the window, I don't get any exceptions. Any hints where the difference is?
Try to run curl a few times. Sometimes the first run works and the 2nd or 3rd kills it.

Screen 1: 
```
> tput cols
120
> LANG=en_US.UTF8 LC_ALL=en_US.ISO-8859-1 ./mitmproxy  --listen-host 127.0.0.1
```

Screen 2:
```
> curl -k -x http://127.0.0.1:8080 "https://www.google.com"
> curl -k -x http://127.0.0.1:8080 "https://www.google.com"
> curl -k -x http://127.0.0.1:8080 "https://www.google.com"
```

Screen 1:
```
Traceback (most recent call last):
  File "mitmproxy/master.py", line 86, in run_loop
  File "site-packages/urwid/main_loop.py", line 286, in run
  File "site-packages/urwid/main_loop.py", line 384, in _run
  File "site-packages/urwid/main_loop.py", line 1484, in run
  File "site-packages/urwid/compat.py", line 58, in reraise
  File "asyncio/events.py", line 88, in _run
  File "site-packages/urwid/main_loop.py", line 1443, in faux_idle_callback
  File "site-packages/urwid/main_loop.py", line 572, in entering_idle
  File "site-packages/urwid/main_loop.py", line 586, in draw_screen
  File "site-packages/urwid/widget.py", line 144, in cached_render
  File "site-packages/urwid/container.py", line 1086, in render
  File "site-packages/urwid/widget.py", line 144, in cached_render
  File "site-packages/urwid/decoration.py", line 226, in render
  File "site-packages/urwid/widget.py", line 144, in cached_render
  File "site-packages/urwid/container.py", line 1086, in render
  File "site-packages/urwid/widget.py", line 144, in cached_render
  File "site-packages/urwid/listbox.py", line 501, in render
  File "site-packages/urwid/widget.py", line 144, in cached_render
  File "site-packages/urwid/widget.py", line 1765, in render
  File "site-packages/urwid/widget.py", line 144, in cached_render
  File "site-packages/urwid/container.py", line 1523, in render
  File "site-packages/urwid/widget.py", line 144, in cached_render
  File "site-packages/urwid/container.py", line 2087, in render
  File "site-packages/urwid/widget.py", line 144, in cached_render
  File "mitmproxy/tools/console/common.py", line 173, in render
  File "site-packages/urwid/canvas.py", line 358, in __init__
urwid.canvas.CanvasError: Canvas text is wider than the maxcol specified
15
[17]
[b'\xe2\x80\xa6ntent missing]']
```

It does not crash if "tput cols" is less than 100, or if I don't add "LC_ALL=en_US.ISO-8859-1".




Doing exactly that works just fine for me. :(

![image](https://user-images.githubusercontent.com/1019198/71752350-094d3300-2e7f-11ea-8521-2441706cd023.png)

Do you have that locale installed?  "locale -a" ?

```
env -i LANG=C.UTF-8 LC_ALL=en_US ./mitmproxy  --listen-host 127.0.0.1
```
Will also crash for me, but lowercase US does not:
```
env -i LANG=C.UTF-8 LC_ALL=en_us ./mitmproxy  --listen-host 127.0.0.1
```

Crashes on first try with microsoft, second try with google.
```
 curl -k -x http://127.0.0.1:8080 "https://www.microsoft.com"
```
Thanks for the instructions - I was able to replicate it. @mhils I can take a look at this
Thank you so much! I can finally reproduce this by forcing mitmproxy to use UTF8 symbols here:
https://github.com/mitmproxy/mitmproxy/blob/184384af57e81bce09469e2fefe6dbb134eda6ce/mitmproxy/tools/console/common.py#L98
and then running
```
env -i LANG=C.UTF-8 LC_ALL=C ~/venv/bin/mitmproxy
```
I have no idea how any of this works, but it looks like we need to adjust our UTF8 check: https://github.com/mitmproxy/mitmproxy/blob/184384af57e81bce09469e2fefe6dbb134eda6ce/mitmproxy/tools/_main.py#L21-L34

🤷‍♂ 
`mitmproxy --version`
``` bash
Mitmproxy: 5.2
Python:    3.7.7+
OpenSSL:   OpenSSL 1.1.1g  21 Apr 2020
Platform:  Linux-4.19.150-1-MANJARO-x86_64-with-arch-Manjaro-Linux
```
`locale -a` 

``` bash
C
en_US.utf8
ja_JP.utf8
POSIX
zh_CN.utf8
```
`tput cols`
``` bash
240
```

``` bash
Traceback (most recent call last):
  File "/home/jimmy/.pyenv/versions/3.7-dev/lib/python3.7/site-packages/mitmproxy/master.py", line 86, in run_loop
    loop()
  File "/home/jimmy/.pyenv/versions/3.7-dev/lib/python3.7/site-packages/urwid/main_loop.py", line 287, in run
    self._run()
  File "/home/jimmy/.pyenv/versions/3.7-dev/lib/python3.7/site-packages/urwid/main_loop.py", line 385, in _run
    self.event_loop.run()
  File "/home/jimmy/.pyenv/versions/3.7-dev/lib/python3.7/site-packages/urwid/main_loop.py", line 1494, in run
    reraise(*exc_info)
  File "/home/jimmy/.pyenv/versions/3.7-dev/lib/python3.7/site-packages/urwid/compat.py", line 58, in reraise
    raise value
  File "/home/jimmy/.pyenv/versions/3.7-dev/lib/python3.7/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/home/jimmy/.pyenv/versions/3.7-dev/lib/python3.7/site-packages/urwid/main_loop.py", line 1454, in faux_idle_callback
    callback()
  File "/home/jimmy/.pyenv/versions/3.7-dev/lib/python3.7/site-packages/urwid/main_loop.py", line 574, in entering_idle
    self.draw_screen()
  File "/home/jimmy/.pyenv/versions/3.7-dev/lib/python3.7/site-packages/urwid/main_loop.py", line 588, in draw_screen
    canvas = self._topmost_widget.render(self.screen_size, focus=True)
  File "/home/jimmy/.pyenv/versions/3.7-dev/lib/python3.7/site-packages/urwid/widget.py", line 145, in cached_render
    canv = fn(self, size, focus=focus)
  File "/home/jimmy/.pyenv/versions/3.7-dev/lib/python3.7/site-packages/urwid/container.py", line 1086, in render
    focus and self.focus_part == 'body')
  File "/home/jimmy/.pyenv/versions/3.7-dev/lib/python3.7/site-packages/urwid/widget.py", line 145, in cached_render
    canv = fn(self, size, focus=focus)
  File "/home/jimmy/.pyenv/versions/3.7-dev/lib/python3.7/site-packages/urwid/decoration.py", line 226, in render
    canv = self._original_widget.render(size, focus=focus)
  File "/home/jimmy/.pyenv/versions/3.7-dev/lib/python3.7/site-packages/urwid/widget.py", line 145, in cached_render
    canv = fn(self, size, focus=focus)
  File "/home/jimmy/.pyenv/versions/3.7-dev/lib/python3.7/site-packages/urwid/container.py", line 1086, in render
    focus and self.focus_part == 'body')
  File "/home/jimmy/.pyenv/versions/3.7-dev/lib/python3.7/site-packages/urwid/widget.py", line 145, in cached_render
    canv = fn(self, size, focus=focus)
  File "/home/jimmy/.pyenv/versions/3.7-dev/lib/python3.7/site-packages/urwid/listbox.py", line 445, in render
    focus_canvas = focus_widget.render((maxcol,), focus=focus)
  File "/home/jimmy/.pyenv/versions/3.7-dev/lib/python3.7/site-packages/urwid/widget.py", line 145, in cached_render
    canv = fn(self, size, focus=focus)
  File "/home/jimmy/.pyenv/versions/3.7-dev/lib/python3.7/site-packages/urwid/widget.py", line 1761, in render
    canv = get_delegate(self).render(size, focus=focus)
  File "/home/jimmy/.pyenv/versions/3.7-dev/lib/python3.7/site-packages/urwid/widget.py", line 145, in cached_render
    canv = fn(self, size, focus=focus)
  File "/home/jimmy/.pyenv/versions/3.7-dev/lib/python3.7/site-packages/urwid/container.py", line 2125, in render
    focus = focus and self.focus_position == i)
  File "/home/jimmy/.pyenv/versions/3.7-dev/lib/python3.7/site-packages/urwid/widget.py", line 145, in cached_render
    canv = fn(self, size, focus=focus)
  File "/home/jimmy/.pyenv/versions/3.7-dev/lib/python3.7/site-packages/mitmproxy/tools/console/common.py", line 207, in render
    return urwid.TextCanvas([c_text.encode()], [c_attr], maxcol=width)
  File "/home/jimmy/.pyenv/versions/3.7-dev/lib/python3.7/site-packages/urwid/canvas.py", line 358, in __init__
    raise CanvasError("Canvas text is wider than the maxcol specified \n%r\n%r\n%r"%(maxcol,widths,text))
urwid.canvas.CanvasError: Canvas text is wider than the maxcol specified
21
[26]
[b'\xe2\x80\xa6": [Errno 101] \xe7\xbd\x91\xe7\xbb\x9c\xe4\xb8\x8d\xe5\x8f\xaf\xe8\xbe\xbe']

mitmproxy has crashed!
Please lodge a bug report at:
	https://github.com/mitmproxy/mitmproxy
```

Resolving patch:
diff --git a/mitmproxy/tools/console/common.py b/mitmproxy/tools/console/common.py
--- a/mitmproxy/tools/console/common.py
+++ b/mitmproxy/tools/console/common.py
@@ -191,7 +191,7 @@ def render(self, size, focus=False):
             text = text[::-1]
             attr = attr[::-1]
 
-        text_len = len(text)  # TODO: unicode?
+        text_len = urwid.util.calc_width(text, 0, len(text))
         if size is not None and len(size) > 0:
             width = size[0]
         else:
@@ -206,8 +206,10 @@ def render(self, size, focus=False):
                 c_text = text
                 c_attr = attr
         else:
-            visible_len = width - len(SYMBOL_ELLIPSIS)
-            visible_text = text[0:visible_len]
+            trim = urwid.util.calc_trim_text(text, 0, width - 1, 0, width - 1)
+            visible_text = text[0:trim[1]]
+            if trim[3] == 1:
+                visible_text += ' '
             c_text = visible_text + SYMBOL_ELLIPSIS
             c_attr = (urwid.util.rle_subseg(attr, 0, len(visible_text.encode())) +
                       [('focus', len(SYMBOL_ELLIPSIS.encode()))])


Test patch:
diff --git a/test/mitmproxy/tools/console/test_common.py b/test/mitmproxy/tools/console/test_common.py
--- a/test/mitmproxy/tools/console/test_common.py
+++ b/test/mitmproxy/tools/console/test_common.py
@@ -36,3 +36,10 @@ def test_format_keyvals():
             ("aa", wrapped)
         ]
     )
+
+
+def test_truncated_text():
+    half_width_text = common.TruncatedText("Half-width", [])
+    full_width_text = common.TruncatedText("ＦＵＬＬ－ＷＩＤＴＨ", [])
+    assert half_width_text.render((10,))
+    assert full_width_text.render((10,))