Repository: pwndbg/pwndbg

Issue:
utf8 decoding error
### Description
Context is not correctly displayed due to utf8 decoding error.

Traceback (most recent call last):
  File "/home/carstein/Tools/pwndbg/pwndbg/commands/__init__.py", line 109, in __call__
    return self.function(*args, **kwargs)
  File "/home/carstein/Tools/pwndbg/pwndbg/commands/__init__.py", line 200, in _OnlyWhenRunning
    return function(*a, **kw)
  File "/home/carstein/Tools/pwndbg/pwndbg/commands/context.py", line 86, in context
    result.extend(func())
  File "/home/carstein/Tools/pwndbg/pwndbg/commands/context.py", line 98, in context_regs
    return [pwndbg.ui.banner("registers")] + get_regs()
  File "/home/carstein/Tools/pwndbg/pwndbg/ui.py", line 52, in banner
    banner = ljust_colored(banner, width, str(config.banner_separator))
  File "/home/carstein/Tools/pwndbg/pwndbg/color/__init__.py", line 77, in ljust_colored
    return x + ((remaining // len(char) + 1) * char)[:remaining]
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe2 in position 105: unexpected end of data


### Steps to reproduce
1. Load binary into gdb
2. `break main`
3. `run`


### My setup
pwndbg version: 
Gdb:      GNU gdb (GDB) 8.1
Python:   2.7.13 (default, Nov 24 2017, 17:33:09)  [GCC 6.3.0 20170516]
Pwndbg:   1.0.0 build: 869e832
Capstone: 4.0.1024
Unicorn:  1.0.1



Discussion:
Python2 is a bitch, you will need to set proper utf8 LC types, there is a hint somewhere what needs to be done 

@anthraxx you probably meant this:
https://github.com/pwndbg/pwndbg/blob/6a1fdb282ed34fd7c6579b16ea111a8a11de226d/gdbinit.py#L23-L34

@carstein can u check if that's the issue?
Hey, if you are talking about launching it via:
$ LC_ALL=en_US.UTF-8 PYTHONIOENCODING=UTF-8 /opt/bin/gdb
tried, does not help
Are you sure it's enabled in locals and locals were regenerated afterwards? Try exporting the env vars instead, before calling gdb.
Also try setting LC_CTYPE via export.
Try playing around in your environment if Python2 can't use utf8 then it's én environmental issue. 

Checked locales and reganeration before
python is able to use utf8 (tested)
```
>>> print '\xc4\x85'
ą
```

I've started looking into the code more closely and it is interesting in a sense, that pwndbg is able to construct half of the banner for registers 
but fails on the second part (after [ REGISTERS]) in ```colored/__init__.py```
52: banner = ljust_colored(banner, width, str(config.banner_separator))

So, rjust_colored() worked fine, but for some reason ljust_colored() failed. That is bizzare.
Nice info, not so much if my theory is correct. Let me test some things and great a PR, if my theory fails we are back at investigating 😄

@anthraxx that's an issue on our side, investigating too :P
It's a possibly half consumed sequence of bytes, already on it 😉

So Python2 configuration values do not guarantee the type, therefor returning a raw string with Unicode sequence will result in being splitted wrongly in between the sequence instead of logically per character like python3 or Python2 with correct decoded unicode type.
I will test my work around and submit a temporary PR until the Parameter configuration type thingies in #404 land in the tree, they should address such issues and make a workaround obsolete. 


Resolving patch:
diff --git a/pwndbg/color/__init__.py b/pwndbg/color/__init__.py
--- a/pwndbg/color/__init__.py
+++ b/pwndbg/color/__init__.py
@@ -7,6 +7,8 @@
 
 import re
 
+import six
+
 import pwndbg.memoize
 
 NORMAL         = "\x1b[0m"
@@ -72,9 +74,17 @@ def terminateWith(x, color):
     return re.sub('\x1b\\[0m', NORMAL + color, x)
 
 def ljust_colored(x, length, char=' '):
+    # TODO: workaround until issue #404
+    if six.PY2:
+        x = x if isinstance(x, six.text_type) else x.decode('utf8')
+        char = char if isinstance(char, six.text_type) else char.decode('utf8')
     remaining = length - len(strip(x))
     return x + ((remaining // len(char) + 1) * char)[:remaining]
 
 def rjust_colored(x, length, char=' '):
+    # TODO: workaround until issue #404
+    if six.PY2:
+        x = x if isinstance(x, six.text_type) else x.decode('utf8')
+        char = char if isinstance(char, six.text_type) else char.decode('utf8')
     remaining = length - len(strip(x))
     return ((remaining // len(char) + 1) * char)[:remaining] + x


Test patch: