Repository: ipython/ipython

Issue:
Support `raw_input` returning unicode
Hello, I'm trying to port my package https://github.com/Drekin/win-unicode-console to Python 2 and I'd like to be compatible with IPython. My code monkeypatches builtin function `raw_input`, which afterwards returns a unicode object rather than str. Unfortunatelly, IPython is not compatible with this under Python 2: `IPython.terminal.interactiveshell.TerminalInteractiveShell.raw_input` calls `IPython.utils.py3compat.str_to_unicode` (which is mapped to `.decode`) on the result of original `raw_input`. But when `raw_input` returns a unicode object, trying to decode it leads to encoding it first, and if it contains a non-ascii character, the extra encoding step leads to `UnicodeEncodeError` since ascii encoding is assumed. The exception leads to `"You or a %run:ed script called sys.stdin.close() or sys.stdout.close()!\nExiting IPython!\n"` warning and IPython REPL exit. It would be nice if a unicode object was left alone rather then tried to be decoded, i.e. if for example `IPython.utils.py3compat.cast_unicode` was used instead of `.str_to_unicode`.



Discussion:
Seems reasonable. We try to be strict about what is expected to be unicode and what is bytes, and normally the return value of raw_input on Py2 is always bytes. But returning unicode seems like a valid thing for it to do on Windows.

It should use `cast_unicode_py2`, though - Py3 input() should never return bytes.

Feel free to make a PR.

Yeah, I suppose we assume that raw_input actually behaves as Python 2 defines its behavior. I think we used to make raw_input return unicode, but that actually caused problems. Because of that, I might recommend that you not modify raw_input's return type, which will break compatibility with many things (including IPython, here), and wrap the decoding of raw_input str in Python 2 in your own code.

@minrk  - I've come across win-unicode-console before, it's a shim that allows Python to talk to the Unicode native APIs for the Windows console. So, whether it's advisable or not, replacing raw_input is part of the aim, and I think it's very easy for us to support raw_input() returning unicode.

The issue with std\* streams (#8669) is probably trickier, and I'm less sure we should accommodate that.

Actually, replacing raw_input is needed for a specific reason just on Python 2 and making it return unicode or bytes is up to me. I'm still not sure which is better. On one hand, I agree with @minrk that returning bytes is definitely safer, on the other hand I think that returning unicode is more useful and correct. So I first tried returning unicode to see, whether there are problems. Maybe, I'll reconsider…

If you return bytes, we'll decode it using the locale encoding, which will typically be cp1252 on anglophone Windows machines. So you can't represent the full range of unicode characters which the user may enter, as I understand it. I think returning unicode is the right thing to do, we just made an assumption that Python 2 would return bytes.

I think that bytes coming from raw_input should be always decoded using sys.stdin.encoding, and that is what you do by default (https://github.com/ipython/ipython/blob/master/IPython/utils/encoding.py#L51).

Ah, right, I'd forgotten that. So you could set `sys.stdin.encoding = 'utf-16'`. I guess that depends on #8669, though, which would require a bigger change.


Resolving patch:
diff --git a/IPython/terminal/interactiveshell.py b/IPython/terminal/interactiveshell.py
--- a/IPython/terminal/interactiveshell.py
+++ b/IPython/terminal/interactiveshell.py
@@ -525,7 +525,7 @@ def raw_input(self, prompt=''):
         prompt = py3compat.cast_bytes_py2(prompt)
 
         try:
-            line = py3compat.str_to_unicode(self.raw_input_original(prompt))
+            line = py3compat.cast_unicode_py2(self.raw_input_original(prompt))
         except ValueError:
             warn("\n********\nYou or a %run:ed script called sys.stdin.close()"
                  " or sys.stdout.close()!\nExiting IPython!\n")


Test patch: