Package: win_unicode_console
A Python package to enable Unicode input and display when running Python from Windows console.General informationWhen running Python in the standard console on Windows, there are several problems when one tries to enter or display Unicode characters. The relevant issue ishttp://bugs.python.org/issue1602. This package solves some of them.First, when you want to display Unicode characters in Windows console, you have to select a font able to display them. Similarly, if you want to enter Unicode characters, you have to have you keyboard properly configured. This has nothing to do with Python, but is included here for completeness.The standard stream objects (sys.stdin,sys.stdout,sys.stderr) are not capable of reading and displaying Unicode characters in Windows console. This has nothing to do with encoding, since evensys.stdin.buffer.raw.readline()returnsb"?\n"when enteringαand there is no encoding under whichsys.stdout.buffer.raw.writedisplaysα.Thestreamsmodule provides several alternative stream objects.stdin_raw,stdout_raw, andstderr_raware raw stream objects using WinAPI functionsReadConsoleWandWriteConsoleWto interact with Windows console through UTF-16-LE encoded bytes. Thestdin_text,stdout_text, andstderr_textare standard text IO wrappers over standard buffered IO over our raw streams, and are intended to be primary replacements tosys.std*streams. Unfortunately, other wrappers aroundstd*_textare needed (see below), so there are more stream objects instreamsmodule.The functionstreams.enableinstalls chosen stream objects instead of the original ones. By default, it chooses appropriate stream objects itself. The functionstreams.disablerestores the original stream objects (these are stored insys.__std*__attributes by Python).After replacing the stream objects, also usingprintwith a string containing Unicode characters and displaying Unicode characters in the interactive loop works. Forinput, see below.Python interactive loop doesn’t usesys.stdinto read input so fixing it doesn’t help. Also theinputfunction may or may not usesys.stdindepending on whethersys.stdinandsys.stdouthave the standard filenos and whether they are interactive. Seehttp://bugs.python.org/issue17620for more information.To solve this, we install a custom readline hook. Readline hook is a function which is used to read a single line interactively by Python REPL. It may also be used byinputfunction under certain conditions (see above). On Linux, this hook is usually set to GNU readline function, which provides features like autocompletion, history, …The modulereadline_hookprovides our custom readline hook, which usessys.stdinto get the input and is (de)activated by functionsreadline_hook.enable,readline_hook.disable.As we said, readline hook can be called from two places – from the REPL and frominputfunction. In the first case the prompt is encoded usingsys.stdin.encoding, but in the second casesys.stdout.encodingis used. So Python currently makes an assumption that these two encodings are equal.Python tokenizer, which is used when parsing the input from REPL, cannot handle UTF-16 or generally any encoding containing null bytes. Because UTF-16-LE is the encoding of Unicode used by Windows, we have to additionally wrap our text stream objects (std*_text). Thus,streamsmodule contains also stream objectsstdin_text_transcoded,stdout_text_transcoded, andstderr_text_transcoded. They basically just hide the underlying UTF-16-LE encoded buffered IO, and sets encoding to UTF-8. These transcoding wrappers are used by default bystreams.enable.There are additional issues on Python 2.Since default Python 2 strings correspond tobytesrather thanunicode, people are usually callingprintwithbytesargument. Therefore,sys.stdout.writeandsys.stderr.writeshould supportbytesargument. That is why we addstdout_text_strandstderr_text_strstream objects tostreamsmodule. They are used by default on Python 2.When we enter a Unicode literal into interactive interpreter, it gets processed by the Python tokenizer, which is bytes-based. When we enteru"\u03b1"into the interactive interpreter, the tokenizer gets essentiallyb'u"\xce\xb1"'plus the information that the encoding used is UTF-8. The problem is that the tokenizer uses the encoding only ifsys.stdinis a file object (seehttps://hg.python.org/cpython/file/d356e68de236/Parser/tokenizer.c#l797). Hence, we introduce another stream objectstreams.stdin_text_fileobjthat wrapsstdin_text_transcodedand also is structurally compatible with Python file object. This object is used by default on Python 2.The check for interactive streams done byraw_inputunfortunately requires that bothsys.stdinandsys.stdoutare file objects. Besidesstdin_text_fileobjfor stdin we could use alsostdout_text_str_fileobjfor stdout. Unfortunately, that breaksprint.Usingprintstatement or function leads to callingPyFile_WriteObjectwithsys.stdoutas argument. Unfortunately, its genericwritemethod is used only if it isnota file object. Otherwise,PyObject_Printis called, and this function is file-based, so it ends with afprintfcall, which is not something we want. In conclusion, we need stdoutnotto be a file object.Given the situation described, the best solution seems to be reimplementingraw_inputandinputbuiltin functions and monkeypatching__builtins__. This is done by ourraw_inputmodule on Python 2.Similarly to the input from fromsys.stdinthe arguments insys.argvare alsobyteson Python 2 and the original ones may not be reconstructable. To overcome this we addunicode_argvmodule. The functionunicode_argv.get_unicode_argvreturns Unicode version ofsys.argvobtained by WinAPI functionsGetCommandLineWandCommandLineToArgvW. The functionunicode_argv.enablemonkeypatchessys.argvwith the Unicode arguments.InstallationInstall the package from PyPI viapip installwin-unicode-console(recommended), or download the archive and install it from the archive (e.g.pip installwin_unicode_console-0.x.zip), or install the package manually by placing directorywin_unicode_consoleand modulerun.pyfrom the archive to thesite-packagesdirectory of your Python installation.UsageThe top-levelwin_unicode_consolemodule contains a functionenable, which install various fixes offered bywin_unicode_consolemodules, and a functiondisable, which restores the original environment. By default, custom stream objects are installed as well as a custom readline hook. On Python 2,raw_inputandinputfunctions are monkeypatched.sys.argvis not monkeypatched by default since unfortunately some Python 2 code strictly assumesstrinstances insys.argvlist. Useenable(use_unicode_argv=True)if you want the monkeypathcing. For further customization, see the sources. The logic should be clear.Generic usage of the package is just callingwin_unicode_console.enable()whenever the fixes should be applied andwin_unicode_console.disable()to revert all the changes. Note that it should be a responsibility of a Python user on Windows to installwin_unicode_consoleand fix his Python environment regarding Unicode interaction with console, rather than of a third-party developer enablingwin_unicode_consolein his application, which adds a dependency. Our package should be seen as an external patch to Python on Windows rather than a feature package for other packages not directly related to fixing Unicode issues.Different ways of howwin_unicode_consolecan be used to fix a Python environment on Windows follow.Python patch (recommended).Just callwin_unicode_console.enable()in yoursitecustomizeorusercustomizemodule (seehttps://docs.python.org/3/tutorial/appendix.html#the-customization-modulesfor more information). This will enablewin_unicode_consoleon every run of the Python interpreter (unlesssiteis disabled). Doing so should not break executed scripts in any way. Otherwise, it is a bug ofwin_unicode_consolethat should be fixed.Opt-in runner.You may easily run a script withwin_unicode_consoleenabled by using ourrunnermodule and its helperrunscript. To do so, executepy-i-mrun script.pyinstead ofpy-iscript.pyfor interactive mode, and similarlypy-mrun script.pyinstead ofpy script.pyfor non-interactive mode. Of course you may provide arguments to your script:py-i-mrun script.py arg1 arg2. To run the bare interactive interpreter withwin_unicode_consoleenabled, executepy-i-mrun.Opt-out runner.In case you are usingwin_unicode_consoleas Python patch, but you want to run a particular script withwin_unicode_consoledisabled, you can also use the runner. To do so, executepy-i-mrun--init-disablescript.py.Customized runner.To move arbitrary initialization (e.g. enablingwin_unicode_consolewith non-default arguments) fromsitecustomizeto opt-in runner, move it to a separate module and usepy-i-mrun--init-modulemodule script.py. That will import a modulemoduleon startup instead of enablingwin_unicode_consolewith default arguments.Compatibilitywin_unicode_consolepackage was tested on Python 3.4, Python 3.5, and Python 2.7. 32-bit or 64-bit shouldn’t matter. It also interacts well with the following packages:coloramapackage (https://pypi.python.org/pypi/colorama) makes ANSI escape character sequences (for producing colored terminal text and cursor positioning) work under MS Windows. It does so by wrappingsys.stdoutandsys.stderrstreams. Sincewin_unicode_consolereplaces the streams in order to support Unicode,win_unicode_console.enablehas to be called beforecolorama.initso everything works as expected.As ofcoloramav0.3.3, there was an early binding issue (https://github.com/tartley/colorama/issues/32), sowin_unicode_console.enablehas to be called even before importingcolorama. Note that is already the case whenwin_unicode_consoleis used as Python patch or as opt-in runner. The issue was already fixed.pyreadlinepackage (https://pypi.python.org/pypi/pyreadline/2.0)  implements GNU readline features on Windows. It provides its own readline hook, which actually supports Unicode input.win_unicode_console.readline_hookdetects whenpyreadlineis active, and in that case, by default, reuses its readline hook rather than installing its own, so GNU readline features are preserved on top of our Unicode streams.IPython(https://pypi.python.org/pypi/ipython) can be also used  withwin_unicode_console.As ofIPython3.2.1, there is an early binding issue (https://github.com/ipython/ipython/issues/8669), sowin_unicode_console.enablehas to be called even before importingIPython. That is the case whenwin_unicode_consoleis used as Python patch.There was also an issue that IPython was not compatible with the builtin functionraw_inputreturning unicode on Python 2 (https://github.com/ipython/ipython/issues/8670). If you hit this issue, you can makewin_unicode_console.raw_input.raw_inputreturn bytes by enabling it aswin_unicode_console.enable(raw_input__return_unicode=False). This was fixed in IPython 4.Backward incompatibilitySince version 0.4, the signature ofstreams.enablehas been changed because there are now more options for the stream objects to be used. It now accepts a keyword argument for eachstdin,stdout,stderr, setting the corresponding stream.Nonemeans “do not set”,Ellipsismeans “use the default value”.A functionstreams.enable_onlywas added. It works the same way asstreams.enable, but the default value for each parameter isNone.Functionsstreams.enable_reader,streams.enable_writer, andstreams.enable_error_writerhave been removed. Example: instead ofstreams.enable_reader(transcode=True)usestreams.enable_only(stdin=streams.stdin_text_transcoding).There are also corresponding changes in top-levelenablefunction.Since version 0.3, the custom stream objects have the standard filenos, so callinginputdoesn’t handle Unicode without custom readline hook.AcknowledgementsThe code ofstreamsmodule is based on the code submitted tohttp://bugs.python.org/issue1602.The idea of providing custom readline hook and the code ofreadline_hookmodule is based onhttps://github.com/pyreadline/pyreadline.The code related tounicode_argv.get_full_unicode_argvis based onhttp://code.activestate.com/recipes/572200/.The idea of using path hooks and the code related tounicode_argv.argv_setter_hookis based onhttps://mail.python.org/pipermail/python-list/2016-June/710183.html.