Repository: ipython/ipython

Issue:
IPython crashes with UnicodeEncodeError on ascii terminal
Displaying a unicode character on an ascii terminal gives a UnicodeEncodeError, while it would be handy if it just shows a question mark or something.

Entering a unicode character which would be displayed directly makes it crash.

Case 1: displaying a unicode character: (é = \u00E9)
```
In [3]: a = 'é'  # looks like a = '??' on an ascii terminal
In [4]: a
Out[4]: ---------------------------------------------------------------------------
UnicodeEncodeError                        Traceback (most recent call last)
<ipython-input-4-3f786850e387> in <module>
----> 1 a

~/.conda/envs/wrf/lib/python3.6/site-packages/IPython/core/displayhook.py in __call__(self, result)
    259             self.fill_exec_result(result)
    260             if format_dict:
--> 261                 self.write_format_data(format_dict, md_dict)
    262                 self.log_output(format_dict)
    263             self.finish_displayhook()

~/.conda/envs/wrf/lib/python3.6/site-packages/IPython/core/displayhook.py in write_format_data(self, format_dict, md_dict)
    188                 result_repr = '\n' + result_repr
    189 
--> 190         print(result_repr)
    191 
    192     def update_user_ns(self, result):

UnicodeEncodeError: 'ascii' codec can't encode characters in position 1-2: ordinal not in range(128)
```
Case 2: direct display of unicode character:
```
In [5]: 'é'  # looks like '??' on ascii terminal
Out[5]: Traceback (most recent call last):
  File "/home/danie010/.conda/envs/wrf/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3296, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-5-9b7611f0c6d4>", line 1, in <module>
    '\xc3\xa9'
  File "/home/danie010/.conda/envs/wrf/lib/python3.6/site-packages/IPython/core/displayhook.py", line 261, in __call__
    self.write_format_data(format_dict, md_dict)
  File "/home/danie010/.conda/envs/wrf/lib/python3.6/site-packages/IPython/core/displayhook.py", line 190, in write_format_data
    print(result_repr)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 1-2: ordinal not in range(128)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/danie010/.conda/envs/wrf/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3220, in run_ast_nodes
    if (yield from self.run_code(code, result)):
  File "/home/danie010/.conda/envs/wrf/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3313, in run_code
    self.showtraceback(running_compiled_code=True)
  File "/home/danie010/.conda/envs/wrf/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2038, in showtraceback
    self._showtraceback(etype, value, stb)
  File "/home/danie010/.conda/envs/wrf/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2056, in _showtraceback
    print(self.InteractiveTB.stb2text(stb))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 276-277: ordinal not in range(128)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/danie010/.conda/envs/wrf/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2874, in _run_cell
    return runner(coro)
  File "/home/danie010/.conda/envs/wrf/lib/python3.6/site-packages/IPython/core/async_helpers.py", line 67, in _pseudo_sync_runner
    coro.send(None)
  File "/home/danie010/.conda/envs/wrf/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3049, in run_cell_async
    interactivity=interactivity, compiler=compiler, result=result)
  File "/home/danie010/.conda/envs/wrf/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3239, in run_ast_nodes
    self.showtraceback()
  File "/home/danie010/.conda/envs/wrf/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2038, in showtraceback
    self._showtraceback(etype, value, stb)
  File "/home/danie010/.conda/envs/wrf/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2056, in _showtraceback
    print(self.InteractiveTB.stb2text(stb))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 842-843: ordinal not in range(128)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/danie010/.conda/envs/wrf/bin/ipython", line 11, in <module>
    sys.exit(start_ipython())
  File "/home/danie010/.conda/envs/wrf/lib/python3.6/site-packages/IPython/__init__.py", line 125, in start_ipython
    return launch_new_instance(argv=argv, **kwargs)
  File "/home/danie010/.conda/envs/wrf/lib/python3.6/site-packages/traitlets/config/application.py", line 658, in launch_instance
    app.start()
  File "/home/danie010/.conda/envs/wrf/lib/python3.6/site-packages/IPython/terminal/ipapp.py", line 356, in start
    self.shell.mainloop()
  File "/home/danie010/.conda/envs/wrf/lib/python3.6/site-packages/IPython/terminal/interactiveshell.py", line 498, in mainloop
    self.interact()
  File "/home/danie010/.conda/envs/wrf/lib/python3.6/site-packages/IPython/terminal/interactiveshell.py", line 489, in interact
    self.run_cell(code, store_history=True)
  File "/home/danie010/.conda/envs/wrf/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2848, in run_cell
    raw_cell, store_history, silent, shell_futures)
  File "/home/danie010/.conda/envs/wrf/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2879, in _run_cell
    self.showtraceback(running_compiled_code=True)
  File "/home/danie010/.conda/envs/wrf/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2038, in showtraceback
    self._showtraceback(etype, value, stb)
  File "/home/danie010/.conda/envs/wrf/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2056, in _showtraceback
    print(self.InteractiveTB.stb2text(stb))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 842-843: ordinal not in range(128)

If you suspect this is an IPython bug, please report it at:
    https://github.com/ipython/ipython/issues
or send an email to the mailing list at ipython-dev@python.org

You can print a more detailed traceback right now with "%tb", or use "%debug"
to interactively debug it.

Extra-detailed tracebacks for bug-reporting purposes can be enabled via:
    %config Application.verbose_crash=True
```

The character encoding of the terminal:
```
python -c "import sys; print(sys.stdout.encoding)"          
ANSI_X3.4-1968
```

sys_info:
```
python -c "import IPython; print(IPython.sys_info())"
{'commit_hash': 'd774f565b',
 'commit_source': 'installation',
 'default_encoding': 'ANSI_X3.4-1968',
 'ipython_path': '/nfs/home3/danie010/.conda/envs/wrf/lib/python3.6/site-packages/IPython',
 'ipython_version': '7.4.0',
 'os_name': 'posix',
 'platform': 'Linux-3.10.0-957.10.1.el7.x86_64-x86_64-with-redhat-7.6-Maipo',
 'sys_executable': '/home/danie010/.conda/envs/wrf/bin/python',
 'sys_platform': 'linux',
 'sys_version': '3.6.7 | packaged by conda-forge | (default, Feb 20 2019, '
                '02:51:38) \n'
                '[GCC 7.3.0]'}
```


Discussion:
Running ipython with PYTHONIOENCODING=UTF-8 solves the issue.
```
~$ PYTHONIOENCODING=UTF-8 ipython3 
Python 3.6.7 | packaged by conda-forge | (default, Feb 20 2019, 02:51:38)                         
Type 'copyright', 'credits' or 'license' for more information                                     
IPython 7.4.0 -- An enhanced Interactive Python. Type '?' for help.                               
                                                                                                  
In [1]: '\u00E9'                                                                                  
Out[1]: '�'                                                                                       
                                                                                                  
In [2]: 'é'                                                                                      
Out[2]: 'é'                                                                                      

In [3]: import sys; sys.stdout.encoding
Out[3]: 'UTF-8'
```
Cpython displays an encoded character when it isn't supported, that sounds like the way to go.

```
>>> '\u00e9'
'\xe9'
>>> print('\u00e9')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 0: ordinal not in range(128)
>>> import sys; sys.stdout.encoding                                                               
'ANSI_X3.4-1968'
```

It still gives an error when using print, but that's another issue.

Resolving patch:
diff --git a/IPython/core/displayhook.py b/IPython/core/displayhook.py
--- a/IPython/core/displayhook.py
+++ b/IPython/core/displayhook.py
@@ -187,7 +187,12 @@ def write_format_data(self, format_dict, md_dict=None):
                 # But avoid extraneous empty lines.
                 result_repr = '\n' + result_repr
 
-        print(result_repr)
+        try:
+            print(result_repr)
+        except UnicodeEncodeError:
+            # If a character is not supported by the terminal encoding replace
+            # it with its \u or \x representation
+            print(result_repr.encode(sys.stdout.encoding,'backslashreplace').decode(sys.stdout.encoding))
 
     def update_user_ns(self, result):
         """Update user_ns with various things like _, __, _1, etc."""


Test patch: