Repository: python/mypy
Issue category: Bug Report

Issue:
Error when displaying error that contains unicode characters in Windows
<!--
  If you're new to mypy and you're not sure whether what you're experiencing is a mypy bug, please see the "Question and Help" form
  instead.
  Please also consider:

  - checking our common issues page: https://mypy.readthedocs.io/en/stable/common_issues.html
  - searching our issue tracker: https://github.com/python/mypy/issues to see if
    it's already been reported
  - asking on gitter chat: https://gitter.im/python/typing
-->

**Bug Report**

<!--
Note: If the problem you are reporting is about a specific library function, then the typeshed tracker is better suited
for this report: https://github.com/python/typeshed/issues
-->

When displaying a type error about e.g. a variable that contains unicode characters, mypy crashes.

**To Reproduce**
1. Make a file `file.py` containing the line `x=γ`. 
2. Run `mypy.exe --show-column-numbers file.py` through flycheck (python-mypy) in Emacs 

**Expected Behavior**
An error message like `file.py:1:5: error: Name "γ" is not defined`

<!--
  How did you expect your project to behave?
  It’s fine if you’re not sure your understanding is correct.
  Write down what you thought would happen. If you just expected no errors, you can delete this section.
-->

**Actual Behavior**
It crashes and prints a stack trace:
```
File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python39_64\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python39_64\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "c:\Program Files (x86)\Microsoft Visual Studio\Shared\Python39_64\Scripts\mypy.exe\__main__.py", line 7, in <module>
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python39_64\lib\site-packages\mypy\__main__.py", line 15, in console_entry
    main(None, sys.stdout, sys.stderr)
  File "mypy\main.py", line 96, in main
  File "mypy\main.py", line 173, in run_build
  File "mypy\build.py", line 180, in build
  File "mypy\build.py", line 256, in _build
  File "mypy\build.py", line 2717, in dispatch
  File "mypy\build.py", line 3048, in process_graph
  File "mypy\build.py", line 3164, in process_stale_scc
  File "mypy\main.py", line 165, in flush_errors
  File "mypy\main.py", line 199, in show_messages
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python39_64\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u03b3' in position 33: character maps to <undefined>
```

I've fixed it locally by adding 
```
sys.stdout.reconfigure(encoding='utf-8')
sys.stderr.reconfigure(encoding='utf-8')
```

in `mypy/__main__.py`. It works for me, but I don't know whether it's the right thing for mypy. 


<!--
  Did something go wrong?
  Is something broken, or not behaving as you expected?
-->

**Your Environment**

Python 3.9.7, mypy 0.931 on Windows

<!-- Include as many relevant details about the environment you experienced the bug in -->

- Mypy version used: 0.931
- Mypy command-line flags: --show-column-numbers
- Mypy configuration options from `mypy.ini` (and other config files): None, I think
- Python version used:  3.9.7
- Operating system and version: Windows 11

<!--
You can freely edit this text, please remove all the lines
you believe are unnecessary.
-->



Hints:
My 'fix' doesn't really work perfectly. Something in Windows+emacs+flycheck doesn't decode the mypy output as unicode, and what I see in Emacs is `file.py:1:5: error: Name "Î³" is not defined`. But that's probably not a mypy issue.
Update: I tested this with updated mypy 0.950 in Windows and Ubuntu, and couldn't reproduce by calling `mypy.exe --show-column-numbers file.py` in the command line. The issue happens only in flycheck in Emacs. I guess that flycheck's python-mypy runs in a special environment where stderr and stdout are opened as TextIO buffers with a non-utf-8 encoding.
This can still happen anytime the output encoding can't represent a codepoint in the error message. For example, this can be reproduced on a unix system by running
```shell
$ PYTHONIOENCODING=cp1252 mypy -c "x=γ"
Traceback (most recent call last):
    ...
  File "/home/brian/Projects/open-contrib/mypy/mypy/main.py", line 230, in show_messages
    f.write(msg + "\n")
  File "/usr/lib/python3.12/encodings/cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode character '\u03b3' in position 50: character maps to <undefined>
```

If this comes up again, we could look into using a different error handler when writing the output to stdout/stderr (the current default is `"strict"`, which raises an exception). Something like `"backslashreplace"` might make sense. For now, affected environments can try setting a different encoding or error hander via `PYTHONIOENCODING` or any other method. For example:

```shell
$ PYTHONIOENCODING=utf-8 mypy -c "x=γ"
<string>:1: error: Name "γ" is not defined  [name-defined]

$ PYTHONIOENCODING=cp1252:backslashreplace mypy -c "x=γ"
<string>:1: error: Name "\u03b3" is not defined  [name-defined]
```
Setting backslashreplace as the error handler seems like a good idea here.

Patch:
diff --git a/mypy/main.py b/mypy/main.py
index e1c9f20400bc..d2a28a18c6a8 100644
--- a/mypy/main.py
+++ b/mypy/main.py
@@ -9,6 +9,7 @@
 import time
 from collections import defaultdict
 from gettext import gettext
+from io import TextIOWrapper
 from typing import IO, Any, Final, NoReturn, Sequence, TextIO
 
 from mypy import build, defaults, state, util
@@ -74,6 +75,10 @@ def main(
     if args is None:
         args = sys.argv[1:]
 
+    # Write an escape sequence instead of raising an exception on encoding errors.
+    if isinstance(stdout, TextIOWrapper) and stdout.errors == "strict":
+        stdout.reconfigure(errors="backslashreplace")
+
     fscache = FileSystemCache()
     sources, options = process_options(args, stdout=stdout, stderr=stderr, fscache=fscache)
     if clean_exit:


Test patch: