Repository: pytest-dev/pytest
Pull request: Fixes #3630
Fixes #3630

- [x] Create a new changelog file in the `changelog` folder, with a name like `<ISSUE NUMBER>.<TYPE>.rst`. See [changelog/README.rst](/changelog/README.rst) for details.
- [x] Target the `master` branch for bug fixes, documentation updates and trivial changes.
- **(not applicable)** Target the `features` branch for new features and removals/deprecations.
- **(not applicable)** Include documentation when adding new features.
- [x] Include new tests or update existing tests when applicable.

Unless your change is trivial or a small documentation fix (e.g.,  a typo or reword of a small section) please:

- **(trivial change)** Add yourself to `AUTHORS` in alphabetical order;


Problem statement:
problems with unicode characters when saving log to file
## Normal case
I have this very simple test which is just logging some unicode character:

test_log.py
```
import logging

logger = logging.getLogger(__name__)

def test():
    logger.info("Normal")
    logger.info("\u251c")
    logger.info("Also normal")
```

And when I run it using `--log-cli-level=INFO` then I get this terminal output, which is fine:
```
============================= test session starts =============================
platform win32 -- Python 3.6.3, pytest-3.6.2, py-1.5.4, pluggy-0.6.0 -- c:\users\victor\desktop\pytest unicode bug\venv\scripts\python.exe
cachedir: .pytest_cache
rootdir: C:\Users\Victor\Desktop\pytest unicode bug, inifile:
collected 1 item                                                               

test_log.py::test
-------------------------------- live log call --------------------------------
test_log.py                  6 INFO     Normal
test_log.py                  7 INFO     ├
test_log.py                  8 INFO     Also normal
PASSED                                                                   [100%]

========================== 1 passed in 0.02 seconds ===========================


```
## Save to file
However, when I also run with `--log-file=logoutput`, this is what I get:

logoutput contents:
```
test_log.py                  6 INFO     Normal
test_log.py                  8 INFO     Also normal
```

**So it seems that the lines with unicode characters are not getting printed in the log file, which is a problem.**

## Environment details

- pytest 3.6.2
- Windows 10

Hints from issue discussion:
[GitMate.io](https://gitmate.io) thinks possibly related issues are https://github.com/pytest-dev/pytest/issues/3338 (Problem with Custom Subclass of logging.Formatter), https://github.com/pytest-dev/pytest/issues/1131 (Experiencing unicode error inside pytester in Hypothesis test suite), https://github.com/pytest-dev/pytest/issues/25 (Can not enter PDB debugger when file contains the copyright '©' character.), https://github.com/pytest-dev/pytest/issues/531 (Problems with fixture parametrization), and https://github.com/pytest-dev/pytest/issues/2435 (f).
Separated the `pytester` problem as another issue. For this one (about logging to file) I got a PR.

Commit message: Fixed open function with encoding in python 2.7

Patch to review:
diff --git a/changelog/3630.bugfix.rst b/changelog/3630.bugfix.rst
new file mode 100644
index 00000000000..505bcb8fde9
--- /dev/null
+++ b/changelog/3630.bugfix.rst
@@ -0,0 +1 @@
+Log messages with unicode characters would not appear in the output log file.
diff --git a/src/_pytest/logging.py b/src/_pytest/logging.py
index 70b6592c405..ae1a80c50aa 100644
--- a/src/_pytest/logging.py
+++ b/src/_pytest/logging.py
@@ -392,7 +392,9 @@ def __init__(self, config):
                 config, "log_file_date_format", "log_date_format"
             )
             # Each pytest runtests session will write to a clean logfile
-            self.log_file_handler = logging.FileHandler(log_file, mode="w")
+            self.log_file_handler = logging.FileHandler(
+                log_file, mode="w", encoding="UTF-8"
+            )
             log_file_formatter = logging.Formatter(
                 log_file_format, datefmt=log_file_date_format
             )
diff --git a/testing/logging/test_reporting.py b/testing/logging/test_reporting.py
index 0a8a585063c..1f961d94a87 100644
--- a/testing/logging/test_reporting.py
+++ b/testing/logging/test_reporting.py
@@ -1,6 +1,7 @@
 # -*- coding: utf-8 -*-
 import re
 import os
+from io import open
 
 import six
 
@@ -827,6 +828,40 @@ def test_log_file(request):
         assert "This log message won't be shown" not in contents
 
 
+def test_log_file_unicode(testdir):
+    log_file = testdir.tmpdir.join("pytest.log").strpath
+
+    testdir.makeini(
+        """
+        [pytest]
+        log_file={}
+        log_file_level = INFO
+        """.format(
+            log_file
+        )
+    )
+    testdir.makepyfile(
+        """
+        import logging
+        def test_log_file():
+            logging.getLogger('catchlog').info("Normal message")
+            logging.getLogger('catchlog').info("\u251c")
+            logging.getLogger('catchlog').info("Another normal message")
+    """
+    )
+
+    result = testdir.runpytest()
+
+    # make sure that that we get a '0' exit code for the testsuite
+    assert result.ret == 0
+    assert os.path.isfile(log_file)
+    with open(log_file, encoding="utf-8") as rfh:
+        contents = rfh.read()
+        assert "Normal message" in contents
+        assert "\u251c" in contents
+        assert "Another normal message" in contents
+
+
 @pytest.mark.parametrize("has_capture_manager", [True, False])
 def test_live_logging_suspends_capture(has_capture_manager, request):
     """Test that capture manager is suspended when we emitting messages for live logging.


Code review comments:
diff_hunk: @@ -827,6 +828,40 @@ def test_log_file(request):
         assert "This log message won't be shown" not in contents
 
 
+def test_log_file_unicode(testdir):
+    log_file = testdir.tmpdir.join("pytest.log").strpath
+
+    testdir.makeini(
+        """
+        [pytest]
+        log_file={}
+        log_file_level = INFO
+        """.format(
+            log_file
+        )
+    )
+    testdir.makepyfile(
+        """
+        import logging
+        def test_log_file():
+            logging.getLogger('catchlog').info("Normal message")
+            logging.getLogger('catchlog').info("\u251c")
+            logging.getLogger('catchlog').info("Another normal message")
+    """
+    )
+
+    result = testdir.runpytest()
+
+    # make sure that that we get a '0' exit code for the testsuite
+    assert result.ret == 0
+    assert os.path.isfile(log_file)
+    with open(log_file, encoding="utf-8") as rfh:
+        contents = rfh.read()
+        assert "Normal message" in contents
+        assert "\u251c" in contents
path: testing/logging/test_reporting.py
text: @user1:
this is a little misleading, in python2 this will be a literal `"\u251c"` -- in python3 it'll be the character `├`

I suspect this test didn't actually fail on python2 before your patch -- mind changing the test to write unicode in both python2 and python3?

@author:
Thanks, you were right to the point. I updated the test to solve this issue.