Repository: Flexget/Flexget

Issue:
logging crash
Just got this error for some log output, its nothing serious, but just wanted to report it.

Python 3.5.2 + Flexget 2.8.1
```
--- Logging error ---
Traceback (most recent call last):
  File "c:\users\andy\appdata\local\programs\python\python35-32\lib\logging\__init__.py", line 982, in emit
    stream.write(msg)
  File "c:\users\andy\appdata\local\programs\python\python35-32\lib\site-packages\colorclass\windows.py", line 275, in write
    print(segment, file=self._original_stream, end='')
  File "c:\users\andy\appdata\local\programs\python\python35-32\lib\encodings\cp850.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_map)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u2022' in position 67: character maps to <undefined>
Call stack:
  File "c:\users\andy\appdata\local\programs\python\python35-32\lib\threading.py", line 882, in _bootstrap
    self._bootstrap_inner()
  File "c:\users\andy\appdata\local\programs\python\python35-32\lib\threading.py", line 914, in _bootstrap_inner
    self.run()
  File "c:\users\andy\appdata\local\programs\python\python35-32\lib\threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "c:\users\andy\appdata\local\programs\python\python35-32\lib\site-packages\flexget\task_queue.py", line 48, in run
    self.current_task.execute()
  File "c:\users\andy\appdata\local\programs\python\python35-32\lib\site-packages\flexget\task.py", line 72, in wrapper
    return func(self, *args, **kw)
  File "c:\users\andy\appdata\local\programs\python\python35-32\lib\site-packages\flexget\task.py", line 646, in execute
    self._execute()
  File "c:\users\andy\appdata\local\programs\python\python35-32\lib\site-packages\flexget\task.py", line 612, in _execute
    self.__run_task_phase(phase)
  File "c:\users\andy\appdata\local\programs\python\python35-32\lib\site-packages\flexget\task.py", line 453, in __run_task_phase
    response = self.__run_plugin(plugin, phase, args)
  File "c:\users\andy\appdata\local\programs\python\python35-32\lib\site-packages\flexget\task.py", line 483, in __run_plugin
    return method(*args, **kwargs)
  File "c:\users\andy\appdata\local\programs\python\python35-32\lib\site-packages\flexget\event.py", line 23, in __call__
    return self.func(*args, **kwargs)
  File "c:\users\andy\appdata\local\programs\python\python35-32\lib\site-packages\flexget\plugins\filter\quality.py", line 37, in on_task_filter
    entry.reject('%s does not match quality requirement %s' % (entry['quality'], reqs))
  File "c:\users\andy\appdata\local\programs\python\python35-32\lib\site-packages\flexget\entry.py", line 152, in reject
    self.run_hooks('reject', reason=reason, **kwargs)
  File "c:\users\andy\appdata\local\programs\python\python35-32\lib\site-packages\flexget\entry.py", line 80, in run_hooks
    func(self, **kwargs)
  File "c:\users\andy\appdata\local\programs\python\python35-32\lib\site-packages\flexget\plugins\operate\verbose.py", line 34, in verbose_details
    task_log.verbose(msg)
  File "c:\users\andy\appdata\local\programs\python\python35-32\lib\site-packages\flexget\logger.py", line 127, in verbose
    self.log(VERBOSE, msg, *args, **kwargs)
```


Discussion:
does this happen after a cli action?
This happens while trying to do some `log.verbose()` output. Maybe this is because of the way i do log output?
```python
log.verbose('some: %s, stuff: %s' % (stuff1, stuff2))
```
paranoidi noted this should become:
```python
log.verbose('some: %s, stuff: %s', stuff1, stuff2)
```
Still not sure what the exact difference is for the variants?
Is this from existing code, or a new plugin you are making? Are you sure stuff1 and stuff2 are text, and not bytes?
I think it was data from anidb, using the http-api which should return strings for text fields. If this should not happen, just close the bug and i will do a better reporting job next time and also safe the input data that caused the crash.
```logtalk
Traceback (most recent call last):
  File "c:\users\andy\appdata\local\programs\python\python35-32\lib\logging\__init__.py", line 982, in emit
    stream.write(msg)
  File "c:\users\andy\appdata\local\programs\python\python35-32\lib\site-packages\colorclass\windows.py", line 275, in write
    print(segment, file=self._original_stream, end='')
  File "c:\users\andy\appdata\local\programs\python\python35-32\lib\encodings\cp850.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_map)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u2022' in position 67: character maps to <undefined>
```
it looks like the cause of this exception has nothing to do with what you were testing, not sure how to replicate that...
I replicated this, quite easily:
```logtalk
--- Logging error ---
Traceback (most recent call last):
  File "C:\python35\lib\logging\__init__.py", line 982, in emit
    stream.write(msg)
  File "C:\python35\lib\encodings\cp1255.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\xe9' in position 45: character maps to <undefined>
Call stack:
  File "C:\python35\lib\threading.py", line 882, in _bootstrap
    self._bootstrap_inner()
  File "C:\python35\lib\threading.py", line 914, in _bootstrap_inner
    self.run()
  File "C:\python35\lib\threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Flexget\flexget\task_queue.py", line 48, in run
    self.current_task.execute()
  File "C:\Flexget\flexget\task.py", line 72, in wrapper
    return func(self, *args, **kw)
  File "C:\Flexget\flexget\task.py", line 646, in execute
    self._execute()
  File "C:\Flexget\flexget\task.py", line 576, in _execute
    log.debug('executing %s' % self.name)
Message: 'executing Aranyélet_test'
```
while trying to replicate https://github.com/Flexget/Flexget/issues/1553 I got a bunch of those:
http://pastebin.com/h72bbE5D
I'm guessing this is because the encoding of the terminal doesn't support the character. I think somehow we should make our console log output just replace that character with a ? or something when the users terminal can't handle printing the character.
That's what we do with `console()` isn't it? 
Yeah. I'm thinking actually maybe we could do something on these lines:
```
old_stdout = sys.stdout
sys.stdout = io.TextIOWrapper(old_stdout.buffer, encoding=old_stdout.encoding, errors='backslashreplace')
```
Which should solve the issue for both logging and regular prints. I'm not sure if that's the best way to change the 'errors' argument on sys.stdout, it didn't let me do `sys.stdout.errors = 'replace'` directly.
Where will that go, root of `terminal.py`? 

Maybe make a function in terminal.py to set it up (and put the windows color code thing in there too) then call it in `__init__.py` or maybe in manager init?
I added this to `terminal.py`:
```python
def terminal_stuff():
    # Enable terminal colors on windows.
    # pythonw (flexget-headless) does not have a sys.stdout, this command would crash in that case
    if sys.platform == 'win32' and sys.stdout:
        Windows.enable(auto_colors=True)

    if sys.stdout:
        old_stdout = sys.stdout
        sys.stdout = io.TextIOWrapper(old_stdout.buffer, encoding=old_stdout.encoding, errors='backslashreplace')
```
and called it via manager `initialize`. Still get the same error
@liiight Does it happen before our logging gets set up? We need to do it before the logging handler grab sys.stdout
Maybe we should just fix it in our logging handler rather than trying to fix all of sys.stdout

I think i added this after our logging actually, but isn't this fix supposedly handle console output as well? That's the bigger issue here imo, you get reports of unicode errors constantly with different cli actions. 
Our console function is already doing a fix like this though? It's just the logging that doesn't have it right?
Hmm, seems all the cli issues with unicode happen when piping terminal table stuff. So the answer is yes I guess, logger only 

Resolving patch:
diff --git a/flexget/logger.py b/flexget/logger.py
--- a/flexget/logger.py
+++ b/flexget/logger.py
@@ -1,6 +1,7 @@
 from __future__ import unicode_literals, division, absolute_import, print_function
 from builtins import *  # noqa pylint: disable=unused-import, redefined-builtin
 
+import codecs
 import collections
 import contextlib
 import logging
@@ -11,6 +12,7 @@
 import warnings
 
 from flexget import __version__
+from flexget.utils.tools import io_encoding
 
 # A level more detailed than DEBUG
 TRACE = 5
@@ -203,7 +205,13 @@ def start(filename=None, level=logging.INFO, to_console=True, to_file=True):
 
     # without --cron we log to console
     if to_console:
-        console_handler = logging.StreamHandler(sys.stdout)
+        # Make sure we don't send any characters that the current terminal doesn't support printing
+        stdout = sys.stdout
+        if hasattr(stdout, 'buffer'):
+            # On python 3, we need to get the buffer directly to support writing bytes
+            stdout = stdout.buffer
+        safe_stdout = codecs.getwriter(io_encoding)(stdout, 'replace')
+        console_handler = logging.StreamHandler(safe_stdout)
         console_handler.setFormatter(formatter)
         console_handler.setLevel(level)
         logger.addHandler(console_handler)


Test patch: