Repository: spyder-ide/spyder

Issue:
Different output for internal and external console on Windows
## Description of your problem

Depending on if i run my code,  which uses subprocess to execute Windows commandline code, in the internal console ("Execute in new dedicated Python console") or the external console ("Execute  in an external System terminal") i get different output and exception bevaior, if none ascii characters are contained.
This is due to different encodings of the internal console (retrived by `locale.getdefaultlocale('LANG')` in `spyderlib/widgets/externalshell/sitecustomize.py lines 149-165`) and the system console (run the `chcp` command in cmd).
Also since the values of `ctypes.windll.kernel32.SetConsoleCP(_cp)` and `ctypes.windll.kernel32.SetConsoleOutputCP(_cp)` are changed there is no way (or at least none i found), to determine the right encoding.
## What steps will reproduce the problem?

To reproduce the the problem you can execute the following code.
Note that `cp850` is the codepage of my windows and may differ for you same as the printability of the `test_string`.
To reproduce the error u may want to change `test_string` with non ascii character of your language aswell as `cp1252` for the output of `locale.getpreferredencoding()` and `cp850` for the codepage given by running the `chcp` command in cmd.

```
# -*- coding: utf-8 -*-
import subprocess, ctypes, locale


def get_cmd_encoding():
    return_string = ""
    test_string = r"äöüÄÖÜß"
    arglist = ("echo "+test_string).split(" ")
    output = subprocess.Popen(arglist, shell=True, stdout=subprocess.PIPE,
                              stdin=subprocess.PIPE).communicate()[0]
    endocing_list = ["utf-8", "cp1252", "cp850"]
    for encoding in endocing_list:
        print(encoding)
        try:
            return_string = output.decode(encoding).strip()
            print(return_string)
        except:
            print("exception happened")
            return_string = ""

ctypes_cdll = ctypes.cdll.kernel32
ctypes_windll = ctypes.windll.kernel32


get_cmd_encoding()
print("\nretrived ecodings:\n")
print("locale.getdefaultlocale() \t\t: ", locale.getdefaultlocale())
print("locale.getlocale() \t\t\t: ", locale.getlocale())
print("locale.getpreferredencoding() \t: ", locale.getpreferredencoding())
print("ctypes_cdll.GetConsoleCP() \t\t: ", ctypes_cdll.GetConsoleCP())
print("ctypes_cdll.GetConsoleOutputCP()\t: ", ctypes_cdll.GetConsoleOutputCP())
print("ctypes_windll.GetConsoleCP() \t\t: ", ctypes_windll.GetConsoleCP())
print("ctypes_windll.GetConsoleOutputCP()\t: ", ctypes_windll.GetConsoleOutputCP())
print("chcp command\t\t\t: ", subprocess.check_output("chcp", shell=True))
# just to keep the external System Terminal open
input()
```

**Expected output : System terminal**

```
utf-8
exception happened
cp1252
exception happened
cp850
äöüÄÖÜß

retrived ecodings:

locale.getdefaultlocale()       :  ('de_DE', 'cp1252')
locale.getlocale()          :  (None, None)
locale.getpreferredencoding()   :  cp1252
ctypes_cdll.GetConsoleCP()      :  850
ctypes_cdll.GetConsoleOutputCP()    :  850
ctypes_windll.GetConsoleCP()        :  850
ctypes_windll.GetConsoleOutputCP()  :  850
chcp command            :  b'Aktive Codepage: 850.\r\n'
```

**False output :  internal console**

```
utf-8
exception happened
cp1252
äöüÄÖÜß
cp850
õ÷³─Í▄▀

retrived ecodings:

locale.getdefaultlocale()       :  ('de_DE', 'cp1252')
locale.getlocale()          :  (None, None)
locale.getpreferredencoding()   :  cp1252
ctypes_cdll.GetConsoleCP()      :  1252
ctypes_cdll.GetConsoleOutputCP()    :  1252
ctypes_windll.GetConsoleCP()        :  1252
ctypes_windll.GetConsoleOutputCP()  :  1252
chcp command            :  b'Aktive Codepage: 1252.\r\n'
```
## Solution

Simply deleting  lines 149-165 `spyderlib/widgets/externalshell/sitecustomize.py` solved that problem for me. But i have no idea which problems may arrise of this "fix". I hope the person who wrote that part still knows why and knows how to test if this produes a problem (none found here yet).

**`spyderlib/widgets/externalshell/sitecustomize.py` lines 149-165**

```
#==============================================================================
# Setting console encoding (otherwise Python does not recognize encoding)
# for Windows platforms
#==============================================================================
if os.name == 'nt':
    try:
        import locale, ctypes
        _t, _cp = locale.getdefaultlocale('LANG')
        try:
            _cp = int(_cp[2:])
            ctypes.windll.kernel32.SetConsoleCP(_cp)
            ctypes.windll.kernel32.SetConsoleOutputCP(_cp)
        except (ValueError, TypeError):
            # Code page number in locale is not valid
            pass
    except ImportError:
        pass
```

Guess the problem with the encoding is that even the python sourcecode says that `locale.getdefaultlocale` is guessing the encoding.
## Versions and main components
- Spyder Version: 3.0.0b2
- Python Version: 3.5.1 64bit
- Operating system: Windows 7 64bit
## Dependencies

IPython >=3.0      :  4.1.1 (OK)
jedi >=0.8.1;<0.9.0:  0.9.0 (NOK)
matplotlib >=1.0   :  1.5.1 (OK)
numpy >=1.7        :  1.10.4 (OK)
pandas >=0.13.1    :  0.18.0 (OK)
pep8 >=0.6         :  1.7.0 (OK)
pyflakes >=0.6.0   :  1.1.0 (OK)
pygments >=1.6     :  2.1.1 (OK)
pylint >=0.25      :  1.5.4 (OK)
qtconsole >=4.0    :  4.1.1 (OK)
rope >=0.9.4       :  0.9.4-1 (OK)
sphinx >=0.6.6     :  1.3.1 (OK)
sympy >=0.7.3      :  1.0 (OK)
zmq >=13.0.0       :  15.2.0 (OK)



Discussion:
@PierreRaybaut, you were the one who added those lines. Could you give us some light here?


Resolving patch:
diff --git a/spyderlib/widgets/externalshell/sitecustomize.py b/spyderlib/widgets/externalshell/sitecustomize.py
--- a/spyderlib/widgets/externalshell/sitecustomize.py
+++ b/spyderlib/widgets/externalshell/sitecustomize.py
@@ -150,7 +150,7 @@ def write(self, text):
 # Setting console encoding (otherwise Python does not recognize encoding)
 # for Windows platforms
 #==============================================================================
-if os.name == 'nt':
+if os.name == 'nt' and PY2:
     try:
         import locale, ctypes
         _t, _cp = locale.getdefaultlocale('LANG')


Test patch: