Repository: numpy/numpy

Issue:
Build Failure on Windows, Python 3.6 GCC Compilers: UnicodeDecodeError.
updating to the latest master , an `UnicodeDecodeError `occurred even after execute `git clean -xfd`
### Reproducing code example:

No code example.

### Error message:
```
CCompilerOpt.cc_test_flags[989] : testing flags (/WX)
CCompilerOpt.__init__[1137] : feature 'AVX512_KNL' is disabled, MSVC compiler doesn't support it
CCompilerOpt.__init__[1137] : feature 'AVX512_KNM' is disabled, MSVC compiler doesn't support it
CCompilerOpt.__init__[1621] : check requested baseline
CCompilerOpt.cc_test_flags[989] : testing flags (/arch:SSE)
Traceback (most recent call last):
  File "setup.py", line 499, in <module>
    setup_package()
  File "setup.py", line 491, in setup_package
    setup(**metadata)
  File "D:\Source\numpy\numpy\distutils\core.py", line 169, in setup
    return old_setup(**new_attr)
  File "C:\Python36\lib\site-packages\setuptools\__init__.py", line 143, in setup
    return distutils.core.setup(**attrs)
  File "C:\Python36\lib\distutils\core.py", line 148, in setup
    dist.run_commands()
  File "C:\Python36\lib\distutils\dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "C:\Python36\lib\distutils\dist.py", line 974, in run_command
    cmd_obj.run()
  File "D:\Source\numpy\numpy\distutils\command\build_ext.py", line 111, in run
    build_clib.run()
  File "D:\Source\numpy\numpy\distutils\command\build_clib.py", line 126, in run
    cpu_dispatch=self.cpu_dispatch, cache_path=opt_cache_path
  File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 2451, in new_ccompiler_opt
    opt = CCompilerOpt(compiler, **kwargs)
  File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 2055, in __init__
    _Parse.__init__(self, cpu_baseline, cpu_dispatch)
  File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 1624, in __init__
    baseline_names = self.feature_names(cpu_baseline)
  File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 1182, in feature_names
    if self.feature_is_supported(f, force_flags=force_flags):
  File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 829, in cache_wrap_me
    ccb = cb(self, *args, **kwargs)
  File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 1471, in feature_is_supported
    if not self.feature_test(impl, force_flags):
  File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 829, in cache_wrap_me
    ccb = cb(self, *args, **kwargs)
  File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 1432, in feature_test
    force_flags = self.feature_flags(name)
  File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 829, in cache_wrap_me
    ccb = cb(self, *args, **kwargs)
  File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 1411, in feature_flags
    if not f or not self.cc_test_flags(f):
  File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 829, in cache_wrap_me
    ccb = cb(self, *args, **kwargs)
  File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 991, in cc_test_flags
    test = self.dist_test(test_path, flags)
  File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 556, in dist_test
    [source], flags, output_dir=self.conf_tmp_path
  File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 536, in dist_compile
    sources, extra_postargs=flags, **kwargs
  File "C:\Python36\lib\distutils\_msvccompiler.py", line 423, in compile
    self.spawn(args)
  File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 668, in _dist_test_spawn_paths
    self._dist_test_spawn(cmd)
  File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 686, in _dist_test_spawn
    o = o.decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 4: invalid continuation byte
CCompilerOpt._cache_write[786] : write cache to path -> D:\Source\numpy\build\temp.win-amd64-3.6\ccompiler_opt_cache_clib.py
```



Discussion:
Do you have non-ascii characters in your path? Could you stop at the error and try to work out what is going on?
does `o` have an encoding property `o.encoding` ?
I'm sure that no non-ascii characters existed, It's seems like the exception is triggered when testing SSE flags:
```
CCompilerOpt.cc_test_flags[991] : testing flags (/arch:SSE)
decode:b'cl: \xc3\xfc\xc1\xee\xd0\xd0 warning D9002 :\xba\xf6\xc2\xd4\xce\xb4\xd6\xaa\xd1\xa1\xcf\xee\xa1\xb0/arch:SSE\xa1\xb1\r\ntest_flags.c\r\n'
```
o.decode() failed to decode
```
b'cl: \xc3\xfc\xc1\xee\xd0\xd0 warning D9002 :\xba\xf6\xc2\xd4\xce\xb4\xd6\xaa\xd1\xa1\xcf\xee\xa1\xb0/arch:SSE\xa1\xb1\r\ntest_flags.c\r\n'
```
@Qiyu8, your compiler output is encoded in [`GB2312`](https://en.wikipedia.org/wiki/GB_2312) rather than utf8 it seems. The string decodes to
```
cl: 命令行 warning D9002 :忽略未知选项“/arch:SSE”
test_flags.c
```
https://bugs.python.org/issue27179#msg267091 seems slightly related, I haven't read the whole thread.
Maybe a parallel problem, I always build numpy with `python setup.py build_ext -i -j 8`,  turned to `python setup.py build_ext -i` and the build successfully. 
My guess would be that the codepage of the parallel processes does not match the codepage of the parent process.
This problem arose again in my local machine, even with non-parallel build.
There was a [long discussion](https://discuss.python.org/t/pep-597-use-utf-8-for-default-text-file-encoding/1819) and a PEP 597 about changing the default stdout encoding. Bottom line: what does
`locale.getpreferredencoding()` show for you? If you modify the offending line
```
  File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 686, in _dist_test_spawn
    o = o.decode()
```

to use that encoding does it fix the problem?
my encoding result is `cp936`, The build can succeed after modifed to `o.decode(encoding="cp936")`, But there are large amount of `warning D9002 :忽略未知选项“/arch:SSE” ` produced during build.
@mattip I want to provide a PR to solve this issue, modify `decode()` to `decode(locale.getpreferredencoding())`, what do you think?
> modify `decode()` to `decode(locale.getpreferredencoding())`

I think the preferred spelling of this is when calling `check_output` to specify `universal_newlines=True` which will force the output to be text (already encoded). Search for "unversal_newlines" on the [subprocess doc page](https://docs.python.org/3.6/library/subprocess.html#frequently-used-arguments). Could you try it (since we don't really test for non-ascii windows versions)?

As for the multiple warnings, can you show the part of the log that looks like the one in [this comment](https://github.com/numpy/numpy/pull/13516#issuecomment-655579534) ?
The build can succeed after specify `universal_newlines=True`, see the build log of multiple warnings:
```
cl: 命令行 warning D9002 :忽略未知选项“/arch:SSE2”
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -DHAVE_BLAS_ILP64 -DBLAS_SYMBOL_SUFFIX=64_ -Inumpy\core\include -Inumpy\core\include/numpy -Inumpy\core\src\common -Inumpy\core\src -Inumpy\core -Inumpy\core\src\npymath -Inumpy\core\src\multiarray -Inumpy\core\src\umath -Inumpy\core\src\npysort -IC:\Python36\include -IC:\Python36\include -Inumpy\core\src\common -Inumpy\core\src\npymath -IC:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\ATLMFC\include -IC:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\include -IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\shared -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\winrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\cppwinrt /Tcnumpy\linalg\lapack_lite\f2c_s_lapack.c /Fobuild\temp.win-amd64-3.6\Release\numpy\linalg\lapack_lite\f2c_s_lapack.obj /arch:SSE2
cl: 命令行 warning D9002 :忽略未知选项“/arch:SSE2”
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -DHAVE_BLAS_ILP64 -DBLAS_SYMBOL_SUFFIX=64_ -Inumpy\core\include -Inumpy\core\include/numpy -Inumpy\core\src\common -Inumpy\core\src -Inumpy\core -Inumpy\core\src\npymath -Inumpy\core\src\multiarray -Inumpy\core\src\umath -Inumpy\core\src\npysort -IC:\Python36\include -IC:\Python36\include -Inumpy\core\src\common -Inumpy\core\src\npymath -IC:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\ATLMFC\include -IC:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\include -IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\shared -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\winrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\cppwinrt /Tcnumpy\linalg\lapack_lite\f2c_lapack.c /Fobuild\temp.win-amd64-3.6\Release\numpy\linalg\lapack_lite\f2c_lapack.obj /arch:SSE2
cl: 命令行 warning D9002 :忽略未知选项“/arch:SSE2”
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -DHAVE_BLAS_ILP64 -DBLAS_SYMBOL_SUFFIX=64_ -Inumpy\core\include -Inumpy\core\include/numpy -Inumpy\core\src\common -Inumpy\core\src -Inumpy\core -Inumpy\core\src\npymath -Inumpy\core\src\multiarray -Inumpy\core\src\umath -Inumpy\core\src\npysort -IC:\Python36\include -IC:\Python36\include -Inumpy\core\src\common -Inumpy\core\src\npymath -IC:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\ATLMFC\include -IC:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\include -IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\shared -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\winrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\cppwinrt /Tcnumpy\linalg\lapack_lite\f2c_blas.c /Fobuild\temp.win-amd64-3.6\Release\numpy\linalg\lapack_lite\f2c_blas.obj /arch:SSE2
cl: 命令行 warning D9002 :忽略未知选项“/arch:SSE2”
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -DHAVE_BLAS_ILP64 -DBLAS_SYMBOL_SUFFIX=64_ -Inumpy\core\include -Inumpy\core\include/numpy -Inumpy\core\src\common -Inumpy\core\src -Inumpy\core -Inumpy\core\src\npymath -Inumpy\core\src\multiarray -Inumpy\core\src\umath -Inumpy\core\src\npysort -IC:\Python36\include -IC:\Python36\include -Inumpy\core\src\common -Inumpy\core\src\npymath -IC:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\ATLMFC\include -IC:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\include -IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\shared -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\winrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\cppwinrt /Tcnumpy\linalg\lapack_lite\f2c_config.c /Fobuild\temp.win-amd64-3.6\Release\numpy\linalg\lapack_lite\f2c_config.obj /arch:SSE2
cl: 命令行 warning D9002 :忽略未知选项“/arch:SSE2”
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -DHAVE_BLAS_ILP64 -DBLAS_SYMBOL_SUFFIX=64_ -Inumpy\core\include -Inumpy\core\include/numpy -Inumpy\core\src\common -Inumpy\core\src -Inumpy\core -Inumpy\core\src\npymath -Inumpy\core\src\multiarray -Inumpy\core\src\umath -Inumpy\core\src\npysort -IC:\Python36\include -IC:\Python36\include -Inumpy\core\src\common -Inumpy\core\src\npymath -IC:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\ATLMFC\include -IC:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\include -IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\shared -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\winrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\cppwinrt /Tcnumpy\linalg\lapack_lite\f2c.c /Fobuild\temp.win-amd64-3.6\Release\numpy\linalg\lapack_lite\f2c.obj /arch:SSE2
cl: 命令行 warning D9002 :忽略未知选项“/arch:SSE2”
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\Python36\libs /LIBPATH:C:\Python36\PCbuild\amd64 /LIBPATH:build\temp.win-amd64-3.6 /LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\ATLMFC\lib\x64 /LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\lib\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\lib\um\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.17763.0\ucrt\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.17763.0\um\x64 npymath.lib /EXPORT:PyInit__umath_linalg build\temp.win-amd64-3.6\Release\numpy\linalg\umath_linalg.obj build\temp.win-amd64-3.6\Release\numpy\linalg\lapack_lite\python_xerbla.obj build\temp.win-amd64-3.6\Release\numpy\linalg\lapack_lite\f2c_z_lapack.obj build\temp.win-amd64-3.6\Release\numpy\linalg\lapack_lite\f2c_c_lapack.obj build\temp.win-amd64-3.6\Release\numpy\linalg\lapack_lite\f2c_d_lapack.obj build\temp.win-amd64-3.6\Release\numpy\linalg\lapack_lite\f2c_s_lapack.obj 
build\temp.win-amd64-3.6\Release\numpy\linalg\lapack_lite\f2c_lapack.obj build\temp.win-amd64-3.6\Release\numpy\linalg\lapack_lite\f2c_blas.obj build\temp.win-amd64-3.6\Release\numpy\linalg\lapack_lite\f2c_config.obj build\temp.win-amd64-3.6\Release\numpy\linalg\lapack_lite\f2c.obj /OUT:numpy\linalg\_umath_linalg.cp36-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.6\Release\numpy\linalg\_umath_linalg.cp36-win_amd64.lib
```
Good to hear: so the correct fix for this issue is `universal_newlines=True`

As for the warnings: on 64-bit windows, you should not be using `/arch:SSE2`, since it is on by default. This is a separate issue, @seiko2plus might be able to help.
Note that `universal_newlines=True` is equivalent to decoding with `locale.getpreferredencoding(False)` according to https://docs.python.org/3/library/io.html#io.TextIOWrapper.

I still suspect there's a nastier issue here regarding the locale in which parallel vs non-parallel builds are executed in.
> @mattip, This is a separate issue 

yes, my bad. I didn't expect msvc to raise warning messages in another language than English.

@Qiyu8, well to solve this issue you have to replace:
https://github.com/numpy/numpy/blob/86fcce61c9cddbc354ba67b2f818670f5822c688/numpy/distutils/ccompiler_opt.py#L676
with:
```Python
"warning D9002|"  # msvc, it should be work with any language. 
```
And why?
Because microsoft and Intel compiler tending to raise warning messages rather than fatal errros for unkown or unsupported options.




Resolving patch:
diff --git a/numpy/distutils/ccompiler_opt.py b/numpy/distutils/ccompiler_opt.py
--- a/numpy/distutils/ccompiler_opt.py
+++ b/numpy/distutils/ccompiler_opt.py
@@ -673,7 +673,7 @@ def _dist_test_spawn_paths(self, cmd, display=None):
         # intel and msvc compilers don't raise
         # fatal errors when flags are wrong or unsupported
         ".*("
-        "ignoring unknown option|" # msvc
+        "warning D9002|"  # msvc, it should be work with any language. 
         "invalid argument for option" # intel
         ").*"
     )
@@ -681,9 +681,8 @@ def _dist_test_spawn_paths(self, cmd, display=None):
     def _dist_test_spawn(cmd, display=None):
         from distutils.errors import CompileError
         try:
-            o = subprocess.check_output(cmd, stderr=subprocess.STDOUT)
-            if isinstance(o, bytes):
-                o = o.decode()
+            o = subprocess.check_output(cmd, stderr=subprocess.STDOUT,
+                                        universal_newlines=True)
             if o and re.match(_Distutils._dist_warn_regex, o):
                 _Distutils.dist_error(
                     "Flags in command", cmd ,"aren't supported by the compiler"
@@ -697,7 +696,6 @@ def _dist_test_spawn(cmd, display=None):
             s = 127
         else:
             return None
-        o = o.decode()
         _Distutils.dist_error(
             "Command", cmd, "failed with exit status %d output -> \n%s" % (
             s, o


Test patch: