Repository: SCons/scons

Issue:
Encoding issue when running Configure checks on German windows 10
# Encoding issue on german windows 10 when running configure

I discussed the issue with @bdbaddog on the #scons-help channel on discord and he asked me to add a bug report here.

## Table of Contents

* [Problem statement](#problem)
* [Environment](#environment)
* [Analysis](#analysis)
* [Workaround](#workaround)
* [Conclusion](#conclusion)
* [Reproducing the bug](#reproducing)

<a name="problem"></a>
## Problem statement

Consider the following SConstruct

~~~~python
env = DefaultEnvironment(MSVC_VERSION="14.1", TARGET_ARCH="x86")
config_context = Configure(env)
config_context.CheckFunc('filelength')
~~~~

when this is run the `filelength` function is not detected, although it is clearly part of the Visual-C++ runtime library.

<a name="environment"></a>
## Environment

~~~~text
F:\devel\scons-bug>ver
Microsoft Windows [Version 10.0.17763.914]

F:\devel\scons-bug>python --version
Python 3.7.6

F:\devel\scons-bug>scons --version
SCons version SCons by Steven Knight et al.:
        script: v3.1.2.bee7caf9defd6e108fc2998a2520ddb36a967691, 2019-12-17 02:07:09, by bdeegan on octodog
        engine: v3.1.2.bee7caf9defd6e108fc2998a2520ddb36a967691, 2019-12-17 02:07:09, by bdeegan on octodog
        engine path: ['f:\\devel\\scons-bug\\lib\\site-packages\\scons\\SCons']
Copyright (c) 2001 - 2019 The SCons Foundation
~~~~

I tried two different editions of Visual Studio: Visual Studio Express 2017 
and Visual Studio 2019

<a name="analysis"></a>
## Analysis

When looking at the generated config.log file the reason becomes apparent:

~~~~text
cl /Fo.sconf_temp\conftest_0.obj /c .sconf_temp\conftest_0.c /nologo
scons: Configure: Caught exception while building ".sconf_temp\conftest_0.obj":

Traceback (most recent call last):
  File "f:\devel\scons-bug\lib\site-packages\scons\SCons\Job.py", line 201, in start
    task.execute()
  File "f:\devel\scons-bug\lib\site-packages\scons\SCons\SConf.py", line 356, in execute
    raise e
  File "f:\devel\scons-bug\lib\site-packages\scons\SCons\SConf.py", line 330, in execute
    self.targets[0].build()
  File "f:\devel\scons-bug\lib\site-packages\scons\SCons\Node\__init__.py", line 766, in build
    self.get_executor()(self, **kw)
  File "f:\devel\scons-bug\lib\site-packages\scons\SCons\Executor.py", line 397, in __call__
    return _do_execute_map[self._do_execute](self, target, kw)
  File "f:\devel\scons-bug\lib\site-packages\scons\SCons\Executor.py", line 128, in execute_action_list
    status = act(*args, **kw)
  File "f:\devel\scons-bug\lib\site-packages\scons\SCons\Action.py", line 1053, in __call__
    show, execute, chdir, executor)
  File "f:\devel\scons-bug\lib\site-packages\scons\SCons\Action.py", line 1120, in __call__
    return c.__call__(self, target, source, env, *args, **kw)
  File "f:\devel\scons-bug\lib\site-packages\scons\SCons\Action.py", line 708, in __call__
    stat = self.execute(target, source, env, executor=executor)
  File "f:\devel\scons-bug\lib\site-packages\scons\SCons\Action.py", line 952, in execute
    result = spawn(shell, escape, cmd_line[0], cmd_line, ENV)
  File "f:\devel\scons-bug\lib\site-packages\scons\SCons\SConf.py", line 574, in pspawn_wrapper
    return self.pspawn(sh, escape, cmd, args, env, self.logstream, self.logstream)
  File "f:\devel\scons-bug\lib\site-packages\scons\SCons\Platform\win32.py", line 223, in piped_spawn
    stdout.write(tmp.read())
  File "C:\Users\Ganssauge\AppData\Local\Programs\Python\Python37-32\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 110: character maps to <undefined>
scons: Configure: no
~~~~

<a name="workaround"></a>
## Workaround

As a workaround I applied the following patch:

~~~~patch
diff -ru SCons/Platform/win32.py new/SCons/Platform/win32.py
--- SCons/Platform/win32.py     2020-01-07 19:47:25.753801822 +0100
+++ new/SCons/Platform/win32.py        2020-01-07 10:45:38.000000000 +0100
@@ -219,7 +219,7 @@
         # and do clean up stuff
         if stdout is not None and stdoutRedirected == 0:
             try:
-                with open(tmpFileStdout, "r" ) as tmp:
+                with open(tmpFileStdout, "r", encoding='cp437') as tmp:
                     stdout.write(tmp.read())
                 os.remove(tmpFileStdout)
             except (IOError, OSError):
@@ -227,7 +227,7 @@

         if stderr is not None and stderrRedirected == 0:
             try:
-                with open(tmpFileStderr, "r" ) as tmp:
+                with open(tmpFileStderr, "r", encoding='cp437') as tmp:
                     stderr.write(tmp.read())
                 os.remove(tmpFileStderr)
             except (IOError, OSError):
~~~~

<a name="conclusion"></a>
## Conclusion

Another run with the workaround applied yields the following:

~~~~text
Checking for C function filelength()... yes
scons: done reading SConscript files.
scons: Building targets ...
scons: `.' is up to date.
scons: done building targets.
file F:\devel\scons-bug\SConstruct,line 2:
        Configure(confdir = .sconf_temp)
scons: Configure: Checking for C function filelength()...
.sconf_temp\conftest_0.c <-
  |
  |
  |#include <assert.h>
  |
  |#ifdef __cplusplus
  |extern "C"
  |#endif
  |char filelength();
  |
  |#if _MSC_VER && !__INTEL_COMPILER
  |    #pragma function(filelength)
  |#endif
  |
  |int main(void) {
  |#if defined (__stub_filelength) || defined (__stub___filelength)
  |  fail fail fail
  |#else
  |  filelength();
  |#endif
  |
  |  return 0;
  |}
  |
cl /Fo.sconf_temp\conftest_0.obj /c .sconf_temp\conftest_0.c /nologo
conftest_0.c
.sconf_temp\conftest_0.c(11): warning C4163: "filelength": Nicht als systeminterne Funktion verfügbar
link /nologo /OUT:.sconf_temp\conftest_0.exe .sconf_temp\conftest_0.obj
scons: Configure: yes
~~~~

The character **ü** above is actually the character with code 0x81 mentioned in the stacktrace of the first run.

`filelength` not being a function doesn't make it unusable, as the now working check proves.

<a name="reproducing"></a>
## Reproducing the bug

In order to reproduce the bug you need a computer (or a VM) running a localized version of Windows 10 with Visual Studio and Python 3 installed.

The attached zip archive contains a windows batch script `bug.cmd`, the `SConstruct` file, a patched version of `SCons\Platform\win32.py` and this bug report.

As a convenience I added the output of the warning to the file `warning.txt` in original cp850 encoding.

After unpacking the archive into an empty directory, simply run `bug.cmd`.
It creates a virtualenv with the latest scons installed, runs scons to demonstrate the problem, copies a patched version of `SCons.Platform.win32` to the scons directory and runs scons again in order to demonstrate that now the bug is gone.



Discussion:
Once again: zip file mentioned in the issue:
[scons-bug.zip](https://github.com/SCons/scons/files/4093108/scons-bug.zip)

Working on a variety of encoding issues and came across this one again.  Does this change sound reasonable?  It reads the temporary files as bytes, and tries to decode into whatever the codec for the two output streams are, defining an error behavior to avoid bailing on something that can't convert ("backslashreplace" is another option).

```py
diff --git a/SCons/Platform/win32.py b/SCons/Platform/win32.py
index b14582361..c2216eb68 100644
--- a/SCons/Platform/win32.py
+++ b/SCons/Platform/win32.py
@@ -165,16 +165,18 @@ def piped_spawn(sh, escape, cmd, args, env, stdout, stderr):
     # and do clean up stuff
     if stdout is not None and not stdoutRedirected:
         try:
-            with open(tmpFileStdoutName) as tmpFileStdout:
-                stdout.write(tmpFileStdout.read())
+            with open(tmpFileStdoutName, "rb") as tmpFileStdout:
+                output = tmpFileStdout.read()
+                stdout.write(output.decode(sys.stdout.encoding, "replace"))
             os.remove(tmpFileStdoutName)
         except OSError:
             pass
 
     if stderr is not None and not stderrRedirected:
         try:
-            with open(tmpFileStderrName) as tmpFileStderr:
-                stderr.write(tmpFileStderr.read())
+            with open(tmpFileStderrName, "rb") as tmpFileStderr:
+                errors = tmpFileStderr.read()
+                stdout.write(errors.decode(sys.stderr.encoding, "replace"))
             os.remove(tmpFileStderrName)
         except OSError:
             pass
```
@mwichmann looks good. does it work with the tarball they attached?
I don't have a German Windows, so hard to repro. I'll take a look.  Manually following the steps (the objectionable message stored in a file) looks like this. Notice since my Windows is set to a US codepage, *it* doesn't fail on the message, but it displays it wonky; the decode inside Python made it come out okay.
```
C:\Users\mats\Documents\github\scons\exp\encoded-response>type data
.sconf_temp\conftest_0.c(11): warning C4163: "filelength": Nicht als systeminterne Funktion verf├╝gbar

C:\Users\mats\Documents\github\scons\exp\encoded-response>py
Python 3.12.1 (tags/v3.12.1:2305ca5, Dec  7 2023, 22:03:25) [MSC v.1937 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> with open("data", "rb") as f:
...     data = f.read()
...
>>>
>>> sys.stdout.write(data.decode(sys.stdout.encoding, errors="convert"))
.sconf_temp\conftest_0.c(11): warning C4163: "filelength": Nicht als systeminterne Funktion verfügbar
102
>>>
```

Resolving patch:
diff --git a/SCons/Platform/win32.py b/SCons/Platform/win32.py
--- a/SCons/Platform/win32.py
+++ b/SCons/Platform/win32.py
@@ -165,16 +165,18 @@ def piped_spawn(sh, escape, cmd, args, env, stdout, stderr):
     # and do clean up stuff
     if stdout is not None and not stdoutRedirected:
         try:
-            with open(tmpFileStdoutName) as tmpFileStdout:
-                stdout.write(tmpFileStdout.read())
+            with open(tmpFileStdoutName, "rb") as tmpFileStdout:
+                output = tmpFileStdout.read()
+                stdout.write(output.decode(stdout.encoding, "replace"))
             os.remove(tmpFileStdoutName)
         except OSError:
             pass
 
     if stderr is not None and not stderrRedirected:
         try:
-            with open(tmpFileStderrName) as tmpFileStderr:
-                stderr.write(tmpFileStderr.read())
+            with open(tmpFileStderrName, "rb") as tmpFileStderr:
+                errors = tmpFileStderr.read()
+                stderr.write(errors.decode(stderr.encoding, "replace"))
             os.remove(tmpFileStderrName)
         except OSError:
             pass


Test patch: