Repository: mesonbuild/meson

Issue:
Wrong cpu  detection for 'x86_64' windows
In ‘x64 Native Tools Command Prompt for VS 2017’, the cpu detected by meson  is 'x86' ？

``` shell
**********************************************************************
** Visual Studio 2017 Developer Command Prompt v15.9.16
** Copyright (c) 2017 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'
```

```shell
D:\opt\glib-2.62.2\_build>meson .. --buildtype=release --prefix="D:\opt\glib\"
The Meson build system
Version: 0.52.0
Source dir: D:\opt\glib-2.62.2
Build dir: D:\opt\glib-2.62.2\_build
Build type: native build
Project name: glib
Project version: 2.62.2
C compiler for the host machine: cl (msvc 19.16.27034)
C linker for the host machine: link 19.16.27034
C++ compiler for the host machine: cl (msvc 19.16.27034)
C++ linker for the host machine: link 19.16.27034
Host machine cpu family: x86
Host machine cpu: x86
```

And I tried in vs2013 , 2015, got the same result.



Discussion:
I don't see this with VS2019--I didn't try with VS2017.

```
The Meson build system
Version: 0.52.0
Source dir: C:\meson\test cases\common\1 trivial
Build dir: c:\temp\11
Build type: native build
Project name: trivial test
Project version: undefined
C compiler for the host machine: cl (msvc 19.23.28106.4)
C linker for the host machine: link 19.23.28106.4
Host machine cpu family: x86_64
Host machine cpu: x86_64
``
Is there any way to overwrite host_machine?
The way I know of to control host_machine is by [cross compiling](https://mesonbuild.com/Cross-compilation.html). However that shouldn't be necessary here.

Is the correct CPU family detected from Powershell or regular windows command prompt instead of Visual Studio prompt? 

Does anyone else reading this see the same issue from VS2017 or older?
glib CI uses VS2017 and it works there, but it's also using meson 0.49.2: https://gitlab.gnome.org/GNOME/glib/-/jobs/489274
hmm, is this related to #6134...
If you want to debug this, adding choice print statements in `environment.py` function `detect_cpu` and `detect_cpu_family` is fairly simple.

Also note that this may be related to the Python you are using. If you have a 32 bit Python on a 64 bit machine, it _might_ misdetect the CPU in use. Something to keep in mind.
Looking inside environment.py, and using VS2019 command prompt on 64-bit Windows with Python 3.8 32-bit, this issue could happen if environment variable `PROCESSOR_ARCHITEW6432` is not defined

@sz-CHEN  what do you get from your Visual Studio command prompt for
```posh
echo %PROCESSOR_ARCHITEW6432%
```
More specifically, here's what one gets on a 64-bit Windows system using Python 64-bit vs. 32-bit. 
It does not matter if using Visual Studio command prompt or Powershell or whatever:

```python
# 64-bit Python
>>> os.environ['PROCESSOR_ARCHITECTURE']
'AMD64'

# 32-bit Python
>>> os.environ['PROCESSOR_ARCHITECTURE']
'x86'
```
@sz-CHEN  you can determine if you're using 32-bit or 64-bit Python by:

```sh
python -c "import sys; print(sys.maxsize > 2**32)"
```

* 32-bit Python will print `False`
* 64-bit Python will print `True`

[reference](https://docs.python.org/3/library/platform.html#platform.architecture)
The above note may be the key to fix this issue. We are now waiting for feedback from @sz-CHEN to run these checks above.
@scivision Thanks for your help, the following is what I have tried.
```shell
**********************************************************************
** Visual Studio 2017 Developer Command Prompt v15.9.16
** Copyright (c) 2017 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'

C:\Program Files (x86)\Microsoft Visual Studio\2017\Community>D:

D:\>cd opt\glib-2.62.2

D:\opt\glib-2.62.2>python -c "import sys; print(sys.maxsize > 2**32)"
True

D:\opt\glib-2.62.2>python
Python 3.8.0 (tags/v3.8.0:fa919fd, Oct 14 2019, 19:37:50) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> os.environ['PROCESSOR_ARCHITECTURE']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'os' is not defined
>>> import os
>>> os.environ['PROCESSOR_ARCHITECTURE']
'AMD64'
>>> quit()

D:\opt\glib-2.62.2>echo %PROCESSOR_ARCHITEW6432%
%PROCESSOR_ARCHITEW6432%

D:\opt\glib-2.62.2>echo %PROCESSOR_ARCHITECTURE%
AMD64
```
It is 64bit python, but  the variable %PROCESSOR_ARCHITEW6432% was not defined but % PROCESSOR_ARCHITECTURE% is defined as 'AMD64'. 
And I tried set  %PROCESSOR_ARCHITEW6432% as 'AMD64', but got the same result.

Then, I modified the 'environment.py', add `print("compiler.target:"+compiler.target)` in function `detect_windows_arch`, and found that: ***compiler.target:x86***, and the compiler is defined from `def detect_machine_info(compilers: typing.Optional[CompilersDict] = None) -> MachineInfo:` called as ` machines.build = detect_machine_info()`
```shell
C compiler for the host machine: cl (msvc 19.16.27034)
C linker for the host machine: link 19.16.27034
C++ compiler for the host machine: cl (msvc 19.16.27034)
C++ linker for the host machine: link 19.16.27034
compiler.target:x86
compiler.target:x86
Host machine cpu family: x86
Host machine cpu: x86
```
I tried print the compilers in different function.
```shell
OrderedDict([('c', <VisualStudioCCompiler: v19.16.27034 `cl`>), ('cpp', <VisualStudioCPPCompiler: v19.16.27034 `cl`>)])
detect_cpu_family
OrderedDict([('c', <VisualStudioCCompiler: v19.16.27034 `cl`>), ('cpp', <VisualStudioCPPCompiler: v19.16.27034 `cl`>)])
detect_windows_arch
OrderedDict([('c', <VisualStudioCCompiler: v19.16.27034 `cl`>), ('cpp', <VisualStudioCPPCompiler: v19.16.27034 `cl`>)])
arch: amd64
compiler.target:x86
detect_cpu
OrderedDict([('c', <VisualStudioCCompiler: v19.16.27034 `cl`>), ('cpp', <VisualStudioCPPCompiler: v19.16.27034 `cl`>)])
detect_windows_arch
OrderedDict([('c', <VisualStudioCCompiler: v19.16.27034 `cl`>), ('cpp', <VisualStudioCPPCompiler: v19.16.27034 `cl`>)])
arch: amd64
compiler.target:x86
```
To summarize sz-CHEN's findings:

* using 64-bit Python
* Visual Studio 2017 19.16
* `PROCESSOR_ARCHITEW6432` was not defined, but even defining it as `AMD64` does not help

I think it may be helpful to trace line by line to see which exact line makes this x86 decision. Does any dev have vs2017 to try or sz-CHEN. You can insert a `breakpoint()` statement to help
Thank you for looking into this scivision, this is indeed the issue I've found too causing issue [#6134](https://github.com/mesonbuild/meson/issues/6134)

I ran this within meson itself:
message('host_machine.cpu_family():'+host_machine.cpu_family())
gstreamer uses this to determine the version of wine to use, when it believes I'm using x86, this causes meson to pick the wine32 version of wine

Otherwise, same issue with python 2.7.17
```
>>> os.environ['PROCESSOR_ARCHITECTURE']
'AMD64'
>>> os.environ['PROCESSOR_ARCHITEW6432']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python27\lib\os.py", line 425, in __getitem__
    return self.data[key.upper()]
KeyError: 'PROCESSOR_ARCHITEW6432'
```
OK @mczarnek I posted some [questions in that issue](https://github.com/mesonbuild/meson/issues/6134#issuecomment-550404429) 

For @sz-CHEN, can you also copy and paste what Meson outputs from configuring a trivial `meson.build` containing only `project('foo', 'c')` 

I suppose an issue may be that you only have Visual Studio compilers--well let's just try it and see what happens from regular (non-visual studio) command prompt.
Hi! I have the same problem compiling GLib with VS 2019 (using x64 Native Tools Command Prompt for VS 2019). I have tried debugging detect_cpu_family in environment.py with pdb.set_trace(). Here's what turned out:

- [detect_windows_arch()](https://github.com/mesonbuild/meson/blob/32f64aa2cb6eb09ad858275d22fc52439c4f860d/mesonbuild/environment.py#L261) calls detect_native_windows_arch()
- [detect_native_windows_arch()](https://github.com/mesonbuild/meson/blob/32f64aa2cb6eb09ad858275d22fc52439c4f860d/mesonbuild/environment.py#L245) rightly detects "amd64" inspecting the PROCESSOR_ARCHITECTURE environment variable.
- Control flow returns to detect_windows_arch() and runs [that checks](https://github.com/mesonbuild/meson/blob/32f64aa2cb6eb09ad858275d22fc52439c4f860d/mesonbuild/environment.py#L292) and immediately x86 is returned
In fact the first if succeds.

I have `compiler.id == 'msvc' and compiler.target == 'x86'`. But is it right to have compiler.target as 'x86'? Afterall I am not cross compiling. But sure, msvc can compile x86_64 **and** x86
Yes that's the question I have, I need someone more experienced with Meson and VS to know what the outcome should be--is it that VS has the "wrong" target configuration or that Meson should understand
https://github.com/mesonbuild/meson/issues/6134#issuecomment-550465629
@nirbheek do you have insight or know who might into this issue? It seems with any version of Visual Studio, Meson thinks an `x86` target should be built when a 64-bit target is desired.
If I debug on 0.51.2 `compiler.target` is `'x64'`
The problem is this: https://github.com/mesonbuild/meson/blob/4c96aa34cb415ca654c0571280386880e9d14ca3/mesonbuild/environment.py#L993

The RE doesn't match. So `match` is `None` and target gets set to 'x86'.
But why doesn't it match?

```
(Pdb) lookat
'Microsoft (R) C/C++ Optimizing Compiler versione 19.23.28106.4 per x64\nCopyright (C) Microsoft Corporation. Tutti i diritti  sono riservati.\n\n'
(Pdb) lookat.split('\n')[0]
'Microsoft (R) C/C++ Optimizing Compiler versione 19.23.28106.4 per x64'
```
Ahhh OMG. The problem in the RE is that 'for' (an english word) is not present in other languages :)
Here's the relevant commit: https://github.com/mesonbuild/meson/commit/a107eab2949cf9750ed88a634118b31666c16c7a
Assuming English language environment has been an issue previously, maybe on some issues still open. 

It seems time to specifically have a non-English CI environment. Maybe a couple (Latin & non-Latin alphabets)
@sz-CHEN if you switch back to Meson 0.51.2 does that work, but 0.52.0 is broken? are you on a non-English language Windows computer?  (like https://github.com/mesonbuild/meson/issues/6128#issuecomment-550490425)
Is there a cl.exe command line option to force output in en_US?
[`set VSLANG=1033`](https://stackoverflow.com/a/48529386)

or [change codepage](https://stackoverflow.com/a/47828954)

If that helps, the question would be how to do that from Meson...

I think Meson's output is always in English, so depending on general opinion, if either of those work for this issue, maybe that would help avoid this issue in other places in Meson.

> [`set VSLANG=1033`](https://stackoverflow.com/a/48529386)

Great! That works :)
BTW, I think we should throw an exception here:

https://github.com/mesonbuild/meson/blob/4c96aa34cb415ca654c0571280386880e9d14ca3/mesonbuild/environment.py#L997
@jpakkane the solution here for a non-English locale appears to be setting environment variable to force Visual Studio to English output.

Another solution (besides removing the word "for" in the Meson regex code) would be per-language regex.

Since Meson's output is always English (?) maybe detecting this situation (non-English locale, use of MSVC) and setting this environment variable to force English MSVC output from the internal Python code is one possible fix?
yes I think an exception would be a better idea there. We'll surely have less confused users who might otherwise just give up in those cases. 

I'm on a Chinese langauage Windows computer. 
After `set VSLANG=1033`  it works.
Thanks for all of you.
Setting `VSLANG=1033` does not seem to change my `cl` output, either on the Windows `cmd` or using a bash e.g. from MSYS or something.

I should add that I have this problem on the VS Community edition 19.23.28106.4

I think the regular expression may be too brittle. Why not change it to this?

```python
 match = re.search('.* (x86|x64|ARM|ARM64)$', lookat.split('\n')[0]) 
```

- You're already in a block that has found `microsoft` in the output.
- The regex is fairly explicit in looking for line-ending specifiers with a preceding space, which should be display language independent.
Ah, I see master is already ahead.
Hi Jens! Just for information purpose, could you post here the first line of `cl.exe /?` ?
Sure - took a while to get there (different/work computer and all that), but it's attached. Language is German, no matter the value of `VSLANG`.

![output](https://user-images.githubusercontent.com/203885/68576674-cc605380-046e-11ea-8708-a2f866bb7ad6.png)

I think the different detection patter in master is doing it right.

Hi @jfinkhaeuser! Were you able to find a workaround in the meantime?
I don't know if that does matters but I have the Visual Studio English Language Pack installed. In Visual Studio Installer, go to the 'Language Pack' tab and check the 'English' checkbox. Maybe this helps
That might be something I'll have to try. I usually configure everything in English, but don't really run VS - I just need the compiler :) I'll have a look.

For the time being, it's not super urgent - we need Windows builds mainly for running tests; the target platforms are mobile. So for now, we've just switched to 32-bit builds. But longer term, it'd be good to see this fixed.

Which really boils down to: is there an ETA for the detection code in master?
@nirbheek using the VSLANG envvar is not a reliable solution. Can we change the regular expression in some way to work for all locales?
The code in master uses a different regex, that should be ok.
Jens are you sure? Because I had this issue using meson from git.
Here's the link: https://github.com/mesonbuild/meson/blob/master/mesonbuild/environment.py#L996
That might've been changed since last I looked. I saw the entire `if 'Microsoft' in out or 'Microsoft' in err:` block factored out into a function that just looked for a line ending in one of the CPU patterns, without the leading `for `.

Sorry for the confusion!

Resolving patch:
diff --git a/mesonbuild/environment.py b/mesonbuild/environment.py
--- a/mesonbuild/environment.py
+++ b/mesonbuild/environment.py
@@ -993,7 +993,7 @@ def sanitize(p):
                     m = 'Failed to detect MSVC compiler version: stderr was\n{!r}'
                     raise EnvironmentException(m.format(err))
                 cl_signature = lookat.split('\n')[0]
-                match = re.search(' for .*(x86|x64|ARM|ARM64)$', cl_signature)
+                match = re.search('.*(x86|x64|ARM|ARM64)$', cl_signature)
                 if match:
                     target = match.group(1)
                 else:


Test patch: