Repository: pypa/setuptools

Issue:
[BUG] `-requiresAny` is not recognized by  Visual Studio 2017 < v15.6
### setuptools version

setuptools==67.8.0

### Python version

Python 3.09, 3.10, 3.11

### OS

Windows Server 2012r2

### Additional environment information

_No response_

### Description

Hi,

I encountered an error when using setup.py to compile a module related to pybind11. The error message was `pybind11\detail/common.h(86): fatal error C1189: #error: pybind11 2.10+ requires MSVC 2017 or newer`. But I actually had VS2017 installed. Later, I found that the _msvccompiler.py module in setuptools seemed to be highly optimized, causing the _get_vc_env function to always return the environment variables of MSVC14. And I tried to add some print statements in that function, but nothing was printed, proving that the function was never called. Finally, I renamed _get_vc_env to _get_vc_env1, and setup.py worked fine.

I tried to print the name of the _get_env_vc function, and the actual output was “<function msvc14_get_vc_env at 0xxxxxxxx>.”, which looked like a function that was optimized for a specific compiler.




### Expected behavior

setuptools can dected vs2017 environment.

### How to Reproduce

1. Call a python setup.py to compile c++ extension.


### Output

```console
\.py3.11\lib\site-packages\pybind11\include\pybind11\detail/common.h(86): fatal error C1189: #error:  pybind11 2.10+ requires MSVC 2017 or newer
```



Discussion:
Hi @AndrewTsao, thank you very much for writing this report.

Is there any chance you can expand a little bit more in the reproducer. Right now it is a bit difficult to understand what is going on. A good methodology for how to write reproducers is the following: https://stackoverflow.com/help/minimal-reproducible-example.

---

Also could you please include the commands of how you are building your package? Does it behave the same way if you use `python -m build` (calling `python setup.py` directly is deprecated).

---

Please note that the naming MSVC14 used in setuptools is not exactly related to the year of release of the toolkit.

You can see in the file below that this number seems to be related to the [runtime library version instead](https://en.wikipedia.org/wiki/Microsoft_Visual_C++#Internal_version_numbering): https://github.com/pypa/setuptools/blob/b545fc778583f644d6c331773dbe0ea53bfa41af/setuptools/msvc.py#L2-L11

---

This is probably how setuptools is finding the latest installation of the toolkit:

https://github.com/pypa/setuptools/blob/b545fc778583f644d6c331773dbe0ea53bfa41af/setuptools/msvc.py#L71-L104

Do you know what the `vswhere` command is returning in your system (and if the `VC/Auxiliary/Build` directory exists inside of it)?




My virtualenv:

![e7f60e52ba1fb64c0c26b583faa77d6](https://github.com/pypa/setuptools/assets/133179/b1d76edf-b7ca-438e-baf1-f59c7c7e9b5e)

My setup.py file:

```py
from setuptools import setup, Extension
import numpy as np

__version__ = '0.0.1'

requires = ['numpy']

setup_requires = ['pybind11','wheel']

class get_pybind_include(object):
    """Helper class to determine the pybind11 include path
    The purpose of this class is to postpone importing pybind11
    until it is actually installed, so that the ``get_include()``
    method can be invoked. """

    def __init__(self, user=False):
        self.user = user

    def __str__(self):
        import pybind11
        return pybind11.get_include(self.user)

ext_modules = [
    Extension(
        'example',
        ['src/main.cpp'],
        include_dirs=[
            # Path to pybind11 headers
            get_pybind_include(),
            get_pybind_include(user=True),
            np.get_include()
        ],
        language='c++',
        # 如果是调试模式，添加以下选项。
        extra_compile_args=['/Od', '/Zi'],
        extra_link_args=['/DEBUG:FULL']
    ),
]

setup(
    name='example',
    ext_modules=ext_modules,
    install_requires=requires,
    setup_requires=setup_requires,
    platforms='win_amd64',
    python_requires='>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*'
)
```

`src/main.cc'
```c++
#include <Python.h>
#define PY_ARRAY_UNIQUE_SYMBOL MYLIBRARY_ARRAY_API
#define PY_UFUNC_UNIQUE_SYMBOL MYLIBRARY_UFUNC_API
#include <numpy/arrayobject.h>
#include <pybind11/numpy.h>
#include <pybind11/pybind11.h>
#include <time.h>

#include <iostream>
#include <string>
#include <vector>
namespace py = pybind11;
using namespace py::literals;

struct Point
{
  int x;
  int y;
  double z;
};

struct Points
{
  Points()
  {
    points.resize(10);
    for (int i = 0; i < 10; i++) {
      points[i].x = i;
      points[i].y = i;
      points[i].z = i * 6.6;
    }
  }

  ~Points()
  {
    std::cout << (void*)this << ": Points bye, p:" << (void*)points.data() << std::endl;
  }

  std::vector<Point> points;
  py::buffer_info Buffer();
};

py::buffer_info Points::Buffer()
{
  auto buf = py::buffer_info(
      points.data(),                          /* Pointer to buffer */
      sizeof(Point),                          /* Size of one scalar */
      py::format_descriptor<Point>::format(), /* Python struct-style format descriptor */
      1,                                      /* Number of dimensions */
      { 10 },                 /* Buffer dimensions */
      {             /* Strides (in bytes) for each index */
        sizeof(Point) }
  );
  return buf;
}

PYBIND11_MODULE(example, m) {

  PYBIND11_NUMPY_DTYPE(Point, x, y, z);
  py::class_<Points>(m, "Points", py::buffer_protocol())
   .def_buffer([](Points &m) -> py::buffer_info {return m.Buffer();});

  m.doc() = "test pybind11 module";
  import_array1();
}
```

build command: `python setup.py bdist`, get error:

![image](https://github.com/pypa/setuptools/assets/133179/faba9854-f4a7-468c-adad-05b4e1665f42)



Interactive call _get_vc_env and _find_vc2017.

![image](https://github.com/pypa/setuptools/assets/133179/0667391e-5214-43b8-8ce3-e6bc744f4563)

After I rename _get_vc_env to _get_vc_env1, call it in python console

![image](https://github.com/pypa/setuptools/assets/133179/aa2de714-31de-43e9-95c6-9cbbb44ead43)

![image](https://github.com/pypa/setuptools/assets/133179/9b862dc6-5d81-4931-b3a4-0177a9e1bce5)

this is result before rename _get_vc_env,

![image](https://github.com/pypa/setuptools/assets/133179/3f279dcb-29b0-4221-bc07-831623d2d52e)

![image](https://github.com/pypa/setuptools/assets/133179/4b387398-1852-41c0-900e-f330bdf42fe9)


Hi @AndrewTsao, can you try running `python -m build`[^1] instead of `python setup.py bdist`? I don't think this command is supposed to work like that after the implementations of PEP 517.

[^1]: You need to install https://pypa-build.readthedocs.io/en/stable/ and configure `pyproject.toml` accordingly (https://pybind11.readthedocs.io/en/stable/compiling.html).

The thing about renaming `_get_vc_env` is more complicated... `setuptools._distutils` is not supposed to be accessed directly. The functions there are patched with the functions in `setuptools.msvc`. If you want to debug these functions, please run `import setuptools; import distutils._msvccompiler` and use the functions from `distutils` not `setuptools._distutils`, this will guarantee you are getting the same thing setuptools uses during the build.

The patch for `_get_vc_env`, is the following: https://github.com/pypa/setuptools/blob/b545fc778583f644d6c331773dbe0ea53bfa41af/setuptools/msvc.py#L154-L187

Here you can see that setuptools will use a `_msvc14_find_vcvarsall` instead of `_find_vcvarsall`...
The functions in both implementations (`setuptools` and `distutils`), differ in very subtle ways, but in general `setuptools` considers more edge cases that `distutils`. Probably the key here is the different `vswhere` commands that `setuptools` and `distutils` run.

If you compare `distuils._msvccompiler._find_vc2017` and `setuptools.msvc._msvc14_find_vcvarsall`, you can see that `setuptools` include a few extra flags to `vswhere` (`-requiresAny -requires Microsoft.VisualStudio.Workload.WDExpress`). There are also other conditions that setuptools checks...




Yes, you are right!

I try call setuptools.msvc module functions, maybe get reason.

![image](https://github.com/pypa/setuptools/assets/133179/a936c1e5-5ab9-4bd1-b496-73dbd73214c3)

I build a vswhere command and run, got results.

![image](https://github.com/pypa/setuptools/assets/133179/66384488-e265-4e0d-8a79-0db61c2e38ed)



It seems that the `-requiresAny` parameter was added in version 2.3 (according to this text: https://github.com/Microsoft/vswhere/wiki/Find-VSTest).

So probably what is happening is that `setuptools` is currently not supporting the version of `vswhere` you have installed in your system.

Now we have to trace down why the author of the patch in `setuptools` added this parameter and what edge cases it covers... If we can simply remove it, or if it is worth to implement a workaround.

I don't know what is the relationship between the versions of the VC toolkit and `vswhere` and how you ended up with an older version installed.
It seems the `-requiresAny` argument was added in https://github.com/pypa/setuptools/pull/2663 to support Visual Studio Express.
If we see the release pages for `vswhere`, we find out that `-requiresAny` was added in [v2.3.2](https://github.com/microsoft/vswhere/releases/tag/2.3.2), released on 23 Jan 2018.

By correlating the date with the Visual Studio 2017 releases, we can say that the earliest possible Visual Studio to include this (or a later) version of `vswhere` (supporting `-requiresAny`) is version 15.5.5. But it is more likely that the update was made in version 15.6.

This means that `setuptools` currently is not compatible with Visual Studio 2017 with versions prior to 15.6.

I asked the original author of the PR about suggestions for how to tackle this problem.

Resolving patch:
diff --git a/setuptools/msvc.py b/setuptools/msvc.py
--- a/setuptools/msvc.py
+++ b/setuptools/msvc.py
@@ -15,6 +15,7 @@
 from io import open
 from os import listdir, pathsep
 from os.path import join, isfile, isdir, dirname
+from subprocess import CalledProcessError
 import contextlib
 import platform
 import itertools
@@ -83,25 +84,28 @@ def _msvc14_find_vc2017():
     if not root:
         return None, None
 
-    try:
-        path = subprocess.check_output([
-            join(root, "Microsoft Visual Studio", "Installer", "vswhere.exe"),
-            "-latest",
-            "-prerelease",
-            "-requiresAny",
-            "-requires", "Microsoft.VisualStudio.Component.VC.Tools.x86.x64",
-            "-requires", "Microsoft.VisualStudio.Workload.WDExpress",
-            "-property", "installationPath",
-            "-products", "*",
-        ]).decode(encoding="mbcs", errors="strict").strip()
-    except (subprocess.CalledProcessError, OSError, UnicodeDecodeError):
-        return None, None
-
-    path = join(path, "VC", "Auxiliary", "Build")
-    if isdir(path):
-        return 15, path
-
-    return None, None
+    suitable_components = (
+        "Microsoft.VisualStudio.Component.VC.Tools.x86.x64",
+        "Microsoft.VisualStudio.Workload.WDExpress",
+    )
+
+    for component in suitable_components:
+        # Workaround for `-requiresAny` (only available on VS 2017 > 15.6)
+        with contextlib.suppress(CalledProcessError, OSError, UnicodeDecodeError):
+            path = subprocess.check_output([
+                join(root, "Microsoft Visual Studio", "Installer", "vswhere.exe"),
+                "-latest",
+                "-prerelease",
+                "-requires", component,
+                "-property", "installationPath",
+                "-products", "*",
+            ]).decode(encoding="mbcs", errors="strict").strip()
+
+            path = join(path, "VC", "Auxiliary", "Build")
+            if isdir(path):
+                return 15, path
+
+    return None, None  # no suitable component found
 
 
 PLAT_SPEC_TO_RUNTIME = {


Test patch: