Repository: pypa/virtualenv

Issue:
Cant find Python.h inside a virtualenv
**Issue**

I try to install a package for python2 inside a virtualenv and the compilation fails mentioning that it can't find Python.h. 

But, if I install the same package at the system level it can find that issue

I tested this with the lastest virutalenv release (20.0.12)

**Environment**

You can tests the build using this Dockerfile

```
FROM amazonlinux:1

RUN yum -y install python27 python27-devel python36 python36-devel python27-pip git gcc

RUN pip-3.6 install --upgrade virtualenv

RUN cd /root && virtualenv --python python2.7 /tmp/bar

RUN source /tmp/bar/bin/activate \
    && pip install cython numpy \
    && pip install cython-bbox
 ```
And the error when building that dockerfile is:

```
    gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/tmp/bar/lib64/python2.7/site-packages/numpy/core/include -I/tmp/bar/include/python2.7 -c src/cython_bbox.c -o build/temp.linux-x86_64-2.7/src/cython_bbox.o -Wno-cpp
    src/cython_bbox.c:26:20: fatal error: Python.h: No such file or directory
     #include "Python.h"
                        ^
    compilation terminated.
    error: command 'gcc' failed with exit status 1
```


**Output of the virtual environment creation**

Make sure to run the creation with `-vvv --with-traceback`:

```console
70 setup logging to NOTSET [DEBUG report:43]
71 created app data folder /root/.local/share/virtualenv [DEBUG app_data:52]
78 find interpreter for spec PythonSpec(implementation=CPython, major=2, minor=7) [INFO builtin:44]
79 proposed PythonInfo(spec=CPython3.6.10.final.0-64, exe=/usr/bin/python3.6, platform=linux, version='3.6.10 (default, Feb 10 2020, 19:55:14) \n[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]', encoding_fs_io=ascii-ANSI_X3.4-1968) [INFO builtin:50]
79 discover PATH[0]=/usr/local/sbin [DEBUG builtin:84]
79 filesystem is case-sensitive [DEBUG info:28]
80 discover PATH[1]=/usr/local/bin [DEBUG builtin:84]
80 discover PATH[2]=/usr/sbin [DEBUG builtin:84]
81 discover PATH[3]=/usr/bin [DEBUG builtin:84]
81 Attempting to acquire lock 139953546058720 on /root/.local/share/virtualenv/py_info/20.0.12/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [DEBUG filelock:270]
81 Lock 139953546058720 acquired on /root/.local/share/virtualenv/py_info/20.0.12/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [INFO filelock:274]
81 get interpreter info via cmd: /usr/bin/python2.7 /usr/local/lib/python3.6/site-packages/virtualenv/discovery/py_info.py [DEBUG cached_py_info:108]
124 write PythonInfo to /root/.local/share/virtualenv/py_info/20.0.12/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.json for /usr/bin/python2.7 [DEBUG cached_py_info:94]
125 Attempting to release lock 139953546058720 on /root/.local/share/virtualenv/py_info/20.0.12/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [DEBUG filelock:315]
125 Lock 139953546058720 released on /root/.local/share/virtualenv/py_info/20.0.12/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [INFO filelock:318]
125 proposed PathPythonInfo(spec=CPython2.7.16.final.0-64, exe=/usr/bin/python2.7, platform=linux2, version='2.7.16 (default, Feb 10 2020, 18:54:57) \n[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]', encoding_fs_io=ANSI_X3.4-1968-None) [INFO builtin:50]
125 accepted PathPythonInfo(spec=CPython2.7.16.final.0-64, exe=/usr/bin/python2.7, platform=linux2, version='2.7.16 (default, Feb 10 2020, 18:54:57) \n[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]', encoding_fs_io=ANSI_X3.4-1968-None) [DEBUG builtin:52]
204 create virtual environment via CPython2Posix(dest=/tmp/bar, clear=False, global=False) [INFO session:52]
204 no include folders as can't find include marker /usr/local/include/python2.7/Python.h [DEBUG cpython2:53]
205 create folder /tmp/bar/bin [DEBUG _sync:21]
205 create folder /tmp/bar/lib/python2.7/site-packages [DEBUG _sync:21]
205 create folder /tmp/bar/lib64/python2.7/site-packages [DEBUG _sync:21]
206 write /tmp/bar/pyvenv.cfg [DEBUG pyenv_cfg:34]
206 	home = /usr [DEBUG pyenv_cfg:38]
206 	implementation = CPython [DEBUG pyenv_cfg:38]
206 	version_info = 2.7.16.final.0 [DEBUG pyenv_cfg:38]
206 	virtualenv = 20.0.12 [DEBUG pyenv_cfg:38]
206 	include-system-site-packages = false [DEBUG pyenv_cfg:38]
206 	base-prefix = /usr [DEBUG pyenv_cfg:38]
206 	base-exec-prefix = /usr [DEBUG pyenv_cfg:38]
206 	base-executable = /usr/bin/python2.7 [DEBUG pyenv_cfg:38]
206 copy /usr/bin/python2.7 to /tmp/bar/bin/python [DEBUG _sync:48]
207 symlink /usr/lib64/python2.7/os.py to /tmp/bar/lib64/python2.7/os.py [DEBUG _sync:40]
207 symlink /usr/lib64/python2.7/os.pyc to /tmp/bar/lib64/python2.7/os.pyc [DEBUG _sync:40]
207 symlink directory /usr/lib64/python2.7/lib-dynload to /tmp/bar/lib64/python2.7/lib-dynload [DEBUG _sync:40]
208 create /tmp/bar/lib/python2.7/site-packages/_virtualenv.py [DEBUG api:78]
208 create virtualenv import hook file /tmp/bar/lib/python2.7/site-packages/_virtualenv.pth [DEBUG api:81]
208 ============================== target debug ============================== [DEBUG session:54]
209 debug via /tmp/bar/bin/python /usr/local/lib/python3.6/site-packages/virtualenv/create/debug.py [DEBUG creator:187]
208 {
  "sys": {
    "executable": "/tmp/bar/bin/python",
    "_base_executable": null,
    "prefix": "/tmp/bar",
    "base_prefix": "/usr",
    "real_prefix": "/usr",
    "exec_prefix": "/tmp/bar",
    "base_exec_prefix": "/usr",
    "path": [
      "/usr/lib/python27.zip",
      "/usr/lib64/python2.7",
      "/usr/lib64/python2.7/plat-linux2",
      "/usr/lib64/python2.7/lib-tk",
      "/usr/lib64/python2.7/lib-old",
      "/usr/lib64/python2.7/lib-dynload",
      "/tmp/bar/lib64/python2.7/site-packages",
      "/tmp/bar/lib/python2.7/site-packages"
    ],
    "meta_path": [
      "<class '_virtualenv._VirtualenvImporter'>"
    ],
    "fs_encoding": "ANSI_X3.4-1968",
    "io_encoding": null
  },
  "version": "2.7.16 (default, Feb 10 2020, 18:54:57) \n[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]",
  "os": "<module 'os' from '/usr/lib64/python2.7/os.pyc'>",
  "site": "<module 'site' from '/usr/lib64/python2.7/site.pyc'>",
  "datetime": "<module 'datetime' from '/usr/lib64/python2.7/lib-dynload/datetime.so'>",
  "math": "<module 'math' from '/usr/lib64/python2.7/lib-dynload/math.so'>",
  "json": "<module 'json' from '/usr/lib64/python2.7/json/__init__.pyc'>"
} [DEBUG session:55]
233 add seed packages via FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/root/.local/share/virtualenv/seed-app-data/v1.0.1) [INFO session:59]
234 Attempting to acquire lock 139953453730056 on /root/.local/share/virtualenv/seed-app-data/v1.0.1/2.7/wheels.lock [DEBUG filelock:270]
234 Lock 139953453730056 acquired on /root/.local/share/virtualenv/seed-app-data/v1.0.1/2.7/wheels.lock [INFO filelock:274]
235 get bundled wheel /usr/local/lib/python3.6/site-packages/virtualenv/seed/embed/wheels/pip-20.0.2-py2.py3-none-any.whl [DEBUG acquire:51]
235 get bundled wheel /usr/local/lib/python3.6/site-packages/virtualenv/seed/embed/wheels/setuptools-44.0.0-py2.py3-none-any.whl [DEBUG acquire:51]
235 get bundled wheel /usr/local/lib/python3.6/site-packages/virtualenv/seed/embed/wheels/wheel-0.34.2-py2.py3-none-any.whl [DEBUG acquire:51]
238 install wheel from wheel /root/.local/share/virtualenv/seed-app-data/v1.0.1/2.7/wheels/wheel-0.34.2-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:46]
239 install setuptools from wheel /root/.local/share/virtualenv/seed-app-data/v1.0.1/2.7/wheels/setuptools-44.0.0-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:46]
239 build install image to /root/.local/share/virtualenv/seed-app-data/v1.0.1/2.7/image/CopyPipInstall/wheel-0.34.2-py2.py3-none-any of wheel-0.34.2-py2.py3-none-any.whl [DEBUG base:57]
239 install pip from wheel /root/.local/share/virtualenv/seed-app-data/v1.0.1/2.7/wheels/pip-20.0.2-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:46]
239 build install image to /root/.local/share/virtualenv/seed-app-data/v1.0.1/2.7/image/CopyPipInstall/setuptools-44.0.0-py2.py3-none-any of setuptools-44.0.0-py2.py3-none-any.whl [DEBUG base:57]
239 build install image to /root/.local/share/virtualenv/seed-app-data/v1.0.1/2.7/image/CopyPipInstall/pip-20.0.2-py2.py3-none-any of pip-20.0.2-py2.py3-none-any.whl [DEBUG base:57]
262 changing mode of /tmp/tmph18mndm1/wheel-2.7 to 755 [INFO util:566]
263 changing mode of /tmp/tmph18mndm1/wheel to 755 [INFO util:566]
263 changing mode of /tmp/tmph18mndm1/wheel2 to 755 [INFO util:566]
269 copy directory /root/.local/share/virtualenv/seed-app-data/v1.0.1/2.7/image/CopyPipInstall/wheel-0.34.2-py2.py3-none-any/wheel-0.34.2.dist-info to /tmp/bar/lib/python2.7/site-packages/wheel-0.34.2.dist-info [DEBUG _sync:48]
274 copy /root/.local/share/virtualenv/seed-app-data/v1.0.1/2.7/image/CopyPipInstall/wheel-0.34.2-py2.py3-none-any/wheel-0.34.2.virtualenv to /tmp/bar/lib/python2.7/site-packages/wheel-0.34.2.virtualenv [DEBUG _sync:48]
275 copy directory /root/.local/share/virtualenv/seed-app-data/v1.0.1/2.7/image/CopyPipInstall/wheel-0.34.2-py2.py3-none-any/wheel to /tmp/bar/lib/python2.7/site-packages/wheel [DEBUG _sync:48]
288 changing mode of /tmp/bar/bin/wheel-2.7 to 755 [INFO util:566]
289 changing mode of /tmp/bar/bin/wheel to 755 [INFO util:566]
289 changing mode of /tmp/bar/bin/wheel2 to 755 [INFO util:566]
289 generated console scripts wheel2 wheel-2.7 wheel [DEBUG base:53]
316 changing mode of /tmp/tmp3hgjfz5o/easy_install-2.7 to 755 [INFO util:566]
316 changing mode of /tmp/tmp3hgjfz5o/easy_install to 755 [INFO util:566]
316 changing mode of /tmp/tmp3hgjfz5o/easy_install2 to 755 [INFO util:566]
324 copy /root/.local/share/virtualenv/seed-app-data/v1.0.1/2.7/image/CopyPipInstall/setuptools-44.0.0-py2.py3-none-any/setuptools-44.0.0.virtualenv to /tmp/bar/lib/python2.7/site-packages/setuptools-44.0.0.virtualenv [DEBUG _sync:48]
325 copy directory /root/.local/share/virtualenv/seed-app-data/v1.0.1/2.7/image/CopyPipInstall/setuptools-44.0.0-py2.py3-none-any/setuptools to /tmp/bar/lib/python2.7/site-packages/setuptools [DEBUG _sync:48]
348 copy directory /root/.local/share/virtualenv/seed-app-data/v1.0.1/2.7/image/CopyPipInstall/setuptools-44.0.0-py2.py3-none-any/pkg_resources to /tmp/bar/lib/python2.7/site-packages/pkg_resources [DEBUG _sync:48]
352 copy /root/.local/share/virtualenv/seed-app-data/v1.0.1/2.7/image/CopyPipInstall/setuptools-44.0.0-py2.py3-none-any/easy_install.py to /tmp/bar/lib/python2.7/site-packages/easy_install.py [DEBUG _sync:48]
353 copy directory /root/.local/share/virtualenv/seed-app-data/v1.0.1/2.7/image/CopyPipInstall/setuptools-44.0.0-py2.py3-none-any/setuptools-44.0.0.dist-info to /tmp/bar/lib/python2.7/site-packages/setuptools-44.0.0.dist-info [DEBUG _sync:48]
355 changing mode of /tmp/bar/bin/easy_install-2.7 to 755 [INFO util:566]
356 changing mode of /tmp/bar/bin/easy_install to 755 [INFO util:566]
356 changing mode of /tmp/bar/bin/easy_install2 to 755 [INFO util:566]
356 generated console scripts easy_install2 easy_install easy_install-2.7 [DEBUG base:53]
389 changing mode of /tmp/tmp1447pgjv/pip to 755 [INFO util:566]
389 changing mode of /tmp/tmp1447pgjv/pip2 to 755 [INFO util:566]
389 changing mode of /tmp/tmp1447pgjv/pip-2.7 to 755 [INFO util:566]
389 changing mode of /tmp/tmp1447pgjv/pip2.7 to 755 [INFO util:566]
415 copy /root/.local/share/virtualenv/seed-app-data/v1.0.1/2.7/image/CopyPipInstall/pip-20.0.2-py2.py3-none-any/pip-20.0.2.virtualenv to /tmp/bar/lib/python2.7/site-packages/pip-20.0.2.virtualenv [DEBUG _sync:48]
416 copy directory /root/.local/share/virtualenv/seed-app-data/v1.0.1/2.7/image/CopyPipInstall/pip-20.0.2-py2.py3-none-any/pip-20.0.2.dist-info to /tmp/bar/lib/python2.7/site-packages/pip-20.0.2.dist-info [DEBUG _sync:48]
417 copy directory /root/.local/share/virtualenv/seed-app-data/v1.0.1/2.7/image/CopyPipInstall/pip-20.0.2-py2.py3-none-any/pip to /tmp/bar/lib/python2.7/site-packages/pip [DEBUG _sync:48]
470 changing mode of /tmp/bar/bin/pip to 755 [INFO util:566]
470 changing mode of /tmp/bar/bin/pip2 to 755 [INFO util:566]
470 changing mode of /tmp/bar/bin/pip-2.7 to 755 [INFO util:566]
471 changing mode of /tmp/bar/bin/pip2.7 to 755 [INFO util:566]
471 generated console scripts pip pip2 pip2.7 pip-2.7 [DEBUG base:53]
471 Attempting to release lock 139953453730056 on /root/.local/share/virtualenv/seed-app-data/v1.0.1/2.7/wheels.lock [DEBUG filelock:315]
471 Lock 139953453730056 released on /root/.local/share/virtualenv/seed-app-data/v1.0.1/2.7/wheels.lock [INFO filelock:318]
471 add activators for Bash, CShell, Fish, PowerShell, Python [INFO session:65]
473 write /tmp/bar/pyvenv.cfg [DEBUG pyenv_cfg:34]
473 	home = /usr [DEBUG pyenv_cfg:38]
473 	implementation = CPython [DEBUG pyenv_cfg:38]
473 	version_info = 2.7.16.final.0 [DEBUG pyenv_cfg:38]
473 	virtualenv = 20.0.12 [DEBUG pyenv_cfg:38]
473 	include-system-site-packages = false [DEBUG pyenv_cfg:38]
473 	base-prefix = /usr [DEBUG pyenv_cfg:38]
473 	base-exec-prefix = /usr [DEBUG pyenv_cfg:38]
473 	base-executable = /usr/bin/python2.7 [DEBUG pyenv_cfg:38]
473 created virtual environment CPython2.7.16.final.0-64 in 404ms
  creator CPython2Posix(dest=/tmp/bar, clear=False, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/root/.local/share/virtualenv/seed-app-data/v1.0.1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator [WARNING __main__:21]

```

Thank you for the amazing work



Discussion:
Here's the relevant error:

```
204 no include folders as can't find include marker /usr/local/include/python2.7/Python.h [DEBUG cpython2:53]
```
Where are the include headers stored? Would be helpful to also see ``/root/.local/share/virtualenv/py_info/20.0.12/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.json`` that specifies the include header location for ``/usr/bin/python2.7``
The headers are stored at:
```
$ locate Python.h
/usr/include/python2.7/Python.h
/usr/include/python3.6m/Python.h
```

and the output of the JSON file is: 
```json

{
  "st_mtime": 1581360923.0,
  "path": "/usr/bin/python2.7",
  "content": {
    "base_exec_prefix": null,
    "system_stdlib": "/usr/lib/python2.7",
    "has_venv": false,
    "prefix": "/usr",
    "stdout_encoding": null,
    "executable": "/usr/bin/python2.7",
    "implementation": "CPython",
    "exec_prefix": "/usr",
    "platform": "linux2",
    "version": "2.7.16 (default, Feb 10 2020, 18:54:57) \n[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]",
    "sysconfig_paths": {
      "platstdlib": "{platbase}/lib64/python{py_version_short}",
      "platlib": "{platbase}/local/lib64/python{py_version_short}/site-packages",
      "purelib": "{base}/local/lib/python{py_version_short}/site-packages",
      "stdlib": "{base}/lib/python{py_version_short}",
      "scripts": "{base}/local/bin",
      "include": "{base}/local/include/python{py_version_short}",
      "data": "{base}/local"
    },
    "base_prefix": null,
    "system_stdlib_platform": "/usr/lib64/python2.7",
    "_creators": null,
    "file_system_encoding": "ANSI_X3.4-1968",
    "version_info": {
      "major": 2,
      "minor": 7,
      "micro": 16,
      "releaselevel": "final",
      "serial": 0
    },
    "sysconfig_vars": {
      "base": "/usr",
      "platbase": "/usr",
      "PYTHONFRAMEWORK": "",
      "py_version_short": "2.7"
    },
    "path": [
      "/usr/local/lib/python3.6/site-packages/virtualenv/discovery",
      "/usr/lib/python27.zip",
      "/usr/lib64/python2.7",
      "/usr/lib64/python2.7/plat-linux2",
      "/usr/lib64/python2.7/lib-tk",
      "/usr/lib64/python2.7/lib-old",
      "/usr/lib64/python2.7/lib-dynload",
      "/usr/local/lib64/python2.7/site-packages",
      "/usr/local/lib/python2.7/site-packages",
      "/usr/lib64/python2.7/site-packages",
      "/usr/lib/python2.7/site-packages",
      "/usr/lib64/python2.7/dist-packages",
      "/usr/lib/python2.7/dist-packages"
    ],
    "max_size": 9223372036854775807,
    "real_prefix": null,
    "distutils_install": {
      "purelib": "lib/python2.7/site-packages",
      "headers": "include/python2.7/UNKNOWN",
      "platlib": "lib64/python2.7/site-packages",
      "data": "",
      "scripts": "bin"
    },
    "architecture": 64,
    "original_executable": "/usr/bin/python2.7",
    "os": "posix",
    "system_executable": "/usr/bin/python2.7"
  }
}
```
Then that python is just broken 😢 The interpreter tells us that it should be inside

```json
    "sysconfig_paths": {
      "include": "{base}/local/include/python{py_version_short}",
    },
    "sysconfig_vars": {
      "base": "/usr",
      "py_version_short": "2.7"
    },
```

Which resolves to ``/usr/local/include/python2.7`` 🤔 Why install them then under ``/usr/include/python2.7``? Who packaged this Python 🤔 

Resolving patch:
diff --git a/src/virtualenv/discovery/py_info.py b/src/virtualenv/discovery/py_info.py
--- a/src/virtualenv/discovery/py_info.py
+++ b/src/virtualenv/discovery/py_info.py
@@ -159,10 +159,16 @@ def creators(self, refresh=False):
 
     @property
     def system_include(self):
-        return self.sysconfig_path(
+        path = self.sysconfig_path(
             "include",
             {k: (self.system_prefix if v.startswith(self.prefix) else v) for k, v in self.sysconfig_vars.items()},
         )
+        if not os.path.exists(path):  # some broken packaging don't respect the sysconfig, fallback to distutils path
+            # the pattern include the distribution name too at the end, remove that via the parent call
+            fallback = os.path.join(self.prefix, os.path.dirname(self.distutils_install["headers"]))
+            if os.path.exists(fallback):
+                path = fallback
+        return path
 
     @property
     def system_prefix(self):


Test patch: