Repository: pypa/cibuildwheel

Issue:
Changing `PATH` in `CIBW_ENVIRONMENT` can cause wrong Python to be used
See https://github.com/samuelcolvin/rtoml/pull/8#issuecomment-580880863: if someone changes `PATH` in `CIBW_ENVIRONMENT`, the newly installed Python (from nuget on Windows, but I'm guessing the same is possible on macOS or Linux; still need to check) is not used, but some other Python version of the system is used.

So I think we should add some kind of check after applying `CIBW_ENVIRONMENT` that makes sure we're using the `python` executable we think we're using.

(I'm happy to actually do it myself, actually, if no one else is eager to do this; but I currently don't have time to do it immediately, so I thought I'd make sure to report it.)


Discussion:
on linux there is not such problem because of different order of command execution. Before build is evaluated before extending path by python location. 

MacOS has same command order like windows.  

I think that solution from linux is best. 

```bash
if [ ! -z {before_build} ]; then
    PATH="$PYBIN:$PATH" sh -c {before_build}
fi

# Build the wheel
rm -rf /tmp/built_wheel
mkdir /tmp/built_wheel
PATH="$PYBIN:$PATH" "$PYBIN/pip" wheel . -w /tmp/built_wheel --no-deps {build_verbosity_flag}
built_wheel=(/tmp/built_wheel/*.whl)
```
> on linux there is not such problem because of different order of command execution. Before build is evaluated before extending path by python location.

Hmmmm, that's not consistent. I thought we'd made different platforms the same before.

> I think that solution from linux is best.

I don't know about that. It might be good to have access to the `python` that will be used when evaluating environment variables (e.g., `PYTHON_VERSION=$(python --version)` or so)? @joerick Any thoughts on the (implicitly) assumed order, here?
Actually; something weirder is going on: `CIBW_ENVIRONMENT` is only evaluated once (per architecture/manylinux image) on Linux, but it's reevaluated each time on macOS and Windows.

The documentation isn't clear on what's happening; it just says

> A space-separated list of environment variables to set during the build.

I would argue to do it for every build, after `python` is added to `PATH`, máybe after `pip` has been added (?), and before `CIBW_BEFORE_BUILD`. I would argue that the location of `python` is already part of the environment you're adapting.

But ... I don't really know. Maybe there's better arguments, so some input would be nice.

Afterthought: why do we do it before `CIBW_BEFORE_BUILD`? Actually, if you need environment variables in `CIBW_BEFORE_BUILD`, you can still set set them in there. The main reason we need `CIBW_ENVIRONMENT` is because `CIBW_BEFORE_BUILD` is run in a subshell. So would it make sense if in some use case `CIBW_BEFORE_BUILD` prepares a dependency, after which `CIBW_ENVIRONMENT` adds it to the `PATH`/environment?
> I would argue to do it for every build, after python is added to PATH, máybe after pip has been added (?), and before CIBW_BEFORE_BUILD. I would argue that the location of python is already part of the environment you're adapting.

I think I agree with you here @YannickJadoul, in fact just today I noticed that linux has different behaviour - by coincidence, that is fixed in #256 as a side effect of my changes.

It's important for the correct `python` and `pip` to be accessible at the top of the PATH, because user scripts or build systems might be using it. So while the old linux 'belt and braces' approach seems more reliable, it could be masking errors that'll cause weirder errors down the line.

So I'm in favour of your original suggestion - asserting that python and pip are correct after CIBW_ENVIRONMENT is evaluated. And once #256 is merged, the behaviour should be consistent between platforms.
Thanks, @joerick. I'll make a PR; shouldn't take too long. If #256 gets merged before, it should be easy enough to rebase :-)

Resolving patch:
diff --git a/cibuildwheel/linux.py b/cibuildwheel/linux.py
--- a/cibuildwheel/linux.py
+++ b/cibuildwheel/linux.py
@@ -63,17 +63,30 @@ def build(project_dir, output_dir, test_command, test_requires, test_extras, bef
             mkdir /output
             cd /project
 
-            {environment_exports}
+            for PYBIN in {pybin_paths}; do (
+                # Temporary hack/workaround, putting loop body in subshell; fixed in PR #256
+
+                export PATH="$PYBIN:$PATH"
+                {environment_exports}
+
+                # check the active python and pip are in PYBIN
+                if [ "$(which pip)" != "$PYBIN/pip" ]; then
+                    echo "cibuildwheel: python available on PATH doesn't match our installed instance. If you have modified PATH, ensure that you don't overwrite cibuildwheel's entry or insert python above it."
+                    exit 1
+                fi
+                if [ "$(which python)" != "$PYBIN/python" ]; then
+                    echo "cibuildwheel: pip available on PATH doesn't match our installed instance. If you have modified PATH, ensure that you don't overwrite cibuildwheel's entry or insert pip above it."
+                    exit 1
+                fi
 
-            for PYBIN in {pybin_paths}; do
                 if [ ! -z {before_build} ]; then
-                    PATH="$PYBIN:$PATH" sh -c {before_build}
+                    sh -c {before_build}
                 fi
 
                 # Build the wheel
                 rm -rf /tmp/built_wheel
                 mkdir /tmp/built_wheel
-                PATH="$PYBIN:$PATH" "$PYBIN/pip" wheel . -w /tmp/built_wheel --no-deps {build_verbosity_flag}
+                pip wheel . -w /tmp/built_wheel --no-deps {build_verbosity_flag}
                 built_wheel=(/tmp/built_wheel/*.whl)
 
                 # repair the wheel
@@ -92,9 +105,9 @@ def build(project_dir, output_dir, test_command, test_requires, test_extras, bef
                 if [ ! -z {test_command} ]; then
                     # Set up a virtual environment to install and test from, to make sure
                     # there are no dependencies that were pulled in at build time.
-                    "$PYBIN/pip" install virtualenv
+                    pip install virtualenv
                     venv_dir=`mktemp -d`/venv
-                    "$PYBIN/python" -m virtualenv "$venv_dir"
+                    python -m virtualenv "$venv_dir"
 
                     # run the tests in a subshell to keep that `activate`
                     # script from polluting the env
@@ -123,7 +136,7 @@ def build(project_dir, output_dir, test_command, test_requires, test_extras, bef
                     )
                     # exit if tests failed (needed for older bash versions)
                     if [ $? -ne 0 ]; then
-                      exit 1;
+                        exit 1;
                     fi
 
                     # clean up
@@ -133,7 +146,7 @@ def build(project_dir, output_dir, test_command, test_requires, test_extras, bef
                 # we're all done here; move it to output
                 mv "${{repaired_wheels[@]}}" /output
                 for repaired_wheel in "${{repaired_wheels[@]}}"; do chown {uid}:{gid} "/output/$(basename "$repaired_wheel")"; done
-            done
+            ) done
         '''.format(
             pybin_paths=' '.join(c.path + '/bin' for c in platform_configs),
             test_requires=' '.join(test_requires),
diff --git a/cibuildwheel/macos.py b/cibuildwheel/macos.py
--- a/cibuildwheel/macos.py
+++ b/cibuildwheel/macos.py
@@ -2,6 +2,7 @@
 import shlex
 import shutil
 import subprocess
+import sys
 import tempfile
 from collections import namedtuple
 from glob import glob
@@ -137,11 +138,20 @@ def build(project_dir, output_dir, test_command, test_requires, test_extras, bef
         # check what version we're on
         call(['which', 'python'], env=env)
         call(['python', '--version'], env=env)
+        which_python = subprocess.check_output(['which', 'python'], env=env, universal_newlines=True).strip()
+        if which_python != '/tmp/cibw_bin/python':
+            print("cibuildwheel: python available on PATH doesn't match our installed instance. If you have modified PATH, ensure that you don't overwrite cibuildwheel's entry or insert python above it.", file=sys.stderr)
+            exit(1)
 
         # install pip & wheel
         call(['python', get_pip_script], env=env, cwd="/tmp")
         assert os.path.exists(os.path.join(installation_bin_path, 'pip'))
+        call(['which', 'pip'], env=env)
         call(['pip', '--version'], env=env)
+        which_pip = subprocess.check_output(['which', 'pip'], env=env, universal_newlines=True).strip()
+        if which_pip != '/tmp/cibw_bin/pip':
+            print("cibuildwheel: pip available on PATH doesn't match our installed instance. If you have modified PATH, ensure that you don't overwrite cibuildwheel's entry or insert pip above it.", file=sys.stderr)
+            exit(1)
         call(['pip', 'install', '--upgrade', 'setuptools', 'wheel', 'delocate'], env=env)
 
         # setup target platform, only required for python 3.5
diff --git a/cibuildwheel/windows.py b/cibuildwheel/windows.py
--- a/cibuildwheel/windows.py
+++ b/cibuildwheel/windows.py
@@ -1,6 +1,7 @@
 import os
 import shutil
 import subprocess
+import sys
 import tempfile
 from collections import namedtuple
 from glob import glob
@@ -141,11 +142,19 @@ def build(project_dir, output_dir, test_command, test_requires, test_extras, bef
         simple_shell(['where', 'python'], env=env)
         simple_shell(['python', '--version'], env=env)
         simple_shell(['python', '-c', '"import struct; print(struct.calcsize(\'P\') * 8)"'], env=env)
+        where_python = subprocess.check_output(['where', 'python'], env=env, universal_newlines=True).splitlines()[0].strip()
+        if where_python != os.path.join(installation_path, 'python.exe'):
+            print("cibuildwheel: python available on PATH doesn't match our installed instance. If you have modified PATH, ensure that you don't overwrite cibuildwheel's entry or insert python above it.", file=sys.stderr)
+            exit(1)
 
         # make sure pip is installed
         if not os.path.exists(os.path.join(installation_path, 'Scripts', 'pip.exe')):
             simple_shell(['python', get_pip_script], env=env, cwd="C:\\cibw")
         assert os.path.exists(os.path.join(installation_path, 'Scripts', 'pip.exe'))
+        where_pip = subprocess.check_output(['where', 'pip'], env=env, universal_newlines=True).splitlines()[0].strip()
+        if where_pip.strip() != os.path.join(installation_path, 'Scripts', 'pip.exe'):
+            print("cibuildwheel: pip available on PATH doesn't match our installed instance. If you have modified PATH, ensure that you don't overwrite cibuildwheel's entry or insert pip above it.", file=sys.stderr)
+            exit(1)
 
         # prepare the Python environment
         simple_shell(['python', '-m', 'pip', 'install', '--upgrade', 'pip'], env=env)


Test patch:
diff --git a/test/05_environment/cibuildwheel_test.py b/test/05_environment/cibuildwheel_test.py
--- a/test/05_environment/cibuildwheel_test.py
+++ b/test/05_environment/cibuildwheel_test.py
@@ -1,5 +1,6 @@
 import os
-
+import pytest
+import subprocess
 import utils
 
 
@@ -17,3 +18,15 @@ def test():
     # also check that we got the right wheels built
     expected_wheels = utils.expected_wheels('spam', '0.1.0')
     assert set(actual_wheels) == set(expected_wheels)
+
+
+def test_overridden_path(tmp_path):
+    project_dir = os.path.dirname(__file__)
+
+    # mess up PATH, somehow
+    with pytest.raises(subprocess.CalledProcessError):
+        utils.cibuildwheel_run(project_dir, output_dir=tmp_path, add_env={
+            'CIBW_ENVIRONMENT': '''SOMETHING="$(mkdir new_path && touch new_path/python)" PATH="$(realpath new_path):$PATH"''',
+            'CIBW_ENVIRONMENT_WINDOWS': '''SOMETHING="$(mkdir new_path && type nul > new_path/python.exe)" PATH="$CD\\new_path;$PATH"''',
+        })
+    assert len(os.listdir(str(tmp_path))) == 0