Repository: pypa/cibuildwheel

Issue:
The "/project/" path isn't there anymore for Linux in version 1.5.3
Upgraded from 1.5.1 to 1.5.3 in a branch here: https://github.com/Chia-Network/bls-signatures/tree/up-cibuildwheel

In 1.5.1 we would install cmake into `/project/cmake-3.17.3-Linux-x86_64` and build various dependencies from there as we add `/project/cmake-3.17.3-Linux-x86_64` to the front of PATH:
https://github.com/Chia-Network/bls-signatures/runs/880546025?check_suite_focus=true#step:10:299

In 1.5.3 cmake gets installed into `//cmake-3.17.3-Linux-x86_64` so the PATH we preset doesn't work and the build fails.
https://github.com/Chia-Network/bls-signatures/runs/887691441?check_suite_focus=true#step:10:284

Operative workflow here: https://github.com/Chia-Network/bls-signatures/blob/up-cibuildwheel/.github/workflows/build.yml

I can work around but seems like either an issue that needs a fix or some mention in documentation. I'm sure it came out of the move to docker - which otherwise looks cool.


Discussion:
Confirmed the same behavior with version 1.5.4 as well.
While you're at it: how easy is it to try 1.5.2? It would be nice to pinpoint the exact release :-)
Trivial - give me 5 minutes.
Thanks. I assume it's going to be #386, so that would mean 1.5.2 should still be OK.
Indeed 1.5.2 works just fine - https://github.com/Chia-Network/bls-signatures/runs/887756210?check_suite_focus=true#step:10:297
OK, then I think that the problem is that `CIBW_BEFORE_BUILD` etc isn't executed inside of `/project` or with `/project` as `cwd`. Could you check this, @joerick?

I guess that would mean that passing `--prefix=/project` would solve your problem, @hoffmang9? Running `pwd` would confirm whether that would work as workaround.

But this seems like a regression (or at least an unintended change), so let me see if I can make a PR for this.
Queued up to push was a quick `echo $PATH` and pwd for as soon as the link above completed.
Two interesting things.

First pwd is `/` so definitely not `/project` has it's historically been (or we've been luckily misusing...)

Second, our trick to find the cmake path - 
```
CIBW_ENVIRONMENT_LINUX: "PATH=/project/cmake-3.17.3-Linux-`uname -m`/bin:$PATH"
```
no longer works in 1.5.4. Not critical but interesting.
> Second, our trick to find the cmake path -
> 
> ```
> CIBW_ENVIRONMENT_LINUX: "PATH=/project/cmake-3.17.3-Linux-`uname -m`/bin:$PATH"
> ```
> 
> no longer works in 1.5.4. Not critical but interesting.

Well, that's the same issue, or do you mean `x86_64` isn't interpolated anymore? Again, we just changed some things (#408 and #403), so that's also possible. Do you have a link, perhaps?
> OK, then I think that the problem is that CIBW_BEFORE_BUILD etc isn't executed inside of /project or with /project as pwd. Could you check this, @joerick?

Erm, wow, okay! That's a pretty big regression! My apologies. I can't deal with it now, and I probably won't have time tomorrow either, so I'm thinking the best thing might be to pull 1.5.4 and 1.5.3 from PyPI for now, until we can look at these regressions properly. Would that be okay with you @YannickJadoul ?
> Two interesting things.
> 
> 
> 
> First pwd is `/` so definitely not `/project` has it's historically been (or we've been luckily misusing...)
> 

No, cwd should be /project , as you had assumed :) that's a bug 

> 
> 
> Second, our trick to find the cmake path - 
> 
> ```
> 
> CIBW_ENVIRONMENT_LINUX: "PATH=/project/cmake-3.17.3-Linux-`uname -m`/bin:$PATH"
> 
> ```
> 
> no longer works in 1.5.4. Not critical but interesting.

Also sounds like a bug!
> Would that be okay with you @YannickJadoul ?

I'll leave version management up to you, actually. It's not thát broken, because we pass the absolute path to `pip wheel`. Not sure it's worth pulling things of PyPI for, but again... up to you.

I was just looking, and it should be as easy as adding a few times `cwd='/project'` (or similar) to the `docker.call` calls. And a few tests. If this is intended (just saw your next comment pop up, so it was intended), I can quickly make a PR, still?
> If this is intended (just saw your next comment pop up, so it was intended)

Also, `CIBW_BEFORE_ALL` wás not executed from `/project` before. Was that intended, or should it also be run from `/project` for consistency?
@hoffmang9 After quick look to code I see one comment: ` # don't build i686 targets, can't seem to find cmake for these` You can get cmake from https://pypi.org/project/cmake/#files Then You need to play a little with PATH  but should work. 

Second think is tat you should `CIBW_BEFORE_BUILD` for external dependences (like cmake). 

@YannickJadoul @joerick looking on this project `{project}` substitution should also be applied to environment. 

I think that this line contains bug 

https://github.com/joerick/cibuildwheel/blob/c0cb43800b6740677e373889705b61b17b879a8d/cibuildwheel/docker_container.py#L119

It should be:
`chdir = f'cd {cwd}' if cwd else  'cd /project'`
FYI I've yanked those versions. 

> I was just looking, and it should be as easy as adding a few times cwd='/project' (or similar) to the docker.call calls. And a few tests. If this is intended (just saw your next comment pop up, so it was intended), I can quickly make a PR, still?

Please do! 

And BEFORE_ALL should be from /project, too. 
> Second think is tat you should `CIBW_BEFORE_BUILD` for external dependences (like cmake).

He is doing that, I saw. `CIBW_BEFORE_ALL` could be nicer, even, doing it only once?

> I think that this line contains bug
> 
> https://github.com/joerick/cibuildwheel/blob/c0cb43800b6740677e373889705b61b17b879a8d/cibuildwheel/docker_container.py#L119

No, this should be out of `docker_container.py`. Nothing else in there refers to the specific situation we use it for, so this should probably also be in `linux.py`.

> @hoffmang9 After quick look to code I see one comment: ` # don't build i686 targets, can't seem to find cmake for these` You can get cmake from https://pypi.org/project/cmake/#files Then You need to play a little with PATH but should work.

That comment is ancient and wrong. We only build on x86_64 for uint128 reasons...

> Second think is tat you should `CIBW_BEFORE_BUILD` for external dependences (like cmake).

Only linux needs cmake installed so we're using CIBW_BEFORE_BUILD_LINUX. Building before everything would blow up MacOS and Windows.

Now that does lead me to mention the strong desire to be able to cache a phase of builds before going on to cp38 - but that's an enhancement for another issue... Installing cmake and compiling gmp and sodium twice - is no fun...
@YannickJadoul @hoffmang9 My bad. Yes. 
Instead using CIBW_BEFORE_BUILD_LINUX you should use CIBW_BEFORE_ALL_LINUX
The thanks I have for the suggestion goes way beyond just a heart emoji!

Resolving patch:
diff --git a/cibuildwheel/docker_container.py b/cibuildwheel/docker_container.py
--- a/cibuildwheel/docker_container.py
+++ b/cibuildwheel/docker_container.py
@@ -27,12 +27,14 @@ class DockerContainer:
     bash_stdin: IO[bytes]
     bash_stdout: IO[bytes]
 
-    def __init__(self, docker_image: str, simulate_32_bit=False):
+    def __init__(self, docker_image: str, simulate_32_bit: bool = False, cwd: Optional[Union[str, PathLike]] = None):
         self.docker_image = docker_image
         self.simulate_32_bit = simulate_32_bit
+        self.cwd = cwd
 
     def __enter__(self) -> 'DockerContainer':
         self.name = f'cibuildwheel-{uuid.uuid4()}'
+        cwd_args = ['-w', str(self.cwd)] if self.cwd else []
         shell_args = ['linux32', '/bin/bash'] if self.simulate_32_bit else ['/bin/bash']
         subprocess.run(
             [
@@ -41,6 +43,7 @@ def __enter__(self) -> 'DockerContainer':
                 '--name', self.name,
                 '-i',
                 '-v', '/:/host',  # ignored on CircleCI
+                *cwd_args,
                 self.docker_image,
                 *shell_args
             ],
diff --git a/cibuildwheel/linux.py b/cibuildwheel/linux.py
--- a/cibuildwheel/linux.py
+++ b/cibuildwheel/linux.py
@@ -110,8 +110,8 @@ def build(options: BuildOptions) -> None:
             continue
 
         try:
-            with DockerContainer(docker_image, simulate_32_bit=platform_tag.endswith('i686')) as docker:
-                docker.copy_into(Path.cwd(), Path('/project'))
+            with DockerContainer(docker_image, simulate_32_bit=platform_tag.endswith('i686'), cwd='/project') as docker:
+                docker.copy_into(Path.cwd(), PurePath('/project'))
 
                 if options.before_all:
                     env = docker.get_environment()


Test patch:
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -17,6 +17,7 @@ jobs:
       matrix:
         os: [ubuntu-18.04, windows-latest, macos-latest]
         python_version: ['3.7']
+    timeout-minutes: 180
     steps:
     - uses: actions/checkout@v2
     - uses: actions/setup-python@v2
diff --git a/test/test_before_all.py b/test/test_before_all.py
--- a/test/test_before_all.py
+++ b/test/test_before_all.py
@@ -1,4 +1,5 @@
-import os
+import pytest
+import subprocess
 import textwrap
 
 from . import utils
@@ -21,7 +22,7 @@ def test(tmp_path):
     project_dir = tmp_path / 'project'
     project_with_before_build_asserts.generate(project_dir)
 
-    with open(os.path.join(project_dir, "text_info.txt"), mode='w') as ff:
+    with (project_dir / 'text_info.txt').open(mode='w') as ff:
         print("dummy text", file=ff)
 
     # build the wheels
@@ -33,6 +34,30 @@ def test(tmp_path):
     })
 
     # also check that we got the right wheels
-    os.remove(os.path.join(project_dir, "text_info.txt"))
+    (project_dir / 'text_info.txt').unlink()
+    expected_wheels = utils.expected_wheels('spam', '0.1.0')
+    assert set(actual_wheels) == set(expected_wheels)
+
+
+def test_failing_command(tmp_path):
+    project_dir = tmp_path / 'project'
+    test_projects.new_c_project().generate(project_dir)
+
+    with pytest.raises(subprocess.CalledProcessError):
+        utils.cibuildwheel_run(project_dir, add_env={
+            'CIBW_BEFORE_ALL': 'false',
+            'CIBW_BEFORE_ALL_WINDOWS': 'exit /b 1',
+        })
+
+
+def test_cwd(tmp_path):
+    project_dir = tmp_path / 'project'
+    test_projects.new_c_project().generate(project_dir)
+
+    actual_wheels = utils.cibuildwheel_run(project_dir, add_env={
+        'CIBW_BEFORE_ALL': f'''python -c "import os; assert os.getcwd() == {str(project_dir)!r}"''',
+        'CIBW_BEFORE_ALL_LINUX': '''python -c "import os; assert os.getcwd() == '/project'"''',
+    })
+
     expected_wheels = utils.expected_wheels('spam', '0.1.0')
     assert set(actual_wheels) == set(expected_wheels)
diff --git a/test/test_before_build.py b/test/test_before_build.py
--- a/test/test_before_build.py
+++ b/test/test_before_build.py
@@ -1,3 +1,5 @@
+import pytest
+import subprocess
 import textwrap
 
 from . import utils
@@ -32,14 +34,41 @@ def test(tmp_path):
     project_dir = tmp_path / 'project'
     project_with_before_build_asserts.generate(project_dir)
 
+    before_build = ('''python -c "import sys; open('{output_dir}pythonversion.txt', 'w').write(sys.version)" && '''
+                    '''python -c "import sys; open('{output_dir}pythonexecutable.txt', 'w').write(sys.executable)"''')
+
     # build the wheels
     actual_wheels = utils.cibuildwheel_run(project_dir, add_env={
         # write python version information to a temporary file, this is
         # checked in setup.py
-        'CIBW_BEFORE_BUILD': '''python -c "import sys; open('/tmp/pythonversion.txt', 'w').write(sys.version)" && python -c "import sys; open('/tmp/pythonexecutable.txt', 'w').write(sys.executable)"''',
-        'CIBW_BEFORE_BUILD_WINDOWS': '''python -c "import sys; open('c:\\pythonversion.txt', 'w').write(sys.version)" && python -c "import sys; open('c:\\pythonexecutable.txt', 'w').write(sys.executable)"''',
+        'CIBW_BEFORE_BUILD': before_build.format(output_dir='/tmp/'),
+        'CIBW_BEFORE_BUILD_WINDOWS': before_build.format(output_dir=r'c:\\'),
     })
 
     # also check that we got the right wheels
     expected_wheels = utils.expected_wheels('spam', '0.1.0')
     assert set(actual_wheels) == set(expected_wheels)
+
+
+def test_failing_command(tmp_path):
+    project_dir = tmp_path / 'project'
+    test_projects.new_c_project().generate(project_dir)
+
+    with pytest.raises(subprocess.CalledProcessError):
+        utils.cibuildwheel_run(project_dir, add_env={
+            'CIBW_BEFORE_BUILD': 'false',
+            'CIBW_BEFORE_BUILD_WINDOWS': 'exit /b 1',
+        })
+
+
+def test_cwd(tmp_path):
+    project_dir = tmp_path / 'project'
+    test_projects.new_c_project().generate(project_dir)
+
+    actual_wheels = utils.cibuildwheel_run(project_dir, add_env={
+        'CIBW_BEFORE_BUILD': f'''python -c "import os; assert os.getcwd() == {str(project_dir)!r}"''',
+        'CIBW_BEFORE_BUILD_LINUX': '''python -c "import os; assert os.getcwd() == '/project'"''',
+    })
+
+    expected_wheels = utils.expected_wheels('spam', '0.1.0')
+    assert set(actual_wheels) == set(expected_wheels)
diff --git a/unit_test/docker_container_test.py b/unit_test/docker_container_test.py
--- a/unit_test/docker_container_test.py
+++ b/unit_test/docker_container_test.py
@@ -41,6 +41,13 @@ def test_environment():
         assert container.call(['sh', '-c', 'echo $TEST_VAR'], env={'TEST_VAR': '1'}, capture_output=True) == '1\n'
 
 
+@pytest.mark.docker
+def test_cwd():
+    with DockerContainer(DEFAULT_IMAGE, cwd='/cibuildwheel/working_directory') as container:
+        assert container.call(['pwd'], capture_output=True) == '/cibuildwheel/working_directory\n'
+        assert container.call(['pwd'], capture_output=True, cwd='/opt') == '/opt\n'
+
+
 @pytest.mark.docker
 def test_container_removed():
     with DockerContainer(DEFAULT_IMAGE) as container: