Repository: pypa/cibuildwheel

Issue:
Stderr on docker run at the top
Currently, stdout is captured and placed in the error. But stderr is not, so it's displayed inline. `CalledProcessError` does support stderr too, so should we do the same thing with stderr? (Not sure I'm a big fan of separating stderr and stdout, sometimes they might be best in the original order).

Thoughts?


Discussion:
Here's an example (just the end of the log):

```
...
Building cp39-manylinux_x86_64 wheel
CPython 3.9 manylinux x86_64

Setting up build environment...

    + /opt/python/cp38-cp38/bin/python -c 'import sys, json, os; json.dump(os.environ.copy(), sys.stdout)'
    + which python
    + which pip

                                                              ✓ 0.04s
Running before_build...

    + sh -c 'pip install cython && cmake -DOPENDHT_PYTHON=ON . && make'
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Indexation enabled since it is required for Python support
/project/src/dhtrunner.cpp: In member function ‘void dht::DhtRunner::setProxyServer(const string&, const string&)’:
/project/src/dhtrunner.cpp:1080:72: warning: unused parameter ‘pushNodeId’ [-Wunused-parameter]
 1080 | DhtRunner::setProxyServer(const std::string& proxy, const std::string& pushNodeId)
      |                                                     ~~~~~~~~~~~~~~~~~~~^~~~~~~~~~
/project/src/dhtrunner.cpp: In member function ‘void dht::DhtRunner::setProxyServer(const string&, const string&)’:
/project/src/dhtrunner.cpp:1080:72: warning: unused parameter ‘pushNodeId’ [-Wunused-parameter]
 1080 | DhtRunner::setProxyServer(const std::string& proxy, const std::string& pushNodeId)
      |                                                     ~~~~~~~~~~~~~~~~~~~^~~~~~~~~~
Traceback (most recent call last):
  File "/project/python/setup.py", line 26, in <module>
    setup(name="opendht",
  File "/opt/python/cp39-cp39/lib/python3.9/site-packages/setuptools/__init__.py", line 87, in setup
    return distutils.core.setup(**attrs)
  File "/opt/python/cp39-cp39/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 148, in setup
    return run_commands(dist)
  File "/opt/python/cp39-cp39/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 163, in run_commands
    dist.run_commands()
  File "/opt/python/cp39-cp39/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 967, in run_commands
    self.run_command(cmd)
  File "/opt/python/cp39-cp39/lib/python3.9/site-packages/setuptools/dist.py", line 1229, in run_command
    super().run_command(command)
  File "/opt/python/cp39-cp39/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 985, in run_command
    cmd_obj.ensure_finalized()
  File "/opt/python/cp39-cp39/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 107, in ensure_finalized
    self.finalize_options()
  File "/opt/python/cp39-cp39/lib/python3.9/site-packages/setuptools/_distutils/command/build.py", line 106, in finalize_options
    if self.distribution.has_ext_modules():
  File "/opt/python/cp39-cp39/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 995, in has_ext_modules
    return self.ext_modules and len(self.ext_modules) > 0
TypeError: object of type 'Extension' has no len()
make[2]: *** [python/CMakeFiles/python.dir/build.make:72: python/CMakeFiles/python] Error 1
make[1]: *** [CMakeFiles/Makefile2:300: python/CMakeFiles/python.dir/all] Error 2
make: *** [Makefile:136: all] Error 2
Collecting cython
  Downloading Cython-0.29.30-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (2.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 19.2 MB/s eta 0:00:00
Installing collected packages: cython
Successfully installed cython-0.29.30
-- The C compiler identification is GNU 11.2.1
-- The CXX compiler identification is GNU 11.2.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/rh/gcc-toolset-11/root/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /opt/rh/gcc-toolset-11/root/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found PkgConfig: /usr/bin/pkg-config (found version "1.4.2")
-- Performing Test HAVE_CXX_ATOMICS_WITHOUT_LIB
-- Performing Test HAVE_CXX_ATOMICS_WITHOUT_LIB - Success
-- Performing Test HAVE_CXX_ATOMICS64_WITHOUT_LIB
-- Performing Test HAVE_CXX_ATOMICS64_WITHOUT_LIB - Success
-- Performing Test LLVM_HAS_ATOMICS
-- Performing Test LLVM_HAS_ATOMICS - Success
-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE)
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Found Threads: TRUE
-- Found GnuTLS: /usr/lib64/libgnutls.so (found suitable version "3.6.16", minimum required is "3.3")
-- Checking for one of the modules 'nettle'
-- Found Readline: /usr/lib64/libreadline.so (Required is at least version "6")
-- Checking for one of the modules 'libargon2'
-- Checking for one of the modules 'jsoncpp'
-- Configuring done
-- Generating done
-- Build files have been written to: /project
[  2%] Building CXX object CMakeFiles/opendht-static.dir/src/utils.cpp.o
[  4%] Building CXX object CMakeFiles/opendht-static.dir/src/infohash.cpp.o
[  6%] Building CXX object CMakeFiles/opendht-static.dir/src/crypto.cpp.o
[  8%] Building CXX object CMakeFiles/opendht-static.dir/src/default_types.cpp.o
[ 10%] Building CXX object CMakeFiles/opendht-static.dir/src/node.cpp.o
[ 12%] Building CXX object CMakeFiles/opendht-static.dir/src/value.cpp.o
[ 14%] Building CXX object CMakeFiles/opendht-static.dir/src/dht.cpp.o
[ 16%] Building CXX object CMakeFiles/opendht-static.dir/src/op_cache.cpp.o
[ 18%] Building CXX object CMakeFiles/opendht-static.dir/src/callbacks.cpp.o
[ 20%] Building CXX object CMakeFiles/opendht-static.dir/src/routing_table.cpp.o
[ 22%] Building CXX object CMakeFiles/opendht-static.dir/src/node_cache.cpp.o
[ 25%] Building CXX object CMakeFiles/opendht-static.dir/src/network_engine.cpp.o
[ 27%] Building CXX object CMakeFiles/opendht-static.dir/src/securedht.cpp.o
[ 29%] Building CXX object CMakeFiles/opendht-static.dir/src/dhtrunner.cpp.o
[ 31%] Building CXX object CMakeFiles/opendht-static.dir/src/log.cpp.o
[ 33%] Building CXX object CMakeFiles/opendht-static.dir/src/network_utils.cpp.o
[ 35%] Building CXX object CMakeFiles/opendht-static.dir/src/thread_pool.cpp.o
[ 37%] Building CXX object CMakeFiles/opendht-static.dir/src/peer_discovery.cpp.o
[ 39%] Building CXX object CMakeFiles/opendht-static.dir/src/indexation/pht.cpp.o
[ 41%] Linking CXX static library libopendht.a
[ 41%] Built target opendht-static
[ 43%] Building CXX object CMakeFiles/opendht.dir/src/utils.cpp.o
[ 45%] Building CXX object CMakeFiles/opendht.dir/src/infohash.cpp.o
[ 47%] Building CXX object CMakeFiles/opendht.dir/src/crypto.cpp.o
[ 50%] Building CXX object CMakeFiles/opendht.dir/src/default_types.cpp.o
[ 52%] Building CXX object CMakeFiles/opendht.dir/src/node.cpp.o
[ 54%] Building CXX object CMakeFiles/opendht.dir/src/value.cpp.o
[ 56%] Building CXX object CMakeFiles/opendht.dir/src/dht.cpp.o
[ 58%] Building CXX object CMakeFiles/opendht.dir/src/op_cache.cpp.o
[ 60%] Building CXX object CMakeFiles/opendht.dir/src/callbacks.cpp.o
[ 62%] Building CXX object CMakeFiles/opendht.dir/src/routing_table.cpp.o
[ 64%] Building CXX object CMakeFiles/opendht.dir/src/node_cache.cpp.o
[ 66%] Building CXX object CMakeFiles/opendht.dir/src/network_engine.cpp.o
[ 68%] Building CXX object CMakeFiles/opendht.dir/src/securedht.cpp.o
[ 70%] Building CXX object CMakeFiles/opendht.dir/src/dhtrunner.cpp.o
[ 72%] Building CXX object CMakeFiles/opendht.dir/src/log.cpp.o
[ 75%] Building CXX object CMakeFiles/opendht.dir/src/network_utils.cpp.o
[ 77%] Building CXX object CMakeFiles/opendht.dir/src/thread_pool.cpp.o
[ 79%] Building CXX object CMakeFiles/opendht.dir/src/peer_discovery.cpp.o
[ 81%] Building CXX object CMakeFiles/opendht.dir/src/indexation/pht.cpp.o
[ 83%] Linking CXX shared library libopendht.so
[ 83%] Built target opendht
[ 85%] Building CXX object tools/CMakeFiles/dhtnode.dir/dhtnode.cpp.o
[ 87%] Linking CXX executable dhtnode
[ 87%] Built target dhtnode
[ 89%] Building CXX object tools/CMakeFiles/dhtscanner.dir/dhtscanner.cpp.o
[ 91%] Linking CXX executable dhtscanner
[ 91%] Built target dhtscanner
[ 93%] Building CXX object tools/CMakeFiles/dhtchat.dir/dhtchat.cpp.o
[ 95%] Linking CXX executable dhtchat
[ 95%] Built target dhtchat
[ 97%] Building CXX object tools/CMakeFiles/perftest.dir/perftest.cpp.o
[100%] Linking CXX executable perftest
[100%] Built target perftest
running build

                                                            ✕ 163.52s
```
Hm. Yeah. I have recreated this locally, here is the MWE:

```bash
python -m test.test_projects test.test_0_basic.basic_project /tmp/basic_project
(cd /tmp/basic_project && CIBW_BEFORE_BUILD="python -uc 'import sys; print(1, file=sys.stdout); print(2, file=sys.stderr)'" cibuildwheel --platform linux)
```

In the printout, you see:

```
Running before_build...

    + sh -c 'python -uc '"'"'import sys; print(1, file=sys.stdout); print(2, file=sys.stderr)'"'"''
2
1
```

Whereas running that command locally:

```bash
sh -c 'python -uc '"'"'import sys; print(1, file=sys.stdout); print(2, file=sys.stderr)'"'"''
1
2
```

I'm not sure why we see this behaviour, tbh. We _do_ line-buffer stdout (while stderr passes through untouched), but even with a 1 second delay between the print statements the behaviour is the same. So it's not about how fast we do it. Line-buffering shouldn't wait for the end of the command before printing the output, but it looks like that's what is happening...

Resolving patch:
diff --git a/cibuildwheel/oci_container.py b/cibuildwheel/oci_container.py
--- a/cibuildwheel/oci_container.py
+++ b/cibuildwheel/oci_container.py
@@ -303,9 +303,11 @@ def call(
                 return_code = int(return_code_str)
                 # add the last line to output, without the footer
                 output_io.write(line[0:footer_offset])
+                output_io.flush()
                 break
             else:
                 output_io.write(line)
+                output_io.flush()
 
         if isinstance(output_io, io.BytesIO):
             output = str(output_io.getvalue(), encoding="utf8", errors="surrogateescape")


Test patch: