Repository: conan-io/conan

Issue:
[feature] Flush stdout before executing commands
I run with `print_run_commands = True` 24/7, as does our CI server. I'm really happy the setting exists. Unfortunately, the output is often jumbled because of buffering. I think if you called `sys.stdout.flush()` and `sys.stderr.flush()` before running an external process, that would solve the issue. Probably? Here's an example of what I'm talking about

```
...
...
...
[13:38:41]	[100%] Linking CXX executable ../bin/iodb_read_write_test
[13:38:41]	[100%] Built target iodb_read_write_test
[13:38:41]	[100%] Built target iodb_read_write_test-mocked
[13:38:41]	Running tests...
[13:38:41]	Test project /home/captain/.conan/data/TagIO/1.0.4/wsbu/stable/build/4df8410dd1e5005908c5e4796a6e566752fe7a41
[13:38:41]	    Start 1: conversion_test-mocked
[13:38:41]	1/9 Test #1: conversion_test-mocked ......................   Passed    0.05 sec
[13:38:41]	    Start 2: iodb_address_ranges_test-mocked
[13:38:42]	2/9 Test #2: iodb_address_ranges_test-mocked .............   Passed    0.05 sec
[13:38:42]	    Start 3: tags_csv_file_change_checking_test-mocked
[13:38:43]	3/9 Test #3: tags_csv_file_change_checking_test-mocked ...***Failed    1.35 sec
[13:38:43]	    Start 4: back_end_utility_functions_test-mocked
[13:38:43]	4/9 Test #4: back_end_utility_functions_test-mocked ......   Passed    0.10 sec
[13:38:43]	    Start 5: iodb_read_write_test-mocked
[13:38:43]	5/9 Test #5: iodb_read_write_test-mocked .................   Passed    0.19 sec
[13:38:43]	    Start 6: overloaded_operators_test-mocked
[13:38:43]	6/9 Test #6: overloaded_operators_test-mocked ............   Passed    0.06 sec
[13:38:43]	    Start 7: tag_management_test-mocked
[13:38:47]	7/9 Test #7: tag_management_test-mocked ..................   Passed    3.86 sec
[13:38:47]	    Start 8: tagio_wrapper_test-mocked
[13:38:47]	8/9 Test #8: tagio_wrapper_test-mocked ...................   Passed    0.09 sec
[13:38:47]	    Start 9: tagio_cli_test-mocked
[13:38:48]	9/9 Test #9: tagio_cli_test-mocked .......................   Passed    0.53 sec
[13:38:48]	
[13:38:48]	89% tests passed, 1 tests failed out of 9
[13:38:48]	
[13:38:48]	Total Test time (real) =   6.28 sec
[13:38:48]	
[13:38:48]	The following tests FAILED:
[13:38:48]		  3 - tags_csv_file_change_checking_test-mocked (Failed)
[13:38:48]	Errors while running CTest
[13:38:48]	make: *** [test] Error 8
[13:38:48]	Makefile:129: recipe for target 'test' failed
[13:38:48]	
[13:38:48]	----Running------
[13:38:48]	> cd '/home/captain/.conan/data/TagIO/1.0.4/wsbu/stable/build/4df8410dd1e5005908c5e4796a6e566752fe7a41' && cmake -G "Unix Makefiles" -DCMAKE_BUILD_TYPE="Debug" -DCMAKE_TOOLCHAIN_FILE="/opt/toolchain-linaro-armhf.cmake" -DCONAN_EXPORTED="1" -DCONAN_IN_LOCAL_CACHE="ON" -DCONAN_COMPILER="gcc" -DCONAN_COMPILER_VERSION="4.9" -DCONAN_LIBCXX="libstdc++" -DBUILD_SHARED_LIBS="ON" -DCMAKE_INSTALL_PREFIX="/home/captain/.conan/data/TagIO/1.0.4/wsbu/stable/package/4df8410dd1e5005908c5e4796a6e566752fe7a41" -DCMAKE_INSTALL_BINDIR="bin" -DCMAKE_INSTALL_SBINDIR="bin" -DCMAKE_INSTALL_LIBEXECDIR="bin" -DCMAKE_INSTALL_LIBDIR="lib" -DCMAKE_INSTALL_INCLUDEDIR="include" -DCMAKE_INSTALL_OLDINCLUDEDIR="include" -DCMAKE_INSTALL_DATAROOTDIR="share" -DCMAKE_EXPORT_NO_PACKAGE_REGISTRY="ON" -Wno-dev '/home/captain/.conan/data/TagIO/1.0.4/wsbu/stable/build/4df8410dd1e5005908c5e4796a6e566752fe7a41'
[13:38:48]	-----------------
[13:38:48]	
[13:38:48]	----Running------
[13:38:48]	> cmake --build '/home/captain/.conan/data/TagIO/1.0.4/wsbu/stable/build/4df8410dd1e5005908c5e4796a6e566752fe7a41' '--' '-j8'
[13:38:48]	-----------------
[13:38:48]	
[13:38:48]	----Running------
[13:38:48]	> cmake --build '/home/captain/.conan/data/TagIO/1.0.4/wsbu/stable/build/4df8410dd1e5005908c5e4796a6e566752fe7a41' '--target' 'test'
[13:38:48]	-----------------
[13:38:48]	TagIO/1.0.4@wsbu/stable: 
[13:38:48]	TagIO/1.0.4@wsbu/stable: ERROR: Package '4df8410dd1e5005908c5e4796a6e566752fe7a41' build failed
[13:38:48]	TagIO/1.0.4@wsbu/stable: WARN: Build folder /home/captain/.conan/data/TagIO/1.0.4/wsbu/stable/build/4df8410dd1e5005908c5e4796a6e566752fe7a41
[13:38:48]	ERROR: TagIO/1.0.4@wsbu/stable: Error in build() method, line 35
[13:38:48]		cmake.test()
[13:38:48]		ConanException: Error 512 while executing cmake --build '/home/captain/.conan/data/TagIO/1.0.4/wsbu/stable/build/4df8410dd1e5005908c5e4796a6e566752fe7a41' '--target' 'test'
[13:38:48]	Process exited with code 1
```


Discussion:
Yes, you are right. I'm not sure if we can reproduce this behavior in a test to check that it is happening and then solve it, but I agree that flushing the buffers before calling an external command makes lot of sense.

Thanks for reporting it.
Let's start with the `runner.py` only. (the module running commands with `self.run` in a recipe)

Resolving patch:
diff --git a/conans/client/runner.py b/conans/client/runner.py
--- a/conans/client/runner.py
+++ b/conans/client/runner.py
@@ -11,6 +11,15 @@
 from conans.util.files import decode_text
 
 
+class _UnbufferedWrite(object):
+    def __init__(self, stream):
+        self._stream = stream
+
+    def write(self, *args, **kwargs):
+        self._stream.write(*args, **kwargs)
+        self._stream.flush()
+
+
 class ConanRunner(object):
 
     def __init__(self, print_commands_to_output=False, generate_run_log_file=False,
@@ -32,6 +41,9 @@ def __call__(self, command, output, log_filepath=None, cwd=None, subprocess=Fals
                   "use six.StringIO() instead ***")
 
         stream_output = output if output and hasattr(output, "write") else sys.stdout
+        if hasattr(output, "flush"):
+            # We do not want output from different streams to get mixed (sys.stdout, os.system)
+            stream_output = _UnbufferedWrite(stream_output)
 
         if not self._generate_run_log_file:
             log_filepath = None


Test patch: