Repository: easybuilders/easybuild-framework

Issue:
run_cmd_qa does not capture full output
When WRF build fails as described in https://github.com/easybuilders/easybuild-easyconfigs/issues/12271 no output from the `configure` script is reported in the build log.

Here is the standard output obtained by executing the `configure` script manually:
```
checking for perl5... no
checking for perl... found /usr/bin/perl (perl)
Will use NETCDF in dir: /cvmfs/hc44rs.azure/centos/7.7/software/netCDF/4.7.4-gompi-2020a
Will use HDF5 in dir: /cvmfs/hc44rs.azure/centos/7.7/software/HDF5/1.10.6-gompi-2020a
Will use PHDF5 in dir: /cvmfs/hc44rs.azure/centos/7.7/software/HDF5/1.10.6-gompi-2020a

ERROR ERROR ERROR ERROR ERROR ERROR ERROR
'm4' utility not found! Can not configure.

If on an Ubuntu machine, use the command
         sudo apt-get install m4        
To download and install the 'm4' utility
```

while the standard error shows:
```
./configure: line 621: type: m4: not found
```


Discussion:
I'm looking into what's wrong with `run_cmd_qa` that it's not catching the output.

I've managed to reproduce the issue with a simple test script that basically does `echo hello` + `exit 1`, but I haven't figured a fix for the issue yet... `run_cmd_qa` isn't catching the output if the command completes very quickly, basically...
Incorrect flushing?

Resolving patch:
diff --git a/easybuild/tools/run.py b/easybuild/tools/run.py
--- a/easybuild/tools/run.py
+++ b/easybuild/tools/run.py
@@ -107,7 +107,10 @@ def get_output_from_process(proc, read_size=None, asynchronous=False):
     """
 
     if asynchronous:
-        output = asyncprocess.recv_some(proc)
+        # e=False is set to avoid raising an exception when command has completed;
+        # that's needed to ensure we get all output,
+        # see https://github.com/easybuilders/easybuild-framework/issues/3593
+        output = asyncprocess.recv_some(proc, e=False)
     elif read_size:
         output = proc.stdout.read(read_size)
     else:
@@ -411,6 +414,7 @@ def check_answers_list(answers):
         # - otherwise the stdout/stderr buffer gets filled and it all stops working
         try:
             out = get_output_from_process(proc, asynchronous=True)
+
             if cmd_log:
                 cmd_log.write(out)
             stdout_err += out


Test patch:
diff --git a/test/framework/run.py b/test/framework/run.py
--- a/test/framework/run.py
+++ b/test/framework/run.py
@@ -351,9 +351,27 @@ def test_run_cmd_qa_buffering(self):
         cmd += 'echo "Pick a number: "; read number; echo "Picked number: $number"'
         (out, ec) = run_cmd_qa(cmd, {'Pick a number: ': '42'}, log_all=True, maxhits=5)
 
+        self.assertEqual(ec, 0)
         regex = re.compile("Picked number: 42$")
         self.assertTrue(regex.search(out), "Pattern '%s' found in: %s" % (regex.pattern, out))
 
+        # also test with script run as interactive command that quickly exits with non-zero exit code;
+        # see https://github.com/easybuilders/easybuild-framework/issues/3593
+        script_txt = '\n'.join([
+            "#/bin/bash",
+            "echo 'Hello, I am about to exit'",
+            "echo 'ERROR: I failed' >&2",
+            "exit 1",
+        ])
+        script = os.path.join(self.test_prefix, 'test.sh')
+        write_file(script, script_txt)
+        adjust_permissions(script, stat.S_IXUSR)
+
+        out, ec = run_cmd_qa(script, {}, log_ok=False)
+
+        self.assertEqual(ec, 1)
+        self.assertEqual(out, "Hello, I am about to exit\nERROR: I failed\n")
+
     def test_run_cmd_qa_log_all(self):
         """Test run_cmd_qa with log_output enabled"""
         (out, ec) = run_cmd_qa("echo 'n: '; read n; seq 1 $n", {'n: ': '5'}, log_all=True)