Repository: numpy/numpy

Issue:
BUG: f2py shell behavior
Creating an issue / bookmark for the recent report from @embray in #11937, related to removal of `exec_command()` from `f2py` machinery, which reads:

> Unfortunately this seems to have broken the previously documented assumption that f2py would be called through a shell. In SageMath, this fact was used by using shell I/O redirection (e.g. > <filename>) to capture standard I/O from the f2py call. This have passed shell=True to subprocess.check_output in order to preserve that behavior, unless breaking it was intentional. This was also used to capture stderr, which I think the current code does not do.

> Also discussed at https://trac.sagemath.org/ticket/27000#comment:10


Discussion:
Should be careful to make sure that any fix applied for this does not interfere with the original objective of the set of changes, which was related to another shell configuration matter raised by some NASA engineers.
Thank you for creating the issue.  I've already worked around it in sage by simply not using `f2py.compile()` in the first place, and making my own `subprocess.Popen` with the appropriate error capture.  Perhaps bizarre shell issues like #11824 are indeed better avoided by not using `shell=True`, which will always carry some danger like that.  

In [my workaround](https://gitlab.com/sagemath/sage/blob/develop/src/sage/misc/inline_fortran.py#L179) I just made it possible to capture stderr so that I could include it in any exception messages.  The only reason `shell=True` was needed in the first place was as a hack to capture the stdout and stderr, and really it shouldn't be needed in the first place.
Since the original reporter has a work-around, can we close this?
Yeah ok
@mattip @tylerjereddy please reopen.  Just because I have a workaround doesn't mean it's a good one or there isn't something to fix here.  My workaround would essentially form a basis for a patch to make this a little better (namely making it easier to get stderr from the FORTRAN call--currently this is now impossible with f2py.compile.

I had to implement a workaround *because* there was a regression and that regression has not been addressed.
Looking at your [changes](https://git.sagemath.org/sage.git/commit/?h=81cc6bf6686103a9f4717ac99913f447f90c1821), it seems `f2py.compile(x, name, ...)` used to write to a `name.log` file but no longer does? I don't see that documented in the [`f2py.compile` documentation](http://www.numpy.org/devdocs/f2py/usage.html?highlight=f2py%20compile#numpy.f2py.compile). The warning at the top of the  docstring does state the interface can change. What if we add an option to return a tuple of `(status, stdout, stderr)` from `compile`?
@charris should we remove this from 1.16.1? I don't think this should block the bug-fix release, as it seems a deeper analysis is needed.
@mattip  Sounds reasonable, I'll move it to 1.16.2.
Off to 1.19 this goes. Be nice to resolve it.@embray do you have a suggestion for a fix?
It appears the original link to my alternative broken [so here it is again](https://gitlab.com/sagemath/sage/blob/develop/src/sage/misc/inline_fortran.py#L179).

If I recall correctly, the original code was calling `subprocess.Popen` with `shell=True`, and using non-portable shell constructs for I/O redirection, rather than using `Popen`'s built-in capabilities for that.

I think I like @mattip's suggestion to optionally return a `(status, stdout, stderr)` tuple, and maybe also include an option to disable writing to the log file.  Since the log file feature is also not even documented (and a bit superfluous) I might also leave that as the default behavior, but deprecate it.

The current behavior I think is probably just based on some very old code.
> What if we add an option to return a tuple of (status, stdout, stderr) from compile?

Rather than inventing our own return format, why not return a `subprocess.CompletedProcess` , by using `subprocess.run` instead (assuming we only target 3.x with the eventual fix here)

Again, we could hide this behind a flag.
It's not such an invention; such a tuple format is fairly common in other contexts.  But I certainly have no problem with using `subprocess.run` and returning `CompletedProcess`.  I think the important point here is there's nothing much fancy going on here other than running f2py in a subprocess with some arguments :)
@embray do you have any interest in making a PR with the proposed changes?
Okay, I will offer a PR.

Resolving patch:
diff --git a/numpy/f2py/__init__.py b/numpy/f2py/__init__.py
--- a/numpy/f2py/__init__.py
+++ b/numpy/f2py/__init__.py
@@ -21,7 +21,8 @@ def compile(source,
             extra_args='',
             verbose=True,
             source_fn=None,
-            extension='.f'
+            extension='.f',
+            full_output=False
            ):
     """
     Build extension module from a Fortran 77 source string with f2py.
@@ -55,10 +56,19 @@ def compile(source,
 
         .. versionadded:: 1.11.0
 
+    full_output : bool, optional
+        If True, return a `subprocess.CompletedProcess` containing
+        the stdout and stderr of the compile process, instead of just
+        the status code.
+
+        .. versionadded:: 1.20.0
+
+
     Returns
     -------
-    result : int
-        0 on success
+    result : int or `subprocess.CompletedProcess`
+        0 on success, or a `subprocess.CompletedProcess` if
+        ``full_output=True``
 
     Examples
     --------
@@ -95,23 +105,22 @@ def compile(source,
              '-c',
              'import numpy.f2py as f2py2e;f2py2e.main()'] + args
         try:
-            output = subprocess.check_output(c)
-        except subprocess.CalledProcessError as exc:
-            status = exc.returncode
-            output = ''
+            cp = subprocess.run(c, stdout=subprocess.PIPE,
+                                   stderr=subprocess.PIPE)
         except OSError:
             # preserve historic status code used by exec_command()
-            status = 127
-            output = ''
+            cp = subprocess.CompletedProcess(c, 127, stdout=b'', stderr=b'')
         else:
-            status = 0
-            output = output.decode()
-        if verbose:
-            print(output)
+            if verbose:
+                print(cp.stdout.decode())
     finally:
         if source_fn is None:
             os.remove(fname)
-    return status
+
+    if full_output:
+        return cp
+    else:
+        return cp.returncode
 
 from numpy._pytesttester import PytestTester
 test = PytestTester(__name__)


Test patch: