Repository: numpy/numpy
Pull request: TST: Mark check for f2py script xfail.
Context:

When running the tests via `python [ROOT DIR]/runtests.py` on windows linux subsystem a test fails with respect to f2py, f2py3, f2py3.6.

This is despite the fact that f2py exists in the my path, however I don't have f2py3 or f2py3.6 which shouldn't be required for the tests to pass.

closes gh-10016

Problem statement:
BUG: runtest.py always fails tests_scripts.text_f2py on windows
This is because `numpy.tests.test_scripts.test_f2py` tries to call `$python_dir/Scripts/f2py.py`, when in fact we're running in a (pseudo?) virtual env with its scripts at `$repo_dir/build/testenv/Scripts/f2py.py`.

There's a test there to disable this for `inplace` testing, but it's not clear what that means, and for whatever reason `runtests.py` is not considered "inplace".

Things have been like this for me since at least 1.12, but I figured I should create an issue for other people with the same problem

Hints from issue discussion:
@eric-wieser what is your setup?

I've just tested this on `Win7_x86` and both `python2.7.14` and `python3.6.2` and it seems to be working:

```windows
(C:\Users\IEUser\Miniconda2) C:\Users\IEUser\projects\numpy-master>python runtests.py -v -t numpy\tests\test_scripts.py
Building, see build.log...
    ... build in progress
    ... build in progress
Build OK
Running unit tests for numpy.tests.test_scripts.py
NumPy version 1.14.0.dev0+Unknown
NumPy relaxed strides checking option: True
NumPy is installed in C:\Users\IEUser\projects\numpy-master\build\testenv\Lib\si
te-packages\numpy
Python version 2.7.14 |Anaconda, Inc.| (default, Oct 15 2017, 06:26:11) [MSC v.1
500 32 bit (Intel)]
nose version 1.3.7
numpy.tests.test_scripts.test_f2py ... ok

----------------------------------------------------------------------
Ran 1 test in 0.320s

OK

(C:\Users\IEUser\Miniconda2) C:\Users\IEUser\projects\numpy-master>
```
At a glance - I'm using a pip-installed python environment, not Miniconda like you are.
That test is very sensitive to the installation. I wonder a bit if we shouldn't just issue a warning and not fail.
Ah, that explains an issue I suddenly had (on a Debian system); I had finally decided that I might need to replace `python setup.py install` with `pip install .`, and have been having these annoying error messages since.
Would it make sense to add the install `bin` or `Scripts` directory to `PATH`. I did this already to make my build work. But I'm not sure how to implement it platform independent. For the `site-package` the `distutils.sysconfig.get_python_lib` function provides the necessary information. Is there something for the scripts as well?

Commit message: updating f2py to ensure that at least f2py base exists, and doesn't error out if multiple do not

Patch to review:
diff --git a/numpy/tests/test_scripts.py b/numpy/tests/test_scripts.py
index 26e3ea745326..e4f38a7586fb 100644
--- a/numpy/tests/test_scripts.py
+++ b/numpy/tests/test_scripts.py
@@ -9,6 +9,7 @@
 import pytest
 from os.path import join as pathjoin, isfile, dirname, basename
 from subprocess import Popen, PIPE
+import warnings
 
 import numpy as np
 from numpy.compat.py3k import basestring
@@ -92,7 +93,14 @@ def try_f2py_commands(cmds):
         version = sys.version_info
         major = str(version.major)
         minor = str(version.minor)
+        f2py_success = try_f2py_commands(('f2py',))
         f2py_cmds = ('f2py', 'f2py' + major, 'f2py' + major + '.' + minor)
-        success = try_f2py_commands(f2py_cmds)
-        msg = "Warning: not all of %s, %s, and %s are found in path" % f2py_cmds
-        assert_(success == 3, msg)
+        all_success = try_f2py_commands(f2py_cmds)
+        if all_success == 3:
+            return
+        elif f2py_success == 1:
+            msg = "f2py fails"
+            warnings.warn(msg)
+        else:
+            msg = "No {}, {}, {} found".format(f2py_cmds)
+            assert_(f2py_success > 0, msg)


Code review comments:
diff_hunk: @@ -92,7 +93,14 @@ def try_f2py_commands(cmds):
         version = sys.version_info
         major = str(version.major)
         minor = str(version.minor)
+        f2py_success = try_f2py_commands(('f2py',))
         f2py_cmds = ('f2py', 'f2py' + major, 'f2py' + major + '.' + minor)
-        success = try_f2py_commands(f2py_cmds)
-        msg = "Warning: not all of %s, %s, and %s are found in path" % f2py_cmds
-        assert_(success == 3, msg)
+        all_success = try_f2py_commands(f2py_cmds)
+        if all_success == 3:
+            return
+        elif f2py_success == 1:
+            msg = "f2py fails"
path: numpy/tests/test_scripts.py
text: @user1:
This doesn't look right.

@user1:
And the warning will raise an error in the development branch.

@user1:
There is something special about the windows linux subsystem as this test passes on appveyor. I don't know how to check for the linux subsystem, so it would be better to simply check for `f2py`, or maybe for the presence of at least one of the options, which is what the test used to do.

@user2:
@user1 it fails on multiple other systems, see gh-10016

I think this test does need to be simplified. The logic here is that if all 3 commands are not found, it checks just for `f2py` and if that is found then the test does not fail. I think that's the main thing to agree on first. The rest can be tweaked as needed.

@user3:
This test always fails for me on both WSL and Mac; mac also always fails `test_sdot_bug_8577` in my hands.  My development workflow is often to establish the baseline failure count expected for the full suite before running tests on a new feature branch. 

@user1:
Yep, I think just checking for `f2py` is the way to go. That used to fail for other systems also, but maybe it is better now that we are using entry points. @user3 Does that simplified test work for you?

@user3:
On my mac with this diff on master:
```diff
--- a/numpy/tests/test_scripts.py
+++ b/numpy/tests/test_scripts.py
@@ -92,7 +92,7 @@ def test_f2py():
         version = sys.version_info
         major = str(version.major)
         minor = str(version.minor)
-        f2py_cmds = ('f2py', 'f2py' + major, 'f2py' + major + '.' + minor)
+        f2py_cmds = ('f2py',)
         success = try_f2py_commands(f2py_cmds)
-        msg = "Warning: not all of %s, %s, and %s are found in path" % f2py_cmds
-        assert_(success == 3, msg)
+        msg = "Warning: %s not found in path" % f2py_cmds
+        assert_(success == 1, msg)
```

From `python runtests.py -t "numpy/tests/test_scripts.py" -- -s` I get a failure and `f2py` itself gives:
```
Traceback (most recent call last):
  File "/Users/treddy/miniconda3/envs/numpy_dev_py36/bin/f2py", line 11, in <module>
    load_entry_point('numpy==1.16.0.dev0+025bbdd', 'console_scripts', 'f2py')()
  File "/Users/treddy/miniconda3/envs/mda_dev_py3/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 565, in load_entry_point
  File "/Users/treddy/miniconda3/envs/mda_dev_py3/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 2598, in load_entry_point
  File "/Users/treddy/miniconda3/envs/mda_dev_py3/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 2258, in load
  File "/Users/treddy/miniconda3/envs/mda_dev_py3/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 2264, in resolve
  File "/Users/treddy/.local/lib/python3.6/site-packages/numpy-1.16.0.dev0+5f2a5ae-py3.6-macosx-10.9-x86_64.egg/numpy/f2py/__main__.py", line 4, in <module>
    from f2py2e import main
ModuleNotFoundError: No module named 'f2py2e'
```

The modified test and `f2py` itself can be made to work if I adjust my path on mac os to check the build path first, before site-packages, with: 
`export PATH=/Users/treddy/github_projects/numpy_linked_worktree/build/testenv/bin:$PATH`
I did something similar to get shippable builds all green.

Once the path search order was fixed, I went back and reversed the change to the unit test noted above and the original test passed on my mac, so...

I'm pretty sure @user4 previously commented somewhere about the difficulty of getting the `f2py` path stuff working just right in a portable fashion.