Repository: easybuilders/easybuild-easyblocks

Issue:
also run numpy.test rather than only a simply numpy.dot performance test in EB_numpy.test_step
Note: this will result in a failed test with numpy built with Intel compilers, which is actually a broken test, see https://github.com/numpy/numpy/issues/3844#issuecomment-97910358

problem reported by @olavks



Discussion:
@pescobar: up for tackling this at some point?

With this easyconfig, at least some numpy tests are run, but the failures aren't detected:

```
name = 'numpy'
version = '1.15.0'
versionsuffix = '-Python-%(pyver)s'

homepage = 'http://www.numpy.org'
description = """NumPy is the fundamental package for scientific computing with Python. It contains among other things:
 a powerful N-dimensional array object, sophisticated (broadcasting) functions, tools for integrating C/C++ and Fortran
 code, useful linear algebra, Fourier transform, and random number capabilities. Besides its obvious scientific uses,
 NumPy can also be used as an efficient multi-dimensional container of generic data. Arbitrary data-types can be
 defined. This allows NumPy to seamlessly and speedily integrate with a wide variety of databases."""

toolchain = {'name': 'foss', 'version': '2018b'}

source_urls = [PYPI_SOURCE]
sources = [SOURCE_ZIP]
checksums = [
    'f28e73cf18d37a413f7d5de35d024e6b98f14566a10d82100f9dc491a7d449f9',  # numpy-1.15.0.zip
]

dependencies = [
    ('Python', '3.6.6'),
    ('pytest', '4.4.0', versionsuffix),
]

moduleclass = 'math'
```

Snippet of output:
```
== 2019-06-06 14:44:13,059 run.py:506 INFO cmd "export PYTHONPATH=/scratch/edmondac-admin/eb-7fJRj3/tmp744GE1/lib/python3.6/site-packages:$PYTHONPATH &&cd .. && /rds/bear-apps/devel/2018b/edmondac-eb-6/EL7/EL7-power9/software/Python/3.6.
6-foss-2018b/bin/python -c 'import numpy; numpy.test(verbose=2)'" exited with exit code 0 and output:
NumPy version 1.15.0
NumPy relaxed strides checking option: True
================================================= test session starts =================================================
platform linux -- Python 3.6.6, pytest-4.4.0, py-1.8.0, pluggy-0.9.0
rootdir: /dev/shm/build-edmondac-admin/edmondac-admin-6/numpy/1.15.0/foss-2018b-Python-3.6.6
collected 4973 items / 88 deselected / 4885 selected

compat/tests/test_compat.py .                                                                                   [  0%]
core/tests/test_abc.py .....                                                                                    [  0%]
core/tests/test_api.py .........                                                                                [  0%]
core/tests/test_arrayprint.py ..x.....F................................                                         [  1%]
core/tests/test_datetime.py ................................................................                    [  2%]

<snip>

tests/test_numpy_version.py .                                                                                   [ 99%]
tests/test_reloading.py ..                                                                                      [ 99%]
tests/test_scripts.py .                                                                                         [100%]

======================================================= ERRORS ========================================================
______________________________ ERROR at setup of TestBlockDocString.test_block_docstring ______________________________

<snip>

core/tests/test_umath.py::TestComplexFunctions::test_loss_of_precision_longcomplex
  /scratch/edmondac-admin/eb-7fJRj3/tmp744GE1/lib/python3.6/site-packages/numpy-1.15.0-py3.6-linux-ppc64le.egg/numpy/core/tests/test_umath.py:2514: RuntimeWarning: overflow encountered in arcsinh
    d = np.absolute(np.arcsinh(x)/np.arcsinh(z).real - 1)

-- Docs: https://docs.pytest.org/en/latest/warnings.html
=== 14 failed, 4837 passed, 21 skipped, 88 deselected, 7 xfailed, 2 xpassed, 13 warnings, 4 error in 84.94 seconds ====
```
It then just continues on to 
```
== 2019-06-06 14:44:13,072 run.py:518 WARNING Found 39 errors in command output

<snip>

== 2019-06-06 14:44:13,109 filetools.py:1297 INFO Path /scratch/edmondac-admin/eb-7fJRj3/tmp744GE1 successfully removed.
```

And finally installs "successfully".

(That was on POWER9)
On haswell, I get this - but otherwise the same behaviour:

```
= 4860 passed, 14 skipped, 88 deselected, 7 xfailed, 6 warnings, 4 error in 68.73 seconds =
```
@edmondac Judging from https://github.com/numpy/numpy/issues/13393, Power9 is pretty exotic for numpy, are you sure that the tests *should* pass without tweaking?
Given the age of this thread, it might be some time before we address this...
I agree - and that's sort of the problem here. The build succeeds, even though the tests failed - so I didn't know they'd failed...
> Given the age of this thread, it might be some time before we address this...

Sure - I was just directed here by @boegel so am commenting :-)
I had a look, the testcmd will never error (that's numpy's choice). The only thing we could do is override (further) the test step and parse the output to grab the error count from the summary. 
I should say that it seems we _do_ run numpy.test as this issue requests, but we then ignore the output.
If someone can help me with the regex I can try to implement it.
> If someone can help me with the regex I can try to implement it.

I'm happy to help if I can, just message me on Slack :-)
FYI this is exactly the same for scipy - tests run (if in standalone eb file and pytest 3.8.0 present) but errors are ignored. scipy result from my easybuild log for scipy 1.1.0 on haswell:
```
=== 6 failed, 13057 passed, 1262 skipped, 585 deselected, 86 xfailed, 6 xpassed, 332308 warnings in 439.73 seconds ====

== 2019-06-06 16:41:25,623 run.py:559 INFO parse_log_for_error msg: Command used: export PYTHONPATH=/scratch/edmondac-admin/eb-SMa1yq/tmpiJO5IM/lib/python3.6/site-packages:$PYTHONPATH &&cd .. && /rds/bear-apps/devel/2018b/edmondac-eb-6/EL7/EL7-haswell/software/Python/3.6.6-foss-2018b/bin/python -c 'import numpy; import scipy; scipy.test(verbose=2)'
```
This came to light because I installed `SciPy-bundle-2019.03-foss-2019a.eb` on POWER9 and it crashed out during the tests. This led us to wonder why it hadn't done that for 2018b, and it seems the tests weren't run when scipy was part of `Python-3.6.6-foss-2018b.eb`.

See https://github.com/scipy/scipy/issues/10256
Should be solved in #1744

Resolving patch:
diff --git a/easybuild/easyblocks/n/numpy.py b/easybuild/easyblocks/n/numpy.py
--- a/easybuild/easyblocks/n/numpy.py
+++ b/easybuild/easyblocks/n/numpy.py
@@ -66,7 +66,17 @@ def __init__(self, *args, **kwargs):
         self.sitecfg = None
         self.sitecfgfn = 'site.cfg'
         self.testinstall = True
-        self.testcmd = "cd .. && %(python)s -c 'import numpy; numpy.test(verbose=2)'"
+        if LooseVersion(self.version) >= LooseVersion('1.15'):
+            # Numpy 1.15+ returns a True on success. Hence invert to get a failure value
+            test_code = 'sys.exit(not numpy.test(verbose=2))'
+        else:
+            # Return value is a TextTestResult. Check the errors member for any error
+            test_code = 'sys.exit(len(numpy.test(verbose=2).errors) > 0)'
+        # Prepend imports
+        test_code = "import sys; import numpy; " + test_code
+        # LDFLAGS should not be set when testing numpy/scipy, because it overwrites whatever numpy/scipy sets
+        # see http://projects.scipy.org/numpy/ticket/182
+        self.testcmd = "unset LDFLAGS && cd .. && %%(python)s -c '%s'" % test_code
 
     def configure_step(self):
         """Configure numpy build by composing site.cfg contents."""


Test patch: