Repository: numpy/numpy

Issue:
configuring numpy 1.16.5 fails on FreeBSD with OpenBlas
<!-- Please describe the issue in detail here, and fill in the fields below -->

This issue can also be found in the [FreeBSD bugzilla](https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=243497). 
There it was suggested to open an issue here.

TL;DR

* Problem: Compiling py37-numpy-1.16.5_3,1 with OpenBlas fails in numpy/distutils/system_info.py with an uncaught distutils.errors.LinkError exception.
* Solution: also catch distutils.errors.LinkError on line 1742 of numpy/distutils/system_info.py.

This is the error (only the last part shown for clarity).

    File "/usr/ports/math/py-numpy/work-py37/numpy-1.16.5/numpy/distutils/system_info.py", line 1740, in has_cblas
        extra_postargs=info.get('extra_link_args', []))
    File "/usr/local/lib/python3.7/distutils/ccompiler.py", line 734, in link_executable
        debug, extra_preargs, extra_postargs, None, target_lang)
    File "/usr/local/lib/python3.7/distutils/unixccompiler.py", line 206, in link
        raise LinkError(msg)
    distutils.errors.LinkError: Command "cc /tmp/tmpm3gnb0ev/tmp/tmpm3gnb0ev/source.o -L/usr/local/lib -lblas -o /tmp/tmpm3gnb0ev/a.out" failed with exit status 1
    *** Error code 1

    Stop.
    make[1]: stopped in /usr/ports/math/py-numpy
    *** Error code 1

    Stop.
    make: stopped in /usr/ports/math/py-numpy

This is the relevant code from system_info.py:

        try:
            with open(src, 'wt') as f:
                f.write(s)

            try:
                # check we can compile (find headers)
                obj = c.compile([src], output_dir=tmpdir,
                                include_dirs=self.get_include_dirs())

                # check we can link (find library)
                # some systems have separate cblas and blas libs. First
                # check for cblas lib, and if not present check for blas lib.
                try:
                    c.link_executable(obj, os.path.join(tmpdir, "a.out"),
                                      libraries=["cblas"],
                                      library_dirs=info['library_dirs'],
                                      extra_postargs=info.get('extra_link_args', []))
                    res = "cblas"
                except distutils.ccompiler.LinkError:
                    c.link_executable(obj, os.path.join(tmpdir, "a.out"),
                                      libraries=["blas"],
                                      library_dirs=info['library_dirs'],
                                      extra_postargs=info.get('extra_link_args', []))
                    res = "blas"
            except distutils.ccompiler.CompileError:
                res = None
        finally:
            shutil.rmtree(tmpdir)
        return res

If linking with “cblas” fails, it tries again by linking with “blas”.
However, the last “except” only handles a compilation error but not a linking error.

When we change:

            except distutils.ccompiler.CompileError:

to

            except (distutils.ccompiler.CompileError, distutils.ccompiler.LinkError):

the problem is fixed.

### Reproducing code example:

On a FreeBSD system with an up-to-date ports tree installed:

    cd /usr/ports/math/py-numpy
    make


<!-- Remove these sections for a feature request -->

### Error message:

<!-- If you are reporting a segfault please include a GDB traceback, which you
can generate by following
https://github.com/numpy/numpy/blob/master/doc/source/dev/development_environment.rst#debugging -->

    creating /tmp/tmpoquazeyp/tmp
    creating /tmp/tmpoquazeyp/tmp/tmpoquazeyp
    compile options: '-I/usr/include -I/usr/local/include -I/usr/local/include/suitesparse -c'
    cc: /tmp/tmpoquazeyp/source.c
    cc /tmp/tmpoquazeyp/tmp/tmpoquazeyp/source.o -L/usr/local/lib -lcblas -o /tmp/tmpoquazeyp/a.out
    ld: error: unable to find library -lcblas
    cc: error: linker command failed with exit code 1 (use -v to see invocation)
    cc /tmp/tmpoquazeyp/tmp/tmpoquazeyp/source.o -L/usr/local/lib -lblas -o /tmp/tmpoquazeyp/a.out
    ld: error: undefined symbol: cblas_ddot
    >>> referenced by source.c
    >>>               /tmp/tmpoquazeyp/tmp/tmpoquazeyp/source.o:(main)
    cc: error: linker command failed with exit code 1 (use -v to see invocation)
    Traceback (most recent call last):
    File "/usr/local/lib/python3.7/distutils/unixccompiler.py", line 204, in link
        self.spawn(linker + ld_args)
    File "/usr/ports/math/py-numpy/work-py37/numpy-1.16.5/numpy/distutils/ccompiler.py", line 92, in <lambda>
        m = lambda self, *args, **kw: func(self, *args, **kw)
    File "/usr/ports/math/py-numpy/work-py37/numpy-1.16.5/numpy/distutils/ccompiler.py", line 172, in CCompiler_spawn
        (cmd, s, msg))
    distutils.errors.DistutilsExecError: Command "cc /tmp/tmpoquazeyp/tmp/tmpoquazeyp/source.o -L/usr/local/lib -lcblas -o /tmp/tmpoquazeyp/a.out" failed with exit status 1

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
    File "/usr/ports/math/py-numpy/work-py37/numpy-1.16.5/numpy/distutils/system_info.py", line 1734, in has_cblas
        extra_postargs=info.get('extra_link_args', []))
    File "/usr/local/lib/python3.7/distutils/ccompiler.py", line 734, in link_executable
        debug, extra_preargs, extra_postargs, None, target_lang)
    File "/usr/local/lib/python3.7/distutils/unixccompiler.py", line 206, in link
        raise LinkError(msg)
    distutils.errors.LinkError: Command "cc /tmp/tmpoquazeyp/tmp/tmpoquazeyp/source.o -L/usr/local/lib -lcblas -o /tmp/tmpoquazeyp/a.out" failed with exit status 1

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
    File "/usr/local/lib/python3.7/distutils/unixccompiler.py", line 204, in link
        self.spawn(linker + ld_args)
    File "/usr/ports/math/py-numpy/work-py37/numpy-1.16.5/numpy/distutils/ccompiler.py", line 92, in <lambda>
        m = lambda self, *args, **kw: func(self, *args, **kw)
    File "/usr/ports/math/py-numpy/work-py37/numpy-1.16.5/numpy/distutils/ccompiler.py", line 172, in CCompiler_spawn
        (cmd, s, msg))
    distutils.errors.DistutilsExecError: Command "cc /tmp/tmpoquazeyp/tmp/tmpoquazeyp/source.o -L/usr/local/lib -lblas -o /tmp/tmpoquazeyp/a.out" failed with exit status 1

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "setup.py", line 419, in <module>
        setup_package()
    File "setup.py", line 411, in setup_package
        setup(**metadata)
    File "/usr/ports/math/py-numpy/work-py37/numpy-1.16.5/numpy/distutils/core.py", line 137, in setup
        config = configuration()
    File "setup.py", line 164, in configuration
        config.add_subpackage('numpy')
    File "/usr/ports/math/py-numpy/work-py37/numpy-1.16.5/numpy/distutils/misc_util.py", line 1036, in add_subpackage
        caller_level = 2)
    File "/usr/ports/math/py-numpy/work-py37/numpy-1.16.5/numpy/distutils/misc_util.py", line 1005, in get_subpackage
        caller_level = caller_level + 1)
    File "/usr/ports/math/py-numpy/work-py37/numpy-1.16.5/numpy/distutils/misc_util.py", line 942, in _get_configuration_from_setup_py
        config = setup_module.configuration(*args)
    File "numpy/setup.py", line 10, in configuration
        config.add_subpackage('core')
    File "/usr/ports/math/py-numpy/work-py37/numpy-1.16.5/numpy/distutils/misc_util.py", line 1036, in add_subpackage
        caller_level = 2)
    File "/usr/ports/math/py-numpy/work-py37/numpy-1.16.5/numpy/distutils/misc_util.py", line 1005, in get_subpackage
        caller_level = caller_level + 1)
    File "/usr/ports/math/py-numpy/work-py37/numpy-1.16.5/numpy/distutils/misc_util.py", line 942, in _get_configuration_from_setup_py
        config = setup_module.configuration(*args)
    File "numpy/core/setup.py", line 760, in configuration
        blas_info = get_info('blas_opt', 0)
    File "/usr/ports/math/py-numpy/work-py37/numpy-1.16.5/numpy/distutils/system_info.py", line 448, in get_info
        return cl().get_info(notfound_action)
    File "/usr/ports/math/py-numpy/work-py37/numpy-1.16.5/numpy/distutils/system_info.py", line 641, in get_info
        self.calc_info()
    File "/usr/ports/math/py-numpy/work-py37/numpy-1.16.5/numpy/distutils/system_info.py", line 1661, in calc_info
        blas_info = get_info('blas')
    File "/usr/ports/math/py-numpy/work-py37/numpy-1.16.5/numpy/distutils/system_info.py", line 448, in get_info
        return cl().get_info(notfound_action)
    File "/usr/ports/math/py-numpy/work-py37/numpy-1.16.5/numpy/distutils/system_info.py", line 641, in get_info
        self.calc_info()
    File "/usr/ports/math/py-numpy/work-py37/numpy-1.16.5/numpy/distutils/system_info.py", line 1697, in calc_info
        lib = self.has_cblas(info)
    File "/usr/ports/math/py-numpy/work-py37/numpy-1.16.5/numpy/distutils/system_info.py", line 1740, in has_cblas
        extra_postargs=info.get('extra_link_args', []))
    File "/usr/local/lib/python3.7/distutils/ccompiler.py", line 734, in link_executable
        debug, extra_preargs, extra_postargs, None, target_lang)
    File "/usr/local/lib/python3.7/distutils/unixccompiler.py", line 206, in link
        raise LinkError(msg)
    distutils.errors.LinkError: Command "cc /tmp/tmpoquazeyp/tmp/tmpoquazeyp/source.o -L/usr/local/lib -lblas -o /tmp/tmpoquazeyp/a.out" failed with exit status 1
    *** Error code 1

    Stop.
    make[1]: stopped in /usr/ports/math/py-numpy
    *** Error code 1

    Stop.
    make: stopped in /usr/ports/math/py-numpy


<!-- Full error message, if any (starting from line Traceback: ...) -->

### Numpy/Python version information:

<!-- Output from 'import sys, numpy; print(numpy.__version__, sys.version)' -->

>>> import sys, numpy; print(numpy.__version__, sys.version)
1.16.5 3.7.6 (default, Dec 23 2019, 18:53:23) 
[Clang 8.0.1 (tags/RELEASE_801/final 366581)]

(the above versions were generated with a fixed build of numpy.)


Discussion:
Seems reasonable. Would you like to open a PR? It should have the label "BLD: catch link error" or so.
Sure. Which branch whould I base the PR on? maintenance/1.16.x or master?
@rsmith-nl unless it only affects a release version, everything goes through master first and is then backported.

Resolving patch:
diff --git a/numpy/distutils/system_info.py b/numpy/distutils/system_info.py
--- a/numpy/distutils/system_info.py
+++ b/numpy/distutils/system_info.py
@@ -1742,7 +1742,7 @@ def has_cblas(self, info):
                                       library_dirs=info['library_dirs'],
                                       extra_postargs=info.get('extra_link_args', []))
                     res = "blas"
-            except distutils.ccompiler.CompileError:
+            except (distutils.ccompiler.CompileError, distutils.ccompiler.LinkError):
                 res = None
         finally:
             shutil.rmtree(tmpdir)


Test patch: