Repository: numpy/numpy

Issue:
ENH: Check for buggy Accelerate on MacOS
I recently began, digging deep into numpy, with the intention of contributing to the project. On my Mac development machine, I encountered test failures when I ran tests, on the master commit build (Note: The build had succeeded). The same commit was fine on a linux machine, and the azure pipeline mac build indicated it was passing, digging through issues and PR found two related issues #14644 and #15154 

### Machine/Environment Information

```
Operating System: MacOS Mojave (10.14.6)
C Compiler : Apple LLVM (clang-1001.0.46.4)
Python Version : 3.7.4
Accelerate  and OpenBLAS available on Mac OS
```

### Reproducing code example:


```bash
$ git clone https://github.com/<numpy-fork>/numpy
$ pip install -r test_requirements.txt
$ pip install . <or> python setup.py install
$ python runtests.py -v -m full 
# The build picks up Acclerate API for LAPACK and BLAS
```

<!-- Remove these sections for a feature request -->

### Error message:
The failing tests include, the following

```bash
numpy/lib/tests/test_mixins.py ...........                                                                                                                                                             [ 56%]
numpy/lib/tests/test_nanfunctions.py ...........................................................................................                                                                       [ 56%]
numpy/lib/tests/test_packbits.py ..................................................................................................................................................................... [ 58%]
...................................................................................................................                                                                                    [ 59%]
numpy/lib/tests/test_polynomial.py ........F........                                                                                                                                                   [ 59%]
numpy/lib/tests/test_recfunctions.py .........................................x........                                                                                                                [ 60%]
numpy/lib/tests/test_regression.py ............F..............                                                                                                                                         [ 60%]
numpy/lib/tests/test_shape_base.py ..................................................................                                                                                                  [ 60%]
numpy/lib/tests/test_stride_tricks.py ................                                                                                                                                                 [ 61%]
numpy/lib/tests/test_twodim_base.py ..................................                                                                                                                                 [ 61%]
numpy/lib/tests/test_type_check.py ....................................................                                                                                                                [ 61%]
numpy/lib/tests/test_ufunclike.py ......                                                                                                                                                               [ 61%]
numpy/lib/tests/test_utils.py ............                                                                                                                                                             [ 62%]
numpy/linalg/tests/test_build.py s                                                                                                                                                                     [ 62%]
numpy/linalg/tests/test_deprecations.py .                                                                                                                                                              [ 62%]
numpy/linalg/tests/test_linalg.py ....................................................................................FFFFFF..FFF..................................................................... [ 63%]
........................................................................................................................................s.................xx                                           [ 65%]
numpy/linalg/tests/test_regression.py .........                                                                                                                                                        [ 65%]
numpy/ma/tests/test_core.py .......................................................................................................................................................................... [ 66%]
...................................................................................................................................................................................................... [ 68%]
...................................................................................................................................................................................................... [ 70%]
...................................................................................................................................................................................................... [ 72%]
...................................................................................................................................................................................................... [ 73%]
...................................................................................................................................................................................................... [ 75%]
...................................................................................................................................................................................................... [ 77%]
.....................................................................xx............................................................................................................................... [ 79%]
..............................................................................................................................................................................                         [ 81%]
numpy/ma/tests/test_deprecations.py ......                                                                                                                                                             [ 81%]
numpy/ma/tests/test_extras.py ..............................................................FF.....................                                                                                    [ 81%]
numpy/ma/tests/test_mrecords.py ........................                                                                                                                                               [ 82%]
numpy/ma/tests/test_old_ma.py ......................................                                                                                                                                   [ 82%]
numpy/ma/tests/test_regression.py .............                                                                                                                                                        [ 82%]
numpy/ma/tests/test_subclassing.py ............                                                                                                                                                        [ 82%]
numpy/matrixlib/tests/test_defmatrix.py ...........................................                                                                                                                    [ 83%]
numpy/matrixlib/tests/test_interaction.py .......................                                                                                                                                      [ 83%]
numpy/matrixlib/tests/test_masked_matrix.py ..............                                                                                                                                             [ 83%]
numpy/matrixlib/tests/test_matrix_linalg.py ....................................FF......................                                                                                               [ 83%]
numpy/matrixlib/tests/test_multiarray.py ..                                                                                                                                                            [ 83%]
numpy/matrixlib/tests/test_numeric.py ..                                                                                                                                                               [ 84%]
numpy/matrixlib/tests/test_regression.py ....                                                                                                                                                          [ 84%]
numpy/polynomial/tests/test_chebyshev.py ........................F................                                                                                                                     [ 84%]
numpy/polynomial/tests/test_classes.py ................................................................................................FFFFFF......................................................... [ 85%]
.................................................................................                                                                                                                      [ 86%]
numpy/polynomial/tests/test_hermite.py ......................F...........                                                                                                                              [ 86%]
numpy/polynomial/tests/test_hermite_e.py ......................F...........                                                                                                                            [ 87%]
numpy/polynomial/tests/test_laguerre.py ......................F...........                                                                                                                             [ 87%]
numpy/polynomial/tests/test_legendre.py ......................F...........                                                                                                                             [ 87%]
numpy/polynomial/tests/test_polynomial.py ............................F..                                                                                                                              [ 88%]
numpy/polynomial/tests/test_polyutils.py ......                                                                                                                                                        [ 88%]
numpy/polynomial/tests/test_printing.py ............                                                                                                                                                   [ 88%]
numpy/random/tests/test_direct.py .......................................................................                                                                                              [ 88%]
numpy/random/tests/test_extending.py .s.                                                                                                                                                               [ 89%]
numpy/random/tests/test_generator_mt19937.py ......................................................................................................................................................... [ 90%]
..................................................................                                                                                                                                     [ 91%]
numpy/random/tests/test_generator_mt19937_regressions.py .............                                                                                                                                 [ 91%]
numpy/random/tests/test_random.py ................................................................................................................................                                     [ 92%]
numpy/random/tests/test_randomstate.py ............................................................................................................................................................... [ 93%]
......                                                                                                                                                                                                 [ 93%]
numpy/random/tests/test_randomstate_regression.py ..................                                                                                                                                   [ 94%]
numpy/random/tests/test_regression.py ............                                                                                                                                                     [ 94%]
numpy/random/tests/test_seed_sequence.py .                                                                                                                                                             [ 94%]
numpy/random/tests/test_smoke.py .s................................................................................................................................................................... [ 95%]
....................................................................................................ss................................................................................................ [ 97%]
..............................................................................................                                                                                                         [ 98%]
numpy/testing/tests/test_decorators.py ........                                                                                                                                                        [ 98%]
numpy/testing/tests/test_utils.py ..................................................................................sss........................................                                        [ 99%]
numpy/tests/test_ctypeslib.py .......................                                                                                                                                                  [ 99%]
numpy/tests/test_matlib.py ........                                                                                                                                                                    [ 99%]
numpy/tests/test_numpy_version.py .                                                                                                                                                                    [ 99%]
numpy/tests/test_public_api.py ..........                                                                                                                                                              [ 99%]
numpy/tests/test_reloading.py ..                                                                                                                                                                       [ 99%]
numpy/tests/test_scripts.py XXX.                                                                                                                                                                       [ 99%]
numpy/tests/test_warnings.py .                                                                                                                                                                         [100%]

================================================================================================== FAILURES ============
```

Some test errors include

```bash
numpy/lib/polynomial.py:635: RankWarning
E           numpy.RankWarning: Polyfit may be poorly conditioned

```

```bash
numpy/testing/_private/utils.py:839: AssertionError
>               raise AssertionError(msg)
E               AssertionError: 
E               Arrays are not almost equal to 6 decimals
E               
E               Mismatched elements: 2 / 2 (100%)
E               Max absolute difference: 2.
E               Max relative difference: inf
E                x: array([2., 1.], dtype=float32)
E                y: array([-0., -0.], dtype=float32)
```

### Commit at the time of filing this issue

https://github.com/numpy/numpy/commit/3cdc3ffb779733e7d9136ecb66a22bce31021c6b

### What resolved this issue

As suggested in #14644 , forcing use of openblas via site.cfg resolves the test failures.
 Mac does not provide openblas under /usr/local/lib/, instead it is under /usr/local/opt/openblas/lib

We need to update docs to reflect this issue, or by default look for openblas under additional locations in Mac

Installation [document](https://github.com/rgommers/numpy/blob/e7002a68f529eb6d275f6c0e2ec9b19443da5ad5/INSTALL.rst.txt) briefly mentions, Accelerate being buggy. However, [Building from Source](https://numpy.org/devdocs/user/building.html) provides no insight into this.



I am ready and willing to contribute back by adding  a document warning , or mac build improvement (look for openblas in additional locations), as per the feedback from  the members of the community. Long term, I feel dropping Accelerate support for Macs, seems the right way forward(#14880 ).




Discussion:
Welcome. As per [this comment](https://github.com/numpy/numpy/pull/14880#issuecomment-568437644) on gh-14880, we have not figured out how to disable or even identify that Accelerate is being used. That PR did not achieve its purpose, so was closed. I am a bit sceptical that a warning or note in the documentation will prevent use of Accelerate, we need a more fool-proof way (until the world invents better fools) of blocking it. 

One thing we did not try is creating a test that would reliably identify a buggy lapack implementation, run that test at import, and raise an error if it fails. If the test is fast and doesn't use much memory we may be able to justify it.
I see, the default blas/lapack implementation, in Mac, when openblas isn't found, is Accelerate (via symlink). That makes it tricky indeed. The testing methodology seems a good idea. Let me look into that possibility. Thanks for the feedback and pointers @mattip 
I looked at possible ways to test LAPACK,
Polyfit (one of the earliest test failures mentioned above), calls init_dgseld, which fails, with Accelerate
I framed the initial code which we can add under _sanity_check() in __init__.py

```python
    def _mac_os_check():
        """
        Quick Sanity check for Mac OS look for accelerate build bugs. Testing numpy polyfit calls init_dgelsd(LAPACK)
        """
        try:
            c = array([3., 2., 1.])
            x = linspace(0, 2, 7)
            y = polyval(c, x)
            err = [1, -1, 1, -1, 1, -1, 1]
            m, cov = polyfit(x, y + err, 2, cov=True)
        except ValueError:
            pass

    import platform
    if platform.system() == "Darwin":
        with warnings.catch_warnings(record=True) as w:
            _mac_os_check()
            assert issubclass(w[-1].category, RankWarning)
            msg = ("Polyfit sanity test failed.\nRankWarning:" +
                   str(w[-1].message) +
                   "\nThis is a known Accelerate issue.(https://github.com/numpy/numpy/issues/15647)" +
                   "\nBuild with OpenBLAS or MKL look at site.cfg.example for information")
            raise RuntimeError(msg.format(__file__))

    del _mac_os_check

```

Tested this with accelerate, with default blas/lapack implementation (again symlinked to accelerate). On MacOS Mojave, High Sierra, the test seems to catch the issue.

It seems  to run quick.

Am I in the right direction?

Thoughts @mattip ??
looks to be in the right direction. This should have a success path, it seems to only be able to fail. Refresh my memory: if used with Accelerate, the polyfit call will emit a `RankWarning`? Then you should check `len(w)` to make sure a warning was emitted.

@WarrenWeckesser or @tylerjereddy I think use macOS, maybe they can comment.
Polyfit emits a  rank warning, while init_dgelsv emits a malloc failure, in the background. Following which a ValueError is thrown by the function.

I agree len(w) should be checked as well to provide a success path

If this is the right way forward, would also like suggestions on improving the error message.
I guess we should do something like 
- open a PR with this test
- add a macOS run to azure.yml
  - does not download (via openblas_support) an OpenBLAS, 
  - checks that importing NumPy gives the desired error message
- make sure the test is operating properly with accelerate 
- add back in gh-14880

If you get the ball rolling by starting the PR I can help with the rest
Sounds, good I will start a PR today.
If you haven't already, the [developer guide](https://numpy.org/devdocs/dev/index.html#development-process-summary) is meant to help new contributors. If something isn't clear, or you can suggest improvements, please let us know.

Resolving patch:
diff --git a/numpy/__init__.py b/numpy/__init__.py
--- a/numpy/__init__.py
+++ b/numpy/__init__.py
@@ -254,3 +254,35 @@ def _sanity_check():
 
     _sanity_check()
     del _sanity_check
+
+    def _mac_os_check():
+        """
+        Quick Sanity check for Mac OS look for accelerate build bugs.
+        Testing numpy polyfit calls init_dgelsd(LAPACK)
+        """
+        try:
+            c = array([3., 2., 1.])
+            x = linspace(0, 2, 5)
+            y = polyval(c, x)
+            _ = polyfit(x, y, 2, cov=True)
+        except ValueError:
+            pass
+
+    import sys
+    if sys.platform == "darwin":
+        with warnings.catch_warnings(record=True) as w:
+            _mac_os_check()
+            # Throw runtime error, if the test failed Check for warning and error_message
+            error_message = ""
+            if len(w) > 0:
+                error_message = "{}: {}".format(w[-1].category.__name__, str(w[-1].message))
+                msg = (
+                    "Polyfit sanity test emitted a warning, most likely due "
+                    "to using a buggy Accelerate backend. "
+                    "If you compiled yourself, "
+                    "see site.cfg.example for information. "
+                    "Otherwise report this to the vendor "
+                    "that provided NumPy.\n{}\n".format(
+                        error_message))
+                raise RuntimeError(msg)
+    del _mac_os_check


Test patch: