Repository: scikit-learn/scikit-learn
Pull request: [MRG] Remove BLAS infos from show_versions and build log
Fixes  #14096

- remove BLAS infos from `show_versions()`
- remove BLAS from the dependencies in advanced installations doc
- remove BLAS infos displayed in the build log



Problem statement:
scikit-learn 0.21.x not linking MKL BLAS
While building scikit-learn from sources with MKL BLAS support it is not linking to MKL libraries. Such behavior started from 0.21.0 release (0.21.2 checked also).

> user@pc:~/scikit-learn-0.21.0/build$ egrep -ir 'mkl_rt'
(empty)

> user@pc:~/scikit-learn-0.20.3/build/lib.linux-x86_64-3.6$ egrep -ir 'mkl_rt'
Binary file sklearn/cluster/_k_means.cpython-36m-x86_64-linux-gnu.so matches
Binary file sklearn/metrics/pairwise_fast.cpython-36m-x86_64-linux-gnu.so matches
Binary file sklearn/utils/weight_vector.cpython-36m-x86_64-linux-gnu.so matches
Binary file sklearn/utils/arrayfuncs.cpython-36m-x86_64-linux-gnu.so matches
Binary file sklearn/svm/liblinear.cpython-36m-x86_64-linux-gnu.so matches
Binary file sklearn/linear_model/cd_fast.cpython-36m-x86_64-linux-gnu.so matches

Checking with **ldd** confirms that libraries are not linked to mkl_rt. However both reports in the beginning of building process:

>   FOUND:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/opt/intel/mkl/lib/intel64']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/opt/intel/mkl/include']

>  FOUND:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/opt/intel/mkl/lib/intel64']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/opt/intel/mkl/include']


**sklearn.show_versions()** for both builds return MKL as BLAS:

> System:
    python: 3.6.8 (default, Jan 14 2019, 11:02:34)  [GCC 8.0.1 20180414 (experimental) [trunk revision 259383]]
executable: /usr/bin/python3
   machine: Linux-4.15.0-51-generic-x86_64-with-Ubuntu-18.04-bionic                                                                                                                                                                          
BLAS:                                                                                                                                                                                                                                        
    macros: SCIPY_MKL_H=None, HAVE_CBLAS=None                                                                                                                                                                                                
  lib_dirs: /opt/intel/mkl/lib/intel64                                                                                                                                                                                                       
cblas_libs: mkl_rt, pthread                                                                                                                                                                                                                  
Python deps:                                                                                                                                                                                                                                 
       pip: 19.1.1                                                                                                                                                                                                                           
setuptools: 41.0.1                                                                                                                                                                                                                           
   sklearn: **0.21.2**                                                                                                                                                                                                                           
     numpy: 1.16.4                                                                                                                                                                                                                           
     scipy: 1.3.0                                                                                                                                                                                                                            
    Cython: 0.29.10                                                                                                                                                                                                                          
    pandas: None

> System:
    python: 3.6.8 (default, Jan 14 2019, 11:02:34)  [GCC 8.0.1 20180414 (experimental) [trunk revision 259383]]
executable: /usr/bin/python3
   machine: Linux-4.15.0-51-generic-x86_64-with-Ubuntu-18.04-bionic
BLAS:
    macros: SCIPY_MKL_H=None, HAVE_CBLAS=None
  lib_dirs: /opt/intel/mkl/lib/intel64
cblas_libs: mkl_rt, pthread
Python deps:
       pip: 19.1.1
setuptools: 41.0.1
   sklearn: **0.20.3**
     numpy: 1.16.4
     scipy: 1.3.0
    Cython: 0.29.10
    pandas: None

Hints from issue discussion:
> While building scikit-learn from sources with MKL BLAS support it is not linking to MKL libraries

Since 0.21 scikit-learn is not linked to a BLAS at build time but uses the BLAS linked to scipy.

> However both reports in the beginning of building process

This report shows which BLAS would be linked if we wanted to link a BLAS. It is not useful any more and we should remove it as it can only bring confusion.

> sklearn.show_versions() for both builds return MKL as BLAS

The output is the same as the report at build time. We need to change that as well.
I see, thanks for clarifying this.
Just last question - so from 0.21 there is no point in building scikit-learn from source as builds distributed through pip will also implicitly use MKL if SciPy was compiled with it?
that's right (be careful that scipy installed with pip will be linked to OpenBLAS).
``
I think you can reopen the issue because the fact that we don't provide the right information in `show_version` is an issue.
Tagging as help wanted for contributor to modify `show_versions`
> Tagging as help wanted for contributor to modify show_versions

Do you mean to remove BLAS information from show_version? That only applies for scikit-learn 0.21+ I think? So then this should be tagged 0.21 not 0.20?

Also I realized that the removal of bundled BLAS https://github.com/scikit-learn/scikit-learn/pull/13203 doesn't have a what's new entry. We should add one to v0.21.0.
>  remove BLAS information from show_version

We can update show_version to display which BLAS is loaded. But we need threadpoolctl for that. Also we can have 2 BLAS at the same time. One from numpy and one from scipy, which can be different
We would need to wait until it is vendored for that. It's annoying to ask users to download a dependency when they submit an issue. Also it would not be vendored for 0.21.X, so there probably not showing that information would probably still be an improvement?

Commit message: revert tests modif

Patch to review:
diff --git a/README.rst b/README.rst
index a8188dba6d150..a409972f7b71f 100644
--- a/README.rst
+++ b/README.rst
@@ -58,13 +58,6 @@ scikit-learn 0.21 and later require Python 3.5 or newer.
 For running the examples Matplotlib >= 1.5.1 is required. A few examples
 require scikit-image >= 0.12.3, a few examples require pandas >= 0.18.0.
 
-scikit-learn also uses CBLAS, the C interface to the Basic Linear Algebra
-Subprograms library. scikit-learn comes with a reference implementation, but
-the system CBLAS will be detected by the build system and used if present.
-CBLAS exists in many implementations; see `Linear algebra libraries
-<http://scikit-learn.org/stable/modules/computing#linear-algebra-libraries>`_
-for known issues.
-
 User installation
 ~~~~~~~~~~~~~~~~~
 
diff --git a/doc/developers/advanced_installation.rst b/doc/developers/advanced_installation.rst
index 07668159e7208..3ffccc88b514d 100644
--- a/doc/developers/advanced_installation.rst
+++ b/doc/developers/advanced_installation.rst
@@ -193,55 +193,27 @@ Installing build dependencies
 Linux
 -----
 
-Installing from source requires you to have installed the scikit-learn runtime
-dependencies, Python development headers and a working C/C++ compiler.
-Under Debian-based operating systems, which include Ubuntu::
+Installing from source without conda requires you to have installed the
+scikit-learn runtime dependencies, Python development headers and a working
+C/C++ compiler. Under Debian-based operating systems, which include Ubuntu::
 
     sudo apt-get install build-essential python3-dev python3-setuptools \
-                     python3-numpy python3-scipy \
-                     libatlas-dev libatlas3-base
-
-On recent Debian and Ubuntu (e.g. Ubuntu 14.04 or later) make sure that ATLAS
-is used to provide the implementation of the BLAS and LAPACK linear algebra
-routines::
+                     python3-pip
+    
+and then::
 
-    sudo update-alternatives --set libblas.so.3 \
-        /usr/lib/atlas-base/atlas/libblas.so.3
-    sudo update-alternatives --set liblapack.so.3 \
-        /usr/lib/atlas-base/atlas/liblapack.so.3
+    pip3 install numpy scipy cython
 
 .. note::
 
     In order to build the documentation and run the example code contains in
     this documentation you will need matplotlib::
 
-        sudo apt-get install python-matplotlib
-
-.. note::
-
-    The above installs the ATLAS implementation of BLAS
-    (the Basic Linear Algebra Subprograms library).
-    Ubuntu 11.10 and later, and recent (testing) versions of Debian,
-    offer an alternative implementation called OpenBLAS.
-
-    Using OpenBLAS can give speedups in some scikit-learn modules,
-    but can freeze joblib/multiprocessing prior to OpenBLAS version 0.2.8-4,
-    so using it is not recommended unless you know what you're doing.
-
-    If you do want to use OpenBLAS, then replacing ATLAS only requires a couple
-    of commands. ATLAS has to be removed, otherwise NumPy may not work::
-
-        sudo apt-get remove libatlas3gf-base libatlas-dev
-        sudo apt-get install libopenblas-dev
-
-        sudo update-alternatives  --set libblas.so.3 \
-            /usr/lib/openblas-base/libopenblas.so.0
-        sudo update-alternatives --set liblapack.so.3 \
-            /usr/lib/lapack/liblapack.so.3
+        sudo apt-get install python3-matplotlib
 
 On Red Hat and clones (e.g. CentOS), install the dependencies using::
 
-    sudo yum -y install gcc gcc-c++ numpy python-devel scipy
+    sudo yum -y install gcc gcc-c++ python-devel numpy scipy
 
 
 Windows
diff --git a/sklearn/_build_utils/__init__.py b/sklearn/_build_utils/__init__.py
index 2d872dd378998..c47d12b8d6245 100644
--- a/sklearn/_build_utils/__init__.py
+++ b/sklearn/_build_utils/__init__.py
@@ -9,8 +9,6 @@
 
 from distutils.version import LooseVersion
 
-from numpy.distutils.system_info import get_info
-
 from .openmp_helpers import check_openmp_support
 
 
@@ -19,30 +17,6 @@
 CYTHON_MIN_VERSION = '0.28.5'
 
 
-def get_blas_info():
-    def atlas_not_found(blas_info_):
-        def_macros = blas_info.get('define_macros', [])
-        for x in def_macros:
-            if x[0] == "NO_ATLAS_INFO":
-                # if x[1] != 1 we should have lapack
-                # how do we do that now?
-                return True
-            if x[0] == "ATLAS_INFO":
-                if "None" in x[1]:
-                    # this one turned up on FreeBSD
-                    return True
-        return False
-
-    blas_info = get_info('blas_opt', 0)
-    if (not blas_info) or atlas_not_found(blas_info):
-        cblas_libs = ['cblas']
-        blas_info.pop('libraries', None)
-    else:
-        cblas_libs = blas_info.pop('libraries', [])
-
-    return cblas_libs, blas_info
-
-
 def build_from_c_and_cpp_files(extensions):
     """Modify the extensions to build from the .c and .cpp files.
 
diff --git a/sklearn/setup.py b/sklearn/setup.py
index 5a377043e9e38..53f6d3f6eb30c 100644
--- a/sklearn/setup.py
+++ b/sklearn/setup.py
@@ -5,12 +5,8 @@
 
 def configuration(parent_package='', top_path=None):
     from numpy.distutils.misc_util import Configuration
-    from numpy.distutils.system_info import get_info
     import numpy
 
-    # needs to be called during build otherwise show_version may fail sometimes
-    get_info('blas_opt', 0)
-
     libraries = []
     if os.name == 'posix':
         libraries.append('m')
diff --git a/sklearn/utils/_show_versions.py b/sklearn/utils/_show_versions.py
index ecba9ec68bea5..cfb7b92879303 100644
--- a/sklearn/utils/_show_versions.py
+++ b/sklearn/utils/_show_versions.py
@@ -8,6 +8,8 @@
 import platform
 import sys
 import importlib
+import numpy
+import scipy
 
 
 def _get_sys_info():
@@ -70,49 +72,22 @@ def get_version(module):
     return deps_info
 
 
-def _get_blas_info():
-    """Information on system BLAS
-
-    Uses the `scikit-learn` builtin method
-    :func:`sklearn._build_utils.get_blas_info` which may fail from time to time
-
-    Returns
-    -------
-    blas_info: dict
-        system BLAS information
-
-    """
-    from .._build_utils import get_blas_info
-
-    cblas_libs, blas_dict = get_blas_info()
-
-    macros = ['{key}={val}'.format(key=a, val=b)
-              for (a, b) in blas_dict.get('define_macros', [])]
-
-    blas_blob = [
-        ('macros', ', '.join(macros)),
-        ('lib_dirs', ':'.join(blas_dict.get('library_dirs', ''))),
-        ('cblas_libs', ', '.join(cblas_libs)),
-    ]
-
-    return dict(blas_blob)
-
-
 def show_versions():
     "Print useful debugging information"
 
     sys_info = _get_sys_info()
     deps_info = _get_deps_info()
-    blas_info = _get_blas_info()
 
     print('\nSystem:')
     for k, stat in sys_info.items():
         print("{k:>10}: {stat}".format(k=k, stat=stat))
 
-    print('\nBLAS:')
-    for k, stat in blas_info.items():
-        print("{k:>10}: {stat}".format(k=k, stat=stat))
-
     print('\nPython deps:')
     for k, stat in deps_info.items():
         print("{k:>10}: {stat}".format(k=k, stat=stat))
+
+    print('\nNumpy BLAS/LAPACK:')
+    numpy.show_config()
+
+    print('\nScipy BLAS/LAPACK:')
+    scipy.show_config()


Code review comments:
diff_hunk: @@ -193,55 +193,27 @@ Installing build dependencies
 Linux
 -----
 
-Installing from source requires you to have installed the scikit-learn runtime
-dependencies, Python development headers and a working C/C++ compiler.
-Under Debian-based operating systems, which include Ubuntu::
+Installing from source without conda requires you to have installed the
+scikit-learn runtime dependencies, Python development headers and a working
+C/C++ compiler. Under Debian-based operating systems, which include Ubuntu::
 
     sudo apt-get install build-essential python3-dev python3-setuptools \
-                     python3-numpy python3-scipy \
-                     libatlas-dev libatlas3-base
-
-On recent Debian and Ubuntu (e.g. Ubuntu 14.04 or later) make sure that ATLAS
-is used to provide the implementation of the BLAS and LAPACK linear algebra
-routines::
+                     python3-pip
+    
+and then::
 
-    sudo update-alternatives --set libblas.so.3 \
-        /usr/lib/atlas-base/atlas/libblas.so.3
-    sudo update-alternatives --set liblapack.so.3 \
-        /usr/lib/atlas-base/atlas/liblapack.so.3
+    pip3 install numpy scipy cython
 
 .. note::
 
     In order to build the documentation and run the example code contains in
     this documentation you will need matplotlib::
 
-        sudo apt-get install python-matplotlib
-
-.. note::
-
-    The above installs the ATLAS implementation of BLAS
-    (the Basic Linear Algebra Subprograms library).
-    Ubuntu 11.10 and later, and recent (testing) versions of Debian,
-    offer an alternative implementation called OpenBLAS.
-
-    Using OpenBLAS can give speedups in some scikit-learn modules,
-    but can freeze joblib/multiprocessing prior to OpenBLAS version 0.2.8-4,
-    so using it is not recommended unless you know what you're doing.
-
-    If you do want to use OpenBLAS, then replacing ATLAS only requires a couple
-    of commands. ATLAS has to be removed, otherwise NumPy may not work::
-
-        sudo apt-get remove libatlas3gf-base libatlas-dev
-        sudo apt-get install libopenblas-dev
-
-        sudo update-alternatives  --set libblas.so.3 \
-            /usr/lib/openblas-base/libopenblas.so.0
-        sudo update-alternatives --set liblapack.so.3 \
-            /usr/lib/lapack/liblapack.so.3
+        sudo apt-get install python3-matplotlib
path: doc/developers/advanced_installation.rst
text: @user1:
why system matplotlib? Seems inconsistent with numpy/scipy above.

diff_hunk: @@ -70,49 +72,22 @@ def get_version(module):
     return deps_info
 
 
-def _get_blas_info():
-    """Information on system BLAS
-
-    Uses the `scikit-learn` builtin method
-    :func:`sklearn._build_utils.get_blas_info` which may fail from time to time
-
-    Returns
-    -------
-    blas_info: dict
-        system BLAS information
-
-    """
-    from .._build_utils import get_blas_info
-
-    cblas_libs, blas_dict = get_blas_info()
-
-    macros = ['{key}={val}'.format(key=a, val=b)
-              for (a, b) in blas_dict.get('define_macros', [])]
-
-    blas_blob = [
-        ('macros', ', '.join(macros)),
-        ('lib_dirs', ':'.join(blas_dict.get('library_dirs', ''))),
-        ('cblas_libs', ', '.join(cblas_libs)),
-    ]
-
-    return dict(blas_blob)
-
-
 def show_versions():
     "Print useful debugging information"
 
     sys_info = _get_sys_info()
     deps_info = _get_deps_info()
-    blas_info = _get_blas_info()
 
     print('\nSystem:')
     for k, stat in sys_info.items():
         print("{k:>10}: {stat}".format(k=k, stat=stat))
 
-    print('\nBLAS:')
-    for k, stat in blas_info.items():
-        print("{k:>10}: {stat}".format(k=k, stat=stat))
-
     print('\nPython deps:')
     for k, stat in deps_info.items():
         print("{k:>10}: {stat}".format(k=k, stat=stat))
+
+    print('\nNumpy BLAS/LAPACK:')
+    numpy.show_config()
+
+    print('\nScipy BLAS/LAPACK:')
+    scipy.show_config()
path: sklearn/utils/_show_versions.py
text: @user1:
This prints half a page of text, we don't really want it in issue reports I think? Also, wouldn't it suffer from the same limitations as  the previous `_get_blas_info` ?

@author:
I think `show_config` is more accurate in general. It's based on infos gathered when building numpy, whereas the previous `_get_blas_info` takes its infos from `numpy.distutils.system_info.get_info` which gather infos at run time.

But I'm really not sure how both work so I might be completely wrong...

diff_hunk: @@ -193,55 +193,27 @@ Installing build dependencies
 Linux
 -----
 
-Installing from source requires you to have installed the scikit-learn runtime
-dependencies, Python development headers and a working C/C++ compiler.
-Under Debian-based operating systems, which include Ubuntu::
+Installing from source without conda requires you to have installed the
+scikit-learn runtime dependencies, Python development headers and a working
+C/C++ compiler. Under Debian-based operating systems, which include Ubuntu::
 
     sudo apt-get install build-essential python3-dev python3-setuptools \
-                     python3-numpy python3-scipy \
-                     libatlas-dev libatlas3-base
-
-On recent Debian and Ubuntu (e.g. Ubuntu 14.04 or later) make sure that ATLAS
-is used to provide the implementation of the BLAS and LAPACK linear algebra
-routines::
+                     python3-pip
+    
+and then::
 
-    sudo update-alternatives --set libblas.so.3 \
-        /usr/lib/atlas-base/atlas/libblas.so.3
-    sudo update-alternatives --set liblapack.so.3 \
-        /usr/lib/atlas-base/atlas/liblapack.so.3
+    pip3 install numpy scipy cython
 
 .. note::
 
     In order to build the documentation and run the example code contains in
     this documentation you will need matplotlib::
 
-        sudo apt-get install python-matplotlib
-
-.. note::
-
-    The above installs the ATLAS implementation of BLAS
-    (the Basic Linear Algebra Subprograms library).
-    Ubuntu 11.10 and later, and recent (testing) versions of Debian,
-    offer an alternative implementation called OpenBLAS.
-
-    Using OpenBLAS can give speedups in some scikit-learn modules,
-    but can freeze joblib/multiprocessing prior to OpenBLAS version 0.2.8-4,
-    so using it is not recommended unless you know what you're doing.
-
-    If you do want to use OpenBLAS, then replacing ATLAS only requires a couple
-    of commands. ATLAS has to be removed, otherwise NumPy may not work::
-
-        sudo apt-get remove libatlas3gf-base libatlas-dev
-        sudo apt-get install libopenblas-dev
-
-        sudo update-alternatives  --set libblas.so.3 \
-            /usr/lib/openblas-base/libopenblas.so.0
-        sudo update-alternatives --set liblapack.so.3 \
-            /usr/lib/lapack/liblapack.so.3
+        sudo apt-get install python3-matplotlib
 
 On Red Hat and clones (e.g. CentOS), install the dependencies using::
 
-    sudo yum -y install gcc gcc-c++ numpy python-devel scipy
+    sudo yum -y install gcc gcc-c++ python-devel numpy scipy
path: doc/developers/advanced_installation.rst
text: @user1:
again, inconsistent with above.
I don't care that much which way we go but we should be consistent. I slightly prefer the pip way.