Repository: pyinstaller/pyinstaller

Issue:
New numpy hook for intel mkl libraries
Hello - anaconda and winpython build numpy against Intel's mkl libraries. Building someone that uses numpy will need to have those libraries.

Here is a winpython hook:
http://stackoverflow.com/a/35853001

and I adapted that for anaconda & python3:
https://github.com/maqifrnswa/scimpy/blob/master/pyinstaller-hooks/hook-numpy.py



Discussion:
@maqifrnswa thanks, could you please create a pull request from those two?
- combine them into one
- use `if is_darwin` to choose the right path for OSX and other OSes

Hi,
I am using pyinstaller 3.1 in conda 4.0.4 to generate a onefile executable for a program in python that needs multiple scientific libraries, such as numpy or scipy. Previously we had all libraries installed directly on the machine, but we have switched to conda for practical reasons and after the machine in which we were compiling had to be reinstalled. However, in this new setup, on some machines, at runtime, the executable produced raises an intel mkl error, previously reported for numpy:

Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.

Both libraries are inside the anaconda installation, in the lib folder, but they are not being included in the wrapped executable, and if the program is searching for them in the machine it is not having success. Looking for a solution I found this thread, I guess that what I will need to try is to use the hook provided by maqifrnswa?

Hi, I did try that, using the hook described as runtime-hook, but got the following error:

Traceback (most recent call last):
  File "<string>", line 9, in <module>
  File "/cri4/claudia/anaconda/envs/arcimboldo/lib/python2.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
  File "site-packages/PyInstaller/**init**.py", line 61, in <module>
  File "site-packages/pkg_resources.py", line 336, in get_distribution
  File "site-packages/pkg_resources.py", line 215, in get_provider
  File "site-packages/pkg_resources.py", line 696, in require
  File "site-packages/pkg_resources.py", line 594, in resolve
pkg_resources.DistributionNotFound: PyInstaller
hook_numpy returned -1

 I see in the log that it has included it (18880 INFO: Including custom run-time hook 'hook_numpy.py'),  but the error happens each time and I see many missing modules from numpy in the warnings folder. I read the answer on http://stackoverflow.com/questions/35478526/pyinstaller-numpy-intel-mkl-fatal-error-cannot-load-mkl-intel-thread-dll and one of them suggest that problems when using the hook might come from using different Python package management schemes. In my case, some of the packages where installed directly by conda and have a built string of py27_*, some others with pip, and in particular, biopython, matplotlib and scipy have one called np110py27. Could that be a problem?

I have also tried to include the libraries as binaries in the spec file and then compile from the spec, but it keeps happening. Someone in the scikit-learn repo told me that this could be related with the numpy version, as mentioned in https://github.com/scikit-learn/scikit-learn/pull/6508#issuecomment-194908920. However, I do have the version 1.10.4, so apparently that should not be a problem.

Just for the record, browsing through github scikit-learn project issues, I found the post at Anaconda blog where they describe how to inactivate the mkl versions of the libraries (https://www.continuum.io/blog/developer-blog/anaconda-25-release-now-mkl-optimizations) and now  I can compile a working version. I don't think this is a problem at all with pyinstaller but with some of the libraries I call, so sorry for all the mess! At least I have learnt about other Pyinstaller options that I had never used, such as the binaries addition or the runtime or linktime hooks. I am pretty new to Pyinstaller so that was good :) Thanks anyway!

@clacri Thanks for those links, that is very helpful! I didn't know about turning off mkl optimizations.

@matysek Looking through the Anaconda EULA https://docs.continuum.io/anaconda/eula it says re-distribution is allowed, as long as it is redistributed "with anaconda." Since **an app frozen with** PyInstaller includes a subset of the anaconda distribution (and thus is a derivative of anaconda), redistribution of mkl should be acceptable as long as the terms of the anaconda license are also met. Is there some document in PyInstaller to let users of specific distributions know that there are licensing terms they need to follow? If so I can update it to include info on the mkl libraries. I hope to update and test hook within a couple of days.

(edits for clarification are in **bold**)

@maqifrnswa IMHO this EULA is not quite clear about the mkl-library licence Would you please be so kind clarifying this with continum.io and issue a pull request? Thanks

BTW: This is plain wrong:

> Since PyInstaller includes a subset of the anaconda distribution (and thus is a derivative of anaconda),

The _frozen application_ does include "a subset of the anaconda distribution", but not PyInstaller. PyInstaller is only the tool or creating this subset.

Thank you - that's what I meant, the pyinstaller application includes a distribution - not pyinstaller. I've edited my previous statement to be clear. I'll do the pull request and ask for clarification from continuum. Busy with day job, might not get to the pull request for a couple of days.

I'm also having trouble bundling numpy from the anaconda distribution due to missing MKL runtime libraries.
For the numpy hook `numpy.__config__` / `numpy.__config__.show()` could be used to detect if numpy is built with MKL support.

What platform? The development version of Pyinstaller should now bundle these on Windows.

The current numpy.core hook in PyInstaller only looks for DLLs in the numpy folder, but it doesn't install the MKL DLLs from the python `/Library/bin` folder. MKL seems to load those on demand. When calling certain numpy functions a bundled application crashes with the message

```
Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.
```

Platform is Windows. Tested with current Anaconda distribution and the PyInstaller develop branch.

On Windows, MKL is looking for `.so` files? That makes no sense...

Ah sorry, I just lazily copy pasted the error message from one of the first comments above. But the same problem occurs when bundling from Anaconda for windows, the library names are just different of course (`mkl_avx2.dll` and `mkl_def.dll`, also located in `Library/bin`).

I'm hacking up a bit of code that might help. Would you be able to test when it's ready?


Resolving patch:
diff --git a/PyInstaller/hooks/hook-numpy.core.py b/PyInstaller/hooks/hook-numpy.core.py
--- a/PyInstaller/hooks/hook-numpy.core.py
+++ b/PyInstaller/hooks/hook-numpy.core.py
@@ -6,22 +6,41 @@
 #
 # The full license is in the file COPYING.txt, distributed with this software.
 #-----------------------------------------------------------------------------
-# On Windows, numpy depends on a set of dynamically-detemined DLLs, which means
-# that PyInstaller's static analysis can't find them. See https://github.com/pyinstaller/pyinstaller/issues/1969
-# for more information. The typical error message: ``Intel MKL FATAL ERROR:
-# Cannot load mkl_intel_thread.dll.``
+# If numpy is built with MKL support it depends on a set of libraries loaded
+# at runtime. Since PyInstaller's static analysis can't find them they must be
+# included manually.
 #
-# So, include them manually.
+# See
+# https://github.com/pyinstaller/pyinstaller/issues/1881
+# https://github.com/pyinstaller/pyinstaller/issues/1969
+# for more information
 import os
 import os.path
+import re
 from PyInstaller.utils.hooks import get_package_paths
+from PyInstaller import log as logging 
+from PyInstaller import compat
 
+binaries = []
+
+# look for libraries in numpy package path
 pkg_base, pkg_dir = get_package_paths('numpy.core')
-# Walk through all files in ``numpy.core``, looking for DLLs.
-datas = []
-for f in os.listdir(pkg_dir):
-    extension = os.path.splitext(f)[1]
-    if extension == '.dll':
-        # Produce the tuple ('/abs/path/to/libs/numpy/core/file.dll', '')
-        source = os.path.join(pkg_dir, f)
-        datas.append((source, ''))
+re_anylib = re.compile(r'\w+\.(?:dll|so)', re.IGNORECASE)
+dlls_pkg = [f for f in os.listdir(pkg_dir) if re_anylib.match(f)]
+binaries += [(os.path.join(pkg_dir, f), '') for f in dlls_pkg]
+
+# look for MKL libraries in pythons lib directory
+# TODO: check numpy.__config__ if numpy is actually depending on MKL
+# TODO: determine which directories are searched by the os linker
+if compat.is_win:
+    lib_dir = os.path.join(compat.base_prefix, "Library", "bin")
+else:
+    lib_dir = os.path.join(compat.base_prefix, "lib")
+if os.path.isdir(lib_dir):
+    re_mkllib = re.compile(r'^(?:lib)?mkl\w+\.(?:dll|so)', re.IGNORECASE)
+    dlls_mkl = [f for f in os.listdir(lib_dir) if re_mkllib.match(f)]
+    if dlls_mkl:
+        logger = logging.getLogger(__name__)
+        logger.info("MKL libraries found when importing numpy. Adding MKL to binaries")
+        binaries += [(os.path.join(lib_dir, f), '') for f in dlls_mkl]
+


Test patch: