Repository: pyinstaller/pyinstaller

Issue:
Pyinstaller can't find libpython on Anaconda
I am using Linux(Ubuntu) and Anaconda 5.3 and the latest version of pyInstaller, 
which I installed yesterday.

Pyinstaller seems not to be able to search and find the libpython3.7.so.1.0 file.

Here is the log:

```
496 INFO: PyInstaller: 3.4
497 INFO: Python: 3.7.0
514 INFO: Platform: Linux-4.15.0-20-generic-x86_64-with-debian-buster-sid
520 INFO: wrote /root/anaconda3/bin/qmlview.spec
533 INFO: UPX is not available.
560 INFO: Extending PYTHONPATH with paths
['/media/sf_Buildozer_Shared_folder/Qmlview', '/root/anaconda3/bin']
560 INFO: checking Analysis
561 INFO: Building Analysis because Analysis-00.toc is non existent
561 INFO: Initializing module dependency graph...
617 INFO: Initializing module graph hooks...
710 INFO: Analyzing base_library.zip ...
11272 INFO: running Analysis Analysis-00.toc
11359 INFO: Caching module hooks...
11366 INFO: Analyzing /media/sf_Buildozer_Shared_folder/Qmlview/qmlview.py
11507 INFO: Loading module hooks...
11507 INFO: Loading module hook "hook-PyQt5.QtGui.py"...
12752 INFO: Loading module hook "hook-pydoc.py"...
12773 INFO: Loading module hook "hook-xml.py"...
13446 INFO: Loading module hook "hook-PyQt5.py"...
13500 INFO: Loading module hook "hook-PyQt5.QtCore.py"...
13547 INFO: Loading module hook "hook-encodings.py"...
13683 INFO: Loading module hook "hook-PyQt5.QtQml.py"...
14173 INFO: Loading module hook "hook-PyQt5.QtNetwork.py"...
14318 INFO: Looking for ctypes DLLs
14318 INFO: Analyzing run-time hooks ...
14324 INFO: Including run-time hook 'pyi_rth_qt5.py'
14344 INFO: Looking for dynamic libraries
21091 INFO: Looking for eggs
21092 INFO: Python library not in binary dependencies. Doing additional searching...
Traceback (most recent call last):
  File "/root/anaconda3/bin/pyinstaller", line 11, in <module>
    sys.exit(run())
  File "/root/anaconda3/lib/python3.7/site-packages/PyInstaller/__main__.py", line 111, in run
    run_build(pyi_config, spec_file, **vars(args))
  File "/root/anaconda3/lib/python3.7/site-packages/PyInstaller/__main__.py", line 63, in run_build
    PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
  File "/root/anaconda3/lib/python3.7/site-packages/PyInstaller/building/build_main.py", line 838, in main
    build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))
  File "/root/anaconda3/lib/python3.7/site-packages/PyInstaller/building/build_main.py", line 784, in build
    exec(text, spec_namespace)
  File "<string>", line 17, in <module>
  File "/root/anaconda3/lib/python3.7/site-packages/PyInstaller/building/build_main.py", line 241, in __init__
    self.__postinit__()
  File "/root/anaconda3/lib/python3.7/site-packages/PyInstaller/building/datastruct.py", line 158, in __postinit__
    self.assemble()
  File "/root/anaconda3/lib/python3.7/site-packages/PyInstaller/building/build_main.py", line 573, in assemble
    self._check_python_library(self.binaries)
  File "/root/anaconda3/lib/python3.7/site-packages/PyInstaller/building/build_main.py", line 676, in _check_python_library
    raise IOError(msg)
OSError: Python library not found: libpython3.7m.so.1.0, libpython3.7.so.1.0, libpython3.7mu.so.1.0
This would mean your Python installation doesn't come with proper library files.
This usually happens by missing development package, or unsuitable build parameters of Python installation.

* On Debian/Ubuntu, you would need to install Python development packages
  * apt-get install python3-dev
  * apt-get install python-dev
* If you're building Python by yourself, please rebuild your Python with `--enable-shared` (or, `--enable-framework` on Darwin)
```




Discussion:
Same issue with python 3.5

`pyinstaller 3.5.dev0+0ffa574bf`

copying the file `libpython3.5m.so.1.0` to `libpython3.5m.so` works (as I see in another site)

to get the error I execute: `pyinstaller myscript.py` and several --hidden-import and I don't have anaconda

thankyou so much for your work ;) 
I'm also seeing this problem on Ubuntu 18.04.1 and Conda 4.5.12 installed with miniconda. Created an environment with Python 3.7, activated it, then got this error when running PyInstaller 3.4 against one of my scripts:

```
...
15408 INFO: Looking for eggs
15409 INFO: Python library not in binary dependencies. Doing additional searching...
Traceback (most recent call last):
  File "/home/vagrant/miniconda3/envs/seaflowpy/bin/pyinstaller", line 11, in <module>
    sys.exit(run())
  File "/home/vagrant/miniconda3/envs/seaflowpy/lib/python3.7/site-packages/PyInstaller/__main__.py", line 111, in run
    run_build(pyi_config, spec_file, **vars(args))
  File "/home/vagrant/miniconda3/envs/seaflowpy/lib/python3.7/site-packages/PyInstaller/__main__.py", line 63, in run_build
    PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
  File "/home/vagrant/miniconda3/envs/seaflowpy/lib/python3.7/site-packages/PyInstaller/building/build_main.py", line 838, in main
    build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))
  File "/home/vagrant/miniconda3/envs/seaflowpy/lib/python3.7/site-packages/PyInstaller/building/build_main.py", line 784, in build
    exec(text, spec_namespace)
  File "<string>", line 17, in <module>
  File "/home/vagrant/miniconda3/envs/seaflowpy/lib/python3.7/site-packages/PyInstaller/building/build_main.py", line 241, in __init__
    self.__postinit__()
  File "/home/vagrant/miniconda3/envs/seaflowpy/lib/python3.7/site-packages/PyInstaller/building/datastruct.py", line 158, in __postinit__
    self.assemble()
  File "/home/vagrant/miniconda3/envs/seaflowpy/lib/python3.7/site-packages/PyInstaller/building/build_main.py", line 573, in assemble
    self._check_python_library(self.binaries)
  File "/home/vagrant/miniconda3/envs/seaflowpy/lib/python3.7/site-packages/PyInstaller/building/build_main.py", line 676, in _check_python_library
    raise IOError(msg)
OSError: Python library not found: libpython3.7.so.1.0, libpython3.7mu.so.1.0, libpython3.7m.so.1.0
This would mean your Python installation doesn't come with proper library files.
This usually happens by missing development package, or unsuitable build parameters of Python installation.
...
```

These are all the `libpython*` files in that environment's `lib` folder.

```
/home/vagrant/miniconda3/envs/seaflowpy/lib/libpython3.7m.a
/home/vagrant/miniconda3/envs/seaflowpy/lib/libpython3.7m.so -> libpython3.7m.so.1.0
/home/vagrant/miniconda3/envs/seaflowpy/lib/libpython3.7m.so.1 -> libpython3.7m.so.1.0
/home/vagrant/miniconda3/envs/seaflowpy/lib/libpython3.7m.so.1.0
```

@amoh-godwin @jesusjda @ctberthiaume This is of course a hack, but setting `LD_LIBRARY_PATH` worked for me. So instead of

```bash
pyinstaller ...
```
run
```
LD_LIBRARY_PATH=path_to_python_libs pyinstaller ...
```
You all can help finishing #3889 which is going to solve the issue :-)

Main points there is: Why does [`compat.is_conda()` return `False`](https://github.com/pyinstaller/pyinstaller/pull/3889#discussion_r243635751) and what does `sys.version` report on your installation?
I'll take a look at the pull request to see if there's anyway I can contribute. 

In the meantime, I have tried setting `LD_LIBRARY_PATH`, which brings up a different error message. To make this as simple as possible I used the base environment in anaconda and ran PyInstaller against a one-line script with only `print('hello world')`.

```
(base) vagrant@ubuntu-bionic:~/helloworld$ LD_LIBRARY_PATH=~/miniconda3/lib/ pyinstaller helloworld.py
25 INFO: PyInstaller: 3.4
25 INFO: Python: 3.7.1
26 INFO: Platform: Linux-4.15.0-43-generic-x86_64-with-debian-buster-sid
26 INFO: wrote /home/vagrant/helloworld/helloworld.spec
27 INFO: UPX is not available.
28 INFO: Extending PYTHONPATH with paths
['/home/vagrant/helloworld', '/home/vagrant/helloworld']
28 INFO: checking Analysis
28 INFO: Building Analysis because Analysis-00.toc is non existent
29 INFO: Initializing module dependency graph...
29 INFO: Initializing module graph hooks...
30 INFO: Analyzing base_library.zip ...
2397 INFO: running Analysis Analysis-00.toc
2405 INFO: Caching module hooks...
2407 INFO: Analyzing /home/vagrant/helloworld/helloworld.py
2410 INFO: Loading module hooks...
2410 INFO: Loading module hook "hook-encodings.py"...
2441 INFO: Loading module hook "hook-pydoc.py"...
2441 INFO: Loading module hook "hook-xml.py"...
2637 INFO: Looking for ctypes DLLs
2637 INFO: Analyzing run-time hooks ...
2640 INFO: Looking for dynamic libraries
3018 INFO: Looking for eggs
3018 INFO: Python library not in binary dependencies. Doing additional searching...
Traceback (most recent call last):
  File "/home/vagrant/miniconda3/bin/pyinstaller", line 11, in <module>
    sys.exit(run())
  File "/home/vagrant/miniconda3/lib/python3.7/site-packages/PyInstaller/__main__.py", line 111, in run
    run_build(pyi_config, spec_file, **vars(args))
  File "/home/vagrant/miniconda3/lib/python3.7/site-packages/PyInstaller/__main__.py", line 63, in run_build
    PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
  File "/home/vagrant/miniconda3/lib/python3.7/site-packages/PyInstaller/building/build_main.py", line 838, in main
    build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))
  File "/home/vagrant/miniconda3/lib/python3.7/site-packages/PyInstaller/building/build_main.py", line 784, in build
    exec(text, spec_namespace)
  File "<string>", line 17, in <module>
  File "/home/vagrant/miniconda3/lib/python3.7/site-packages/PyInstaller/building/build_main.py", line 241, in __init__
    self.__postinit__()
  File "/home/vagrant/miniconda3/lib/python3.7/site-packages/PyInstaller/building/datastruct.py", line 158, in __postinit__
    self.assemble()
  File "/home/vagrant/miniconda3/lib/python3.7/site-packages/PyInstaller/building/build_main.py", line 573, in assemble
    self._check_python_library(self.binaries)
  File "/home/vagrant/miniconda3/lib/python3.7/site-packages/PyInstaller/building/build_main.py", line 661, in _check_python_library
    python_lib = bindepend.get_python_library_path()
  File "/home/vagrant/miniconda3/lib/python3.7/site-packages/PyInstaller/depend/bindepend.py", line 879, in get_python_library_path
    python_libname = findLibrary(name)
  File "/home/vagrant/miniconda3/lib/python3.7/site-packages/PyInstaller/depend/bindepend.py", line 825, in findLibrary
    return os.path.join(dir, _get_so_name(lib))
  File "/home/vagrant/miniconda3/lib/python3.7/site-packages/PyInstaller/depend/bindepend.py", line 836, in _get_so_name
    m = re.search(r'\s+SONAME\s+([^\s]+)', compat.exec_command(*cmd))
  File "/home/vagrant/miniconda3/lib/python3.7/site-packages/PyInstaller/compat.py", line 363, in exec_command
    out = subprocess.Popen(cmdargs, stdout=subprocess.PIPE, **kwargs).communicate()[0]
  File "/home/vagrant/miniconda3/lib/python3.7/subprocess.py", line 769, in __init__
    restore_signals, start_new_session)
  File "/home/vagrant/miniconda3/lib/python3.7/subprocess.py", line 1516, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'objdump': 'objdump'
```
@ctberthiaume You wnt to read this: https://pyinstaller.readthedocs.io/en/stable/requirements.html#linux
My bad, I should have done a better job consulting the clear documentation before reporting that error. Installing binutils + setting `LD_LIBRARY_PATH` resolved everything.
> You all can help finishing #3889 which is going to solve the issue :-)
> 
> Main points there is: Why does [`compat.is_conda()` return `False`](https://github.com/pyinstaller/pyinstaller/pull/3889#discussion_r243635751) and what does `sys.version` report on your installation?
 
`compat.is_conda()` returns False in Linux because the `sys.version` appears to have changed in Anaconda Python, at least with conda version 4.5.12. `compat.is_conda` checks for strings 'conda' or 'Continuum', but conda Python 3.7.2 `sys.version` produces '3.7.2 (default, Dec 29 2018, 06:19:36) \n[GCC 7.3.0]'. This is similar to Linux system Python, which is '3.6.7 (default, Oct 22 2018, 11:32:17) \n[GCC 8.2.0]'.

The only place I could find any indication that this Python was created by Anaconda was in the interactive shell banner, which reads:
```
Python 3.7.2 (default, Dec 29 2018, 06:19:36) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
```

However, I can't find anyway to get this information from inside Python, e.g. in `sys`. You could maybe open a subprocess that starts the Python interactive shell and feed `'quit()' + os.linesep` to stdin to close the shell, then look for 'Anaconda' in stdout. Looking at `compat.exec_` functions it's not clear how to do this in a way that works for all supported Python versions without code changes there.
@ctberthiaume Thanks for this analysis. My aversion against Anaconda is growing further (they changed other things causing quite some issues). Spawning a subprocess just to set some `is_conda` variable is too much overhead IMO as this would be done on each invocation on each Unix system.
What about this (taken from https://stackoverflow.com/questions/47610844#47610844):
```
is_conda = os.path.isdir(os.path.join(sys.prefix, 'conda-meta'))
```
(We should use `compat.base_prefix` to escape any virtual env.)
Looks like that would work. To fix this issue I'm not sure `is_conda` is strictly necessary, although `is_conda` should certainly be fixed to work correctly. PyInstaller currently works on MacOS with Anaconda without checking `is_conda` because the final fallback for MacOS when looking for Python dynamic libraries is to search `compat.base_prefix`, which covers Anaconda.

https://github.com/pyinstaller/pyinstaller/blob/d5a902d9eac1478944bdc7e9bd36662fabe58b10/PyInstaller/depend/bindepend.py#L883-L902

This could also be the final fallback for Unix by changing `elif is_darwin:` here to `if is_darwin or is_unix:`. It doesn't seem like this would ever be the wrong thing to do on Unix if all other attempts to find the library fail. I'm not that familiar with this project so I don't know all the possible implications, but it fixes this issue for me.
> Looks like that would work

Is it `conda-meta` documented somewhere?

> the final fallback for MacOS when looking for Python dynamic libraries is to search compat.base_prefix, [...] This could also be the final fallback for Unix.

Yes and no. When adding this here it would only work for libpython, while adding it to `findLibrary()` would also cover all other dynamic libraries. Maybe we should do both.

Or maybe even better unify some code between unix and osx along these lines:
- Implement an option like `ignore_ldconfig_cache` to `findLibrary()` (I suggest this options since I'm afraid searching libpython in ldconfig-cache and DYNLIB_... might return the wrong lib on OS X)
- Unconditionally add $base_prefix and $base_prefix/lib to path in line 772 (this would cover OS X then)
- Think about reworking the code to calculate the paths to be based on prefixes (sys.prefix, compat.base_prefix, "/", "/usr", "/usr/local") and then appending all these arch- and multi-arch-dependent lib-paths to each of the prefixes.
- Also think about caching the calculation as it will not change during run-time. Hmm, when caching it might be worth the effort testing whether an directory actually exists prior to adding it to the list. Gieb all this, it might also be a good ide to move this into a sub-function.
- In `get_python_library_path()` unify the `is_unix` and `is_osx` cases.

What do you think?
`conda-meta` looks like it will be a reliable way to detect Conda installations for a while. It's referenced in the API and user guide docs.

* https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-pkgs.html#preventing-packages-from-updating-pinning
* https://docs.conda.io/projects/conda/en/latest/api/api.html?highlight=conda-meta

Also, there's a Change Log entry referencing it for Conda 3 in Jan 2014 (current is Conda 4). So it's been established for at least 5 years and has survived at least one major version change.

* https://docs.conda.io/projects/conda/en/latest/release-notes.html?highlight=%27conda-meta%27#id344

And looking at the Conda github project I don't see a branch working on version 5. Even if `conda-meta` disappears in the next major version release, it doesn't look like this is on the horizon.

I'll take a closer look other comments when I get some time.
> * Implement an option like `ignore_ldconfig_cache` to `findLibrary()` (I suggest this options since I'm afraid searching libpython in ldconfig-cache and DYNLIB_... might return the wrong lib on OS X)

Yes, because `ldconfig` cache is searched first in `findLibrary()` it would find the wrong version of the Python shared library in my use case, i.e. If I have Python 3.6 installed system-wide, and I have created a Conda environment with Python 3.6, the system `libpython3.6m.so.1.0` will be found and used instead of the one in my Conda env. But even ignoring the `ldconfig` cache isn't enough because the system location (`/usr/lib`) is searched before `compat.base_prefix` is.

> * Unconditionally add $base_prefix and $base_prefix/lib to path in line 772 (this would cover OS X then)

This, combined with the above `ignore_ldconfig_cache`, or just moving this search before `ldconfig` cache search, would solve the above problem.

> * Think about reworking the code to calculate the paths to be based on prefixes (sys.prefix, compat.base_prefix, "/", "/usr", "/usr/local") and then appending all these arch- and multi-arch-dependent lib-paths to each of the prefixes.

Sounds reasonable. Deferring to the Python executable's self-reported library location is probably better than hardcoding a search for system locations first before finally looking in `sys.*prefix` locations.

> * In `get_python_library_path()` unify the `is_unix` and `is_osx` cases.

If `findLibrary()` is changed to search in `sys.*prefix` first then  the `is_darwin` case will already be covered. The `if is_unix:` test can be changed to `if is_unix or is_darwin:`, `findLibrary()` will do the right thing, and the old `is_darwin` case can be removed.


Same with python 3.6.6:

```
3805 INFO: Python library not in binary dependencies. Doing additional searching...
Traceback (most recent call last):
  File "/home/george/anaconda3/envs/pytorch/bin/pyinstaller", line 11, in <module>
    sys.exit(run())
  File "/home/george/anaconda3/envs/pytorch/lib/python3.6/site-packages/PyInstaller/__main__.py", line 111, in run
    run_build(pyi_config, spec_file, **vars(args))
  File "/home/george/anaconda3/envs/pytorch/lib/python3.6/site-packages/PyInstaller/__main__.py", line 63, in run_build
    PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
  File "/home/george/anaconda3/envs/pytorch/lib/python3.6/site-packages/PyInstaller/building/build_main.py", line 838, in main
    build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))
  File "/home/george/anaconda3/envs/pytorch/lib/python3.6/site-packages/PyInstaller/building/build_main.py", line 784, in build
    exec(text, spec_namespace)
  File "<string>", line 17, in <module>
  File "/home/george/anaconda3/envs/pytorch/lib/python3.6/site-packages/PyInstaller/building/build_main.py", line 241, in __init__
    self.__postinit__()
  File "/home/george/anaconda3/envs/pytorch/lib/python3.6/site-packages/PyInstaller/building/datastruct.py", line 158, in __postinit__
    self.assemble()
  File "/home/george/anaconda3/envs/pytorch/lib/python3.6/site-packages/PyInstaller/building/build_main.py", line 573, in assemble
    self._check_python_library(self.binaries)
  File "/home/george/anaconda3/envs/pytorch/lib/python3.6/site-packages/PyInstaller/building/build_main.py", line 676, in _check_python_library
    raise IOError(msg)
OSError: Python library not found: libpython3.6.so.1.0, libpython3.6m.so.1.0, libpython3.6mu.so.1.0
This would mean your Python installation doesn't come with proper library files.
This usually happens by missing development package, or unsuitable build parameters of Python installation.

* On Debian/Ubuntu, you would need to install Python development packages
  * apt-get install python3-dev
  * apt-get install python-dev
* If you're building Python by yourself, please rebuild your Python with `--enable-shared` (or, `--enable-framework` on Darwin)

```
Does anyone have some hack to resolve it?

Thank you for your efforts anyway!

Resolving patch:
diff --git a/PyInstaller/compat.py b/PyInstaller/compat.py
--- a/PyInstaller/compat.py
+++ b/PyInstaller/compat.py
@@ -204,8 +204,8 @@ def __getattr__(self, name):
 # Conda environments sometimes have different paths or apply patches to
 # packages that can affect how a hook or package should access resources.
 # Method for determining conda taken from:
-# https://stackoverflow.com/a/21318941/433202
-is_conda = 'conda' in sys.version or 'Continuum' in sys.version
+# https://stackoverflow.com/questions/47610844#47610844
+is_conda = os.path.isdir(os.path.join(base_prefix, 'conda-meta'))
 
 # In Python 3.4 module 'imp' is deprecated and there is another way how
 # to obtain magic value.
diff --git a/PyInstaller/depend/bindepend.py b/PyInstaller/depend/bindepend.py
--- a/PyInstaller/depend/bindepend.py
+++ b/PyInstaller/depend/bindepend.py
@@ -22,7 +22,8 @@
 
 from .. import compat
 from ..compat import (is_win, is_unix, is_aix, is_solar, is_cygwin, is_hpux,
-                      is_darwin, is_freebsd, is_venv, base_prefix, PYDYLIB_NAMES)
+                      is_darwin, is_freebsd, is_venv, is_conda, base_prefix,
+                      PYDYLIB_NAMES)
 from . import dylib, utils
 
 from .. import log as logging
@@ -866,6 +867,14 @@ def get_python_library_path():
     Darwin custom builds could possibly also have non-framework style libraries,
     so this method also checks for that variant as well.
     """
+    def _find_lib_in_libdirs(*libdirs):
+        for libdir in libdirs:
+            for name in PYDYLIB_NAMES:
+                full_path = os.path.join(libdir, name)
+                if os.path.exists(full_path):
+                    return full_path
+        return None
+
     # Try to get Python library name from the Python executable. It assumes that Python
     # library is not statically linked.
     dlls = getImports(sys.executable)
@@ -880,9 +889,17 @@ def get_python_library_path():
                 return filename
 
     # Python library NOT found. Resume searching using alternative methods.
-    # Applies only to non Windows platforms.
+    # Applies only to non Windows platforms and conda.
 
-    if is_unix:
+    if is_conda:
+        # Conda needs to be the first here since it overrules the operating
+        # system specific paths.
+        python_libname = _find_lib_in_libdirs(
+            os.path.join(compat.base_prefix, 'lib'))
+        if python_libname:
+            return python_libname
+
+    elif is_unix:
         for name in PYDYLIB_NAMES:
             python_libname = findLibrary(name)
             if python_libname:
@@ -900,14 +917,9 @@ def get_python_library_path():
         # and exec_prefix. That's why we can use just sys.prefix.
         # In virtualenv PyInstaller is not able to find Python library.
         # We need special care for this case.
-        # Anaconda places the python library in the lib directory, so
-        # we search this one as well.
-        prefixes = [compat.base_prefix, os.path.join(compat.base_prefix, 'lib')]
-        for prefix in prefixes:
-            for name in PYDYLIB_NAMES:
-                full_path = os.path.join(prefix, name)
-                if os.path.exists(full_path):
-                    return full_path
+        python_libname = _find_lib_in_libdirs(compat.base_prefix)
+        if python_libname:
+            return python_libname
 
     # Python library NOT found. Return just None.
     return None


Test patch: