Repository: pyinstaller/pyinstaller

Issue:
Missing some shared libraries when building PySide2 project inside of a debian based docker container.
## Description of the issue

Building a project that uses PySide2 in a Debian stretch docker container misses several required libraries which can cause the resulting binary to fail. In my project, I am missing `libpng16.so.16` from the archive which causes PySide2 to use my local version of the library which is not compatible with other libraries included in the archive. If I run pyinstaller locally (outside of the docker container), all of the expected libraries are there. Comparing the output of both show a large number of missing libraries in the one run inside of the container.

### Context information (for bug reports)

* Output of `pyinstaller --version`: 4.3
* Version of Python: 3.6.12
* Platform: Debian Stretch docker container, Arch host
* How you installed Python: pyenv
* Did you also try this on another platform? Does it work there?: Tried with docker containers of debian bullseye, ubuntu 14.04, 16.04, 18.04, 20.04, and 21.04. All failed similarly.
* try the latest development version: No change in behavior

### Make sure [everything is packaged correctly](https://github.com/pyinstaller/pyinstaller/wiki/How-to-Report-Bugs#make-sure-everything-is-packaged-correctly)

  * [x] start with clean installation
  * [x] use the latest development version
  * [x] Run your frozen program **from a command window (shell)** — instead of double-clicking on it
  * [x] Package your program in **--onedir mode**
  * [x] Package **without UPX**, say: use the option `--noupx` or set `upx=False` in your .spec-file
  * [x] Repackage you application in **verbose/debug mode**. For this, pass the option `--debug` to `pyi-makespec` or `pyinstaller` or use `EXE(..., debug=1, ...)` in your .spec file.


### A minimal example program which shows the error

https://github.com/achow101/pyinstaller-pyside2-docker-failure

### Stacktrace / full error message

Build: [build.log](https://github.com/pyinstaller/pyinstaller/files/6549892/build.log)
Execution: [exec.log](https://github.com/pyinstaller/pyinstaller/files/6549893/exec.log)




Discussion:
Whoaho! Look at this @rokm - the issue template fully filled in! I haven't seen an issue like this on here for months...

I can confirm that I can reproduce this.
I notice that trying to load PySide2 with plain Python in the docker image also fails although with a different output.
```bash
> docker run builder:latest python -c "import PySide2.QtWidgets"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: libGL.so.1: cannot open shared object file: No such file or directory
```
I'm not sure if that implies that what we are attempting is impossible.
> Whoaho! Look at this @rokm - the issue template fully filled in! I haven't seen an issue like this on here for months...

I suppose this is the exception that confirms the rule. :+1: 

----

Alas, the issue itself can probably be filed under `We cannot collect what is not there.` A slim version of debian container is used, and it does not provide proper (full) environment that would be required to run PySide2. 

In this case, `libpng16.so` is indeed missing and the system one cannot be loaded because on the other hand, `libz.so`is collected but is too old. However, `libpng16.so`, while installed, is not collected from the build container, because it is not direct dependency of anything that we collect.

For example, looking at dependencies of `libqxcb.so` plugin inside the container:

<details><summary>Output of the ldd command</summary>
<p>

```
root@f4cfc581d545:/opt# ldd /root/.pyenv/versions/3.6.12/lib/python3.6/site-packages/PySide2/Qt/plugins/platforms/libqxcb.so 
	linux-vdso.so.1 (0x00007fff827fc000)
	libQt5XcbQpa.so.5 => /root/.pyenv/versions/3.6.12/lib/python3.6/site-packages/PySide2/Qt/plugins/platforms/../../lib/libQt5XcbQpa.so.5 (0x00007f89377be000)
	libfontconfig.so.1 => not found
	libfreetype.so.6 => not found
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f89375a4000)
	libQt5Gui.so.5 => /root/.pyenv/versions/3.6.12/lib/python3.6/site-packages/PySide2/Qt/plugins/platforms/../../lib/libQt5Gui.so.5 (0x00007f8936c73000)
	libQt5DBus.so.5 => /root/.pyenv/versions/3.6.12/lib/python3.6/site-packages/PySide2/Qt/plugins/platforms/../../lib/libQt5DBus.so.5 (0x00007f89369e7000)
	libQt5Core.so.5 => /root/.pyenv/versions/3.6.12/lib/python3.6/site-packages/PySide2/Qt/plugins/platforms/../../lib/libQt5Core.so.5 (0x00007f89361f1000)
	libGL.so.1 => not found
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f8935fd4000)
	libX11-xcb.so.1 => not found
	libxcb-icccm.so.4 => not found
	libxcb-image.so.0 => not found
	libxcb-shm.so.0 => not found
	libxcb-util.so.1 => not found
	libxcb-keysyms.so.1 => not found
	libxcb-randr.so.0 => not found
	libxcb-render-util.so.0 => not found
	libxcb-render.so.0 => not found
	libxcb-shape.so.0 => not found
	libxcb-sync.so.1 => not found
	libxcb-xfixes.so.0 => not found
	libxcb-xinerama.so.0 => not found
	libxcb-xkb.so.1 => not found
	libxcb.so.1 => /usr/lib/x86_64-linux-gnu/libxcb.so.1 (0x00007f8935dac000)
	libXext.so.6 => /usr/lib/x86_64-linux-gnu/libXext.so.6 (0x00007f8935b9a000)
	libX11.so.6 => /usr/lib/x86_64-linux-gnu/libX11.so.6 (0x00007f893585a000)
	libxkbcommon-x11.so.0 => not found
	libxkbcommon.so.0 => not found
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f8935656000)
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f89352d4000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f8934fd0000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f8934db9000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f8934a1a000)
	libfontconfig.so.1 => not found
	libfreetype.so.6 => not found
	libGL.so.1 => not found
	libX11-xcb.so.1 => not found
	libxcb-icccm.so.4 => not found
	libxcb-image.so.0 => not found
	libxcb-shm.so.0 => not found
	libxcb-util.so.1 => not found
	libxcb-keysyms.so.1 => not found
	libxcb-randr.so.0 => not found
	libxcb-render-util.so.0 => not found
	libxcb-render.so.0 => not found
	libxcb-shape.so.0 => not found
	libxcb-sync.so.1 => not found
	libxcb-xfixes.so.0 => not found
	libxcb-xinerama.so.0 => not found
	libxcb-xkb.so.1 => not found
	libxkbcommon-x11.so.0 => not found
	libxkbcommon.so.0 => not found
	libgthread-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgthread-2.0.so.0 (0x00007f8934818000)
	libglib-2.0.so.0 => /lib/x86_64-linux-gnu/libglib-2.0.so.0 (0x00007f8934504000)
	libGL.so.1 => not found
	libdbus-1.so.3 => not found
	libicui18n.so.56 => /root/.pyenv/versions/3.6.12/lib/python3.6/site-packages/PySide2/Qt/plugins/platforms/../../lib/libicui18n.so.56 (0x00007f893406b000)
	libicuuc.so.56 => /root/.pyenv/versions/3.6.12/lib/python3.6/site-packages/PySide2/Qt/plugins/platforms/../../lib/libicuuc.so.56 (0x00007f8933cb3000)
	libicudata.so.56 => /root/.pyenv/versions/3.6.12/lib/python3.6/site-packages/PySide2/Qt/plugins/platforms/../../lib/libicudata.so.56 (0x00007f89322d0000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f8937d1b000)
	libXau.so.6 => /usr/lib/x86_64-linux-gnu/libXau.so.6 (0x00007f89320cc000)
	libXdmcp.so.6 => /usr/lib/x86_64-linux-gnu/libXdmcp.so.6 (0x00007f8931ec6000)
	libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f8931c53000)
	libbsd.so.0 => /lib/x86_64-linux-gnu/libbsd.so.0 (0x00007f8931a3e000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f8931836000)
```

</p></details>


So the environment is missing a lot of libraries (although some of them would explicitly not be collected by pyinstaller). 

Now compare this to list of dependencies on the host machine (in my case, Fedora):


<details><summary>Output of the ldd command</summary>
<p>

```
[rok@alpha pyinstaller-pyside2-docker-failure]$ LD_LIBRARY_PATH=./dist/main/ ldd dist/main/PySide2/plugins/platforms/libqxcb.so 
dist/main/PySide2/plugins/platforms/libqxcb.so: ./dist/main/libz.so.1: version `ZLIB_1.2.9' not found (required by /lib64/libpng16.so.16)
	linux-vdso.so.1 (0x00007ffe877fc000)
	libQt5XcbQpa.so.5 => ./dist/main/libQt5XcbQpa.so.5 (0x00007f35e5e16000)
	libfontconfig.so.1 => /lib64/libfontconfig.so.1 (0x00007f35e5da8000)
	libfreetype.so.6 => /lib64/libfreetype.so.6 (0x00007f35e5ce4000)
	libz.so.1 => ./dist/main/libz.so.1 (0x00007f35e5aca000)
	libQt5Gui.so.5 => ./dist/main/libQt5Gui.so.5 (0x00007f35e5199000)
	libQt5DBus.so.5 => ./dist/main/libQt5DBus.so.5 (0x00007f35e4f0b000)
	libQt5Core.so.5 => ./dist/main/libQt5Core.so.5 (0x00007f35e4715000)
	libGL.so.1 => /lib64/libGL.so.1 (0x00007f35e468e000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f35e466d000)
	libX11-xcb.so.1 => /lib64/libX11-xcb.so.1 (0x00007f35e4668000)
	libxcb-icccm.so.4 => /lib64/libxcb-icccm.so.4 (0x00007f35e4661000)
	libxcb-image.so.0 => /lib64/libxcb-image.so.0 (0x00007f35e4659000)
	libxcb-shm.so.0 => /lib64/libxcb-shm.so.0 (0x00007f35e4654000)
	libxcb-util.so.1 => /lib64/libxcb-util.so.1 (0x00007f35e464b000)
	libxcb-keysyms.so.1 => /lib64/libxcb-keysyms.so.1 (0x00007f35e4646000)
	libxcb-randr.so.0 => /lib64/libxcb-randr.so.0 (0x00007f35e4633000)
	libxcb-render-util.so.0 => /lib64/libxcb-render-util.so.0 (0x00007f35e462c000)
	libxcb-render.so.0 => /lib64/libxcb-render.so.0 (0x00007f35e461a000)
	libxcb-shape.so.0 => /lib64/libxcb-shape.so.0 (0x00007f35e4615000)
	libxcb-sync.so.1 => /lib64/libxcb-sync.so.1 (0x00007f35e460b000)
	libxcb-xfixes.so.0 => /lib64/libxcb-xfixes.so.0 (0x00007f35e4601000)
	libxcb-xinerama.so.0 => /lib64/libxcb-xinerama.so.0 (0x00007f35e45fc000)
	libxcb-xkb.so.1 => /lib64/libxcb-xkb.so.1 (0x00007f35e45de000)
	libxcb.so.1 => /lib64/libxcb.so.1 (0x00007f35e45b1000)
	libXext.so.6 => ./dist/main/libXext.so.6 (0x00007f35e439f000)
	libX11.so.6 => ./dist/main/libX11.so.6 (0x00007f35e405f000)
	libxkbcommon-x11.so.0 => /lib64/libxkbcommon-x11.so.0 (0x00007f35e4054000)
	libxkbcommon.so.0 => /lib64/libxkbcommon.so.0 (0x00007f35e400b000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f35e4004000)
	libstdc++.so.6 => ./dist/main/libstdc++.so.6 (0x00007f35e3c80000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f35e3b3c000)
	libgcc_s.so.1 => ./dist/main/libgcc_s.so.1 (0x00007f35e3925000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f35e3756000)
	libgthread-2.0.so.0 => ./dist/main/libgthread-2.0.so.0 (0x00007f35e3554000)
	libglib-2.0.so.0 => ./dist/main/libglib-2.0.so.0 (0x00007f35e3240000)
	libxml2.so.2 => /lib64/libxml2.so.2 (0x00007f35e30b5000)
	libbz2.so.1 => /lib64/libbz2.so.1 (0x00007f35e30a2000)
	libpng16.so.16 => /lib64/libpng16.so.16 (0x00007f35e3069000)
	libharfbuzz.so.0 => /lib64/libharfbuzz.so.0 (0x00007f35e2f99000)
	libbrotlidec.so.1 => /lib64/libbrotlidec.so.1 (0x00007f35e2f8b000)
	libdbus-1.so.3 => /lib64/libdbus-1.so.3 (0x00007f35e2f37000)
	libicui18n.so.56 => ./dist/main/libicui18n.so.56 (0x00007f35e2a9e000)
	libicuuc.so.56 => ./dist/main/libicuuc.so.56 (0x00007f35e26e6000)
	libicudata.so.56 => ./dist/main/libicudata.so.56 (0x00007f35e0d03000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f35e6375000)
	libGLX.so.0 => /lib64/libGLX.so.0 (0x00007f35e0cd1000)
	libGLdispatch.so.0 => /lib64/libGLdispatch.so.0 (0x00007f35e0c17000)
	libXau.so.6 => ./dist/main/libXau.so.6 (0x00007f35e0a13000)
	libpcre.so.3 => ./dist/main/libpcre.so.3 (0x00007f35e07a0000)
	liblzma.so.5 => ./dist/main/liblzma.so.5 (0x00007f35e057a000)
	libgraphite2.so.3 => /lib64/libgraphite2.so.3 (0x00007f35e0557000)
	libbrotlicommon.so.1 => /lib64/libbrotlicommon.so.1 (0x00007f35e0534000)
	libsystemd.so.0 => /lib64/libsystemd.so.0 (0x00007f35e0470000)
	librt.so.1 => /lib64/librt.so.1 (0x00007f35e0465000)
	libzstd.so.1 => /lib64/libzstd.so.1 (0x00007f35e036f000)
	liblz4.so.1 => /lib64/liblz4.so.1 (0x00007f35e0349000)
	libcap.so.2 => /lib64/libcap.so.2 (0x00007f35e033f000)
	libselinux.so.1 => ./dist/main/libselinux.so.1 (0x00007f35e0117000)
	libgcrypt.so.20 => /lib64/libgcrypt.so.20 (0x00007f35dffdc000)
	libgpg-error.so.0 => /lib64/libgpg-error.so.0 (0x00007f35dffb6000)
```

</p></details>


The above list contains both direct dependencies, as well as their dependencies. `libpng16.so.16` is now on the list, because at least one of direct dependencies (e.g., `libfontconfig.so.1`) depends on it.

So `libpng16.so.16` not being collected is consequence of missing `libfontconfig` and `freetype` in the container.

But as a side note, moving `fontconfig` and `freetype` libraries across different distributions might cause other errors due to incompatibilities and distribution-specific behavior (see #4985).






Thanks for looking into this. Based on your findings, I was able to resolve the issue by adding `qt5-default` to the list of installed packages. After doing this, `ldd libqxcb.so` does show most of the libraries as being found (`libxcb-util.so.1` is missing, but it does not seem to have affected functionality).

But this leads me to wonder how come PyInstaller doesn't have any warnings or errors when libraries are not found.

Another thing to note is that this container (without the `qt5-default` package) did work previously to build my actual project with all of the libraries. But I'll just chalk that up to the base container being updated and changed some time between then and now.
> But this leads me to wonder how come PyInstaller doesn't have any warnings or errors when libraries are not found.

Yeah, that is rather odd considering how much noise it makes on other OSes in similar situations...

I think the culprit is the fact that regex used for parsing `ldd`  output does not account for `=> not found` option:
https://github.com/pyinstaller/pyinstaller/blob/fbf58d88ed35c8ac46c447a0ea19413484754732/PyInstaller/depend/bindepend.py#L567

So the lines in `ldd` output that correspond to missing libraries end up being quietly dropped...


Resolving patch:
diff --git a/PyInstaller/depend/bindepend.py b/PyInstaller/depend/bindepend.py
--- a/PyInstaller/depend/bindepend.py
+++ b/PyInstaller/depend/bindepend.py
@@ -527,11 +527,8 @@ def selectImports(pth, xtrapath=None):
                 logger.debug("Adding %s dependency of %s from %s",
                              lib, os.path.basename(pth), npth)
                 rv.append((lib, npth))
-        else:
-            # Don't spew out false warnings on win 10 and UCRT (see issue
-            # #1566).
-            if not (compat.is_win_10 and lib.startswith("api-ms-win-crt")):
-                logger.warning("lib not found: %s dependency of %s", lib, pth)
+        elif dylib.warn_missing_lib(lib):
+            logger.warning("lib not found: %s dependency of %s", lib, pth)
 
     return rv
 
@@ -601,9 +598,18 @@ def _getImports_ldd(pth):
                 # Add lib if it is not already found.
                 if lib not in rslt:
                     rslt.add(lib)
-            else:
-                logger.error('Can not find %s in path %s (needed by %s)',
-                             name, lib, pth)
+            elif dylib.warn_missing_lib(name):
+                logger.warning('Cannot find %s in path %s (needed by %s)',
+                               name, lib, pth)
+        elif line.endswith("not found"):
+            # On glibc-based linux distributions, missing libraries
+            # are marked with name.so => not found
+            tokens = line.split('=>')
+            if len(tokens) != 2:
+                continue
+            name = tokens[0].strip()
+            if dylib.warn_missing_lib(name):
+                logger.warning('Cannot find %s (needed by %s)', name, pth)
     return rslt
 
 
@@ -697,9 +703,9 @@ def _getImports_macholib(pth):
                     final_lib = os.path.abspath(os.path.join(run_path, lib))
                     rslt.add(final_lib)
                     break
-            # Log error if no existing file found.
-            if not final_lib:
-                logger.error('Can not find path %s (needed by %s)', lib, pth)
+            # Log warning if no existing file found.
+            if not final_lib and dylib.warn_missing_lib(lib):
+                logger.warning('Cannot find path %s (needed by %s)', lib, pth)
 
         # Macholib has to be used to get absolute path to libraries.
         else:
@@ -715,9 +721,9 @@ def _getImports_macholib(pth):
                 # Starting with Big Sur, system libraries are hidden. And
                 # we do not collect system libraries on any macOS version
                 # anyway, so suppress the corresponding error messages.
-                if not in_system_path(lib):
-                    logger.error('Can not find path %s (needed by %s)',
-                                 lib, pth)
+                if not in_system_path(lib) and dylib.warn_missing_lib(lib):
+                    logger.warning('Cannot find path %s (needed by %s)',
+                                   lib, pth)
 
     return rslt
 
diff --git a/PyInstaller/depend/dylib.py b/PyInstaller/depend/dylib.py
--- a/PyInstaller/depend/dylib.py
+++ b/PyInstaller/depend/dylib.py
@@ -26,7 +26,7 @@
 import re
 
 
-from PyInstaller.compat import is_win, is_unix, is_aix, is_darwin
+from PyInstaller import compat
 
 
 import PyInstaller.log as logging
@@ -167,13 +167,13 @@
 }
 
 
-if is_win:
+if compat.is_win:
     _includes |= _win_includes
     _excludes |= _win_excludes
-elif is_aix:
+elif compat.is_aix:
     # The exclude list for AIX differs from other *nix platforms.
     _excludes |= _aix_excludes
-elif is_unix:
+elif compat.is_unix:
     # Common excludes for *nix platforms -- except AIX.
     _excludes |= _unix_excludes
 
@@ -206,7 +206,7 @@ def search(self, libname):
 include_list = IncludeList()
 
 
-if is_darwin:
+if compat.is_darwin:
     # On Mac use macholib to decide if a binary is a system one.
     from macholib import util
 
@@ -227,7 +227,7 @@ def search(self, libname):
 
     exclude_list = MacExcludeList(exclude_list)
 
-elif is_win:
+elif compat.is_win:
     class WinExcludeList(object):
         def __init__(self, global_exclude_list):
             self._exclude_list = global_exclude_list
@@ -270,6 +270,49 @@ def include_library(libname):
         return True
 
 
+# Patterns for suppressing warnings about missing dynamically linked
+# libraries
+_warning_suppressions = [
+    # We fail to discover shiboken2 (PySide2) and shiboken6 (PySide6)
+    # shared libraries due to the way the packages set up the search
+    # path to the library, which is located in a separate package.
+    # Suppress the harmless warnings to avoid confusion.
+    r'(lib)?shiboken.*',
+]
+
+# On some systems (e.g., openwrt), libc.so might point to ldd. Suppress
+# warnings about it.
+if compat.is_linux:
+    _warning_suppressions.append(r'ldd')
+
+# Suppress false warnings on win 10 and UCRT (see issue #1566).
+if compat.is_win_10:
+    _warning_suppressions.append(r'api-ms-win-crt.*')
+
+
+class MissingLibWarningSuppressionList:
+    def __init__(self):
+        self.regex = re.compile('|'.join(_warning_suppressions), re.I)
+
+    def search(self, libname):
+        # Running re.search() on '' regex never returns None.
+        if _warning_suppressions:
+            return self.regex.match(os.path.basename(libname))
+        else:
+            return False
+
+
+missing_lib_warning_suppression_list = MissingLibWarningSuppressionList()
+
+
+def warn_missing_lib(libname):
+    """
+    Check if a missing-library warning should be displayed for the
+    given library name (or full path).
+    """
+    return not missing_lib_warning_suppression_list.search(libname)
+
+
 def mac_set_relative_dylib_deps(libname, distname):
     """
     On Mac OS X set relative paths to dynamic library dependencies


Test patch: