Repository: pyinstaller/pyinstaller

Issue:
Broken ctypes dependency scanning on Kivy
<!--
Welcome to the PyInstaller issue tracker! Before creating an issue, please heed the following:

1. This tracker should only be used to report bugs and request features / enhancements to PyInstaller
    - For questions and general support, use the mailing list, see
	  <http://www.pyinstaller.org/support.html#mailing-list>
2. Use the search function before creating a new issue. Duplicates will be closed and directed to
   the original discussion.
3. When making a bug report, make sure you provide all required information. The easier it is for
   maintainers to reproduce, the faster it'll be fixed.
-->

+++ ONLY TEXT +++ DO NOT POST IMAGES +++

## Description of the issue

[kivy.clock](https://github.com/kivy/kivy/blob/9c69f30742e1df8190ae9ce1293edf6d6b0c9f7e/kivy/clock.py#L507) contains a reference to `libc.dylib` which isn't ignored on Linux-based systems. Due to a recent change in Python, looking for libc via `ctypes.util.find_library('libc')` will [cause an exception to be thrown](https://stackoverflow.com/questions/65410697/python-scapy-all-file-not-found) and stops the build process. Add a try...except block around [this find_library call](https://github.com/pyinstaller/pyinstaller/blob/7d77825f42a5929d08b3337fd7b4550bc5387f42/PyInstaller/depend/utils.py#L307) works around the issue but there will still be a warning about it can't find libc.dylib.

## Context information (for bug reports)

* Output of `pyinstaller --version`: ```5.0.dev0``` (7d77825f42a5929d08b3337fd7b4550bc5387f42)
* Version of Python: 3.9
* Platform: GNU/Linux
* Did you also try this on another platform? Does it work there?

No but should still work on Windows since PyInstaller excludes all *.dylib on that platform, and on OSX for obvious reasons.

* try the latest development version, using the following command:

```shell
pip install https://github.com/pyinstaller/pyinstaller/archive/develop.zip
```

* follow *all* the instructions in our "If Things Go Wrong" Guide
  (https://github.com/pyinstaller/pyinstaller/wiki/If-Things-Go-Wrong) and

### Make sure [everything is packaged correctly](https://github.com/pyinstaller/pyinstaller/wiki/How-to-Report-Bugs#make-sure-everything-is-packaged-correctly)

  * [X] start with clean installation
  * [X] use the latest development version (7d77825f42a5929d08b3337fd7b4550bc5387f42)
  * [ ] Run your frozen program **from a command window (shell)** — instead of double-clicking on it
  * [ ] Package your program in **--onedir mode**
  * [ ] Package **without UPX**, say: use the option `--noupx` or set `upx=False` in your .spec-file
  * [ ] Repackage you application in **verbose/debug mode**. For this, pass the option `--debug` to `pyi-makespec` or `pyinstaller` or use `EXE(..., debug=1, ...)` in your .spec file.


### A minimal example program which shows the error

Any kivy example program on Python 3.9 should be able to reproduce this issue.

### Stacktrace / full error message

```
17603 INFO: Looking for ctypes DLLs
 Traceback (most recent call last):
17667 WARNING: stderr: Traceback (most recent call last):
   File "<REDACTED VENV PATH>/bin/pyinstaller", line 8, in <module>
17667 WARNING: stderr:   File "<REDACTED VENV PATH>/bin/pyinstaller", line 8, in <module>
     sys.exit(run())
17667 WARNING: stderr:     sys.exit(run())
   File "<REDACTED VENV PATH>/lib/python3.9/site-packages/PyInstaller/__main__.py", line 114, in run
17667 WARNING: stderr:   File "<REDACTED VENV PATH>/lib/python3.9/site-packages/PyInstaller/__main__.py", line 114, in run
     run_build(pyi_config, spec_file, **vars(args))
17668 WARNING: stderr:     run_build(pyi_config, spec_file, **vars(args))
   File "<REDACTED VENV PATH>/lib/python3.9/site-packages/PyInstaller/__main__.py", line 65, in run_build
17668 WARNING: stderr:   File "<REDACTED VENV PATH>/lib/python3.9/site-packages/PyInstaller/__main__.py", line 65, in run_build
     PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
17668 WARNING: stderr:     PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
   File "<REDACTED VENV PATH>/lib/python3.9/site-packages/PyInstaller/building/build_main.py", line 725, in main
17668 WARNING: stderr:   File "<REDACTED VENV PATH>/lib/python3.9/site-packages/PyInstaller/building/build_main.py", line 725, in main
     build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))
17669 WARNING: stderr:     build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))
   File "<REDACTED VENV PATH>/lib/python3.9/site-packages/PyInstaller/building/build_main.py", line 672, in build
17669 WARNING: stderr:   File "<REDACTED VENV PATH>/lib/python3.9/site-packages/PyInstaller/building/build_main.py", line 672, in build
     exec(code, spec_namespace)
17669 WARNING: stderr:     exec(code, spec_namespace)
   File "<REDACTED SPEC FILE>.spec", line 7, in <module>
17669 WARNING: stderr:   File "<REDACTED SPEC FILE>.spec", line 7, in <module>
     a = Analysis(['src/start.py'],
17669 WARNING: stderr:     a = Analysis(['src/start.py'],
   File "<REDACTED VENV PATH>/lib/python3.9/site-packages/PyInstaller/building/build_main.py", line 242, in __init__
17669 WARNING: stderr:   File "<REDACTED VENV PATH>/lib/python3.9/site-packages/PyInstaller/building/build_main.py", line 242, in __init__
     self.__postinit__()
17670 WARNING: stderr:     self.__postinit__()
   File "<REDACTED VENV PATH>/lib/python3.9/site-packages/PyInstaller/building/datastruct.py", line 160, in __postinit__
17670 WARNING: stderr:   File "<REDACTED VENV PATH>/lib/python3.9/site-packages/PyInstaller/building/datastruct.py", line 160, in __postinit__
     self.assemble()
17670 WARNING: stderr:     self.assemble()
   File "<REDACTED VENV PATH>/lib/python3.9/site-packages/PyInstaller/building/build_main.py", line 438, in assemble
17670 WARNING: stderr:   File "<REDACTED VENV PATH>/lib/python3.9/site-packages/PyInstaller/building/build_main.py", line 438, in assemble
     ctypes_binaries = scan_code_for_ctypes(co)
17671 WARNING: stderr:     ctypes_binaries = scan_code_for_ctypes(co)
   File "<REDACTED VENV PATH>/lib/python3.9/site-packages/PyInstaller/depend/utils.py", line 151, in scan_code_for_ctypes
17671 WARNING: stderr:   File "<REDACTED VENV PATH>/lib/python3.9/site-packages/PyInstaller/depend/utils.py", line 151, in scan_code_for_ctypes
     binaries = _resolveCtypesImports(binaries)
17671 WARNING: stderr:     binaries = _resolveCtypesImports(binaries)
   File "<REDACTED VENV PATH>/lib/python3.9/site-packages/PyInstaller/depend/utils.py", line 308, in _resolveCtypesImports
17671 WARNING: stderr:   File "<REDACTED VENV PATH>/lib/python3.9/site-packages/PyInstaller/depend/utils.py", line 308, in _resolveCtypesImports
     cpath = find_library(os.path.splitext(cbin)[0])
17671 WARNING: stderr:     cpath = find_library(os.path.splitext(cbin)[0])
   File "/usr/lib/python3.9/ctypes/util.py", line 330, in find_library
17671 WARNING: stderr:   File "/usr/lib/python3.9/ctypes/util.py", line 330, in find_library
     _get_soname(_findLib_gcc(name)) or _get_soname(_findLib_ld(name))
17672 WARNING: stderr:     _get_soname(_findLib_gcc(name)) or _get_soname(_findLib_ld(name))
   File "/usr/lib/python3.9/ctypes/util.py", line 147, in _findLib_gcc
17672 WARNING: stderr:   File "/usr/lib/python3.9/ctypes/util.py", line 147, in _findLib_gcc
     if not _is_elf(file):
17672 WARNING: stderr:     if not _is_elf(file):
   File "/usr/lib/python3.9/ctypes/util.py", line 99, in _is_elf
17672 WARNING: stderr:   File "/usr/lib/python3.9/ctypes/util.py", line 99, in _is_elf
     with open(filename, 'br') as thefile:
17672 WARNING: stderr:     with open(filename, 'br') as thefile:
 FileNotFoundError: [Errno 2] No such file or directory: b'liblibc.a'
17672 WARNING: stderr: FileNotFoundError: [Errno 2] No such file or directory: b'liblibc.a'
```

Please also see <https://github.com/pyinstaller/pyinstaller/wiki/How-to-Report-Bugs>
for more about what would use to solve the issue.



Discussion:
Hmm, I saw this with libc too. There's a bug in Python's ctypes itself which I've been meaning to report but I need to find a docker image it can be reproduced on because it doesn't happen on most Linuxes. It should silently return None and move on.
```python
>>> from ctypes.util import find_library
>>> find_library("libc.so")
>>> find_library("libc")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/brenainn/.pyenv/versions/3.9.1/lib/python3.9/ctypes/util.py", line 330, in find_library
    _get_soname(_findLib_gcc(name)) or _get_soname(_findLib_ld(name))
  File "/home/brenainn/.pyenv/versions/3.9.1/lib/python3.9/ctypes/util.py", line 147, in _findLib_gcc
    if not _is_elf(file):
  File "/home/brenainn/.pyenv/versions/3.9.1/lib/python3.9/ctypes/util.py", line 99, in _is_elf
    with open(filename, 'br') as thefile:
FileNotFoundError: [Errno 2] No such file or directory: b'liblibc.a'
```
I've only been able to reproduce this on Manjaro linux. Can I have your full platform name? i.e. `uname -a`.
Hmm, on Fedora (python 3.9.2) calling `find_library("libc")` also returns `None`. And the python code involved does not seem to be patched in any way...

@bwoodsend, can you try running `gcc -Wl,-t -o prog_tmp -llibc` on your Majaro system and paste the console output (compilation should fail, but it's the output we're interested in)?
```bash
brenainn@Ostrich-manjaro /e/n/pyinstaller (cross-compile-freebsd)> gcc -Wl,-t -o prog_tmp -llibc
/usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../lib/Scrt1.o
/usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../lib/crti.o
/usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/crtbeginS.o
/usr/bin/ld: cannot find -llibc
/usr/bin/ld: note to link with /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../lib/libc.a use -l:libc.a or rename it to liblibc.a
/usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/libgcc.a
/usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../lib/libgcc_s.so
/usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../lib/libgcc_s.so.1
/usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/libgcc.a
collect2: error: ld returned 1 exit status
```
Aha, so it's compiler-dependent. This diagnostic note:

> /usr/bin/ld: note to link with /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../lib/libc.a use -l:libc.a or rename it to liblibc.a

is being matched by regex in `_findLib_gcc`, and yields `liblibc.a`.
I think the best way to deal with this is to simply enclose the `find_library` call in try/catch block, as the OP suggested. This will yield warning about `library libc.dylib required via ctypes not found`, but that's the current behavior on other linux distributions anyway...

We could also perform some additional checks on the library names and skip incompatible ones for particular OS (e.g., `.dylib` suffix is valid only on macOS, `.dll` suffix only on Windows; and on linux, there should be no suffix at all).
> I think the best way to deal with this is to simply enclose the find_library call in try/catch block, as the OP suggested.

Yeah, I'll do this now.

>  and on linux, there should be no suffix at all).

Except on Alpine. For some reason the suffix is mandatory there:
```python
brenainn@Ostrich-manjaro /e/n/pyinstaller $ docker run -it python:3.9.0-alpine 
Python 3.9.0 (default, Nov 25 2020, 02:36:55) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes
>>> ctypes.CDLL("libc")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.9/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: Error loading shared library libc: No such file or directory
>>> ctypes.CDLL("libc.so")
<CDLL 'libc.so', handle 7fc69d0a3b40 at 0x7fc69cb78a30>
>>> ctypes.CDLL("libm")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.9/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: Error loading shared library libm: No such file or directory
>>> ctypes.CDLL("libm.so")
<CDLL 'libm.so', handle 7fc69d0a3b40 at 0x7fc69cb78a60>
```
Ah, right, our ctype-handling codepath is catering to both `ctypes.CDLL()` and `find_library()` behavior...

Resolving patch:
diff --git a/PyInstaller/depend/utils.py b/PyInstaller/depend/utils.py
--- a/PyInstaller/depend/utils.py
+++ b/PyInstaller/depend/utils.py
@@ -304,7 +304,13 @@ def _restorePaths(old):
     # local paths to library search paths, then replaces original values.
     old = _setPaths()
     for cbin in cbinaries:
-        cpath = find_library(os.path.splitext(cbin)[0])
+        try:
+            # There is an issue with find_library() where it can run into
+            # errors trying to locate the library. See #5734.
+            cpath = find_library(os.path.splitext(cbin)[0])
+        except FileNotFoundError:
+            # In these cases, find_library() should return None.
+            cpath = None
         if is_unix:
             # CAVEAT: find_library() is not the correct function. Ctype's
             # documentation says that it is meant to resolve only the filename


Test patch:
diff --git a/tests/functional/test_regression.py b/tests/functional/test_regression.py
--- a/tests/functional/test_regression.py
+++ b/tests/functional/test_regression.py
@@ -92,3 +92,13 @@ def test_issue_4141(pyi_builder):  #script_dir,
     pyi_builder.test_script('pyi_issue_4141.py',
                             app_name="main", run_from_path=True,
                             pyi_args=['--path', str(extra_path)])
+
+
+def test_5734():
+    """
+    In a regression this will raise a:
+        FileNotFoundError: [Errno 2] No such file or directory: b'liblibc.a'
+    on some Linux/gcc combinations.
+    """
+    from PyInstaller.depend.utils import _resolveCtypesImports
+    _resolveCtypesImports(["libc"])