Repository: pyinstaller/pyinstaller

Issue:
Pyinstaller3.X does not find libpython3.6m.so (AIX)
Hi. New to PyInstaller. Is this a bug?

Pyinstaller3,4 finds libpython3.6m.so but does not recognize it as a valid library.

After I create a symbolic link from /usr/lib/libpython3.6.so to libpython3.6m.so it works fine.

FYI: platform is AIX, Python is Python3-3.6.8 (see [https://github.com/aixtools/cpython/tree/aix-v3.6.8](https://github.com/aixtools/cpython/tree/aix-v3.6.8)) - I have a patch to ctypes (which is in v3.7, but not backported to v3.6) to fix ctype.find_library() and also to allow load of a MEMBER file of an AIX archive.
For this particular issue finding an archive member and/or dlload() of an archive(member) is not relevant as the default shared library packaging creates a .so file, rather than an archive containing a .so file.

```
root@x067:[/home/root/python/hello]ldd /opt/bin/python3
/opt/bin/python3 needs:
         /usr/lib/libc.a(shr.o)
         /usr/lib/libpthreads.a(shr_xpg5.o)
         /usr/lib/libpython3.6m.so
         /unix
         /usr/lib/libcrypt.a(shr.o)
         /usr/lib/libpthreads.a(shr_comm.o)
         /usr/lib/libdl.a(shr.o)

```




Discussion:
I left this bit out - I added a few print statements to build_main.py to track what is (not) happening.

```
15886 INFO: Analyzing run-time hooks ...
15908 INFO: Looking for dynamic libraries
ldd: /usr/lib/libssl.a: File is an archive.
ldd: /usr/lib/libcrypto.a: File is an archive.
34516 INFO: Looking for eggs
libpython3.6m.so /usr/lib/libpython3.6m.so BINARY
resource /opt/lib/python3.6/lib-dynload/resource.so EXTENSION
_pickle /opt/lib/python3.6/lib-dynload/_pickle.so EXTENSION
_datetime /opt/lib/python3.6/lib-dynload/_datetime.so EXTENSION
_posixsubprocess /opt/lib/python3.6/lib-dynload/_posixsubprocess.so EXTENSION
_ssl /opt/lib/python3.6/lib-dynload/_ssl.so EXTENSION
unicodedata /opt/lib/python3.6/lib-dynload/unicodedata.so EXTENSION
_codecs_hk /opt/lib/python3.6/lib-dynload/_codecs_hk.so EXTENSION
_codecs_kr /opt/lib/python3.6/lib-dynload/_codecs_kr.so EXTENSION
_codecs_cn /opt/lib/python3.6/lib-dynload/_codecs_cn.so EXTENSION
_codecs_jp /opt/lib/python3.6/lib-dynload/_codecs_jp.so EXTENSION
_codecs_tw /opt/lib/python3.6/lib-dynload/_codecs_tw.so EXTENSION
_codecs_iso2022 /opt/lib/python3.6/lib-dynload/_codecs_iso2022.so EXTENSION
_multibytecodec /opt/lib/python3.6/lib-dynload/_multibytecodec.so EXTENSION
_heapq /opt/lib/python3.6/lib-dynload/_heapq.so EXTENSION
_opcode /opt/lib/python3.6/lib-dynload/_opcode.so EXTENSION
pyexpat /opt/lib/python3.6/lib-dynload/pyexpat.so EXTENSION
termios /opt/lib/python3.6/lib-dynload/termios.so EXTENSION
_sha1 /opt/lib/python3.6/lib-dynload/_sha1.so EXTENSION
_md5 /opt/lib/python3.6/lib-dynload/_md5.so EXTENSION
_sha256 /opt/lib/python3.6/lib-dynload/_sha256.so EXTENSION
_sha512 /opt/lib/python3.6/lib-dynload/_sha512.so EXTENSION
_blake2 /opt/lib/python3.6/lib-dynload/_blake2.so EXTENSION
_sha3 /opt/lib/python3.6/lib-dynload/_sha3.so EXTENSION
_hashlib /opt/lib/python3.6/lib-dynload/_hashlib.so EXTENSION
_bisect /opt/lib/python3.6/lib-dynload/_bisect.so EXTENSION
_random /opt/lib/python3.6/lib-dynload/_random.so EXTENSION
select /opt/lib/python3.6/lib-dynload/select.so EXTENSION
zlib /opt/lib/python3.6/lib-dynload/zlib.so EXTENSION
_bz2 /opt/lib/python3.6/lib-dynload/_bz2.so EXTENSION
_lzma /opt/lib/python3.6/lib-dynload/_lzma.so EXTENSION
grp /opt/lib/python3.6/lib-dynload/grp.so EXTENSION
binascii /opt/lib/python3.6/lib-dynload/binascii.so EXTENSION
_socket /opt/lib/python3.6/lib-dynload/_socket.so EXTENSION
math /opt/lib/python3.6/lib-dynload/math.so EXTENSION
_struct /opt/lib/python3.6/lib-dynload/_struct.so EXTENSION
libssl.a /usr/lib/libssl.a BINARY
libcrypto.a /usr/lib/libcrypto.a BINARY
34518 INFO: Python library not in binary dependencies. Doing additional searching...
35339 INFO: Using Python library /lib/libpython3.6.so

```
After removing the symbolic links this is what I get:

```
14808 INFO: Looking for ctypes DLLs
14808 INFO: Looking for ctypes DLLs - finished
14809 INFO: Analyzing run-time hooks ...
14829 INFO: Looking for dynamic libraries
ldd: /usr/lib/libssl.a: File is an archive.
ldd: /usr/lib/libcrypto.a: File is an archive.
41147 INFO: Looking for eggs
libpython3.6m.so /opt/lib/libpython3.6m.so BINARY
_datetime /opt/lib/python3.6/lib-dynload/_datetime.so EXTENSION
resource /opt/lib/python3.6/lib-dynload/resource.so EXTENSION
_pickle /opt/lib/python3.6/lib-dynload/_pickle.so EXTENSION
_posixsubprocess /opt/lib/python3.6/lib-dynload/_posixsubprocess.so EXTENSION
_ssl /opt/lib/python3.6/lib-dynload/_ssl.so EXTENSION
_codecs_hk /opt/lib/python3.6/lib-dynload/_codecs_hk.so EXTENSION
_codecs_tw /opt/lib/python3.6/lib-dynload/_codecs_tw.so EXTENSION
_codecs_cn /opt/lib/python3.6/lib-dynload/_codecs_cn.so EXTENSION
_codecs_kr /opt/lib/python3.6/lib-dynload/_codecs_kr.so EXTENSION
_codecs_iso2022 /opt/lib/python3.6/lib-dynload/_codecs_iso2022.so EXTENSION
unicodedata /opt/lib/python3.6/lib-dynload/unicodedata.so EXTENSION
_codecs_jp /opt/lib/python3.6/lib-dynload/_codecs_jp.so EXTENSION
_multibytecodec /opt/lib/python3.6/lib-dynload/_multibytecodec.so EXTENSION
_heapq /opt/lib/python3.6/lib-dynload/_heapq.so EXTENSION
_opcode /opt/lib/python3.6/lib-dynload/_opcode.so EXTENSION
pyexpat /opt/lib/python3.6/lib-dynload/pyexpat.so EXTENSION
termios /opt/lib/python3.6/lib-dynload/termios.so EXTENSION
_sha1 /opt/lib/python3.6/lib-dynload/_sha1.so EXTENSION
_md5 /opt/lib/python3.6/lib-dynload/_md5.so EXTENSION
_sha256 /opt/lib/python3.6/lib-dynload/_sha256.so EXTENSION
_sha512 /opt/lib/python3.6/lib-dynload/_sha512.so EXTENSION
_blake2 /opt/lib/python3.6/lib-dynload/_blake2.so EXTENSION
_sha3 /opt/lib/python3.6/lib-dynload/_sha3.so EXTENSION
_hashlib /opt/lib/python3.6/lib-dynload/_hashlib.so EXTENSION
_bisect /opt/lib/python3.6/lib-dynload/_bisect.so EXTENSION
_random /opt/lib/python3.6/lib-dynload/_random.so EXTENSION
zlib /opt/lib/python3.6/lib-dynload/zlib.so EXTENSION
_bz2 /opt/lib/python3.6/lib-dynload/_bz2.so EXTENSION
_lzma /opt/lib/python3.6/lib-dynload/_lzma.so EXTENSION
grp /opt/lib/python3.6/lib-dynload/grp.so EXTENSION
binascii /opt/lib/python3.6/lib-dynload/binascii.so EXTENSION
_socket /opt/lib/python3.6/lib-dynload/_socket.so EXTENSION
_struct /opt/lib/python3.6/lib-dynload/_struct.so EXTENSION
select /opt/lib/python3.6/lib-dynload/select.so EXTENSION
math /opt/lib/python3.6/lib-dynload/math.so EXTENSION
libssl.a /usr/lib/libssl.a BINARY
libcrypto.a /usr/lib/libcrypto.a BINARY
41150 INFO: Python library not in binary dependencies. Doing additional searching...
Traceback (most recent call last):
  File "/opt/bin/pyinstaller", line 11, in <module>
    load_entry_point('PyInstaller==3.4', 'console_scripts', 'pyinstaller')()
  File "/opt/lib/python3.6/site-packages/PyInstaller-3.4-py3.6.egg/PyInstaller/__main__.py", line 111, in run
    run_build(pyi_config, spec_file, **vars(args))
  File "/opt/lib/python3.6/site-packages/PyInstaller-3.4-py3.6.egg/PyInstaller/__main__.py", line 63, in run_build
    PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
  File "/opt/lib/python3.6/site-packages/PyInstaller-3.4-py3.6.egg/PyInstaller/building/build_main.py", line 840, in main
    build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))
  File "/opt/lib/python3.6/site-packages/PyInstaller-3.4-py3.6.egg/PyInstaller/building/build_main.py", line 786, in build
    exec(text, spec_namespace)
  File "<string>", line 17, in <module>
  File "/opt/lib/python3.6/site-packages/PyInstaller-3.4-py3.6.egg/PyInstaller/building/build_main.py", line 241, in __init__
    self.__postinit__()
  File "/opt/lib/python3.6/site-packages/PyInstaller-3.4-py3.6.egg/PyInstaller/building/datastruct.py", line 158, in __postinit__
    self.assemble()
  File "/opt/lib/python3.6/site-packages/PyInstaller-3.4-py3.6.egg/PyInstaller/building/build_main.py", line 574, in assemble
    self._check_python_library(self.binaries)
  File "/opt/lib/python3.6/site-packages/PyInstaller-3.4-py3.6.egg/PyInstaller/building/build_main.py", line 678, in _check_python_library
    raise IOError(msg)
OSError: Python library not found: libpython3.6.a, libpython3.6.so
This would mean your Python installation doesn't come with proper library files.
This usually happens by missing development package, or unsuitable build parameters of Python installation.

Please note /usr/lib/libssl.a and usr/lib/libcrypto.a are archives that contain "shared" members.

One example:
root@x067:[/home/root/python/hello]ldd /usr/lib/libssl.a
ldd: /usr/lib/libssl.a: File is an archive.
root@x067:[/home/root/python/hello]ar tv /usr/lib/libssl.a
rwxr-xr-x     0/0     729117 Nov 27 06:51 2018 libssl.so
rwxr-xr-x     0/0     510766 Nov 27 06:51 2018 libssl.so.0.9.8
rwxr-xr-x     0/0     729117 Nov 27 06:51 2018 libssl.so.1.0.0
rwxr-xr-x     0/0     729117 Nov 27 05:57 2018 libssl.so.1.0.2

Note: the 'error' above goes away after:

root@x067:[/home/root/python/hello]ln -s /opt/lib/libpython3.6m.so /usr/lib/libpython3.6.so

...
39215 INFO: Python library not in binary dependencies. Doing additional searching...
41530 INFO: Using Python library /lib/libpython3.6.so
41543 INFO: Warnings written to /home/root/python/hello/build/hello/warn-hello.txt
41633 INFO: Graph cross-reference written to /home/root/python/hello/build/hello/xref-hello.html
41680 INFO: checking PYZ
41684 INFO: Building because toc changed
41684 INFO: Building PYZ (ZlibArchive) /home/root/python/hello/build/hello/PYZ-00.pyz
42597 INFO: Building PYZ (ZlibArchive) /home/root/python/hello/build/hello/PYZ-00.pyz completed successfully.
42606 INFO: checking PKG
42607 INFO: Building because toc changed
42608 INFO: Building PKG (CArchive) PKG-00.pkg
73816 INFO: Building PKG (CArchive) PKG-00.pkg completed successfully.
76145 INFO: Bootloader /opt/lib/python3.6/site-packages/PyInstaller-3.4-py3.6.egg/PyInstaller/bootloader/AIX-32bit/run
76145 INFO: checking EXE
76147 INFO: Building because toc changed
76148 INFO: Building EXE from EXE-00.toc
76156 INFO: Appending archive to EXE /home/root/python/hello/dist/hello
78951 INFO: Building EXE from EXE-00.toc completed successfully.

FYI: /lib is a symbolic link to /usr/lib
root@x067:[/home/root/python/hello]ls -l /lib
lrwxrwxrwx    1 bin      bin               8 May 01 13:09 /lib -> /usr/lib

root@x067:[/home/root/python/hello]ls -l /lib/libpython*
lrwxrwxrwx    1 root     system           25 May 06 05:58 /lib/libpython3.6.so -> /opt/lib/libpython3.6m.so

root@x067:[/home/root/python/hello]cat hello.py
print("Hello World")

Suggestions on where to add debug statements is appreciated!




```
Would like to look into this later, so hints are still appreciated.

I understand now that the 'm' is there becaue that means Python Malloc is used, rather than "system" malloc (and it does make a difference!). Maybe you know where to look within CPython to see whether the 'm' is valid, and/or needed. That is where I'll look, when I have time to examine. Currently I am setting symbolic links (manually).
It looks asif this will not be relevant after Python3.9 (as I do not notive the library being built as libpython3.9m.so - just as libpython3.9.so.

Better, imho, (and maybe you do already to this) would be to look for the .so object in an archive - like the loader does. In any case, since Python3.7 you should be able to use ctypes.find_library() to find any shared library AIX knows about. Before that - this was broken in CPython as it always returned NULL.

When in an archive ctypes.find_library("c") returns something like "libc.a[shr.o]"

e.g.,
>>> import ctypes.util
>>> ctypes.util.find_library("c")
'libc.a(shr.o)'
>>> ctypes.util.find_library("ssl")
'libssl.a(libssl.so)'

Please help me understand what the best approach is to test/document.

Michael
Is this still valid in PyInstaller 4.0dev?
I think it is resolved in PyInstaller 3.6 - but I need to do a virgin install to be sure. I may still have "adjustments" made to my environment so that I do not notice it.

PyInstaller 4.0dev does not build (I tried that yesterday with (nearly) the very latest off of develop branch. 

Also, iirc, starting with Python3.9 this issue also goes away because it appears Python is no longer adding the (m) for "PythonMalloc used" to the library name.
Python 3.9 is unsupported - so is 3.8. If you can check this is still valid in a clean Env etc please do so.
Was not asking for support on either 3.8 or 3.9, just commenting the 'm' character is disappearing (gone in 3.8 in any case).

fyi: when working with pyinstaller I am using Python 3.6 and 3.7
@aixtools that's useful to know. I've figured out the source of this; https://github.com/pyinstaller/pyinstaller/blob/96897fc78d9cbb5b8be9d353499b25295badbc9f/PyInstaller/compat.py#L73. If you can submit a PR which adds an extra name to the list etc that fixes it, please do. 
With pleasure - I'll make a PR. But which branch should I submit it against. Against "develop" I can submit a PR, but there are additional issues with "develop" that block a build. Still need to dig. (Just as build of PyInstaller3.6 against Python3-3.8.2, for mysterious reasons is not working).
Python 3.8 isn't supported; the bootloader needs a major overhaul for that version due to cpython differences. What are the issues with develop? It should be stable - as such.
The initial issue #4732 and #4733 - you fixed already, and closed #4733. That is mainly what I was referencing.

I have a heavy load this week - so I have my doubts that I will get a vanilla system installed to test 3.6 and 3.7 library naming by CPython.
@aixtools when you can test for more issues, please do.
Built Python v3.6 and it only builds one shared library - adding the "m" character.

`-r-xr-xr-x   1 bin      bin         6160233 Mar 09 18:08 /opt/lib/libpython3.6m.so`

so I'll proceed to work on a PR for this. Thx for the pointer on where to work!

Resolving patch:
diff --git a/PyInstaller/compat.py b/PyInstaller/compat.py
--- a/PyInstaller/compat.py
+++ b/PyInstaller/compat.py
@@ -68,10 +68,13 @@
                      'libpython%d.%d.dylib' % _pyver,
                      'libpython%d.%dm.dylib' % _pyver}
 elif is_aix:
-    # Shared libs on AIX are archives with shared object members, thus the ".a" suffix.
-    # However, python 2.7.11 built with XLC produces libpython?.?.so file, too.
+    # Shared libs on AIX may be archives with shared object members,
+    # hence the ".a" suffix. However, starting with python 2.7.11
+    # libpython?.?.so and Python3 libpython?.?m.so files are produced.
     PYDYLIB_NAMES = {'libpython%d.%d.a' % _pyver,
-                     'libpython%d.%d.so' % _pyver}
+                     'libpython%d.%dm.a' % _pyver,
+                     'libpython%d.%d.so' % _pyver,
+                     'libpython%d.%dm.so' % _pyver}
 elif is_freebsd:
     PYDYLIB_NAMES = {'libpython%d.%d.so.1' % _pyver,
                      'libpython%d.%dm.so.1' % _pyver,


Test patch: