Repository: pyinstaller/pyinstaller

Issue:
linux  python3.7 can not found '/tmp/_MEIZLLDaA/libpython3.7m.so‘

* Output of `pyinstaller --version`:  5.1
* Version of Python:  Python 3.7.11
* Platform:  CentOS Linux release 7.9.2009 (Core)
* How you installed Python:  conda
* Did you also try this on another platform? Does it work there? :no

## Description of the issue
1、I have a python file HelloServer.py
2、pyinstaller -F  HelloServer.py --distpath . 
3、generate HelloServer succ

Every time I modify the code HelloServer.py  and repackage  HelloServer 。

The strange  is
sometimes HelloServer run succ with out any error;
sometimes HelloServer run fail with  error like this:

 [21318] Error loading Python lib '/tmp/_MEIZLLDaA/libpython3.7m.so': dlopen: /tmp/_MEIZLLDaA/libpython3.7m.so: cannot open shared object file: No such file or directory_

"/tmp/_MEIZLLDaA/libpython3.7m.so" is not exsit in my system , and the right position is "~/miniconda3/envs/py37/lib/libpython3.7m.so"

Can I get some advice ？



Discussion:
`/tmp/_MEIXXXXXX` is a temporary directory where one-file (-F) build unpacks its content each time it is run. The last six characters in name are randomly generated, so they are different each time; and the temporary directory is cleaned up once the program finishes, so you will see it exist only while the program is running. 

Therefore, `/tmp/_MEIZLLDaA/libpython3.7m.so` is a correct path, because PyInstaller-frozen program should contain a copy of python shared library. You can verify that the library is collected by using `pyi-archive_viewer` command-line utility to list the contents of the programs' embedded archive - you should see the `libpython3.7m.so` among the listed contents.

So if the (same) program runs sometimes but sometimes gives the error (on the same machine), then there might be a problem with unpacking to the temporary directory (insufficient space?), or with some other program/service removing the temporary files (although that's quite a race for removal to happen right at the early stages of program run). Either way, you should probably investigate what is going on with the temporary directory.
> `/tmp/_MEIXXXXXX` is a temporary directory where one-file (-F) build unpacks its content each time it is run. The last six characters in name are randomly generated, so they are different each time; and the temporary directory is cleaned up once the program finishes, so you will see it exist only while the program is running.
> 
> Therefore, `/tmp/_MEIZLLDaA/libpython3.7m.so` is a correct path, because PyInstaller-frozen program should contain a copy of python shared library. You can verify that the library is collected by using `pyi-archive_viewer` command-line utility to list the contents of the programs' embedded archive - you should see the `libpython3.7m.so` among the listed contents.
> 
> So if the (same) program runs sometimes but sometimes gives the error (on the same machine), then there might be a problem with unpacking to the temporary directory (insufficient space?), or with some other program/service removing the temporary files (although that's quite a race for removal to happen right at the early stages of program run). Either way, you should probably investigate what is going on with the temporary directory.

Thank you for your answer~ 

I have checked the space,space is  enough.

when i run 'pyinstaller -F HelloServer.py --distpath .' to generate HelloServer。
If first time HelloServer run succ, then follow-up run is always succ

and i did not change any code,but only  run 'pyinstaller -F HelloServer.py --distpath .' to generate HelloServer again.
If first time HelloServer run fail, then follow-up run is always fail

so the variable is only i run the repackage command ~


In that case, when you get a build that fails with the error, can you list its contents using `pyi-archive_viewer`, and compare it to the contents of a good build?
> 

I package HelloServer for many times and get 4 succ executable , 4 fail executable ( the same machine and with out change code ).

```
succ/HelloServer1     fail/HelloServer1 
succ/HelloServer2     fail/HelloServer2
succ/HelloServer3     fail/HelloServer3
succ/HelloServer4     fail/HelloServer4
```

 use  pyi-archive_viewer to compare each other ,
pyi-archive_viewer succ/HelloServer1  pyi-archive_viewer succ/HelloServer2...
pyi-archive_viewer fail/HelloServer1  pyi-archive_viewer fail/HelloServer2...

the difference at PYZ-00.pyz  :
```
succ/HelloServer1 (29878659, 1002914, 1002914, 0, 'z', 'PYZ-00.pyz')
succ/HelloServer2 (29878659, 1002917, 1002917, 0, 'z', 'PYZ-00.pyz')
succ/HelloServer3 (29878659, 1002917, 1002917, 0, 'z', 'PYZ-00.pyz')
succ/HelloServer4 (29878659, 1002911, 1002911, 0, 'z', 'PYZ-00.pyz')

fail/HelloServer1 (29878659, 1002917, 1002917, 0, 'z', 'PYZ-00.pyz')
fail/HelloServer2 (29878659, 1002916, 1002916, 0, 'z', 'PYZ-00.pyz')
fail/HelloServer3 (29878659, 1002914, 1002914, 0, 'z', 'PYZ-00.pyz')
fail/HelloServer4 (29878659, 1002916, 1002916, 0, 'z', 'PYZ-00.pyz')

```
But there seems to be no obvious difference , only the package size ..

And 
I copy the  succ executable to other machine ,it run succ; 
I copy the  fail executable to other machine ,it run fail;






`PYZ-00.pyz` being different should have no impact on the `libpython3.7m.so` error; but you can try to ensure that builds are reproducible by settings `PYTHONHASHSEED`, as described [here](https://pyinstaller.org/en/stable/advanced-topics.html#creating-a-reproducible-build). So `libpython3.7m.so` is collected in all cases, and has the same size?

Just to be sure, can you use `pyi-archive_viewer` to extract `libpython3.7m.so` from both a successful and failing program, and compare them (e.g., via some checksum value)?

Does it make any difference if you relocate the temporary directory? For example, create a `test-tmp` in user's home directory (`mkdir ~/test-tmp`) and then run the failing program as `TMPDIR=~/test-tmp ./fail/HelloServer1`?



> `PYZ-00.pyz` being different should have no impact on the `libpython3.7m.so` error; but you can try to ensure that builds are reproducible by settings `PYTHONHASHSEED`, as described [here](https://pyinstaller.org/en/stable/advanced-topics.html#creating-a-reproducible-build). So `libpython3.7m.so` is collected in all cases, and has the same size?
> 
> Just to be sure, can you use `pyi-archive_viewer` to extract `libpython3.7m.so` from both a successful and failing program, and compare them (e.g., via some checksum value)?
> 
> Does it make any difference if you relocate the temporary directory? For example, create a `test-tmp` in user's home directory (`mkdir ~/test-tmp`) and then run the failing program as `TMPDIR=~/test-tmp ./fail/HelloServer1`?

By your advice :

I set the PYTHONHASHSEED  to "Creating a Reproducible Build " , and I use md5sum(HelloServer) to confirm the effectiveness

```
    export PYTHONHASHSEED=1  HelloServer1  (md5)007ff8e8fd1e74faa708589195a454c8   succ
    export PYTHONHASHSEED=2  HelloServer2  (md5)af6bf0acc08c85fe2177351ffd9e47a7    succ
    export PYTHONHASHSEED=3  HelloServer3  (md5)6503274955e1354ef52a3f1cf795c4c2   succ
    export PYTHONHASHSEED=4  HelloServer4  (md5)91771915247b1f3e7914bf2dfb531e44   fail
```
    
As shown above ,when i set PYTHONHASHSEED=4 to genarate the programme ,it is always run failure 

I use pyi-archive_viewer to extract libpython3.7m.so from both a successful and failing program, they are the same (use md5 to confire).


"For example, create a test-tmp in user's home directory (mkdir ~/test-tmp) and then run the failing program as TMPDIR=~/test-tmp ./fail/HelloServer1"

```
   **run succ:**
    (py37) [upc@localhost all]$ TMPDIR=~/test-tmp ./HelloServer3
    start server with config, for example: exe --config=config.conf
    
   **run fail**
    (py37) [upc@localhost all]$ TMPDIR=~/test-tmp ./HelloServer4
    [25193] Error loading Python lib '/home/upc/test-tmp/_MEIwlbdXn/libpython3.7m.so': dlopen: /home/upc/test-tmp/_MEIwlbdXn/libpython3.7m.so: cannot open shared object file: No such file or directory
```

Any other suggestions ?
    
Hmmm... I did some testing in a VM with CentOS 7.9 and anaconda, and it seems that sometimes, we pick up `libpython3.7m.so.1.0` as the python shared library, and sometimes we pick up `libpython3.7m.so` (which should be a symlink to the former in the original conda's lib directory). In my case, both combinations seem to work, though.

Can you check if this is the difference? Are successful builds using `libpython3.7m.so.1.0` as the library name, and failing ones `libpython3.7m.so`?

If you re-build the program again with PYTHONHASHSEED=1 and PYTHONHASHSEED=4,  check the build log for messages like these:
```
[...]
3504 INFO: Looking for eggs
3504 INFO: Python library not in binary dependencies. Doing additional searching...
3526 INFO: Using Python library [...]/lib/libpython3.7m.so
[...]
```

Then, check the `build/<name>/PKG-00.toc` and find the entry for `libpython*`. Finally, check the generated executable with `pyi-archive-viewer` to see what the name of collected `libpython*` is there. Do the names match? And lastly, do they match the name of library reported in the error when the failing build fails to run?

---

On the other hand, does the same problem occur if you try to build a different program? For example, a simple "Hello world" script? If it does, can you create a failing build of such a simple script and upload it somewhere so I can take a look at it?
> PYTHONHASHSEED=1

I've compared the building process before ,i compared each build files , it seems no obvious differance 


/home/upc/miniconda3/envs/py37/lib
lrwxrwxrwx.  1 upc upc       20 10月 26 2021 libpython3.7m.so -> libpython3.7m.so.1.0
-rwxrwxr-x.  2 upc upc  3808528 7月  27 2021 libpython3.7m.so.1.0 

succ build:
...
3080 INFO: Using Python library /home/upc/miniconda3/envs/py37/lib/libpython3.7m.so.1.0
...

    (py37) [upc@localhost HelloServer]$ cat PKG-00.toc|grep libpython3.7m.so
      ('libpython3.7m.so.1.0',
       '/home/upc/miniconda3/envs/py37/lib/libpython3.7m.so.1.0',

    pyi-archive_viewer all/HelloServer1
     (5259723, 1402372, 3808528, 1, 'b', 'libpython3.7m.so.1.0'),

fail build:
...
3063 INFO: Using Python library /home/upc/miniconda3/envs/py37/lib/libpython3.7m.so.1.0
...
   
    (py37) [upc@localhost HelloServer]$ cat PKG-00.toc|grep libpython3.7m.so
    ('libpython3.7m.so.1.0',
    '/home/upc/miniconda3/envs/py37/lib/libpython3.7m.so.1.0',
    
    pyi-archive_viewer all/HelloServer4
    (5259723, 1402372, 3808528, 1, 'b', 'libpython3.7m.so.1.0'),
   

And I even deleted the symbol link , then copy the libpython3.7m.so.1.0 to libpython3.7m.so, it also has no effect

    (py37) [upchina@localhost lib]$ ll|grep libpython3.7m.so
    -rwxrwxr-x.  1 upchina upchina  3808528 5月  23 09:42 libpython3.7m.so
    -rwxrwxr-x.  2 upchina upchina  3808528 7月  27 2021 libpython3.7m.so.1.0
       

And i use a simple program only print "Hello Word!" to reproduce the problem (set PYTHONHASHSEED from 1 to 20), it is always run succ 。     
    HelloWorld.py
```

        import os
        print ("workdir :" , os.getcwd() )
        print ("HelloWorld!")
```
   
So it can be confirmed that the case of responsible search led to this problem.
My program is a little complicated. It refers to a so file compiled by C + + based on pybind11 (https://github.com/pybind/pybind11) 
And i have mailed the build diffrence to you (rok.mandeljc@gmail.com)

> My program is a little complicated. It refers to a so file compiled by C + + based on pybind11

Aha, that's the missing piece of the puzzle. :)

What's going on is, the python shared library name that bootloader tries to load is determined here:
https://github.com/pyinstaller/pyinstaller/blob/dc093332c49da0a1693384e0f2fd7e37ad2324d4/PyInstaller/building/api.py#L290-L291
This name is stored at the end of generated `PKG` file (which is added to the `EXE`), but it is not visible to the `pyi-archive-viewer`, nor is it visible in the build log. And, according to my earlier tests, we sometimes pick up `libpython3.7m.so.1.0` and sometimes `libpython3.7m.so`.

However, we ensure that the python library is actually collected in another part of the code - earlier, in `Analysis`:
https://github.com/pyinstaller/pyinstaller/blob/dc093332c49da0a1693384e0f2fd7e37ad2324d4/PyInstaller/building/build_main.py#L643-L661
As long as python shared library has not been already collected, we use the same `bindepend.get_python_library_path()` function, whose output within the same program run seems to be stable (i.e., always gives back the same name). So the name of collected shared library and expected name of the shared library match, and everything works.

But a pybind11-based extension is probably also linked against python shared library, which always causes `libpython3.7m.so.1.0` to be collected. But the `bindepend.get_python_library_path()` call is not aware of that, which (on some runs) results in a mismatch between the actual collected library name and the expected one, resulting in the error.

---

The cleanest way to fix this would probably be to propagate the determined shared library name from `Analysis` to other objects (i..e, `EXE` and `PKG`). But that requires modification of the templates, which means that it won't work for already-generated specs (though we could make it so that nothing changes for existing specs with default value of additional parameters being `None`).

We could also have `bindepend.get_python_library_path()` resolve symlinks, so instead of `libpython3.X.so` it returns `libpython3.X.so.1.0`. This should help in cases when other binaries are also linked against the python shared library, because the linker should resolve against versioned .so's (to enforce ABI compatibility). It would not help if someone also (manually) collected `libpython3.X.so`, though.

~And perhaps we should also be dropping the [un-versioned .so patterns from `compat.PYDYLIB_NAMES`](https://github.com/pyinstaller/pyinstaller/blob/dc093332c49da0a1693384e0f2fd7e37ad2324d4/PyInstaller/compat.py#L127-L128), as those should be symlinks to versioned .so's anyway.~ EDIT: we shouldn't, because that will break [RH SCL python](https://github.com/pyinstaller/pyinstaller/commit/49880cccc30707e9138569ff9bca976f0c8703bf).




Resolving patch:
diff --git a/PyInstaller/depend/bindepend.py b/PyInstaller/depend/bindepend.py
--- a/PyInstaller/depend/bindepend.py
+++ b/PyInstaller/depend/bindepend.py
@@ -854,8 +854,15 @@ def _find_lib_in_libdirs(*libdirs):
         for libdir in libdirs:
             for name in compat.PYDYLIB_NAMES:
                 full_path = os.path.join(libdir, name)
-                if os.path.exists(full_path):
-                    return full_path
+                if not os.path.exists(full_path):
+                    continue
+                # Resolve potential symbolic links to achieve consistent results with linker-based search; e.g., on
+                # POSIX systems, linker resolves unversioned library names (python3.X.so) to versioned ones
+                # (libpython3.X.so.1.0) due to former being symbolic linkes to the latter. See #6831.
+                full_path = os.path.realpath(full_path)
+                if not os.path.exists(full_path):
+                    continue
+                return full_path
         return None
 
     # If this is Microsoft App Store Python, check the compat.base_path first. While compat.python_executable resolves


Test patch: