Repository: pyinstaller/pyinstaller

Issue:
Windows executble using cryptography won't run
## Description of the issue

I build my application (https://github.com/taers232c/GAMADV-XTD3) with GitHub Actions and have been doing this
successfully for years. Starting about a week ago, the Windows build produced an executable that won't run. The Linux
and MacOS builds produce working executables.

### Context information (for bug reports)

Python 3.12.3
pyinstaller 6.7.0
setuptools 70.0.0
cryptography 42.0.7

### Stacktrace / full error message

This occurs when the Windows executable is run.
```
Traceback (most recent call last):
  File "__main__.py", line 24, in <module>
  File "PyInstaller\loader\pyimod02_importers.py", line 419, in exec_module
  File "__init__.py", line 97, in <module>
  File "PyInstaller\loader\pyimod02_importers.py", line 419, in exec_module
  File "cryptography\x509\__init__.py", line 7, in <module>
  File "PyInstaller\loader\pyimod02_importers.py", line 419, in exec_module
  File "cryptography\x509\certificate_transparency.py", line 11, in <module>
ImportError: DLL load failed while importing _rust: The specified module could not be found.
[4508] Failed to execute script '__main__' due to unhandled exception!
```
```
File "__main__.py", line 24:   My code: import gam
File "__init__.py", line 97: My code: from cryptography import x509
File "cryptography\x509\__init__.py", line 7: from cryptography.x509 import certificate_transparency, verification
File "cryptography\x509\certificate_transparency.py": from cryptography.hazmat.bindings._rust import x509 as rust_x509
File "cryptography\hazmat\bindings\_rust\x509.pyi"
```

Any suggestions on how to proceed will be much appreciated.

Thanks,

Ross


Discussion:
Any chance you can generate an artifact from that failing Windows build?
I think we fail to collect `python3.dll` from that custom python build of yours.
Rok,

I'll remove running the excutable and the build should finish.

Ross
----
Ross Scroggs
***@***.***



> On May 26, 2024, at 11:58 AM, Rok Mandeljc ***@***.***> wrote:
> 
> 
> Any chance you can generate an artifact from that failing Windows build?
> 
> —
> Reply to this email directly, view it on GitHub <https://github.com/pyinstaller/pyinstaller/issues/8565#issuecomment-2132364260>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACCTYLZ3FO5RSNG2ULPZDQDZEIWDTAVCNFSM6AAAAABIKAKQ36VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZSGM3DIMRWGA>.
> You are receiving this because you authored the thread.
> 


How would I get the artifact to you?
----
Ross Scroggs
***@***.***



> On May 26, 2024, at 11:58 AM, Rok Mandeljc ***@***.***> wrote:
> 
> 
> Any chance you can generate an artifact from that failing Windows build?
> 
> —
> Reply to this email directly, view it on GitHub <https://github.com/pyinstaller/pyinstaller/issues/8565#issuecomment-2132364260>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACCTYLZ3FO5RSNG2ULPZDQDZEIWDTAVCNFSM6AAAAABIKAKQ36VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZSGM3DIMRWGA>.
> You are receiving this because you authored the thread.
> 


I could get it off the pipeline directly, I think? But you can retrieve it yourself and confirm that there is no `python3.dll` in the frozen application bundle.
I've forked your repository and instrumented CI a bit.

Like I said earlier, we fail to collect `python3.dll`, presumably because `sys.base_prefix` does not point to where python executable and DLLs are:

https://github.com/rokm/GAMADV-XTD3/actions/runs/9246031018/job/25433117029#step:31:81

I think adding the directory with python executable and DLLs to `PATH`  ([here](https://github.com/taers232c/GAMADV-XTD3/blob/b4c37b65a5e3fcd837f78a3f171de4c4b6a0c73b/.github/workflows/build.yml#L470-L472)) should work around this.
Rok,

Thanks.

Is this what I want?
            export PATH=$("${PYTHON}" -c "import sys; import os; print(sys.base_prefix)")

Ross
----
Ross Scroggs
***@***.***



> On May 26, 2024, at 2:14 PM, Rok Mandeljc ***@***.***> wrote:
> 
> 
> I've forked your repository and instrumented CI a bit.
> 
> Like I said earlier, we fail to collect python3.dll, presumably because sys.base_prefix does not point to where python executable and DLLs are:
> 
> https://github.com/rokm/GAMADV-XTD3/actions/runs/9246031018/job/25433117029#step:31:81
> 
> I think adding the directory with python executable and DLLs to PATH (here <https://github.com/taers232c/GAMADV-XTD3/blob/b4c37b65a5e3fcd837f78a3f171de4c4b6a0c73b/.github/workflows/build.yml#L470-L472>) should work around this.
> 
> —
> Reply to this email directly, view it on GitHub <https://github.com/pyinstaller/pyinstaller/issues/8565#issuecomment-2132397912>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACCTYL5N6LFMEY32SAIIXTLZEJGEXAVCNFSM6AAAAABIKAKQ36VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZSGM4TOOJRGI>.
> You are receiving this because you authored the thread.
> 


> Is this what I want? 
> export PATH=$("${PYTHON}" -c "import sys; import os; print(sys.base_prefix)")

No, because `sys.base_prefix` is NOT pointing to directory where python executable and DLLs are (it seems to be pointing to source directory, `D:\a\GAMADV-XTD3\GAMADV-XTD3\src\cpython`). We are [already adding `sys.base_prefix` to search path for Windows DLLs](https://github.com/pyinstaller/pyinstaller/blob/fc05dfd44f7c39ab46fb8a08b473e706561e9611/PyInstaller/building/build_main.py#L139-L142) specifically to catch `python3.dll` - but in this case, it does not help.

You probably want `export PATH="$(dirname ${PYTHON}):/usr/bin"`


Or this:
export PATH=$("${PYTHON}" -c "import sys; import os; print(sys.base_prefix)")+':'+$PATH
----
Ross Scroggs
***@***.***



> On May 26, 2024, at 2:47 PM, Ross Scroggs ***@***.***> wrote:
> 
> Rok,
> 
> Thanks.
> 
> Is this what I want?
>             export PATH=$("${PYTHON}" -c "import sys; import os; print(sys.base_prefix)")
> 
> Ross
> ----
> Ross Scroggs
> ***@***.***
> 
> 
> 
>> On May 26, 2024, at 2:14 PM, Rok Mandeljc ***@***.***> wrote:
>> 
>> 
>> I've forked your repository and instrumented CI a bit.
>> 
>> Like I said earlier, we fail to collect python3.dll, presumably because sys.base_prefix does not point to where python executable and DLLs are:
>> 
>> https://github.com/rokm/GAMADV-XTD3/actions/runs/9246031018/job/25433117029#step:31:81
>> 
>> I think adding the directory with python executable and DLLs to PATH (here <https://github.com/taers232c/GAMADV-XTD3/blob/b4c37b65a5e3fcd837f78a3f171de4c4b6a0c73b/.github/workflows/build.yml#L470-L472>) should work around this.
>> 
>> —
>> Reply to this email directly, view it on GitHub <https://github.com/pyinstaller/pyinstaller/issues/8565#issuecomment-2132397912>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACCTYL5N6LFMEY32SAIIXTLZEJGEXAVCNFSM6AAAAABIKAKQ36VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZSGM4TOOJRGI>.
>> You are receiving this because you authored the thread.
>> 
> 


Thanks,

Here is an artifact: https://github.com/taers232c/GAMADV-XTD3/releases/tag/Rok

Ross
----
Ross Scroggs
***@***.***



> On May 26, 2024, at 2:53 PM, Rok Mandeljc ***@***.***> wrote:
> 
> 
> Is this what I want?
> export PATH=$("${PYTHON}" -c "import sys; import os; print(sys.base_prefix)")
> 
> No, because sys.base_prefix is NOT pointing to directory where python executable and DLLs are (it seems to be pointing to source directory, D:\a\GAMADV-XTD3\GAMADV-XTD3\src\cpython). We are already adding sys.base_prefix to search path for Windows DLLs <https://github.com/pyinstaller/pyinstaller/blob/fc05dfd44f7c39ab46fb8a08b473e706561e9611/PyInstaller/building/build_main.py#L139-L142> specifically to catch python3.dll - but in this case, it does not help.
> 
> You probably want export PATH="$(dirname ${PYTHON}):/usr/bin"
> 
> —
> Reply to this email directly, view it on GitHub <https://github.com/pyinstaller/pyinstaller/issues/8565#issuecomment-2132407332>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACCTYLYKILFKZLXR5HCDD2LZEJKW7AVCNFSM6AAAAABIKAKQ36VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZSGQYDOMZTGI>.
> You are receiving this because you authored the thread.
> 


> Here is an artifact: https://github.com/taers232c/GAMADV-XTD3/releases/tag/Rok

Yeah, `python3.dll` is missing.

And `export PATH="$(dirname ${PYTHON}):/usr/bin"` fixes the problem:

https://github.com/rokm/GAMADV-XTD3/actions/runs/9246444465/job/25433996451

---

I'll try to build python from source in my local Windows VM to see what the deal with prefix and paths is. But I suppose we should add `os.path.dirname(sys.executable)` to the list of paths to search.
Rok,

All is now working. Your support has been superb.

Thank you very much!

Ross
----
Ross Scroggs
***@***.***



> On May 26, 2024, at 3:33 PM, Rok Mandeljc ***@***.***> wrote:
> 
> 
> Here is an artifact: https://github.com/taers232c/GAMADV-XTD3/releases/tag/Rok
> 
> Yeah, python3.dll is missing.
> 
> And export PATH="$(dirname ${PYTHON}):/usr/bin" fixes the problem:
> 
> https://github.com/rokm/GAMADV-XTD3/actions/runs/9246444465/job/25433996451
> 
> I'll try to build python from source in my local Windows VM to see what the deal with prefix and paths is. But I suppose we should add os.path.dirname(sys.executable) to the list of paths to search.
> 
> —
> Reply to this email directly, view it on GitHub <https://github.com/pyinstaller/pyinstaller/issues/8565#issuecomment-2132416602>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACCTYL3YPVN5RJC54IIR2A3ZEJPLBAVCNFSM6AAAAABIKAKQ36VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZSGQYTMNRQGI>.
> You are receiving this because you authored the thread.
> 



Resolving patch:
diff --git a/PyInstaller/building/build_main.py b/PyInstaller/building/build_main.py
--- a/PyInstaller/building/build_main.py
+++ b/PyInstaller/building/build_main.py
@@ -141,6 +141,12 @@ def find_binary_dependencies(binaries, import_packages):
         # `python3X.dll`), regardless of other python installations that might be present in the PATH.
         extra_libdirs.append(compat.base_prefix)
 
+        # When using python built from source, `sys.base_prefix` does not point to the directory that contains python
+        # executable, `python3X.dll`, and `python3.dll`. To accommodate such case, also add the directory in which
+        # python executable is located to the extra search paths. On the off-chance that this is a combination of venv
+        # and python built from source, prefer `sys._base_executable` over `sys.executable`.
+        extra_libdirs.append(os.path.dirname(getattr(sys, '_base_executable', sys.executable)))
+
         # If `pywin32` is installed, resolve the path to the `pywin32_system32` directory. Most `pywin32` extensions
         # reference the `pywintypes3X.dll` in there. Based on resolved `pywin32_system32` directory, also add other
         # `pywin32` directory, in case extensions in different directories reference each other (the ones in the same


Test patch: