Repository: pyinstaller/pyinstaller

Issue:
Unable to build executable using embedded zip Python
## Description of the issue
The official Python embeddable zip file distributions for Windows from Python.org contain a zip file (pythonXX.zip) that contains the pre-compiled and optimized .pyc files to run Python isolated on a machine. I've installed the embedded Python on a local agent that we use in our azure devops pipeline, following the [Microsoft docs](https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/tool/use-python-version?view=azure-devops#how-can-i-configure-a-self-hosted-agent-to-use-this-task).

However, building on this agent using PyInstaller fails, see Stacktrace below. This failure occurs because the requirements for `base_library.zip` are located inside a zip. Thus, PyInstaller tries to do an `os.stat()` on a path inside a zip folder, which fails.

## Context information (for bug reports)

* Output of `pyinstaller --version`: ```3.6```
* Version of Python: 3.7.7
* Platform: Windows 10
* Same failure occurs on newest development version of PyInstaller
* Followed all the instructions in the "If Things Go Wrong" Guide and checked that everything is packaged correctly. The error occurs regardless of PyInstaller settings or spec file content.

### A minimal example program which shows the error
- Download embedded Python from [Python.org](https://www.python.org/downloads/)
- Add pip using [get-pip](https://michlstechblog.info/blog/python-install-python-with-pip-on-windows-by-the-embeddable-zip-file/)
- Install PyInstaller
- Create `helloworld.py`
   ```
   print("Hello World!")
   ```
- Try to run PyInstaller on `helloworld.py` using any settings.

### Stacktrace / full error message
```
==============================================================================
Task         : Command line
Description  : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows
Version      : 2.164.2
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/command-line
==============================================================================
Generating script.
Script contents:
python -m PyInstaller --distpath "%cd%\exe" SPECFILENAME.spec
========================== Starting Command Output ===========================
"C:\WINDOWS\system32\cmd.exe" /D /E:ON /V:OFF /S /C "CALL "agent2\_work\_temp\44e0b0b7-f2f9-4990-a3ee-31bbec9a0c70.cmd""
285 INFO: PyInstaller: 3.6
285 INFO: Python: 3.7.7
285 INFO: Platform: Windows-10-10.0.18362-SP0
291 INFO: UPX is not available.
303 INFO: checking Analysis
307 INFO: Building Analysis because Analysis-00.toc is non existent
307 INFO: Initializing module dependency graph...
313 INFO: Caching module graph hooks...
326 INFO: Analyzing base_library.zip ...
1791 INFO: Processing pre-find module path hook   distutils
1804 INFO: distutils: retargeting to non-venv dir 'agent2\\_work\\_tool\\Python\\3.7.7\\x64\\python37.zip'
2579 INFO: Caching module dependency graph...
2629 ERROR: base_library.zip could not be created!
Traceback (most recent call last):
  File "D:\obj\windows-release\37amd64_Release\msi_python\zip_amd64\runpy.py", line 193, in _run_module_as_main
  File "D:\obj\windows-release\37amd64_Release\msi_python\zip_amd64\runpy.py", line 85, in _run_code
  File "agent2\_work\_tool\Python\3.7.7\x64\lib\site-packages\PyInstaller\__main__.py", line 121, in <module>
    run()
  File "agent2\_work\_tool\Python\3.7.7\x64\lib\site-packages\PyInstaller\__main__.py", line 114, in run
    run_build(pyi_config, spec_file, **vars(args))
  File "agent2\_work\_tool\Python\3.7.7\x64\lib\site-packages\PyInstaller\__main__.py", line 65, in run_build
    PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
  File "agent2\_work\_tool\Python\3.7.7\x64\lib\site-packages\PyInstaller\building\build_main.py", line 734, in main
    build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))
  File "agent2\_work\_tool\Python\3.7.7\x64\lib\site-packages\PyInstaller\building\build_main.py", line 681, in build
    exec(code, spec_namespace)
  File "SPECFILENAME.spec", line 24, in <module>
    noarchive=False)
  File "agent2\_work\_tool\Python\3.7.7\x64\lib\site-packages\PyInstaller\building\build_main.py", line 244, in __init__
    self.__postinit__()
  File "agent2\_work\_tool\Python\3.7.7\x64\lib\site-packages\PyInstaller\building\datastruct.py", line 160, in __postinit__
    self.assemble()
  File "agent2\_work\_tool\Python\3.7.7\x64\lib\site-packages\PyInstaller\building\build_main.py", line 352, in assemble
    create_py3_base_library(libzip_filename, graph=self.graph)
  File "agent2\_work\_tool\Python\3.7.7\x64\lib\site-packages\PyInstaller\depend\utils.py", line 74, in create_py3_base_library
    st = os.stat(mod.filename)
FileNotFoundError: [WinError 3] The system cannot find the path specified: 'agent2\\_work\\_tool\\Python\\3.7.7\\x64\\python37.zip\\encodings\\__init__.pyc'
##[error]Cmd.exe exited with code '1'.
```



Discussion:
I manually adjusted `depends\utils.py` in the PyInstaller package to handle files inside a zip folder. However, this did not solve the issue as PyInstaller would then continue to load the `hook-distutils.py` and attempt to get `Include\pyconfig.h` which does not exist for embedded zip Python:

```
169 INFO: PyInstaller: 3.6
169 INFO: Python: 3.7.7
169 INFO: Platform: Windows-10-10.0.18362-SP0
184 INFO: UPX is not available.
184 INFO: checking Analysis
200 INFO: Building Analysis because Analysis-00.toc is non existent
200 INFO: Initializing module dependency graph...
200 INFO: Caching module graph hooks...
222 INFO: Analyzing base_library.zip ...
1140 INFO: Processing pre-find module path hook   distutils
1156 INFO: distutils: retargeting to non-venv dir 'Python\\3.7.7\\x64\\python37.zip'
1689 INFO: Caching module dependency graph...
1742 INFO: running Analysis Analysis-00.toc
1742 INFO: Adding Microsoft.Windows.Common-Controls to dependent assemblies of final executable
  required by Python\3.7.7\x64\python.exe
1990 INFO: Analyzing FILE_TO_BUILD_TO_EXECUTABLE.py
3078 INFO: Processing pre-safe import module hook   setuptools.extern.six.moves
3627 INFO: Processing pre-find module path hook   site
3633 INFO: site: retargeting to fake-dir 'Python\\3.7.7\\x64\\lib\\site-packages\\PyInstaller\\fake-modules'
7645 INFO: Processing module hooks...
7645 INFO: Loading module hook "hook-distutils.py"...
Unable to find "Python\3.7.7\x64\Include\pyconfig.h" when adding binary and data files.This would mean your Python installation doesn't come with proper library files. This usually happens by missing development package, or unsuitable build parameters of Python installation.
* On Debian/Ubuntu, you would need to install Python development packages
  * apt-get install python3-dev
  * apt-get install python-dev
* If you're building Python by yourself, please rebuild your Python with
`--enable-shared` (or, `--enable-framework` on Darwin)
```

I found that it's related to PR #4182 and issue #4181. As mentioned in the PR, this issue was not actually fixed as filesystem operations are still used, even if Python is embedded in a zip file. Furthermore, as shown in this reply, it's unfortunately not as simple as only addressing the filesystem operations.
@Ruud-K this is indeed a bug. I shall fix this in the next few days before the next release.
Hi, is there any temporary o manual workaround to bypass error using embedded python?
@menteora nope - sorry this got buried. I'll do this next - it's now top of my to-do list.
@Ruud-K Do you still have the changes you made to get PyInstaller to handle the zipped libs? The `pyconfig.h` issue should be fixed by #5218 once it goes through. So if we apply your changes afterwards, this issue will hopefully be fixed.
I ran into similar issues when using pyinstaller with embedded python3.7.9. The solution was to use the `--onedir` option and then copy from PYTHONHOME the `python37.zip` into `dist\{appname}\base_library.zip`
This seems to be a long-running issue and the content of `base_library.zip` has been updated several times

- #3397, f101f6f, March 2018, added warnings module 
- #1705, e7fc427, Dec 2015, added functools and a bunch, and holds an informative discussion

I have a naive suggestion here: maybe we can consider using embeddable python.zip directly as base_library.zip? Otherwise we might ends up updating `PY3_BASE_MODULES` once anyone runs into a missing module problem. And because `--onefile` mode exe erase its temporary directory, it's even harder for the user to find out the actual reason of a `Cannot import ... Errorr` .

Also configuring embeddable python alone is already a demanding task and I need to thank the creator of this post https://dev.to/fpim/setting-up-python-s-windows-embeddable-distribution-properly-1081
I'll work on this to fix the bug right away
I  successfully generated the exe file through an ad hoc solution like this: in the embeddable python, there is a pythonxx .zip of the package, extract the package and rename it to the pyhonxx.zip folder, and run the pyinstaller again. Witness the miracle. 
Of course, I still hope to completely solve this bug and facilitate the use of embeddable Python.
still
No definitive solution?

In my case it appears:
FileNotFoundError: [Errno 2] No such file or directory: 'Python_amd64\\python311.zip\\struct.pyc'

Resolving patch:
diff --git a/PyInstaller/building/utils.py b/PyInstaller/building/utils.py
--- a/PyInstaller/building/utils.py
+++ b/PyInstaller/building/utils.py
@@ -24,6 +24,7 @@
 import struct
 import subprocess
 import sys
+import zipfile
 
 from PyInstaller import compat
 from PyInstaller import log as logging
@@ -593,22 +594,38 @@ def get_code_object(modname, filename):
     This is a simplifed non-performant version which circumvents __pycache__.
     """
 
-    try:
-        if filename in ('-', None):
-            # This is a NamespacePackage, modulegraph marks them by using the filename '-'. (But wants to use None, so
-            # check for None, too, to be forward-compatible.)
-            logger.debug('Compiling namespace package %s', modname)
-            txt = '#\n'
-            return compile(txt, filename, 'exec')
+    if filename in ('-', None):
+        # This is a NamespacePackage, modulegraph marks them by using the filename '-'. (But wants to use None, so
+        # check for None, too, to be forward-compatible.)
+        logger.debug('Compiling namespace package %s', modname)
+        txt = '#\n'
+        code_object = compile(txt, filename, 'exec')
+    else:
+        _, ext = os.path.splitext(filename)
+        ext = ext.lower()
+
+        if ext == '.pyc':
+            # The module is available in binary-only form. Read the contents of .pyc file using helper function, which
+            # supports reading from either stand-alone or archive-embedded .pyc files.
+            logger.debug('Reading code object from .pyc file %s', filename)
+            pyc_data = _read_pyc_data(filename)
+            code_object = marshal.loads(pyc_data[16:])
         else:
-            logger.debug('Compiling %s', filename)
+            # Assume this is a source .py file, but allow an arbitrary extension (other than .pyc, which is taken in
+            # the above branch). This allows entry-point scripts to have an arbitrary (or no) extension, as tested by
+            # the `test_arbitrary_ext` in `test_basic.py`.
+            logger.debug('Compiling python script/module file %s', filename)
+
             with open(filename, 'rb') as f:
                 source = f.read()
-            return compile(source, filename, 'exec')
-    except SyntaxError as e:
-        print("Syntax error in ", filename)
-        print(e.args)
-        raise
+
+            try:
+                code_object = compile(source, filename, 'exec')
+            except SyntaxError:
+                logger.warning("Sytnax error while compiling %s", filename)
+                raise
+
+    return code_object
 
 
 def strip_paths_in_code(co, new_filename=None):
@@ -722,12 +739,9 @@ def compile_pymodule(name, src_path, workpath, code_cache=None):
             with open(pyc_path, 'rb') as fp:
                 pyc_data = fp.read()
         elif ext == '.pyc':
-            # The module is available in binary-only form. Read it...
-            with open(src_path, 'rb') as fp:
-                pyc_data = fp.read()
-            # ... verify the python version...
-            if pyc_data[:4] != compat.BYTECODE_MAGIC:
-                raise ValueError(f"The .pyc module {src_path} was compiled for incompatible version of python!")
+            # The module is available in binary-only form. Read the contents of .pyc file using helper function, which
+            # supports reading from either stand-alone or archive-embedded .pyc files.
+            pyc_data = _read_pyc_data(src_path)
         else:
             raise ValueError(f"Invalid python module file {src_path}; unhandled extension {ext}!")
 
@@ -748,6 +762,35 @@ def compile_pymodule(name, src_path, workpath, code_cache=None):
     return pyc_path
 
 
+def _read_pyc_data(filename):
+    """
+    Helper for reading data from .pyc files. Supports both stand-alone and archive-embedded .pyc files. Used by
+    `compile_pymodule` and `get_code_object` helper functions.
+    """
+    src_file = pathlib.Path(filename)
+
+    if src_file.is_file():
+        # Stand-alone .pyc file.
+        pyc_data = src_file.read_bytes()
+    else:
+        # Check if .pyc file is stored in a .zip archive, as is the case for stdlib modules in embeddable
+        # python on Windows.
+        parent_zip_file = misc.path_to_parent_archive(src_file)
+        if parent_zip_file is not None and zipfile.is_zipfile(parent_zip_file):
+            with zipfile.ZipFile(parent_zip_file, 'r') as zip_archive:
+                # NOTE: zip entry names must be in POSIX format, even on Windows!
+                zip_entry_name = str(src_file.relative_to(parent_zip_file).as_posix())
+                pyc_data = zip_archive.read(zip_entry_name)
+        else:
+            raise FileNotFoundError(f"Cannot find .pyc file {filename!r}!")
+
+        # Verify the python version
+        if pyc_data[:4] != compat.BYTECODE_MAGIC:
+            raise ValueError(f"The .pyc module {filename} was compiled for incompatible version of python!")
+
+    return pyc_data
+
+
 def postprocess_binaries_toc_pywin32(binaries):
     """
     Process the given `binaries` TOC list to apply work around for `pywin32` package, fixing the target directory
diff --git a/PyInstaller/utils/misc.py b/PyInstaller/utils/misc.py
--- a/PyInstaller/utils/misc.py
+++ b/PyInstaller/utils/misc.py
@@ -19,6 +19,7 @@
 import re
 import tokenize
 import io
+import pathlib
 
 from PyInstaller import log as logging
 from PyInstaller.compat import is_win
@@ -232,3 +233,15 @@ def is_iterable(arg):
     except TypeError:
         return False
     return True
+
+
+def path_to_parent_archive(filename):
+    """
+    Check if the given file path points to a file inside an existing archive file. Returns first path from the set of
+    parent paths that points to an existing file, or `None` if no such path exists (i.e., file is an actual stand-alone
+    file).
+    """
+    for parent in pathlib.Path(filename).parents:
+        if parent.is_file():
+            return parent
+    return None


Test patch: