Repository: pyinstaller/pyinstaller

Issue:
LoadIconMetric/Ordinal 380 errors when console is False
## Description of the issue

EDIT: Update, this issue seems to be with the default bootloader as well, but in a different manner. See first comment.

When I use pyinstaller on a script with a gcc compiled bootloader (in this case using MSYS2 and Mingw-64 gcc) on Windows 10. The program only launches without error if there is a console enabled. If I set console to False, I get an error.


### Context information (for bug reports)

* Output of `pyinstaller --version`: ```4.5.1```
* Version of Python: 3.6.7
* Platform: Windows 10 20H2
* How you installed Python: python.org (installer)
* Did you also try this on another platform? Does it work there? Only Windows

### A minimal example program which shows the error
I am using the following run_application.py to generate a simple Window, so that I can still see something without a console.

```
import pyglet
window = pyglet.window.Window()
pyglet.app.run()
```

EXE is created using: `pyinstaller .\run_application.py --noconfirm --clean --noconsole`

Bootloader compiled via `python ./waf --gcc all --target-arch=32bit`

### Stacktrace / full error message


Below message shows up in a win32 error dialog message:
```
---------------------------
run_application.exe - Entry Point Not Found
---------------------------
The procedure entry point LoadIconMetric could not be located in the dynamic link library 
```

EDIT: I will note, this works without a console with the default bootloader.



Discussion:
This may be a clue, but if I compile the same script above with the default bootloader, it does work. *however* If I rename the file. Say `run_application.exe` to anything else other than the original (in this case `run_applicatio.exe`, then I get the following error: 
```
---------------------------
run_applicatio.exe - Ordinal Not Found
---------------------------
The ordinal 380 could not be located in the dynamic link library C:\Users\Admin\Desktop\tester\dist\run_application\run_applicatio.exe. 
---------------------------
OK   
---------------------------
```

I googled "ordinal 380" and this has to do with `LoadIconMetric` as well.

This issue did not occur with the default bootloader in pyinstaller 4.2. This also occurs in the dev branch.
For the first issue, you have to use the ``develop`` branch to build the bootloader I guess. See [here](https://github.com/pyinstaller/pyinstaller/discussions/6196#discussioncomment-1302204) and the pull request #6199 .

I don't know about the second issue.
Thanks for the response, I just compiled the bootloader with gcc with the development branch, and the original issue is fixed in that I can now launch it with a custom bootloader. However, the second issue still remains, where renaming the file produces the same error.
I can reproduce the second error
If you rename an `.exe` in `onedir` bundle, you also need to rename the corresponding `.exe.manifest`.
In the previous version I upgraded from (4.2) this was not a requirement and the executable still atleast launched without error.

However, I have tried renaming the manifest file as instructed, but the same error persists.
> In the previous version I upgraded from (4.2) this was not a requirement and the executable still atleast launched without error.

It did, but it did so without manifest active. If you had been using any of manifest-related features (`--uac-admin`, `--uac-uiacess`), that would have stopped working once you renamed the .exe.

With newer versions, the windowed version of pyinstaller bootloader requires the manifest to enable common controls v6, so you can't get away with breaking the manifest anymore.

> However, I have tried renaming the manifest file as instructed, but the same error persists.

It should work if you rename both before you try to launch the exe for the first time (i.e., rebuild the frozen application, rename both files, launch the exe). It seems that at first launch, Windows establishes some sort of link between .exe and its external manifest, and if the manifest is not found then, it will not be found later (even if it becomes available).

EDIT: [This article](https://csi-windows.com/blog/all/27-csi-news-general/245-find-out-why-your-external-manifest-is-being-ignored) explains the external manifests and caching issues, and offers a work around - changing the .exe's modification date. For example, using Power Shell, you can do `(get-item "program.exe").LastWriteTime=("18 September 2021 18:15:00")`.
Thanks for explanation and insight. I tested your suggestion, ran pyinstaller, renamed both files before launching, and it launches correctly. You are right, the manifest becomes tattood into that executable name after the first launch.

I feel like something like this should be documented somewhere, as the error offers no actual explanation to the reason. Is it possible to clarify the error message popup as a possible manifest issue?  Doesn't feel very intuitive that you cannot rename an application (as well it working in a previous pyinstaller version), but that's Windows for you. Side question, is there anyway to embed an internal manifest?

I will look into that workaround about modifying the modification date, it looks like `os.utime` may offer this functionality.

EDIT: Looks like it can cache 'no manifest' as well. If you try running without the manifest, then add one after, it's still bricked.

EDIT: For anyone who ended up here, and needs a python solution to fix a renamed exe or some manifest issue. The following re-enabled usage of the executable after renaming it:
```py
import os
import time
import datetime

path = './/run_application//run_application.exe'

modified_ts = time.mktime(datetime.datetime.now().timetuple())
os.utime(path, (modified_ts, modified_ts))
print("Executable modification time has been updated.")
```
> Thanks for explanation and insight. I tested your suggestion, ran pyinstaller, renamed both files before launching, and it launches correctly. You are right, the manifest becomes tattood into that executable name after the first launch.
> 
> I feel like something like this should be documented somewhere, as the error offers no actual explanation to the reason. Is it possible to clarify the error message popup as a possible manifest issue? 

I hear you, but the truth is that we (I?) became aware of the problem with MinGW-compiled bootloaders only [a week ago](https://github.com/pyinstaller/pyinstaller/discussions/6196#discussioncomment-1302204), and I had no idea that activation context cache was a thing until couple of hours ago.

> Doesn't feel very intuitive that you cannot rename an application (as well it working in a previous pyinstaller version), but that's Windows for you. 

Technically, renaming the executable within the frozen bundle amounts to messing with its internal structure, so as far as I'm concerned, that falls into do-at-your-own-peril area. Sure, it worked in earlier versions, but that was due to specifics of your application (i.e., no use of manifest-related options, and not using package sideload option).

Surely instead of renaming the executable, you could have generated the bundle with the desired name in the first place?

> Side question, is there anyway to embed an internal manifest?

Yes, there is. We actually do that [with `onefile` builds](https://github.com/pyinstaller/pyinstaller/blob/ed046e587520850922263d8f403f4f14459c08bf/PyInstaller/building/api.py#L655-L656), and now I'm not even sure why we are using external manifests with `onedir` bundles...

I suspect there's a historic reason for that (e.g., as implied by cc01cab0bf5d15f6ec1231fa1ec12bfa24e9b453), but it may not be relevant nowadays anymore...

> I will look into that workaround about modifying the modification date, it looks like `os.utime` may offer this functionality.
> 
> EDIT: Looks like it can cache 'no manifest' as well. If you try running without the manifest, then add one after, it's still bricked.

Yes. But missing manifest due to name mismatch is same as manifest not being present there at all.


> I hear you, but the truth is that we (I?) became aware of the problem with MinGW-compiled bootloaders only [a week ago](https://github.com/pyinstaller/pyinstaller/discussions/6196#discussioncomment-1302204), and I had no idea that activation context cache was a thing until couple of hours ago.

Just for clarity, the issue is also present in the default bootloader, the only difference is the default bootloader calls this `ordinal 380` while gcc calls it `LoadIconMetric`. But yeah I have never heard of this manifest caching until now.

> Technically, renaming the executable within the frozen bundle amounts to messing with its internal structure, so as far as I'm concerned, that falls into do-at-your-own-peril area. Sure, it worked in earlier versions, but that was due to specifics of your application (i.e., no use of manifest-related options, and not using package sideload option).

Fair enough.  As someone not too familiar with application creation, in a lot of programs I have tinkered with in the past, removing the manifest never caused any (noticeable) issue. So initially my thought process was more of a 'suggestion' for Windows.

> Surely instead of renaming the executable, you could have generated the bundle with the desired name in the first place?

As far as why I would rename it, this was more of my use case, but was for updating the application to the latest version for an end user. When a new update is pushed out, I could push out a file called like `run_application_v123`, then attempt to launch it. If it launched correctly, then the launcher could rename the old one to `run_application.exe.bak` and update the new one to  `run_application.exe`. Perhaps there is a better way of achieving this, but that is what I came up with.

> Yes, there is. We actually do that with onefile builds, and now I'm not even sure why we are using external manifests with onedir bundles...

Would be interested in a revisit to this sometime in the future; may be beneficial?

> > I hear you, but the truth is that we (I?) became aware of the problem with MinGW-compiled bootloaders only [a week ago](https://github.com/pyinstaller/pyinstaller/discussions/6196#discussioncomment-1302204), and I had no idea that activation context cache was a thing until couple of hours ago.
> 
> Just for clarity, the issue is also present in the default bootloader, the only difference is the default bootloader calls this `ordinal 380` while gcc calls it `LoadIconMetric`. 

Yes, the rename-induced issue is present with both bootloader variants. I was referring to an issue where MinGW-compiled bootloaders wouldn't load external manifest at all, because MinGW already embedded a default one into the executable.

> > Yes, there is. We actually do that with onefile builds, and now I'm not even sure why we are using external manifests with onedir bundles...
> 
> Would be interested in a revisit to this sometime in the future; may be beneficial?

Yeah, I'll add this to my TODO list. I think what we should be doing here is embedding manifests into `onedir` builds by default, and introduce a spec-file/command-line option to switch back to an external manifest, in case someone's workflow depends on that (e.g., some manual or automated modification of the manifest file in post-processing).
Doesn't mean much but I looked at https://github.com/pyinstaller/pyinstaller/commit/cc01cab0bf5d15f6ec1231fa1ec12bfa24e9b453 to see where the manifest was omitted from one dir and changed:

```py
            if self.manifest and not self.exclude_binaries:
                self.manifest.update_resources(tmpnm, [1])
```
to 
```py
            if self.manifest:
                self.manifest.update_resources(tmpnm, [1])
```

It embedded the manifest and I could run and rename the file and/or remove manifest with no issue. Promising for my use case atleast. 👍 Perhaps there is some library that has trouble with this? An option to allow embedding sounds like a good start. 
Because in 4.4, we introduced a new unhandled-error dialog (#5890) for windowed mode, which uses Common Controls V6, and those need to be switched on via the manifest. That codepath is enabled only in windowed/noconsole bootloaders, so those are the ones that are affected by the missing manifest in a clear and obvious way (EDIT: "clear and obvious" as in "refuses to start at all"). 

But if your application used `--uic-admin` for example (to prompt for administrator mode at startup), that would stop working as soon as manifest was broken, regardless of console or non-console, (And even on older PyInstaller versions, such as 4.2).
Or, if the application had to work with long file paths, that would end up broken as well, because `longPathAware` is also turned on via the manifest.
Thanks, that makes sense.

Resolving patch:
diff --git a/PyInstaller/building/api.py b/PyInstaller/building/api.py
--- a/PyInstaller/building/api.py
+++ b/PyInstaller/building/api.py
@@ -338,6 +338,11 @@ def __init__(self, *args, **kwargs):
                 Windows only. Setting to True creates a Manifest with will request elevation upon application start.
             uac_uiaccess
                 Windows only. Setting to True allows an elevated application to work with Remote Desktop.
+            embed_manifest
+                Windows only. Setting to True (the default) embeds the manifest into the executable. Setting to False
+                generates an external .exe.manifest file. Applicable only in onedir mode (exclude_binaries=True); in
+                onefile mode (exclude_binaries=False), the manifest is always embedded in the executable, regardless
+                of this option.
             argv_emulation
                 macOS only. Enables argv emulation in macOS .app bundles (i.e., windowed bootloader). If enabled, the
                 initial open document/URL Apple Events are intercepted by bootloader and converted into sys.argv.
@@ -366,6 +371,7 @@ def __init__(self, *args, **kwargs):
         self.icon = kwargs.get('icon', None)
         self.versrsrc = kwargs.get('version', None)
         self.manifest = kwargs.get('manifest', None)
+        self.embed_manifest = kwargs.get('embed_manifest', True)
         self.resources = kwargs.get('resources', [])
         self.strip = kwargs.get('strip', False)
         self.upx_exclude = kwargs.get("upx_exclude", [])
@@ -458,6 +464,11 @@ def __init__(self, *args, **kwargs):
             self.toc.append(("pyi-macos-argv-emulation", "", "OPTION"))
 
         if is_win:
+            if not self.exclude_binaries:
+                # onefile mode forces embed_manifest=True
+                if not self.embed_manifest:
+                    logger.warning("Ignoring embed_manifest=False setting in onefile mode!")
+                self.embed_manifest = True
             if not self.icon:
                 # --icon not specified; use default from bootloader folder
                 if self.console:
@@ -472,11 +483,11 @@ def __init__(self, *args, **kwargs):
 
             manifest_filename = os.path.basename(self.name) + ".manifest"
 
-            # In the onedir mode, we need to collect the manifest file, so that it is placed next to the executable as
-            # an external manifest. In the onefile mode, we do not need to collect the manifest, as it is embedded into
-            # the final executable by the assembly pipeline.
-            if self.exclude_binaries:
-                self.toc.append((manifest_filename, filename, 'BINARY'))  # Onedir mode.
+            # If external manifest file is requested (supported only in onedir mode), add the file to the TOC in order
+            # for it to be collected as an external manifest file. Otherwise, the assembly pipeline will embed the
+            # manifest into the executable later on.
+            if not self.embed_manifest:
+                self.toc.append((manifest_filename, filename, 'BINARY'))
 
             if self.versrsrc:
                 if not isinstance(self.versrsrc, versioninfo.VSVersionInfo) and not os.path.isabs(self.versrsrc):
@@ -513,6 +524,7 @@ def __init__(self, *args, **kwargs):
         ('uac_admin', _check_guts_eq),
         ('uac_uiaccess', _check_guts_eq),
         ('manifest', _check_guts_eq),
+        ('embed_manifest', _check_guts_eq),
         ('append_pkg', _check_guts_eq),
         ('argv_emulation', _check_guts_eq),
         ('target_arch', _check_guts_eq),
@@ -650,9 +662,8 @@ def assemble(self):
                             resfile,
                             exc_info=1
                         )
-            # In onefile mode, we need to embed the manifest into the executable in order for manifest-related options
-            # (e.g., uac-admin) to work.
-            if self.manifest and not self.exclude_binaries:
+            # Embed the manifest into the executable.
+            if self.embed_manifest:
                 self.manifest.update_resources(tmpnm, [1])
             trash.append(tmpnm)
             exe = tmpnm
diff --git a/PyInstaller/building/makespec.py b/PyInstaller/building/makespec.py
--- a/PyInstaller/building/makespec.py
+++ b/PyInstaller/building/makespec.py
@@ -479,6 +479,13 @@ def __add_options(parser):
         metavar="<FILE or XML>",
         help="Add manifest FILE or XML to the exe.",
     )
+    g.add_argument(
+        "--no-embed-manifest",
+        dest="embed_manifest",
+        action="store_false",
+        help="Generate an external .exe.manifest file instead of embedding the manifest into the exe. Applicable only "
+        "to onedir mode; in onefile mode, the manifest is always embedded, regardless of this option.",
+    )
     g.add_argument(
         "-r",
         "--resource",
@@ -610,6 +617,7 @@ def main(
     binaries=None,
     icon_file=None,
     manifest=None,
+    embed_manifest=True,
     resources=None,
     bundle_identifier=None,
     hiddenimports=None,
@@ -687,6 +695,8 @@ def main(
         else:
             # Assume filename
             exe_options += "\n    manifest='%s'," % escape_win_filepath(manifest)
+    if not embed_manifest:
+        exe_options += "\n    embed_manifest=False,"
     if resources:
         resources = list(map(escape_win_filepath, resources))
         exe_options += "\n    resources=%s," % repr(resources)


Test patch: