Repository: pyinstaller/pyinstaller
Pull request: Fix exe being rebuilt every time on windows
Fixes https://github.com/pyinstaller/pyinstaller/issues/2239

Problem statement:
PyInstaller rebuilds exe every time
PyInstaller on Windows keeps rebuilding the executable on every invocation even if nothing has changed. It claims `Building because manifest changed`.

I invoke PyInstaller with

```
pyinstaller --noupx                  ^
            --icon "%iconfile,0"     ^
            --onefile                ^
            --workpath build\windows ^
            --specpath build\windows ^
            --distpath build         ^
            %1
```

```
[F:\...\script] tcc> mkexe3.btm script.py
202 INFO: PyInstaller: 3.2
203 INFO: Python: 3.5.2
204 INFO: Platform: Windows-10-10.0.10586-SP0
207 INFO: wrote build\windows\script.spec
255 INFO: UPX is available.
259 INFO: Extending PYTHONPATH with paths
['F:\\Documents\\dev\\python\\script',
 'F:\\Documents\\dev\\python\\script\\build\\windows']
260 INFO: checking Analysis
339 INFO: checking PYZ
378 INFO: checking PKG
384 INFO: Building because build\windows\script\script.exe.manifest changed
385 INFO: Building PKG (CArchive) out00-PKG.pkg
3311 INFO: Bootloader F:\PortableApps\Python35\lib\site-packages\PyInstaller\bootloader\Windows-64bit\run.exe
3312 INFO: checking EXE
3317 INFO: Building because manifest changed
3318 INFO: Building EXE from out00-EXE.toc
3375 INFO: SRCPATH [('\\PortableApps\\Python3x\\python.exe', 0)]
3375 INFO: Updating icons from \PortableApps\Python3x\python.exe, 0 to F:\PortableApps\ConEmu\Data\Temp\tmp87fgkiez
3410 INFO: Appending archive to EXE build\script.exe
```

The first `building PKG` refers to line 68 in `_check_guts_toc_mtime` in `utils.py`:

```
for (nm, fnm, typ) in old:
    if misc.mtime(fnm) > last_build:
        logger.info("Building because %s changed", fnm)
```

The second `building EXE` refers to

```
def _check_guts_eq(attr, old, new, last_build):
    """
    rebuild is required if values differ
    """
    if old != new:

        logger.info("Building because %s changed", attr)
```

`old` and `new` contain the content of the manifest. They have the same `supportedOS Id` attributes but the order is different. `new` has the same order as the manifest, `old` a different one.

Hints from issue discussion:
The manual discusses this problem under [Creating a reproducible build](http://pyinstaller.readthedocs.io/en/stable/advanced-topics.html#creating-a-reproducible-build)

I'm not sure if I understand this correctly. I don't want to create identical builds. I want PyInstaller to skip building if nothing has changed. That's how it works on Linux,

Attaching the log from Linux from the same source. No PKG or EXE is built on second run:

```
51 INFO: PyInstaller: 3.2
51 INFO: Python: 3.5.2
52 INFO: Platform: Linux-4.4.0-45-generic-x86_64-with-Ubuntu-16.04-xenial
56 INFO: wrote build/linux/script.spec
63 INFO: UPX is available.
70 INFO: Extending PYTHONPATH with paths
['/mnt/hgfs/drive-f/Documents/dev/python/script',
 '/mnt/hgfs/drive-f/Documents/dev/python/script/build/linux']
70 INFO: checking Analysis
116 INFO: checking PYZ
127 INFO: checking PKG
137 INFO: Bootloader /usr/local/lib/python3.5/dist-packages/PyInstaller/bootloader/Linux-64bit/run
137 INFO: checking EXE
```

Setting `PYTHONSEED` prevents the ``building PKG` but the executable is still rebuilt:

```
207 INFO: PyInstaller: 3.2
208 INFO: Python: 3.5.2
209 INFO: Platform: Windows-10-10.0.10586-SP0
211 INFO: wrote build\windows\script.spec
256 INFO: UPX is available.
258 INFO: Extending PYTHONPATH with paths
['F:\\Documents\\dev\\python\\script',
 'F:\\Documents\\dev\\python\\script\\build\\windows']
259 INFO: checking Analysis
337 INFO: checking PYZ
371 INFO: checking PKG
376 INFO: Bootloader F:\PortableApps\Python35\lib\site-packages\PyInstaller\bootloader\Windows-64bit\run.exe
376 INFO: checking EXE
379 INFO: Rebuilding out00-EXE.toc because mtimes don't match
379 INFO: Building EXE from out00-EXE.toc
416 INFO: SRCPATH [('\\PortableApps\\Python3x\\python.exe', 0)]
416 INFO: Updating icons from \PortableApps\Python3x\python.exe, 0 to F:\PortableApps\ConEmu\Data\Temp\tmp6jqs8zzg
451 INFO: Appending archive to EXE build\script.exe
```

It would helpful to know which mtimes it is unhappy with:

```
379 INFO: Rebuilding out00-EXE.toc because mtimes don't match
```

Line 452 in `api.py`:

```
def _check_guts(self, data, last_build):
[...]
    mtm = data['mtm']
    if mtm != misc.mtime(self.name):
        logger.info("Rebuilding %s because mtimes don't match", self.tocbasename)
```

The rebuild is actually triggered every second time: no rebuild, rebuild, no rebuild, rebuild.

> old and new contain the content of the manifest. They have the same supportedOS Id attributes but the order is different. new has the same order as the manifest, old a different one.

Please post the content of one of these. This `supportedOS` is Windoof specific so I can not check it on my systems.

> Setting PYTHONSEED prevents the `building PKG but the executable is still rebuilt:

Did you rebuild _twice_ with PYTHONSEED set?

@stephenrauch: Well spotted!

file `build\windows\script\script.exe.manifest`:

``` xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<assembly manifestVersion="1.0" xmlns="urn:schemas-microsoft-com:asm.v1">
  <assemblyIdentity name="script" processorArchitecture="amd64" type="win32" version="1.0.0.0"/>
  <compatibility xmlns="urn:schemas-microsoft-com:compatibility.v1">
    <application>
      <supportedOS Id="{1f676c76-80e1-4239-95bb-83d0f6d0da78}"/>
      <supportedOS Id="{e2011457-1546-43c5-a5fe-008deee3d3f0}"/>
      <supportedOS Id="{35138b9a-5d96-4fbd-8e2d-a2440225f93a}"/>
      <supportedOS Id="{4a2f28e3-53b9-4441-ba9c-d69d4a4a6e38}"/>
      <supportedOS Id="{8e0f7a12-bfb3-4fe8-b9a5-48fd50a15a9a}"/>
    </application>
  </compatibility>
</assembly>
```

> > the rebuild is actually triggered every second time: no rebuild, rebuild, no rebuild, rebuild.
> 
> Did you rebuild twice with PYTHONSEED set?

Actually I've rebuilt about a dozen times with `PYTHONSEED` set.

By the way: Microsoft offers virtual Windows images for developers: https://developer.microsoft.com/en-us/microsoft-edge/tools/vms/

Stale

Commit message: Fix check for windows manifest change

This check failing meant the PKG toc guts check failed every time
causing unnecessary rebuilds
see https://github.com/pyinstaller/pyinstaller/issues/2239

Patch to review:
diff --git a/PyInstaller/building/api.py b/PyInstaller/building/api.py
index 4b4a513bd14..31518a12791 100644
--- a/PyInstaller/building/api.py
+++ b/PyInstaller/building/api.py
@@ -476,6 +476,13 @@ def __init__(self, *args, **kwargs):
             self.toc.append(("pyi-disable-windowed-traceback", "", "OPTION"))
 
         if is_win:
+            if not self.icon:
+                # --icon not specified; use default from bootloader folder
+                self.icon = os.path.join(
+                    os.path.dirname(os.path.dirname(__file__)),
+                    'bootloader', 'images',
+                    'icon-console.ico' if self.console else 'icon-windowed.ico')
+
             filename = os.path.join(CONF['workpath'], CONF['specnm'] + ".exe.manifest")
             self.manifest = winmanifest.create_manifest(filename, self.manifest,
                 self.console, self.uac_admin, self.uac_uiaccess)
@@ -607,12 +614,6 @@ def assemble(self):
             os.close(fd)
             self._copyfile(exe, tmpnm)
             os.chmod(tmpnm, 0o755)
-            if not self.icon:
-                # --icon not specified; use default from bootloader folder
-                self.icon = os.path.join(
-                    os.path.dirname(os.path.dirname(__file__)),
-                    'bootloader', 'images',
-                    'icon-console.ico' if self.console else 'icon-windowed.ico')
             if self.icon != "NONE":
                 icon.CopyIcons(tmpnm, self.icon)
             if self.versrsrc:
diff --git a/PyInstaller/utils/win32/winmanifest.py b/PyInstaller/utils/win32/winmanifest.py
index b6aacd3a326..e2ca3aa3b0b 100644
--- a/PyInstaller/utils/win32/winmanifest.py
+++ b/PyInstaller/utils/win32/winmanifest.py
@@ -1085,7 +1085,7 @@ def create_manifest(filename, manifest, console, uac_admin=False, uac_uiaccess=F
     # only write a new manifest if it is different from the old
     need_new = not os.path.exists(filename)
     if not need_new:
-        old_xml = ManifestFromXMLFile(filename).toprettyxml()
+        old_xml = ManifestFromXMLFile(filename).toprettyxml().replace('\r','')
         new_xml = manifest.toprettyxml().replace('\r','')
 
         # this only works if PYTHONHASHSEED is set in environment


Code review comments:
diff_hunk: @@ -1085,7 +1085,7 @@ def create_manifest(filename, manifest, console, uac_admin=False, uac_uiaccess=F
     # only write a new manifest if it is different from the old
     need_new = not os.path.exists(filename)
     if not need_new:
-        old_xml = ManifestFromXMLFile(filename).toprettyxml()
+        old_xml = ManifestFromXMLFile(filename).toprettyxml().replace('\r','')
         new_xml = manifest.toprettyxml().replace('\r','')
path: PyInstaller/utils/win32/winmanifest.py
text: @author:
```suggestion
        new_xml = manifest.toprettyxml().replace('\r', '')
```

diff_hunk: @@ -476,6 +476,13 @@ def __init__(self, *args, **kwargs):
             self.toc.append(("pyi-disable-windowed-traceback", "", "OPTION"))
 
         if is_win:
+            if not self.icon:
+                # --icon not specified; use default from bootloader folder
+                self.icon = os.path.join(
+                    os.path.dirname(os.path.dirname(__file__)),
+                    'bootloader', 'images',
+                    'icon-console.ico' if self.console else 'icon-windowed.ico')
+
path: PyInstaller/building/api.py
text: @author:
```suggestion
            if not self.icon:
                # --icon not specified; use default from bootloader folder
                if self.console:
                    icon = 'icon-console.ico'
                else:
                    icon = 'icon-windowed.ico'
                self.icon = os.path.join(
                    os.path.dirname(os.path.dirname(__file__)),
                    'bootloader', 'images', icon)
```

diff_hunk: @@ -1085,7 +1085,7 @@ def create_manifest(filename, manifest, console, uac_admin=False, uac_uiaccess=F
     # only write a new manifest if it is different from the old
     need_new = not os.path.exists(filename)
     if not need_new:
-        old_xml = ManifestFromXMLFile(filename).toprettyxml()
+        old_xml = ManifestFromXMLFile(filename).toprettyxml().replace('\r','')
path: PyInstaller/utils/win32/winmanifest.py
text: @author:
```suggestion
        old_xml = ManifestFromXMLFile(filename).toprettyxml().replace('\r', '')
```