Repository: pyinstaller/pyinstaller

Issue:
Packaging fails on Windows x86 (32-bit) systems, prompting a memory error.
My attempt to package on a Windows x86 (32-bit) system failed, prompting a memory error. The python version is 3.9.7 x86. At the same time, I try to package on Windows x64 (64 bit) system. The python version is 3.9.7 x64, but the same program can be packaged successfully.
The final size of the EXE program should be about 700MB.At the same time, after testing, if the size of the packaged program is less than 500MB, there is basically no problem.

![image](https://user-images.githubusercontent.com/75734559/172138403-57781e74-4deb-4cf9-b7e6-9a53dfa5496b.png)
![image](https://user-images.githubusercontent.com/75734559/172138812-00aec219-bbcc-433c-96d1-ec3b6eae00d3.png)



Discussion:
This looks like memory shortage problem. How much RAM do you have available on your Windows x86 system?
> 这看起来像内存短缺问题。您的 Windows x86 系统上有多少可用 RAM？

16G.
![image](https://user-images.githubusercontent.com/75734559/172181880-16916882-d642-4af8-882f-922bee2f7f94.png)

This is 64-bit system, is it not? Is the 32-bit build failing on native 32-bit system, or are you running 32-bit python on this system as well?
> 这是64位系统，不是吗？32 位构建是否在本机 32 位系统上失败，或者您是否也在此系统上运行 32 位 python？

I installed Python 3.7.9 x86 on my computer. I am packing on a 64 bit system, that is, the computer in the figure.
![image](https://user-images.githubusercontent.com/75734559/172194823-bbe3eae4-0374-4172-adbd-8f84630262bd.png)

OK. In that case, the memory limit (presumably for the whole process) should still be around 2 GB... Do you have any way of checking what the memory consumption is like during the build and before the error is hit (even using Task Manager, if there's no better tool available)?

---

I suppose a work-around would be to comment this line in your 32-bit PyInstaller installation:
https://github.com/pyinstaller/pyinstaller/blob/2df8314ffaedd95ddc9e2871237e2f2188d3735e/PyInstaller/building/api.py#L768

and see if the build succeeds. 

If it does, you can manually fix-up the exe headers by opening python console (in 64-bit python, or even 32-bit one, if there is less memory pressure) and running:

```python
import time
from PyInstaller.utils.win32 import winutils
winutils.fixup_exe_headers(r'dist\program_name.exe`, time.time())
```


> ```python
> winutils.fixup_exe_headers(r'dist\program_name.exe`, time.time())
> ```

The above method is OK. I found that after commenting out the line "winutils.fixup\u exe\u headers (build\u name, build\u timestamp)", the packaging speed of the whole program is very fast. In addition, even if you do not use the code to manually repair the EXE header, the packaged program can run normally. What is the reason?
> > ```python
> > winutils.fixup_exe_headers(r'dist\program_name.exe`, time.time())
> > ```
> 
> The above method is OK. I found that after commenting out the line "winutils.fixup\u exe\u headers (build\u name, build\u timestamp)", the packaging speed of the whole program is very fast. In addition, even if you do not use the code to manually repair the EXE header, the packaged program can run normally. What is the reason?

If you execute the repair exe code, it will still report an error!
![image](https://user-images.githubusercontent.com/75734559/172437110-9cc70bab-e0cc-4bcb-b5c4-94ae26508091.png)

> > ```python
> > winutils.fixup_exe_headers(r'dist\program_name.exe`, time.time())
> > ```
> 
> The above method is OK. I found that after commenting out the line "winutils.fixup\u exe\u headers (build\u name, build\u timestamp)", the packaging speed of the whole program is very fast. 

Commenting out this line should not have any effect on the speed of other building steps (but on subsequent build runs, the build might be re-using cached results, unless you specify `--clean`). On the other hand, in order to fixup the headers, the whole exe needs to be read (so that PE checksum can be computed) - but I imagine that is also not the speed-up you are observing, because that `MemoryError` occurs while reading the file... Or are you talking about speed-up of 64-bit build?

> In addition, even if you do not use the code to manually repair the EXE header, the packaged program can run normally. What is the reason?

The headers fix-up does two things:
- it sets the build timestamp to current time (as opposed to the time when pre-built bootloaders were compiled)
- it recomputes the PE checksum (which should help against the exe being flagged as malware by anti-virus programs, at least in theory...)

Neither of these are critical, and the program can run without them being corrected/recomputed.
> If you execute the repair exe code, it will still report an error!

And if you do it from 64-bit python?
> > If you execute the repair exe code, it will still report an error!
> 
> And if you do it from 64-bit python?

That's all right.
> > > If you execute the repair exe code, it will still report an error!
> > 
> > 
> > And if you do it from 64-bit python?
> 
> That's all right.

Only x86 will report an error.
> > > > If you execute the repair exe code, it will still report an error!
> > > 
> > > 
> > > And if you do it from 64-bit python?
> > 
> > 
> > That's all right.
> 
> Only x86 will report an error.

I guess that means that 32-bit python process does not get enough memory to load and process an exe of that size.
> > > > > If you execute the repair exe code, it will still report an error!
> > > > 
> > > > 
> > > > And if you do it from 64-bit python?
> > > 
> > > 
> > > That's all right.
> > 
> > 
> > Only x86 will report an error.
> 
> I guess that means that 32-bit python process does not get enough memory to load and process an exe of that size.

Maybe so.

Resolving patch:
diff --git a/PyInstaller/building/api.py b/PyInstaller/building/api.py
--- a/PyInstaller/building/api.py
+++ b/PyInstaller/building/api.py
@@ -764,8 +764,10 @@ def assemble(self):
         if is_win:
             # Set checksum to appease antiviral software. Also set build timestamp to current time to increase entropy
             # (but honor SOURCE_DATE_EPOCH environment variable for reproducible builds).
+            logger.info("Fixing EXE headers")
             build_timestamp = int(os.environ.get('SOURCE_DATE_EPOCH', time.time()))
-            winutils.fixup_exe_headers(build_name, build_timestamp)
+            winutils.set_exe_build_timestamp(build_name, build_timestamp)
+            winutils.update_exe_pe_checksum(build_name)
         elif is_darwin:
             # If the version of macOS SDK used to build bootloader exceeds that of macOS SDK used to built Python
             # library (and, by extension, bundled Tcl/Tk libraries), force the version declared by the frozen executable
diff --git a/PyInstaller/utils/win32/winutils.py b/PyInstaller/utils/win32/winutils.py
--- a/PyInstaller/utils/win32/winutils.py
+++ b/PyInstaller/utils/win32/winutils.py
@@ -148,18 +148,19 @@ def convert_dll_name_to_str(dll_name):
         return dll_name
 
 
-def fixup_exe_headers(exe_path, timestamp=None):
+def set_exe_build_timestamp(exe_path, timestamp):
     """
-    Set executable's checksum and build timestamp in its headers.
-
-    This optional checksum is supposed to protect the executable against corruption but some anti-viral software have
-    taken to flagging anything without it set correctly as malware. See issue #5579.
+    Modifies the executable's build timestamp by updating values in the corresponding PE headers.
     """
     import pefile
-    pe = pefile.PE(exe_path, fast_load=False)  # full load because we need all headers
-    # Set build timestamp.
-    # See: https://0xc0decafe.com/malware-analyst-guide-to-pe-timestamps
-    if timestamp is not None:
+
+    with pefile.PE(exe_path, fast_load=True) as pe:
+        # Manually perform a full load. We need it to load all headers, but specifying it in the constructor triggers
+        # byte statistics gathering that takes forever with large files. So we try to go around that...
+        pe.full_load()
+
+        # Set build timestamp.
+        # See: https://0xc0decafe.com/malware-analyst-guide-to-pe-timestamps
         timestamp = int(timestamp)
         # Set timestamp field in FILE_HEADER
         pe.FILE_HEADER.TimeDateStamp = timestamp
@@ -169,7 +170,189 @@ def fixup_exe_headers(exe_path, timestamp=None):
         for debug_entry in debug_entries:
             if debug_entry.struct.TimeDateStamp:
                 debug_entry.struct.TimeDateStamp = timestamp
-    # Set PE checksum
-    pe.OPTIONAL_HEADER.CheckSum = pe.generate_checksum()
-    pe.close()
-    pe.write(exe_path)
+
+        # Generate updated EXE data
+        data = pe.write()
+
+    # Rewrite the exe
+    with open(exe_path, 'wb') as fp:
+        fp.write(data)
+
+
+def update_exe_pe_checksum(exe_path):
+    """
+    Compute the executable's PE checksum, and write it to PE headers.
+
+    This optional checksum is supposed to protect the executable against corruption but some anti-viral software have
+    taken to flagging anything without it set correctly as malware. See issue #5579.
+    """
+    import pefile
+
+    # Compute checksum using our equivalent of the MapFileAndCheckSumW - for large files, it is significantly faster
+    # than pure-pyton pefile.PE.generate_checksum(). However, it requires the file to be on disk (i.e., cannot operate
+    # on a memory buffer).
+    try:
+        checksum = compute_exe_pe_checksum(exe_path)
+    except Exception as e:
+        raise RuntimeError("Failed to compute PE checksum!") from e
+
+    # Update the checksum
+    with pefile.PE(exe_path, fast_load=True) as pe:
+        pe.OPTIONAL_HEADER.CheckSum = checksum
+
+        # Generate updated EXE data
+        data = pe.write()
+
+    # Rewrite the exe
+    with open(exe_path, 'wb') as fp:
+        fp.write(data)
+
+
+def compute_exe_pe_checksum(exe_path):
+    """
+    This is a replacement for the MapFileAndCheckSumW function. As noted in MSDN documentation, the Microsoft's
+    implementation of MapFileAndCheckSumW internally calls its ASCII variant (MapFileAndCheckSumA), and therefore
+    cannot handle paths that contain characters that are not representable in the current code page.
+    See: https://docs.microsoft.com/en-us/windows/win32/api/imagehlp/nf-imagehlp-mapfileandchecksumw
+
+    This function is based on Wine's implementation of MapFileAndCheckSumW, and due to being based entirely on
+    the pure widechar-API functions, it is not limited by the current code page.
+    """
+    # ctypes bindings for relevant win32 API functions
+    import ctypes
+    from ctypes import windll, wintypes
+
+    INVALID_HANDLE = wintypes.HANDLE(-1).value
+
+    GetLastError = ctypes.windll.kernel32.GetLastError
+    GetLastError.argtypes = ()
+    GetLastError.restype = wintypes.DWORD
+
+    CloseHandle = windll.kernel32.CloseHandle
+    CloseHandle.argtypes = (
+        wintypes.HANDLE,  # hObject
+    )
+    CloseHandle.restype = wintypes.BOOL
+
+    CreateFileW = windll.kernel32.CreateFileW
+    CreateFileW.argtypes = (
+        wintypes.LPCWSTR,  # lpFileName
+        wintypes.DWORD,  # dwDesiredAccess
+        wintypes.DWORD,  # dwShareMode
+        wintypes.LPVOID,  # lpSecurityAttributes
+        wintypes.DWORD,  # dwCreationDisposition
+        wintypes.DWORD,  # dwFlagsAndAttributes
+        wintypes.HANDLE,  # hTemplateFile
+    )
+    CreateFileW.restype = wintypes.HANDLE
+
+    CreateFileMappingW = windll.kernel32.CreateFileMappingW
+    CreateFileMappingW.argtypes = (
+        wintypes.HANDLE,  # hFile
+        wintypes.LPVOID,  # lpSecurityAttributes
+        wintypes.DWORD,  # flProtect
+        wintypes.DWORD,  # dwMaximumSizeHigh
+        wintypes.DWORD,  # dwMaximumSizeLow
+        wintypes.LPCWSTR,  # lpName
+    )
+    CreateFileMappingW.restype = wintypes.HANDLE
+
+    MapViewOfFile = windll.kernel32.MapViewOfFile
+    MapViewOfFile.argtypes = (
+        wintypes.HANDLE,  # hFileMappingObject
+        wintypes.DWORD,  # dwDesiredAccess
+        wintypes.DWORD,  # dwFileOffsetHigh
+        wintypes.DWORD,  # dwFileOffsetLow
+        wintypes.DWORD,  # dwNumberOfBytesToMap
+    )
+    MapViewOfFile.restype = wintypes.LPVOID
+
+    UnmapViewOfFile = windll.kernel32.UnmapViewOfFile
+    UnmapViewOfFile.argtypes = (
+        wintypes.LPCVOID,  # lpBaseAddress
+    )
+    UnmapViewOfFile.restype = wintypes.BOOL
+
+    GetFileSizeEx = windll.kernel32.GetFileSizeEx
+    GetFileSizeEx.argtypes = (
+        wintypes.HANDLE,  # hFile
+        wintypes.PLARGE_INTEGER,  # lpFileSize
+    )
+
+    CheckSumMappedFile = windll.imagehlp.CheckSumMappedFile
+    CheckSumMappedFile.argtypes = (
+        wintypes.LPVOID,  # BaseAddress
+        wintypes.DWORD,  # FileLength
+        wintypes.PDWORD,  # HeaderSum
+        wintypes.PDWORD,  # CheckSum
+    )
+    CheckSumMappedFile.restype = wintypes.LPVOID
+
+    # Open file
+    hFile = CreateFileW(
+        ctypes.c_wchar_p(exe_path),
+        0x80000000,  # dwDesiredAccess = GENERIC_READ
+        0x00000001 | 0x00000002,  # dwShareMode = FILE_SHARE_READ | FILE_SHARE_WRITE,
+        None,  # lpSecurityAttributes = NULL
+        3,  # dwCreationDisposition = OPEN_EXISTING
+        0x80,  # dwFlagsAndAttributes = FILE_ATTRIBUTE_NORMAL
+        None  # hTemplateFile = NULL
+    )
+    if hFile == INVALID_HANDLE:
+        err = GetLastError()
+        raise RuntimeError(f"Failed to open file {exe_path}! Error code: {err}")
+
+    # Query file size
+    fileLength = wintypes.LARGE_INTEGER(0)
+    if GetFileSizeEx(hFile, fileLength) == 0:
+        err = GetLastError()
+        CloseHandle(hFile)
+        raise RuntimeError(f"Failed to query file size file! Error code: {err}")
+    fileLength = fileLength.value
+    if fileLength > (2**32 - 1):
+        raise RuntimeError("Executable size exceeds maximum allowed executable size on Windows (4 GiB)!")
+
+    # Map the file
+    hMapping = CreateFileMappingW(
+        hFile,
+        None,  # lpFileMappingAttributes = NULL
+        0x02,  # flProtect = PAGE_READONLY
+        0,  # dwMaximumSizeHigh = 0
+        0,  # dwMaximumSizeLow = 0
+        None  # lpName = NULL
+    )
+    if not hMapping:
+        err = GetLastError()
+        CloseHandle(hFile)
+        raise RuntimeError(f"Failed to map file! Error code: {err}")
+
+    # Create map view
+    baseAddress = MapViewOfFile(
+        hMapping,
+        4,  # dwDesiredAccess = FILE_MAP_READ
+        0,  # dwFileOffsetHigh = 0
+        0,  # dwFileOffsetLow = 0
+        0  # dwNumberOfBytesToMap = 0
+    )
+    if baseAddress == 0:
+        err = GetLastError()
+        CloseHandle(hMapping)
+        CloseHandle(hFile)
+        raise RuntimeError(f"Failed to create map view! Error code: {err}")
+
+    # Finally, compute the checksum
+    headerSum = wintypes.DWORD(0)
+    checkSum = wintypes.DWORD(0)
+    ret = CheckSumMappedFile(baseAddress, fileLength, ctypes.byref(headerSum), ctypes.byref(checkSum))
+    if ret is None:
+        err = GetLastError()
+
+    # Cleanup
+    UnmapViewOfFile(baseAddress)
+    CloseHandle(hMapping)
+    CloseHandle(hFile)
+
+    if ret is None:
+        raise RuntimeError(f"CheckSumMappedFile failed! Error code: {err}")
+
+    return checkSum.value


Test patch: