Repository: pyinstaller/pyinstaller

Issue:
PyInstaller fails to read Windows versioninfo files with byteorder markers (BOM)
I was freezing my simple program using pyinstaller...

just then:

THIS HAPPENED:
```
  File "e:\programming\SECRET\environment\lib\site-packages\PyInstaller\utils\win32\versioninfo.py", line 632, in SetVersion
    vs = eval(txt)
  File "<string>", line 1
    ﻿# UTF-8
    ^
SyntaxError: invalid character in identifier
```

the whole log:
```
190 INFO: PyInstaller: 4.5.1
190 INFO: Python: 3.8.10
230 INFO: Platform: Windows-7-6.1.7601-SP1
230 INFO: wrote E:\Programming\SECRET\environment\Scripts\SECRET 3.4.spec
260 INFO: UPX is not available.
270 INFO: Extending PYTHONPATH with paths
['E:\\Programming\\SECRET\\environment',
 'E:\\Programming\\SECRET\\environment\\Scripts']
970 INFO: checking Analysis
980 INFO: Building Analysis because Analysis-00.toc is non existent
980 INFO: Initializing module dependency graph...
980 INFO: Caching module graph hooks...
1050 INFO: Analyzing base_library.zip ...
10248 INFO: Processing pre-find module path hook distutils from 'e:\\programming\\SECRET\\environment\\lib\\site-packages\\PyInstaller\\hooks\\pre_find_module_path\\hook-distutils.py'.
10258 INFO: distutils: retargeting to non-venv dir 'e:\\programming\\python38-32\\lib'
17268 INFO: Caching module dependency graph...
18038 INFO: running Analysis Analysis-00.toc
18088 INFO: Adding Microsoft.Windows.Common-Controls to dependent assemblies of final executable
  required by e:\programming\python38-32\python.exe
18628 INFO: Analyzing E:\Programming\SECRET\environment\main.py
20308 INFO: Processing pre-find module path hook site from 'e:\\programming\\SECRET\\environment\\lib\\site-packages\\PyInstaller\\hooks\\pre_find_module_path\\hook-site.py'.
20308 INFO: site: retargeting to fake-dir 'e:\\programming\\SECRET\\environment\\lib\\site-packages\\PyInstaller\\fake-modules'
26418 INFO: Processing module hooks...
26418 INFO: Loading module hook 'hook-pycparser.py' from 'e:\\programming\\SECRET\\environment\\lib\\site-packages\\_pyinstaller_hooks_contrib\\hooks\\stdhooks'...
26418 INFO: Loading module hook 'hook-difflib.py' from 'e:\\programming\\SECRET\\environment\\lib\\site-packages\\PyInstaller\\hooks'...
26418 INFO: Loading module hook 'hook-distutils.py' from 'e:\\programming\\SECRET\\environment\\lib\\site-packages\\PyInstaller\\hooks'...
26428 INFO: Loading module hook 'hook-distutils.util.py' from 'e:\\programming\\SECRET\\environment\\lib\\site-packages\\PyInstaller\\hooks'...
26428 INFO: Loading module hook 'hook-encodings.py' from 'e:\\programming\\SECRET\\environment\\lib\\site-packages\\PyInstaller\\hooks'...
26758 INFO: Loading module hook 'hook-heapq.py' from 'e:\\programming\\SECRET\\environment\\lib\\site-packages\\PyInstaller\\hooks'...
26758 INFO: Loading module hook 'hook-lib2to3.py' from 'e:\\programming\\SECRET\\environment\\lib\\site-packages\\PyInstaller\\hooks'...
27248 INFO: Loading module hook 'hook-multiprocessing.util.py' from 'e:\\programming\\SECRET\\environment\\lib\\site-packages\\PyInstaller\\hooks'...
27248 INFO: Loading module hook 'hook-pickle.py' from 'e:\\programming\\SECRET\\environment\\lib\\site-packages\\PyInstaller\\hooks'...
27258 INFO: Loading module hook 'hook-pkg_resources.py' from 'e:\\programming\\SECRET\\environment\\lib\\site-packages\\PyInstaller\\hooks'...
28108 INFO: Processing pre-safe import module hook win32com from 'e:\\programming\\SECRET\\environment\\lib\\site-packages\\_pyinstaller_hooks_contrib\\hooks\\pre_safe_import_module\\hook-win32com.py'.
28848 WARNING: Hidden import "pkg_resources.py2_warn" not found!
28848 WARNING: Hidden import "pkg_resources.markers" not found!
28858 INFO: Loading module hook 'hook-pygments.py' from 'e:\\programming\\SECRET\\environment\\lib\\site-packages\\PyInstaller\\hooks'...
34688 INFO: Loading module hook 'hook-setuptools.py' from 'e:\\programming\\SECRET\\environment\\lib\\site-packages\\PyInstaller\\hooks'...
36578 INFO: Loading module hook 'hook-sysconfig.py' from 'e:\\programming\\SECRET\\environment\\lib\\site-packages\\PyInstaller\\hooks'...
36578 INFO: Loading module hook 'hook-win32ctypes.core.py' from 'e:\\programming\\SECRET\\environment\\lib\\site-packages\\PyInstaller\\hooks'...
37448 INFO: Loading module hook 'hook-xml.dom.domreg.py' from 'e:\\programming\\SECRET\\environment\\lib\\site-packages\\PyInstaller\\hooks'...
37458 INFO: Loading module hook 'hook-xml.etree.cElementTree.py' from 'e:\\programming\\SECRET\\environment\\lib\\site-packages\\PyInstaller\\hooks'...
37458 INFO: Loading module hook 'hook-xml.py' from 'e:\\programming\\SECRET\\environment\\lib\\site-packages\\PyInstaller\\hooks'...
37468 INFO: Loading module hook 'hook-_tkinter.py' from 'e:\\programming\\SECRET\\environment\\lib\\site-packages\\PyInstaller\\hooks'...
38288 INFO: checking Tree
38288 INFO: Building Tree because Tree-00.toc is non existent
38298 INFO: Building Tree Tree-00.toc
38458 INFO: checking Tree
38458 INFO: Building Tree because Tree-01.toc is non existent
38458 INFO: Building Tree Tree-01.toc
38758 INFO: checking Tree
38758 INFO: Building Tree because Tree-02.toc is non existent
38758 INFO: Building Tree Tree-02.toc
38778 INFO: Loading module hook 'hook-setuptools.msvc.py' from 'e:\\programming\\SECRET\\environment\\lib\\site-packages\\PyInstaller\\hooks'...
38888 INFO: Looking for ctypes DLLs
39008 INFO: Analyzing run-time hooks ...
39028 INFO: Including run-time hook 'e:\\programming\\SECRET\\environment\\lib\\site-packages\\PyInstaller\\hooks\\rthooks\\pyi_rth_pkgutil.py'
39038 INFO: Including run-time hook 'e:\\programming\\SECRET\\environment\\lib\\site-packages\\PyInstaller\\hooks\\rthooks\\pyi_rth_multiprocessing.py'
39048 INFO: Including run-time hook 'e:\\programming\\SECRET\\environment\\lib\\site-packages\\PyInstaller\\hooks\\rthooks\\pyi_rth_inspect.py'
39058 INFO: Including run-time hook 'e:\\programming\\SECRET\\environment\\lib\\site-packages\\PyInstaller\\hooks\\rthooks\\pyi_rth_pkgres.py'
39078 INFO: Including run-time hook 'e:\\programming\\SECRET\\environment\\lib\\site-packages\\PyInstaller\\hooks\\rthooks\\pyi_rth__tkinter.py'
39108 INFO: Looking for dynamic libraries
39648 INFO: Looking for eggs
39648 INFO: Using Python library e:\programming\python38-32\python38.dll
39648 INFO: Found binding redirects:
[]
39658 INFO: Warnings written to E:\Programming\SECRET\environment\Scripts\build\SECRET 3.4\warn-SECRET 3.4.txt
39978 INFO: Graph cross-reference written to E:\Programming\SECRET\environment\Scripts\build\SECRET 3.4\xref-SECRET 3.4.html
40108 INFO: Appending 'datas' from .spec
40118 INFO: checking PYZ
40128 INFO: Building PYZ because PYZ-00.toc is non existent
40128 INFO: Building PYZ (ZlibArchive) E:\Programming\SECRET\environment\Scripts\build\SECRET 3.4\PYZ-00.pyz
42788 INFO: Building PYZ (ZlibArchive) E:\Programming\SECRET\environment\Scripts\build\SECRET 3.4\PYZ-00.pyz completed successfully.
42848 INFO: checking PKG
42848 INFO: Building PKG because PKG-00.toc is non existent
42848 INFO: Building PKG (CArchive) PKG-00.pkg
49314 INFO: Building PKG (CArchive) PKG-00.pkg completed successfully.
49394 INFO: Bootloader e:\programming\SECRET\environment\lib\site-packages\PyInstaller\bootloader\Windows-32bit\runw.exe
49394 INFO: checking EXE
49394 INFO: Building EXE because EXE-00.toc is non existent
49394 INFO: Building EXE from EXE-00.toc
49404 INFO: Copying icons from ['E:\\Programming\\SECRET\\environment\\DATA\\icons\\favicon.v3.ico']
49404 INFO: Writing RT_GROUP_ICON 0 resource with 20 bytes
49404 INFO: Writing RT_ICON 1 resource with 5903 bytes
Traceback (most recent call last):
  File "e:\programming\python38-32\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "e:\programming\python38-32\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "E:\Programming\SECRET\environment\Scripts\pyinstaller.exe\__main__.py", line 7, in <module>
  File "e:\programming\SECRET\environment\lib\site-packages\PyInstaller\__main__.py", line 126, in run
    run_build(pyi_config, spec_file, **vars(args))
  File "e:\programming\SECRET\environment\lib\site-packages\PyInstaller\__main__.py", line 65, in run_build
    PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
  File "e:\programming\SECRET\environment\lib\site-packages\PyInstaller\building\build_main.py", line 815, in main
    build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))
  File "e:\programming\SECRET\environment\lib\site-packages\PyInstaller\building\build_main.py", line 762, in build
    exec(code, spec_namespace)
  File "E:\Programming\SECRET\environment\Scripts\SECRET 3.4.spec", line 23, in <module>
    exe = EXE(pyz,
  File "e:\programming\SECRET\environment\lib\site-packages\PyInstaller\building\api.py", line 524, in __init__
    self.__postinit__()
  File "e:\programming\SECRET\environment\lib\site-packages\PyInstaller\building\datastruct.py", line 159, in __postinit__
    self.assemble()
  File "e:\programming\SECRET\environment\lib\site-packages\PyInstaller\building\api.py", line 622, in assemble
    versioninfo.SetVersion(tmpnm, self.versrsrc)
  File "e:\programming\SECRET\environment\lib\site-packages\PyInstaller\utils\win32\versioninfo.py", line 632, in SetVersion
    vs = eval(txt)
  File "<string>", line 1
    ﻿# UTF-8
    ^
SyntaxError: invalid character in identifier
```

the command i ran:
```
pyinstaller --noconfirm --onefile --windowed --icon "E:\Programming\SECRET\environment\DATA\icons\favicon.ico" --name "tHE gOOD aPPLICATION" --version-file "E:/Programming/SECRET/vfile.txt" --add-data "E:/Programming/Python38-32/Lib/site-packages/tkcode;./tkcode/" --add-data "E:/Programming/SECRET/environment/utils.py;."  "E:/Programming/SECRET/environment/main.py"
```

well.. My program has all good characters (not weird unicode) that too all typed by me ONLY

btw, i am using an virtual env for the project..

**Then

I tried another file and it worked fine..


Discussion:
I dug more and found that it is caused by my `vfile`..

its contents:

```
# UTF-8
#
# For more details about fixed file info 'ffi' see:
# http://msdn.microsoft.com/en-us/library/ms646997.aspx

VSVersionInfo(
  ffi=FixedFileInfo(
    # filevers and prodvers should be always a tuple with four items: (1, 2, 3, 4)
    # Set not needed items to zero 0. Must always contain 4 elements.
    filevers=(2,0,0,0),
    prodvers=(2,0,0,0),
    # Contains a bitmask that specifies the valid bits 'flags'r
    mask=0x3f,
    # Contains a bitmask that specifies the Boolean attributes of the file.
    flags=0x0,
    # The operating system for which this file was designed.
    # 0x4 - NT and there is no need to change it.
    OS=0x40004,
    # The general type of file.
    # 0x1 - the file is an application.
    fileType=0x1,
    # The function of the file.
    # 0x0 - the function is not defined for this fileType
    subtype=0x0,
    # Creation date and time stamp.
    date=(0, 0)
    ),
  kids=[
    StringFileInfo(
      [
      StringTable(
        u'040904B0',
        [StringStruct(u'CompanyName', u'whMSFT (Whirlpool-programmer)'),
        StringStruct(u'FileDescription', u'software 3.4'),
        StringStruct(u'FileVersion', u'3.4.0'),
        StringStruct(u'InternalName', u'software'),
        StringStruct(u'LegalCopyright', u'© 2020-21 whmsft. All rights reserved.'),
        StringStruct(u'OriginalFilename', u'software.exe'),
        StringStruct(u'ProductName', u'software'),
        StringStruct(u'ProductVersion', u'3.4.0')])
      ]), 
    VarFileInfo([VarStruct(u'Translation', [1033, 1200])])
  ]
)

```

I myself didn't made it.. but i used a python module to generate it..
how may i workaround it?
Can you share your vfile as an attachment instead of copy/pasting it in so that we can verify that it is UTF-8 as it claims to be? It seems to be implying that comments passed to `eval()` are a syntax error but that can be trivially disproved (see below) so it must be something else:
```python
>>> eval("# distracting comment\n1 + 1  # more comments")
2
```
Your `vfile` probably has a BOM at the beginning, which is causing the error. 

Use an editor with proper encodings support (e.g., Notepad++) and convert it from "UTF-8-BOM" to "UTF-8".
[WhirlEdit_2.txt](https://github.com/pyinstaller/pyinstaller/files/7273293/WhirlEdit_2.txt)

                                                                                   
I tried changing this file and the file after (my) changes ^

> I myself didn't made it.. but i used a python module to generate it..

Which module did you use?
It's as Rok says, there's a BOM at the start of the file:
```python
>>> open("WhirlEdit_2.txt", "rb").read(50)
b'\xef\xbb\xbfVSVersionInfo(\n  ffi=FixedFileInfo(\n    filever'
```
I guess we could make that function BOM tollerant without too much hassle.
> I guess we could make that function BOM tollerant without too much hassle.

Indeed.
> > I myself didn't made it.. but i used a python module to generate it..
> 
> Which module did you use?

Well.. i forgot..

I made that file months ago.. since then i have been editing it to keep up with the next versions of my software..

Resolving patch:
diff --git a/PyInstaller/utils/misc.py b/PyInstaller/utils/misc.py
--- a/PyInstaller/utils/misc.py
+++ b/PyInstaller/utils/misc.py
@@ -17,6 +17,10 @@
 import pprint
 import py_compile
 import sys
+import codecs
+import re
+import tokenize
+import io
 
 from PyInstaller import log as logging
 from PyInstaller.compat import BYTECODE_MAGIC, is_win
@@ -283,3 +287,28 @@ def is_file_qt_plugin(filename):
             return False
 
         return True
+
+
+BOM_MARKERS_TO_DECODERS = {
+    codecs.BOM_UTF32_LE: codecs.utf_32_le_decode,
+    codecs.BOM_UTF32_BE: codecs.utf_32_be_decode,
+    codecs.BOM_UTF32: codecs.utf_32_decode,
+    codecs.BOM_UTF16_LE: codecs.utf_16_le_decode,
+    codecs.BOM_UTF16_BE: codecs.utf_16_be_decode,
+    codecs.BOM_UTF16: codecs.utf_16_decode,
+    codecs.BOM_UTF8: codecs.utf_8_decode,
+}
+BOM_RE = re.compile(rb"\A(%s)?(.*)" % b"|".join(map(re.escape, BOM_MARKERS_TO_DECODERS)), re.DOTALL)
+
+
+def decode(raw: bytes):
+    """
+    Decode bytes to string, respecting and removing any byte-order marks if present, or respecting but not removing any
+    PEP263 encoding comments (# encoding: cp1252).
+    """
+    bom, raw = BOM_RE.match(raw).groups()
+    if bom:
+        return BOM_MARKERS_TO_DECODERS[bom](raw)[0]
+
+    encoding, _ = tokenize.detect_encoding(io.BytesIO(raw).readline)
+    return raw.decode(encoding)
diff --git a/PyInstaller/utils/win32/versioninfo.py b/PyInstaller/utils/win32/versioninfo.py
--- a/PyInstaller/utils/win32/versioninfo.py
+++ b/PyInstaller/utils/win32/versioninfo.py
@@ -642,8 +642,10 @@ def SetVersion(exenm, versionfile):
     if isinstance(versionfile, VSVersionInfo):
         vs = versionfile
     else:
-        with codecs.open(versionfile, 'r', 'utf-8') as fp:
-            txt = fp.read()
+        # Read and parse the version file. It may have a byte order marker or encoding cookie - respect it if it does.
+        from PyInstaller.utils.misc import decode
+        with open(versionfile, 'rb') as fp:
+            txt = decode(fp.read())
         vs = eval(txt)
 
     # Remember overlay


Test patch:
diff --git a/tests/unit/test_miscutils.py b/tests/unit/test_miscutils.py
--- a/tests/unit/test_miscutils.py
+++ b/tests/unit/test_miscutils.py
@@ -163,3 +163,30 @@ def read_file_version_info(filename, *attributes):
 
     values = read_file_version_info(test_file, 'FileDescription', 'ProductName', 'ProductVersion')
     assert values == [FILE_DESCRIPTION, PRODUCT_NAME, PRODUCT_VERSION]
+
+
+CHINESE_LOREM_IPSUM = """
+索感歴新策理通幻月続本初迷検価泉属速棄者。料流重者県作軽渡獲県行選見。意鑑民供変介画同表後院苦面例紙約度電心
+時盤止野断希茶術同懲上権策全崎査応校匹国。真村辺名下周一場庭戸原図会正前対校米利。岩録開教純止表乗都偉別際政難然始京国京
+校県水路中旅続球是校高約止性振下上派若福。案神整細番公流関閲元弊導説大実枝際確導
+"""
+
+
+def test_decode():
+    """
+    Test PyInstaller.utils.misc.decode().
+    """
+    import codecs
+    from PyInstaller.utils.misc import decode
+
+    # Test UTF8 with or without the BOM.
+    assert decode(CHINESE_LOREM_IPSUM.encode()) == CHINESE_LOREM_IPSUM
+    assert decode(codecs.BOM_UTF8 + CHINESE_LOREM_IPSUM.encode()) == CHINESE_LOREM_IPSUM
+
+    # Test non-default UTF variants specified via a BOM.
+    assert decode(codecs.BOM_UTF32_LE + codecs.utf_32_le_encode(CHINESE_LOREM_IPSUM)[0]) == CHINESE_LOREM_IPSUM
+    assert decode(codecs.BOM_UTF16_BE + codecs.utf_16_be_encode(CHINESE_LOREM_IPSUM)[0]) == CHINESE_LOREM_IPSUM
+
+    # Test using the encoding comment.
+    with_cookie = "# encoding: gb18030\n" + CHINESE_LOREM_IPSUM
+    assert decode(with_cookie.encode("GB18030")) == with_cookie