Repository: pyinstaller/pyinstaller

Issue:
Error running Streamlit on Windows
## v4.3 is the last version which works with Streamlit on Windows

macOS seems to be working but something triggers errors in Windows on versions after v4.3.

There is a Github action, you can check the output for v4.5.1 vs v4.3:
https://github.com/aljosa/pyinstaller-streamlit-issues/actions/runs/1150131459

Please note that I use `timeout` to shutdown the Streamlit server and that's why both versions exit w/ error.

This is the error w/ versions after v4.3:
```

Exception in thread ScriptRunner.scriptThread:
Traceback (most recent call last):
  File "linecache.py", line 95, in updatecache
FileNotFoundError: [WinError 2] The system cannot find the file specified: 'D:\\a\\pyinstaller-streamlit-issues\\pyinstaller-streamlit-issues\\dist\\main\\streamlit\\elements\\empty.pyc'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "streamlit\script_runner.py", line 350, in _run_script
  File "D:\a\pyinstaller-streamlit-issues\pyinstaller-streamlit-issues\hello.py", line 4, in <module>
    ui.init()
  File "D:\a\pyinstaller-streamlit-issues\pyinstaller-streamlit-issues\ui.py", line 7, in init
    TEXT = st.empty()
  File "streamlit\elements\empty.py", line 66, in empty
  File "streamlit\delta_generator.py", line 372, in _enqueue
  File "streamlit\__init__.py", line 501, in _maybe_print_use_warning
  File "streamlit\env_util.py", line 42, in is_repl
  File "inspect.py", line 1526, in stack
  File "inspect.py", line 1503, in getouterframes
  File "inspect.py", line 1477, in getframeinfo
  File "inspect.py", line 794, in findsource
  File "linecache.py", line 47, in getlines
  File "linecache.py", line 103, in updatecache
  File "PyInstaller\loader\pyimod03_importers.py", line 343, in get_source
  File "encodings\cp1252.py", line 23, in decode
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8f in position 1438: character maps to <undefined>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "linecache.py", line 95, in updatecache
FileNotFoundError: [WinError 3] The system cannot find the path specified: 'streamlit\\elements\\empty.py'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "threading.py", line 932, in _bootstrap_inner
  File "threading.py", line 870, in run
  File "streamlit\script_runner.py", line 172, in _process_request_queue
  File "streamlit\script_runner.py", line 359, in _run_script
  File "streamlit\error_util.py", line 46, in handle_uncaught_app_exception
  File "traceback.py", line 167, in format_exc
  File "traceback.py", line 120, in format_exception
  File "traceback.py", line 508, in __init__
  File "traceback.py", line 366, in extract
  File "traceback.py", line 288, in line
  File "linecache.py", line 16, in getline
  File "linecache.py", line 47, in getlines
  File "linecache.py", line 103, in updatecache
  File "PyInstaller\loader\pyimod03_importers.py", line 343, in get_source
  File "encodings\cp1252.py", line 23, in decode
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8f in position 1438: character maps to <undefined>
```





Discussion:
Hmm, so at its core, this *is* a bug in our loader implementation.

The encoding error is triggered here:
https://github.com/pyinstaller/pyinstaller/blob/5a02f55c696f16b98f23a8b487f3daa8f644a8d2/PyInstaller/loader/pyimod03_importers.py#L342-L343
because the source file in question, `streamlit/elements/empty.py`, contains [unicode characters](https://github.com/streamlit/streamlit/blob/21f87fcd75dcd6f57339959d2b1a84c4dab910b5/lib/streamlit/elements/empty.py#L42-L44).
As we do not specify `encoding` with `open()`, it uses `locale.getpreferredencoding(False)`, which on Windows returns `'cp1252'`.

Explicitly specifying `encoding='utf8'` with `open()` should fix the issue.

-----

On the other hand, the issue can also be avoided by not using brute-force collection via `collect_all` in those custom hooks (they end up collecting the source .py files in the first place...).

Instead, the test application in question can be built using the following CLI switches (or their equivalents in .spec file and/or hooks):
```
pyinstaller --clean --noconfirm main.py \
    --copy-metadata streamlit \  # should go in streamlit hook
    --collect-data pandas.io.formats --hiddenimport jinja2 \  # unnecessary since 4.5 due to added pandas.io.formats.style hook
    --collect-data altair  # should go in altair hook
```





Resolving patch:
diff --git a/PyInstaller/compat.py b/PyInstaller/compat.py
--- a/PyInstaller/compat.py
+++ b/PyInstaller/compat.py
@@ -602,6 +602,7 @@ def importlib_load_source(name, pathname):
     'sre_compile',
     'sre_constants',
     'sre_parse',
+    'tokenize',  # used by loader/pymod03_importers.py
     'traceback',  # for startup errors
     'types',
     'weakref',
diff --git a/PyInstaller/loader/pyimod03_importers.py b/PyInstaller/loader/pyimod03_importers.py
--- a/PyInstaller/loader/pyimod03_importers.py
+++ b/PyInstaller/loader/pyimod03_importers.py
@@ -56,6 +56,23 @@ def load_module(self, fullname):
         return self._importer.load_module(fullname, self._entry_name)
 
 
+def _decode_source(source_bytes):
+    """
+    Decode bytes representing source code and return the string. Universal newline support is used in the decoding.
+    Based on CPython's implementation of the same functionality:
+    https://github.com/python/cpython/blob/3.9/Lib/importlib/_bootstrap_external.py#L679-L688
+    """
+    # Local imports to avoid bootstrap issues
+    # NOTE: both modules are listed in compat.PY3_BASE_MODULES and collected into base_library.zip.
+    import io
+    import tokenize
+
+    source_bytes_readline = io.BytesIO(source_bytes).readline
+    encoding = tokenize.detect_encoding(source_bytes_readline)
+    newline_decoder = io.IncrementalNewlineDecoder(decoder=None, translate=True)
+    return newline_decoder.decode(source_bytes.decode(encoding[0]))
+
+
 class FrozenImporter(object):
     """
     Load bytecode of Python modules from the executable created by PyInstaller.
@@ -290,11 +307,13 @@ def get_source(self, fullname):
         Return None, unless the corresponding source file was explicitly collected to the filesystem.
         """
         if fullname in self.toc:
-            # Try loading .py file from the filesystem
+            # Try loading the .py file from the filesystem (only for collected modules)
             filename = pyi_os_path.os_path_join(SYS_PREFIX, fullname.replace('.', pyi_os_path.os_sep) + '.py')
             try:
-                with open(filename, 'r') as fp:
-                    return fp.read()
+                # Read in binary mode, then decode
+                with open(filename, 'rb') as fp:
+                    source_bytes = fp.read()
+                return _decode_source(source_bytes)
             except FileNotFoundError:
                 pass
             return None


Test patch:
diff --git a/tests/functional/modules/module_with_utf8_emoji.py b/tests/functional/modules/module_with_utf8_emoji.py
new file mode 100644
--- /dev/null
+++ b/tests/functional/modules/module_with_utf8_emoji.py
@@ -0,0 +1,14 @@
+#-----------------------------------------------------------------------------
+# Copyright (c) 2021, PyInstaller Development Team.
+#
+# Distributed under the terms of the GNU General Public License (version 2
+# or later) with exception for distributing the bootloader.
+#
+# The full license is in the file COPYING.txt, distributed with this software.
+#
+# SPDX-License-Identifier: (GPL-2.0-or-later WITH Bootloader-exception)
+#-----------------------------------------------------------------------------
+
+x = '🚀'  # 😓
+
+# ⏳✔️ <- These seem to be undecodable by 'cp1252' codec (default on my Windows).
diff --git a/tests/functional/test_import.py b/tests/functional/test_import.py
--- a/tests/functional/test_import.py
+++ b/tests/functional/test_import.py
@@ -156,6 +156,22 @@ def test_module_with_coding_utf8(pyi_builder):
     pyi_builder.test_source("import module_with_coding_utf8")
 
 
+# Test that our FrozenImporter's get_source() method can load source files with utf-8 emoji characters. See issue #6143.
+def test_source_utf8_emoji(pyi_builder):
+    # Collect the module's source as data file
+    datas = os.pathsep.join((os.path.join(_MODULES_DIR, 'module_with_utf8_emoji.py'), os.curdir))
+    pyi_builder.test_source(
+        """
+        import inspect
+
+        import module_with_utf8_emoji
+
+        # Retrieve source code
+        source = inspect.getsource(module_with_utf8_emoji)
+        """, ['--add-data', datas]
+    )
+
+
 def test_hiddenimport(pyi_builder):
     # The script simply does nothing, not even print out a line. The check is done by comparing with
     # logs/test_hiddenimport.toc