Repository: pyinstaller/pyinstaller

Issue:
fails to wrap jaraco.text with Python 3.5
<!--
Welcome to the PyInstaller issue tracker! Before creating an issue, please heed the following:

1. This tracker should only be used to report bugs and request features / enhancements to PyInstaller
    - For questions and general support, use the mailing list, see
	  <http://www.pyinstaller.org/support.html#mailing-list>
2. Use the search function before creating a new issue. Duplicates will be closed and directed to
   the original discussion.
3. When making a bug report, make sure you provide all required information. The easier it is for
   maintainers to reproduce, the faster it'll be fixed.
-->

## Description of the issue

PyInstaller fails to wrap jaraco.text. This was supposedly fixed by #4632 but it is still broken in the latest PyInstaller 4.1, at least when running Python 3.5. This downloads pyinstaller-hooks-contrib-2020.10, which contains https://github.com/pyinstaller/pyinstaller-hooks-contrib/blob/master/src/_pyinstaller_hooks_contrib/hooks/stdhooks/hook-jaraco.text.py, which supposedly fixed the issue in #4632, but still doesn't work.

**Note that this works as expected on Debian 10 using Python 3.7, and works on Ubuntu 18.04 using Python 3.8, but fails on Debian 9 using Python 3.5.**

## Context information (for bug reports)

* Output of `pyinstaller --version`: ```4.1``` (also reproduced with ```4.2.dev0+cc49966ac9```)
* Version of Python: 3.5
* Platform: Debian 9 (from a pristine debian:9 docker image)

### A minimal example program which shows the error

```
# Run in a pristine debian 9 docker image. Note that THE EXAMPLE WORKS if you use debain:10 instead, which uses Python 3.7!!
docker pull debian:9
docker run -it --rm debian:9 bash
apt-get update
apt-get install python3 python3-pip python3-setuptools
python3 -m pip install pyinstaller jaraco.text

cat > test.py <<EOF
import jaraco.text
print("hello")
EOF

pyinstaller test.py
./dist/test/test
# fails with FileNotFoundError: 'Lorem ipsum.txt' resource not found in 'jaraco.text'
```



### Stacktrace / full error message

```
Traceback (most recent call last):
  File "test.py", line 1, in <module>
    import jaraco.text
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
  File "/usr/local/lib/python3.5/dist-packages/PyInstaller/loader/pyimod03_importers.py", line 493, in exec_module
    exec(bytecode, module.__dict__)
  File "jaraco/text/__init__.py", line 232, in <module>
  File "importlib_resources/_py3.py", line 85, in read_text
  File "importlib_resources/_py3.py", line 67, in open_text
  File "importlib_resources/_py3.py", line 58, in open_binary
FileNotFoundError: 'Lorem ipsum.txt' resource not found in 'jaraco.text'
```



Discussion:
What is the output of:

```python
from PyInstaller.utils.hooks import collect_data_files

print(collect_data_files('jaraco.text'))
```

Can you also confirm that using `python -m PyInstaller test.py` instead of `pyinstaller test.py` doesn't make a difference?
Ugh, another smoke-and-mirrors issue. This one has nothing to do with missing data file - the hook fires correctly, and the `Lorem ipsum.txt` is properly collected. Rather, `importlib_resources` in `3.2.0` made some changes in package path resolution, which are incompatible with module spec behavior in pyinstaller-frozen applications (which is probably incorrect, and will need to be fixed).

Only python 3.5 and 3.6 are affected, because `jaraco.text` package pulls in `importlib_resources` only on python older than 3.7. [And even if `importlib_resources` were installed, `jaraco.text` prefers python's `importlib`.](https://github.com/jaraco/jaraco.text/blob/596f8476551bf1adf98020d6a13eb7df9a9dabe1/jaraco/text/__init__.py#L6-L9)

On the affected python versions, downgrading `importlib_resources` to 3.1.1 or earlier should fix the issue.

______________

In 3.2.0, the `read_binary` tries to [resolve the package path like this](https://github.com/python/importlib_resources/blob/00aec71139624c2d8970396c916892975f6f817b/importlib_resources/_py3.py#L27-L58):

<details><summary>Code snippet</summary><p>

```python
def open_binary(package: Package, resource: Resource) -> BinaryIO:
    """Return a file-like object opened for binary reading of the resource."""
    resource = _common.normalize_path(resource)
    package = _common.get_package(package)
    reader = _common.get_resource_reader(package)
    if reader is not None:
        return reader.open_resource(resource)
    # Using pathlib doesn't work well here due to the lack of 'strict'
    # argument for pathlib.Path.resolve() prior to Python 3.6.
    if package.__spec__.submodule_search_locations is not None:
        paths = package.__spec__.submodule_search_locations
    elif package.__spec__.origin is not None:
        paths = [os.path.dirname(os.path.abspath(package.__spec__.origin))]


    for package_path in paths:
        full_path = os.path.join(package_path, resource)
        try:
            return open(full_path, mode='rb')
        except OSError:
            # Just assume the loader is a resource loader; all the relevant
            # importlib.machinery loaders are and an AttributeError for
            # get_data() will make it clear what is needed from the loader.
            loader = cast(ResourceLoader, package.__spec__.loader)
            data = None
            if hasattr(package.__spec__.loader, 'get_data'):
                with suppress(OSError):
                    data = loader.get_data(full_path)
            if data is not None:
                return BytesIO(data)


    raise FileNotFoundError('{!r} resource not found in {!r}'.format(
        resource, package.__spec__.name))
```

</p></details>

In non-frozen application, `package.__spec__.submodule_search_locations` is a list containing the path, whereas in the frozen one, it is an empty list. Because it is not `None`, the elif statement using the `package.__spec__.origin` (which is correctly set) is not executed, and we end up with empty `paths`, and finally with `FileNotFoundError`.


[In 3.1.1](https://github.com/python/importlib_resources/blob/v3.1.1/importlib_resources/_py3.py#L27-L56), `package.__spec__.origin` is used, so it works:

<details><summary>Code snippet</summary><p>

```python
def open_binary(package: Package, resource: Resource) -> BinaryIO:
    """Return a file-like object opened for binary reading of the resource."""
    resource = _common.normalize_path(resource)
    package = _common.get_package(package)
    reader = _common.get_resource_reader(package)
    if reader is not None:
        return reader.open_resource(resource)
    # Using pathlib doesn't work well here due to the lack of 'strict'
    # argument for pathlib.Path.resolve() prior to Python 3.6.
    absolute_package_path = os.path.abspath(
        package.__spec__.origin or 'non-existent file')
    package_path = os.path.dirname(absolute_package_path)
    full_path = os.path.join(package_path, resource)
    try:
        return open(full_path, mode='rb')
    except OSError:
        # Just assume the loader is a resource loader; all the relevant
        # importlib.machinery loaders are and an AttributeError for
        # get_data() will make it clear what is needed from the loader.
        loader = cast(ResourceLoader, package.__spec__.loader)
        data = None
        if hasattr(package.__spec__.loader, 'get_data'):
            with suppress(OSError):
                data = loader.get_data(full_path)
        if data is None:
            package_name = package.__spec__.name
            message = '{!r} resource not found in {!r}'.format(
                resource, package_name)
            raise FileNotFoundError(message)
        return BytesIO(data)
```

</p></details>

So to properly address this issue, we'll need to check and fix the behavior of `.__spec__.submodule_search_locations` for frozen packages.


Resolving patch:
diff --git a/PyInstaller/loader/pyimod03_importers.py b/PyInstaller/loader/pyimod03_importers.py
--- a/PyInstaller/loader/pyimod03_importers.py
+++ b/PyInstaller/loader/pyimod03_importers.py
@@ -288,6 +288,14 @@ def find_spec(self, fullname, path=None, target=None):
         # (e.g. zipimport), that information may be stored in
         # spec.loader_state.
         spec.has_location = True
+
+        # Set submodule_search_locations for packages. Seems to be
+        # required for importlib_resources from 3.2.0 - see issue #5395.
+        if is_pkg:
+            spec.submodule_search_locations = [
+                pyi_os_path.os_path_dirname(self.get_filename(entry_name))
+            ]
+
         return spec
 
     def create_module(self, spec):


Test patch: