Repository: pyinstaller/pyinstaller

Issue:
with setuptools v70.0.0: `ModuleNotFoundError: No module named 'pkg_resources.extern'`
<!--
Welcome to the PyInstaller issue tracker! Before creating an issue, please heed the following:

1. This tracker should only be used to report bugs and request features / enhancements to PyInstaller
    - For questions and general support, use the discussions forum.
2. Use the search function before creating a new issue. Duplicates will be closed and directed to
   the original discussion.
3. When making a bug report, make sure you provide all required information. The easier it is for
   maintainers to reproduce, the faster it'll be fixed.
-->

<!-- +++ ONLY TEXT +++ DO NOT POST IMAGES +++ -->

## Description of the issue

I have added some TODO notes below but wanted to submit this sooner than later for any other users running into this issue today to be able to find it.

This morning I noticed an error in my tests that exercise PyInstaller generated Windows .exe's in CI.

https://github.com/Chia-Network/chia-blockchain/actions/runs/9175546125/job/25229722015?pr=16898
```
Traceback (most recent call last):
  File "Lib\site-packages\PyInstaller\hooks\rthooks\pyi_rth_pkgres.py", line 158, in <module>
  File "Lib\site-packages\PyInstaller\hooks\rthooks\pyi_rth_pkgres.py", line 36, in _pyi_rthook
  File "PyInstaller\loader\pyimod02_importers.py", line 419, in exec_module
  File "pkg_resources\__init__.py", line [7](https://github.com/Chia-Network/chia-blockchain/actions/runs/9175546125/job/25229722015?pr=16898#step:6:8)7, in <module>
ModuleNotFoundError: No module named 'pkg_resources.extern'
[2148] Failed to execute script 'pyi_rth_pkgres' due to unhandled exception!
```

First I correlated this with [the release of setuptools v70.0.0](https://pypi.org/project/setuptools/70.0.0/#history) a few hours earlier (and not a new PyInstaller release `:]`).  After looking here and finding no issues reported I checked over at setuptools and found https://github.com/pypa/setuptools/issues/4374.  In that discussion I noted that the issue appears with https://github.com/pypa/setuptools/commit/e9995828311c5e0c843622ca2be85e7f09f1ff0d and not its parent commit.  That commit does indeed change how some of the `pkg_resources.extern` imports are handled inside `pkg_resources`.  Another developer provided an example, though that example has not yet resulted in recreation of the issue.

### Context information (for bug reports)

* Output of `pyinstaller --version`: ```(paste here)``` (TODO: add this to my CI run)
* Version of Python: 3.10
* Platform: Windows (GitHub Actions runner)
* How you installed Python: in-house action https://github.com/chia-network/actions/setup-python that should, in this case, pass through to upstream https://github.com/actions/setup-python
* Did you also try this on another platform? Does it work there?

~Similar Linux (Rocky and Ubuntu) builds as well as macOS (Intel and ARM) builds and tests seem to continue to work fine.  I could afford to review these runs in more detail for other relevant changes, but have not quite yet.~

When forcing the setuptools to be installed, this does happen on all of the platforms we build executables for (Rocky, Ubuntu (Intel and ARM), macOS (Intel and ARM), Windows)

* try the latest development version, using the following command:

https://github.com/Chia-Network/chia-blockchain/actions/runs/9179289212/job/25241848658?pr=18051 shows the failure using `develop`, specifically 676584885f2dfa1f885ab6155a5eda9150892c03.

* follow *all* the instructions in our "If Things Go Wrong" Guide
  (https://github.com/pyinstaller/pyinstaller/wiki/If-Things-Go-Wrong) and

### Make sure [everything is packaged correctly](https://github.com/pyinstaller/pyinstaller/wiki/How-to-Report-Bugs#make-sure-everything-is-packaged-correctly)

  * [x] start with clean installation
  * [ ] use the latest development version
  * [x] Run your frozen program **from a command window (shell)** — instead of double-clicking on it
  * [ ] Package your program in **--onedir mode**
  * [ ] Package **without UPX**, say: use the option `--noupx` or set `upx=False` in your .spec-file
  * [ ] Repackage you application in **verbose/debug mode**. For this, pass the option `--debug` to `pyi-makespec` or `pyinstaller` or use `EXE(..., debug=1, ...)` in your .spec file.


### A minimal example program which shows the error

TODO: I will try to create this

```
(paste text here)
“Minimal“ means: remove everything from your code which is not relevant for this bug,
esp. don't use external programs, remote requests, etc.
A very good example is https://gist.github.com/ronen/024cdae9ff2d50488438. This one helped
us reproducing and fixing a quite complex problem within approx 1 hour.
```

### Stacktrace / full error message


```
(paste text here)
```

Please also see <https://github.com/pyinstaller/pyinstaller/wiki/How-to-Report-Bugs>
for more about what would use to solve the issue.



Discussion:
@LuisHenri, since you already had an example, do you have time to figure out why it didn't reproduce as expected and share it here?
Thanks for the heads up! I can reproduce it on my main linux system, and it's likely that the next CI run will also hit this.

Looks like we just need to add `pkg_resources.extern` to the hidden imports for `setuptools >= 70.0`.
I reviewed some other runs and it seems that yes, this does happen on all platforms.  I have corrected my initial report here.
@altendky Yes, I managed to reproduce it.

1. Create a virtual environment
2. Install pyinstaller[encryption]==4.* and pyarmor==7.*
3. `pip install tendo==0.2.*` (just for the matter of having some import on the main.py file) (for some reason, it works with tendo>=0.3.0)
4. Create a test.py 
    ```python
    import tendo.singleton
    
    def main():
        print("Hello World!")
    
    if __name__ == "__main__":
        me = tendo.singleton.SingleInstance()
        main()
    ```
5. Run `pyarmor pack .\test.py -x "--exclude .venv"`
6. Run the test.exe on `dist/test/`

Output:
```console
Traceback (most recent call last):
  File "Lib\site-packages\PyInstaller\hooks\rthooks\pyi_rth_pkgres.py", line 158, in <module>
  File "Lib\site-packages\PyInstaller\hooks\rthooks\pyi_rth_pkgres.py", line 36, in _pyi_rthook
  File "PyInstaller\loader\pyimod02_importers.py", line 419, in exec_module
  File "pkg_resources\__init__.py", line 77, in <module>
ModuleNotFoundError: No module named 'pkg_resources.extern'
[2288] Failed to execute script 'pyi_rth_pkgres' due to unhandled exception!
```
Yeah, next CI run would be failing, too: https://github.com/rokm/pyinstaller/actions/runs/9179034509/job/25240267353

> Install pyinstaller[encryption]==4.* and pyarmor==7.*

We'll fix this for next PyInstaller release, but since I see PyInstaller 4.x mentioned up there, you'll have to work around this by adding `--hiddenimport pkg_resources.extern` or the equivalent .spec directive.

Resolving patch:
diff --git a/PyInstaller/hooks/hook-pkg_resources.py b/PyInstaller/hooks/hook-pkg_resources.py
--- a/PyInstaller/hooks/hook-pkg_resources.py
+++ b/PyInstaller/hooks/hook-pkg_resources.py
@@ -55,3 +55,9 @@
         'pkg_resources._vendor.jaraco.context',
         'pkg_resources._vendor.jaraco.text',
     ]
+
+# As of setuptools 70.0.0, we need pkg_resources.extern added to hidden imports.
+if check_requirement("setuptools >= 70.0.0"):
+    hiddenimports += [
+        'pkg_resources.extern',
+    ]


Test patch: