Repository: pyinstaller/pyinstaller

Issue:
Pyinstaller assumes that there is a top_level.txt in wheels but top_level.txt is not part of the wheel specification.
<!--
Welcome to the PyInstaller issue tracker! Before creating an issue, please heed the following:

1. This tracker should only be used to report bugs and request features / enhancements to PyInstaller
    - For questions and general support, use the mailing list, see
	  <http://www.pyinstaller.org/support.html#mailing-list>
2. Use the search function before creating a new issue. Duplicates will be closed and directed to
   the original discussion.
3. When making a bug report, make sure you provide all required information. The easier it is for
   maintainers to reproduce, the faster it'll be fixed.
-->

<!-- +++ ONLY TEXT +++ DO NOT POST IMAGES +++ -->

## Description of the issue
One of our dependencies idna recently updated to a new version 3.4, for their 3.4 release they moved to using pyproject.toml. As part of this they moved from setuptools to flit as the build backend. setuptools put a top_level.text in the egg-info, and continued to include this file in the dist-info provided in wheels. However this file is not required as part of the wheel specification and therefore is not required to be present. Looking at toc_conversion.py it appears that on lines 111 and 150 there is an assumption that this file will be present. We get a stacktrace on packaging our executable:

I created a minimum reproducable case with a test.py that simply imports and uses idna 3.4 removing all other code from our project. The stack trace is below and I am including the test.py and setup.py. I did my minimum reproducible case on my wsl box running Python 3.9, but our build box is a Linux VM running 3.8 generating the same exception with our full product.

### Context information (for bug reports)

* Output of `pyinstaller --version`: ```5.1``` also tried 5.4.1 release and 5.4.1 develop
* Version of Python: 3.8
* Platform: Ubuntu 20.04 LTS
* How you installed Python: Python 3.8 is the preinstalled version of python on Ubuntu 20.04 LTS
* Did you also try this on another platform? Does it work there? Also tried on Windows, it appears to work there. Tried on wsl Debian 11 with Python 3.9 and it fails.


* try the latest development version, using the following command:

```shell
pip install https://github.com/pyinstaller/pyinstaller/archive/develop.zip
```

* follow *all* the instructions in our "If Things Go Wrong" Guide
  (https://github.com/pyinstaller/pyinstaller/wiki/If-Things-Go-Wrong) and

### Make sure [everything is packaged correctly](https://github.com/pyinstaller/pyinstaller/wiki/How-to-Report-Bugs#make-sure-everything-is-packaged-correctly)

  * [X] start with clean installation
  * [X] use the latest development version
  * [~] Run your frozen program **from a command window (shell)** — instead of double-clicking on it
  * [X] Package your program in **--onedir mode** (Tried both onefile and onedir, not difference)
  * [X] Package **without UPX**, say: use the option `--noupx` or set `upx=False` in your .spec-file
  * [~] Repackage you application in **verbose/debug mode**. For this, pass the option `--debug` to `pyi-makespec` or `pyinstaller` or use `EXE(..., debug=1, ...)` in your .spec file.


### A minimal example program which shows the error
[test_pyinstaller_error.zip](https://github.com/pyinstaller/pyinstaller/files/9571934/test_pyinstaller_error.zip)
```
See attached zip file with test.py and setup.py

“Minimal“ means: remove everything from your code which is not relevant for this bug,
esp. don't use external programs, remote requests, etc.
A very good example is https://gist.github.com/ronen/024cdae9ff2d50488438. This one helped
us reproducing and fixing a quite complex problem within approx 1 hour.
```

### Stacktrace / full error message


```
(build_env) rick@test:/tmp/test_pyinstaller_error$ python -m PyInstaller --clean --onefile --workpath ../build_path/ --specpath ../build_path/specs --distpath ../build_path/dist -n test.so ./test.py
22 INFO: PyInstaller: 5.4.1
22 INFO: Python: 3.9.2
23 INFO: Platform: Linux-5.10.16.3-microsoft-standard-WSL2-x86_64-with-glibc2.31
24 INFO: wrote ../build_path/specs/test.so.spec
56 INFO: UPX is not available.
56 INFO: Removing temporary files and cleaning cache in /home/rick/.cache/pyinstaller
58 INFO: Extending PYTHONPATH with paths
['/tmp/test_pyinstaller_error']
129 INFO: checking Analysis
129 INFO: Building Analysis because Analysis-00.toc is non existent
129 INFO: Initializing module dependency graph...
129 INFO: Caching module graph hooks...
133 INFO: Analyzing base_library.zip ...
483 INFO: Loading module hook 'hook-encodings.py' from '/tmp/build_env/lib/python3.9/site-packages/PyInstaller/hooks'...
1056 INFO: Loading module hook 'hook-pickle.py' from '/tmp/build_env/lib/python3.9/site-packages/PyInstaller/hooks'...
1484 INFO: Loading module hook 'hook-heapq.py' from '/tmp/build_env/lib/python3.9/site-packages/PyInstaller/hooks'...
1826 INFO: Caching module dependency graph...
1885 INFO: running Analysis Analysis-00.toc
1898 INFO: Analyzing /tmp/test_pyinstaller_error/test.py
2050 INFO: Processing module hooks...
Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/tmp/build_env/lib/python3.9/site-packages/PyInstaller/__main__.py", line 189, in <module>
    run()
  File "/tmp/build_env/lib/python3.9/site-packages/PyInstaller/__main__.py", line 179, in run
    run_build(pyi_config, spec_file, **vars(args))
  File "/tmp/build_env/lib/python3.9/site-packages/PyInstaller/__main__.py", line 60, in run_build
    PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
  File "/tmp/build_env/lib/python3.9/site-packages/PyInstaller/building/build_main.py", line 962, in main
    build(specfile, distpath, workpath, clean_build)
  File "/tmp/build_env/lib/python3.9/site-packages/PyInstaller/building/build_main.py", line 884, in build
    exec(code, spec_namespace)
  File "../build_path/specs/test.so.spec", line 7, in <module>
    a = Analysis(
  File "/tmp/build_env/lib/python3.9/site-packages/PyInstaller/building/build_main.py", line 409, in __init__
    self.__postinit__()
  File "/tmp/build_env/lib/python3.9/site-packages/PyInstaller/building/datastruct.py", line 173, in __postinit__
    self.assemble()
  File "/tmp/build_env/lib/python3.9/site-packages/PyInstaller/building/build_main.py", line 583, in assemble
    self.datas.extend(deps_proc.make_datas_toc())
  File "/tmp/build_env/lib/python3.9/site-packages/PyInstaller/building/toc_conversion.py", line 111, in make_datas_toc
    toplevel = dist.get_metadata('top_level.txt').strip()
  File "/tmp/build_env/lib/python3.9/site-packages/pkg_resources/__init__.py", line 1420, in get_metadata
    value = self._get(path)
  File "/tmp/build_env/lib/python3.9/site-packages/pkg_resources/__init__.py", line 1616, in _get
    with open(path, 'rb') as stream:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/build_env/lib/python3.9/site-packages/idna-3.4-py3.9.egg/EGG-INFO/top_level.txt'
```

Please also see <https://github.com/pyinstaller/pyinstaller/wiki/How-to-Report-Bugs>
for more about what would use to solve the issue.



Discussion:
https://github.com/pyinstaller/pyinstaller/blob/6c8a6a53c359622d5e9f22e934eb3ae71fd0efb5/PyInstaller/building/toc_conversion.py#L111

Eughh! What the hell is that doing there.
I'm just writing here so I will get a notification when this issue gets resolved, so I can update my build environment which "automatically" adds an empty text file to certain packages so pyinstaller will not break.

I can't reproduce this. Can you share your spec file please? And also, can you check which format idna is installed in (i.e. is it in legacy egg/zipped egg format or conventional modern install with a `.dist-info` directory or some kind of editable install)? Or failing that, can you just zip up and upload your copy of idna? The offending code appears to be specific to egg distributions but even if I install idna as an egg I can't get those lines to run.
Hi, maybe I can jump in with a reproduction, even if for me it is a different package that is causing the troubles.
We write an application that uses pg8000.
pg8000 also requires typing_extensions via importlib-metadata.
pyinstaller 5.4.1 on python 3.7 needs the typing_extensions 4.4.0 via importlib-metadata.
Installing pyinstaller from source installs it and the typing_extensions in an egg format.
So doing this:
```
~/AppData/Local/Programs/Python/Python37/python.exe -m venv ./venv
. ./venv/Scripts/activate
# for linux:
# python3.7 -m venv ./venv
# . ./venv/bin/activate
pip install wheel
git clone --depth=1 --single-branch --branch v5.4.1 https://github.com/pyinstaller/pyinstaller
cd pyinstaller
python setup.py install
cd ..
ls -lah venv/Lib/site-packages/typing_extensions*/*
# shows that typing_extensions is an egg package without top_level.txt
pip install pg8000
# typing_extensions requirement is already satisfied by the egg , not installing it again
cat > test.py <<EOF
import pg8000
print(pg8000)
EOF
python -m PyInstaller --clean --onefile --workpath ./build_path/ --specpath ./build_path/specs --distpath ./build_path/dist -n test.so ./test.py
#Boom
```
I can confirm that the `typing_extensions` is one of the modules which gives me the same problems. 
@skelsec Are you also installing `pyinstaller` via the obsolete `python setup.py install`?
always.  
Here is my build script for `aardwolf` package, as you notice there are two additional copy instructions for creating `top_level` otherwise `pyinstaller` will fail.  
```
set hiddenimports= --hidden-import cryptography --hidden-import cffi --hidden-import cryptography.hazmat.backends.openssl --hidden-import cryptography.hazmat.bindings._openssl --hidden-import unicrypto --hidden-import unicrypto.backends.pycryptodome.DES --hidden-import  unicrypto.backends.pycryptodome.TDES --hidden-import unicrypto.backends.pycryptodome.AES --hidden-import unicrypto.backends.pycryptodome.RC4 --hidden-import unicrypto.backends.pure.DES --hidden-import  unicrypto.backends.pure.TDES --hidden-import unicrypto.backends.pure.AES --hidden-import unicrypto.backends.pure.RC4 --hidden-import unicrypto.backends.cryptography.DES --hidden-import  unicrypto.backends.cryptography.TDES --hidden-import unicrypto.backends.cryptography.AES --hidden-import unicrypto.backends.cryptography.RC4 --hidden-import unicrypto.backends.pycryptodomex.DES --hidden-import  unicrypto.backends.pycryptodomex.TDES --hidden-import unicrypto.backends.pycryptodomex.AES --hidden-import unicrypto.backends.pycryptodomex.RC4
set root=%WORKSPACE%
set projectname=aardwolf
set python=C:\Python310\python.exe
set git=C:\BUILDER\git\bin\git.exe
set sevenz=C:\7zip\7z.exe
set resdir=C:\BUILDER\RESULTS\%projectname%\%BUILD_NUMBER%
set pyenv=%root%\env
set repo=%root%\repo
mkdir %resdir%\files
mkdir %repo%
%git% clone https://github.com/skelsec/aardwolf %repo%\aardwolf
%git% clone https://github.com/skelsec/aardwolfgui %repo%\aardwolfgui
%git% clone https://github.com/skelsec/minikerberos %repo%\minikerberos
%git% clone https://github.com/skelsec/asysocks %repo%\asysocks
%python% -m venv %pyenv%
%pyenv%\Scripts\activate.bat &^
pip install pyinstaller &^
pip install unicrypto &^
pip install pyqt5 &^
pip install setuptools_rust &^
pip install Pillow>=9.0.0 &^
cd %repo%\asysocks &^
setup.py install &^
cd %repo%\minikerberos &^
setup.py install &^
cd %repo%\aardwolf &^
pip install . &^
cd %repo%\aardwolfgui &^
pip install . &^
type NUL >  %pyenv%\lib\site-packages\pyqt5-5.12.3-py3.10-win-amd64.egg\EGG-INFO\top_level.txt &^
type NUL >  %pyenv%\lib\site-packages\pyparsing-3.0.9-py3.10.egg\EGG-INFO\top_level.txt &^
cd %repo%\aardwolf\aardwolf\examples\scanners &^
pyinstaller -F __main__.py %hiddenimports% &^
cd %repo%\aardwolf\aardwolf\examples\scanners\dist & copy __main__.exe %resdir%\files\rdpscan.exe &^
cd %repo%\aardwolfgui\aardwolfgui &^
pyinstaller -F aardpclient.py %hiddenimports% &^
```
Actually, for me the original reproduction by @rickkoliser is throwing the error, and much simpler than mine (that does not reproduce it on the develop branch of PyInstaller somehow...). Sorry for the confusion, if there was any.
When I use the provided setup.py and do
```
python3.8 setup.py install
```
it downloads the egg version of idna:
```
[...]
Installed d:\downloads\test_pyinstaller_error2\venv\lib\site-packages\test-0.0.0
-py3.8.egg
Processing dependencies for test==0.0.0
Searching for idna==3.4
Reading https://pypi.org/simple/idna/
Downloading https://files.pythonhosted.org/packages/fc/34/3030de6f1370931b9dbb4d
ad48f6ab1015ab1d32447850b9fc94e60097be/idna-3.4-py3-none-any.whl#sha256=90b77e79
eaa3eba6de819a0c442c0b4ceefc341a7a2ab77d7562bf49f425c5c2
Best match: idna 3.4
Processing idna-3.4-py3-none-any.whl
Installing idna-3.4-py3-none-any.whl to d:\downloads\test_pyinstaller_error2\ven
v\lib\site-packages
Adding idna 3.4 to easy-install.pth file

Installed d:\downloads\test_pyinstaller_error2\venv\lib\site-packages\idna-3.4-p
y3.8.egg
```

which then breaks on
```
$ python3.8 -m PyInstaller --clean --onefile --workpath ./build_path/ --specpath ./build_path/specs --distpath ./build_path/dist -n test.so ./test.py
```
@rokm If `python setup.py install` is obsolete, the docs should be updated here: https://pyinstaller.org/en/v5.6.2/installation.html#installing-from-the-archive
> @rokm If `python setup.py install` is obsolete, the docs should be updated here: https://pyinstaller.org/en/v5.6.2/installation.html#installing-from-the-archive

Ah, thanks - they should indeed. 

And we should probably also override the `bdist_egg` command in `setup.py` to exit with an error message, like [this](https://github.com/jupyter/nbconvert/blob/6107ca1205931b7f86f7dd4a94d8c8356a546a7f/setup.py#L150-L159).
... which won't really help us when some other package is installed via `python setup.py install` (like in skelsec's example). 

So for such cases, we should handle the exception thrown by `dist.get_metadata('top_level.txt').strip()` call. Or perhaps throw out that part of code altogether; we don't attempt to automatically collect data from non-eggs, and I fail to see why eggs should be treated any differently... (EDIT: perhaps because `collect_data_files` et al. fail to pick up contents of egg-ed package?).
Scrapping that part of the codebase would be ideal. I still can't really figure out what it's trying to achieve.

> ... which won't really help us when some other package is installed via python setup.py install (like in skelsec's example).

Now I'm starting to toy with the idea of having PyInstaller error out if it detects any eggs. It would allow us to scrap all our zipped egg compatibility code that's been rotting for years without any test coverage at the expense of users occasionally having to run `pip uninstall $package; pip install wheel; pip install $package`. Pip is finally changing that default of destruction _falling back to legacy setup.py install_ if wheel isn't installed so hopefully accidental egg installations will finally start to disappear soon.
> Scrapping that part of the codebase would be ideal. I still can't really figure out what it's trying to achieve.

It collects "data files" (anything that does not have suffix from `importlib.machinery.all_suffixes()`) from the egg-installed package. For non-zipped eggs, I think this is the same or similar to what a hook with `collect_data_files` would achieve; so if we have a hook for that package, the data should already be collected; if we don't have a hook, the data gets collected only from an egg, which is not ideal.

The best part is that this `toplevel` is not really used anywhere, because
https://github.com/pyinstaller/pyinstaller/blob/d46744f487035aee97f27966847f9a88f209b840/PyInstaller/building/toc_conversion.py#L113-L114
is a no-op (the result of join is not assigned anywhere).

Same thing in `make_zipped_toc` data couple of lines below:
https://github.com/pyinstaller/pyinstaller/blob/d46744f487035aee97f27966847f9a88f209b840/PyInstaller/building/toc_conversion.py#L158-L159

----

So I think the quick and minimal-risk way to solve the problem is to simply remove those couple of lines. And update the docs to stop encouraging bad practices.

---

> Now I'm starting to toy with the idea of having PyInstaller error out if it detects any eggs. It would allow us to scrap all our zipped egg compatibility code that's been rotting for years without any test coverage at the expense of users occasionally having to run `pip uninstall $package; pip install wheel; pip install $package`. Pip is finally changing that default of destruction _falling back to legacy setup.py install_ if wheel isn't installed so hopefully accidental egg installations will finally start to disappear soon.

Yeah, that I was thinking the same. The non-zipped eggs should not be a problem (the collection from them should be handled by regular codepaths, I think?), but we could detect if any module is collected from a zipped egg, and raise an error that lists all problematic eggs.

Then we can jettison all of that extra-data-collection code from `toc_conversion` as well.


Resolving patch:
diff --git a/PyInstaller/building/toc_conversion.py b/PyInstaller/building/toc_conversion.py
--- a/PyInstaller/building/toc_conversion.py
+++ b/PyInstaller/building/toc_conversion.py
@@ -108,10 +108,6 @@ def make_datas_toc(self):
                 and not dist._pyinstaller_info['zip-safe']
             ):
                 # this is a un-zipped, not-zip-safe egg
-                toplevel = dist.get_metadata('top_level.txt').strip()
-                basedir = dist.location
-                if toplevel:
-                    os.path.join(basedir, toplevel)
                 tree = Tree(dist.location, excludes=PY_IGNORE_EXTENSIONS)
                 toc.extend(tree)
         return toc
@@ -146,17 +142,12 @@ def make_zipped_data_toc(self):
         logger.debug('Looking for egg data files...')
         for dist in self._distributions:
             if dist._pyinstaller_info['egg']:
-                # TODO: check in docs if top_level.txt always exists
-                toplevel = dist.get_metadata('top_level.txt').strip()
                 if dist._pyinstaller_info['zipped']:
                     # this is a zipped egg
                     tree = self.__collect_data_files_from_zip(dist.location)
                     toc.extend(tree)
                 elif dist._pyinstaller_info['zip-safe']:
                     # this is an un-zipped, zip-safe egg
-                    basedir = dist.location
-                    if toplevel:
-                        os.path.join(basedir, toplevel)
                     tree = Tree(dist.location, excludes=PY_IGNORE_EXTENSIONS)
                     toc.extend(tree)
                 else:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -250,13 +250,36 @@ def run(self) -> None:
 
 #--
 
+# --- Prevent `python setup.py install` from building and installing eggs ---
+
+if "bdist_egg" not in sys.argv:
+    from setuptools.command.bdist_egg import bdist_egg
+
+    class bdist_egg_disabled(bdist_egg):
+        """
+        Disabled version of bdist_egg, which prevents `setup.py install` from performing setuptools' default
+        easy_install, which is deprecated and should be avoided.
+        """
+        def run(self):
+            raise SystemExit(
+                "Error: Aborting implicit building of eggs. To install from source, use `pip install .` instead of "
+                "`python setup.py install`."
+            )
+
+    bdist_egg_override = {'bdist_egg': bdist_egg_disabled}
+else:
+    bdist_egg_override = {}
+
+#--
+
 setup(
-    setup_requires=["setuptools >= 39.2.0"],
+    setup_requires=["setuptools >= 42.0.0"],
     cmdclass={
         'build_bootloader': build_bootloader,
         'build': MyBuild,
         **wheel_commands,
         'bdist_wheels': bdist_wheels,
+        **bdist_egg_override,
     },
     packages=find_packages(include=["PyInstaller", "PyInstaller.*"]),
     package_data={


Test patch: