Repository: pyinstaller/pyinstaller

Issue:
Windows: Cannot bundle with debug if pkg_resources is a dependency
This issue happens when I try to bundle my project, in the Analysis.assemble phase and only when I try to do it with debug enabled. PyInstaller tries to compile a module that is part of an executable (pyinstaller.exe in this case) which fails because it cannot read the module.

This is with Windows 10, Python 3.6.6 (official from python.org) and PyInstaller 3.5.dev0+51429f8fc (which should be the latest develop version as of today).

Here is the traceback:

```
Traceback (most recent call last):
  File "c:\python36-32\Lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\python36-32\Lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\RMYROY~1\VIRTUA~1\CDDA-G~3\Scripts\pyinstaller.exe\__main__.py", line 9, in <module>
  File "c:\users\rmyroy~1\virtua~1\cdda-g~3\lib\site-packages\PyInstaller\__main__.py", line 111, in run
    run_build(pyi_config, spec_file, **vars(args))
  File "c:\users\rmyroy~1\virtua~1\cdda-g~3\lib\site-packages\PyInstaller\__main__.py", line 63, in run_build
    PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
  File "c:\users\rmyroy~1\virtua~1\cdda-g~3\lib\site-packages\PyInstaller\building\build_main.py", line 846, in main
    build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))
  File "c:\users\rmyroy~1\virtua~1\cdda-g~3\lib\site-packages\PyInstaller\building\build_main.py", line 793, in build
    exec(code, spec_namespace)
  File "launcher.spec", line 17, in <module>
    noarchive=True)
  File "c:\users\rmyroy~1\virtua~1\cdda-g~3\lib\site-packages\PyInstaller\building\build_main.py", line 243, in __init__
    self.__postinit__()
  File "c:\users\rmyroy~1\virtua~1\cdda-g~3\lib\site-packages\PyInstaller\building\datastruct.py", line 158, in __postinit__
    self.assemble()
  File "c:\users\rmyroy~1\virtua~1\cdda-g~3\lib\site-packages\PyInstaller\building\build_main.py", line 599, in assemble
    for name, path, typecode in compile_py_files(new_toc, CONF['workpath']):
  File "c:\users\rmyroy~1\virtua~1\cdda-g~3\lib\site-packages\PyInstaller\utils\misc.py", line 150, in compile_py_files
    with open(obj_fnm, 'rb') as fh:
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\RMYROY~1\\VIRTUA~1\\CDDA-G~3\\Scripts\\pyinstaller.exe\\__main__.pyo'
```

For some reason, the following entry is added in Analysis.pure

```python
('__main__.pyc', 'C:\\Users\\RMYROY~1\\VIRTUA~1\\CDDA-G~3\\Scripts\\pyinstaller.exe\\__main__.py', 'PYMODULE')
```

**That entry is incorrect in that it shouldn't have been added in pure or it shouldn't be compiled in assemble which is the source of this issue.**

Here is my spec file:

```python
# -*- mode: python ; coding: utf-8 -*-

block_cipher = None


a = Analysis(['cddagl\\launcher.py'],
             pathex=['C:\\Program Files (x86)\\Windows Kits\\10\\Redist\\ucrt\\DLLs\\x86\\', 'C:\\Users\\Rémy Roy\\Projects\\CDDA-Game-Launcher'],
             binaries=[],
             datas=[('alembic', 'alembic'), ('data', 'data'), ('cddagl/resources', 'cddagl/resources'), ('cddagl/VERSION', 'cddagl'), ('C:\\Users\\Rémy Roy\\VirtualEnvs\\CDDA-Game-Launcher\\Scripts\\UnRAR.exe', '.'), ('cddagl/locale/en/LC_MESSAGES/cddagl.mo', 'cddagl/locale/en/LC_MESSAGES'), ('cddagl/locale/fr/LC_MESSAGES/cddagl.mo', 'cddagl/locale/fr/LC_MESSAGES'), ('cddagl/locale/it/LC_MESSAGES/cddagl.mo', 'cddagl/locale/it/LC_MESSAGES'), ('cddagl/locale/ja/LC_MESSAGES/cddagl.mo', 'cddagl/locale/ja/LC_MESSAGES'), ('cddagl/locale/ru/LC_MESSAGES/cddagl.mo', 'cddagl/locale/ru/LC_MESSAGES')],
             hiddenimports=['lxml.cssselect', 'babel.numbers'],
             hookspath=[],
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher,
             noarchive=True)
pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)
exe = EXE(pyz,
          a.scripts,
          [('v', None, 'OPTION')],
          exclude_binaries=True,
          name='launcher',
          debug=True,
          bootloader_ignore_signals=False,
          strip=False,
          upx=False,
          console=True , icon='cddagl\\resources\\launcher.ico')
coll = COLLECT(exe,
               a.binaries,
               a.zipfiles,
               a.datas,
               strip=False,
               upx=False,
               upx_exclude=[],
               name='launcher')
```

You can probably reproduce this issue easily by cloning [my project](https://github.com/remyroy/CDDA-Game-Launcher) and issuing the following command:

```
python setup.py freeze --debug=1
```

Here is the full pyinstaller log output: https://gist.github.com/remyroy/37f7f0a912d5d714a947cddfb78769d4

I'll investigate how that entry is added in Analysis to give more context to this issue.


Discussion:
It seems like the erroneous entry is found from the:

```
Package('pkg_resources', 'c:\\users\\rmyroy~1\\virtua~1\\cdda-g~3\\lib\\site-packages\\pkg_resources\\__init__.py', ['c:\\users\\rmyroy~1\\virtua~1\\cdda-g~3\\lib\\site-packages\\pkg_resources'])
```

in PyInstaller.lib.modulegraph.modulegraph.import_hook . So it seems `pyinstaller.exe\__main__.py` is a module found from pkg_resources with modulegraph.

Also, the only reason that the debug freeze crashes and the non-debug freeze does not is because when I select a debug freeze, noarchive is set True and it triggers python file compilation in assemble.
So my temporary workaround (which is a pretty bad hack) is the following:

```diff
diff --git a/PyInstaller/building/build_main.py b/PyInstaller/building/build_main.py
index ef48170b..13090178 100644
--- a/PyInstaller/building/build_main.py
+++ b/PyInstaller/building/build_main.py
@@ -585,6 +585,8 @@ class Analysis(Target):
             new_toc = TOC()
             for name, path, typecode in self.pure:
                 assert typecode == 'PYMODULE'
+                if 'pyinstaller.exe' in path:
+                    continue
                 # Transform a python module name into a file name.
                 name = name.replace('.', os.sep)
                 # Special case: modules have an implied filename to add.
```

You might be able to use this workaround if you are having the same issue. 
It seems related to the use of the pkg_resources package.

Here is a reduced and minimal example that triggers this issue on Windows 10.

*main.py*
```python
import pkg_resources

if __name__ == '__main__':
    print('Hi!')
```

*Command to bundle main.py*

```
pyinstaller -c -d all main.py
```
@remyroy Thanks for posting this issue and the patch workaround. I am having the same issue and was able to use your patch as well.

I would be interested in helping to fix this if someone with more experience with pyinstaller could point me in the right direction.
@remyroy Does this script show the same issue:
```
import pkg_resources
print('Hi!')
```
@htgoebel Yes
Does the import graph (to be found in the build directory) provide any more useful information? Esp. which module or package triggers inclusion of `__main__`.
There are only 2 files in my build directory that are created when this error is reached for the reduced and minimal example: `base_library.zip` which contains the python libraries and `main.exe.manifest` which contains the following windows manifest:

```xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<assembly manifestVersion="1.0" xmlns="urn:schemas-microsoft-com:asm.v1">
  <assemblyIdentity name="main" processorArchitecture="x86" type="win32" version="1.0.0.0"/>
  <dependency>
    <dependentAssembly>
      <assemblyIdentity language="*" name="Microsoft.Windows.Common-Controls" processorArchitecture="*" publicKeyToken="6595b64144ccf1df" type="win32" version="6.0.0.0"/>
      <compatibility xmlns="urn:schemas-microsoft-com:compatibility.v1"/>
    </dependentAssembly>
  </dependency>
  <compatibility xmlns="urn:schemas-microsoft-com:compatibility.v1">
    <application>
      <supportedOS Id="{e2011457-1546-43c5-a5fe-008deee3d3f0}"/>
      <supportedOS Id="{35138b9a-5d96-4fbd-8e2d-a2440225f93a}"/>
      <supportedOS Id="{4a2f28e3-53b9-4441-ba9c-d69d4a4a6e38}"/>
      <supportedOS Id="{1f676c76-80e1-4239-95bb-83d0f6d0da78}"/>
      <supportedOS Id="{8e0f7a12-bfb3-4fe8-b9a5-48fd50a15a9a}"/>
    </application>
  </compatibility>
</assembly>
```

I don't see an *import graph*.
The build directory, not the dist directory.
There should be a `xref-*`.html file in th build directory, Together with quite some `.toc` files, a .pyz file and others.
The xref-* file is not generated before the exception is launched from this issue.

[Here is the xref-* file](https://gist.github.com/remyroy/4862435f8803c9a08d27e6c0143c5e94) with [the workaround](https://github.com/pyinstaller/pyinstaller/issues/4263#issuecomment-499655698).
This says:
```
__main__ SourceModule
imports: PyInstaller.__main__ • re • sys
imported by: pkg_resources 
```
and indeed `pkg_resources/__init__.py` contains this line:
```
            from __main__ import __requires__
```

Now, this is what is happening here:
- When running PyInstaller, `__main__` is the stub script generated by the setup.py console entry-point. (The stub-script also imports `re` and `sys`).
- Now due to the above line, when your program uses `pkg_resources`, `__main__ get added as a dependency.
- I currently can not explain why the stub-scripts imports `PyInstaller.__main__`, but not `pkg_resources`, though. But this might depend on the exact version of setuptools used to isntall your PyInstaller. Different versions create different stub-scripts.

Thus the solution would be to add `__main__` as a "excluded import" for `pgk_resources`. Please try again after adding this line to `PyInstaller/hooks/hook-pkg_resources.py`:
```
excludedimports = ["__main__"]
```

If this works I'd appreciate a pull-request.

Resolving patch:
diff --git a/PyInstaller/hooks/hook-pkg_resources.py b/PyInstaller/hooks/hook-pkg_resources.py
--- a/PyInstaller/hooks/hook-pkg_resources.py
+++ b/PyInstaller/hooks/hook-pkg_resources.py
@@ -11,3 +11,5 @@
 # pkg_resources keeps vendored modules in its _vendor subpackage, and does
 # sys.meta_path based import magic to expose them as pkg_resources.extern.*
 hiddenimports = collect_submodules('pkg_resources._vendor')
+
+excludedimports = ['__main__']


Test patch: