Repository: pyinstaller/pyinstaller
Pull request: Add assertion message to assert statement in ModuleHook
Closes #4626 
Related to #4538 
Supercedes #4684 

Problem statement:
PyInstaller 3.6 AssertionError
Hi, since the latest 3.6 release all my Linux builds are failing with a new `AssertionError`.
This is the log:
```
23 INFO: PyInstaller: 3.6
24 INFO: Python: 2.7.16
25 INFO: Platform: Linux-4.9.184-linuxkit-x86_64-with-debian-squeeze-sid
26 INFO: wrote /simulator.spec
35 INFO: UPX is not available.
36 INFO: Removing temporary files and cleaning cache in /.cache/pyinstaller
37 INFO: Extending PYTHONPATH with paths
['/agent/project/src/bin', '/']
37 INFO: checking Analysis
37 INFO: Building Analysis because Analysis-00.toc is non existent
38 INFO: Initializing module dependency graph...
49 INFO: Caching module graph hooks...
Traceback (most recent call last):
  File "/usr/local/bin/pyinstaller", line 8, in <module>
    sys.exit(run())
  File "/usr/local/lib/python2.7/site-packages/PyInstaller/__main__.py", line 114, in run
    run_build(pyi_config, spec_file, **vars(args))
  File "/usr/local/lib/python2.7/site-packages/PyInstaller/__main__.py", line 65, in run_build
    PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/PyInstaller/building/build_main.py", line 734, in main
    build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))
  File "/usr/local/lib/python2.7/site-packages/PyInstaller/building/build_main.py", line 681, in build
    exec(code, spec_namespace)
  File "/simulator.spec", line 17, in <module>
    noarchive=False)
  File "/usr/local/lib/python2.7/site-packages/PyInstaller/building/build_main.py", line 244, in __init__
    self.__postinit__()
  File "/usr/local/lib/python2.7/site-packages/PyInstaller/building/datastruct.py", line 160, in __postinit__
    self.assemble()
  File "/usr/local/lib/python2.7/site-packages/PyInstaller/building/build_main.py", line 343, in assemble
    excludes=self.excludes, user_hook_dirs=self.hookspath)
  File "/usr/local/lib/python2.7/site-packages/PyInstaller/depend/analysis.py", line 734, in initialize_modgraph
    user_hook_dirs=user_hook_dirs,
  File "/usr/local/lib/python2.7/site-packages/PyInstaller/depend/analysis.py", line 116, in __init__
    self._reset(user_hook_dirs)
  File "/usr/local/lib/python2.7/site-packages/PyInstaller/depend/analysis.py", line 134, in _reset
    self._hooks = self._cache_hooks("")
  File "/usr/local/lib/python2.7/site-packages/PyInstaller/depend/analysis.py", line 215, in _cache_hooks
    return ModuleHookCache(self, hook_dirs)
  File "/usr/local/lib/python2.7/site-packages/PyInstaller/depend/imphook.py", line 100, in __init__
    self._cache_hook_dirs(hook_dirs)
  File "/usr/local/lib/python2.7/site-packages/PyInstaller/depend/imphook.py", line 133, in _cache_hook_dirs
    hook_module_name_prefix=self._hook_module_name_prefix,
  File "/usr/local/lib/python2.7/site-packages/PyInstaller/depend/imphook.py", line 289, in __init__
    assert self.hook_module_name not in HOOKS_MODULE_NAMES
AssertionError
```

PyInstaller: 3.6
OS: Linux & macOS
Python: 2.7

After reverting to version 3.5 it seems to work fine, what can I do to try to find the source of this issue?

Hints from issue discussion:
did simple test shows it died on my custom `gevent` hook, 
remove custom hook pleased PyInstaller.

maybe you can share what died on your setup?

`self.hook_module_name` value is `__PyInstaller_hooks_0_gevent`

mine `hook-gevent.py`
```
# -*- coding: utf-8 -*-

hiddenimports = [
        'gevent.__greenlet_primitives',
        'gevent.__hub_local',
        'gevent.__hub_primitives',
        'gevent.__ident',
        'gevent.__imap',
        'gevent.__semaphore',
        'gevent.__tracer',
        'gevent.__waiter',
        'gevent._event',
        'gevent._greenlet',
        'gevent._local',
        'gevent._queue',

        # info from monkey patch_all
        'gevent.os',
        'gevent.time',
        'gevent.thread',
        # 'gevent.sys',
        'gevent.socket',
        'gevent.select',
        'gevent.ssl',
        # 'gevent.httplib',
        'gevent.subprocess',
        'gevent.builtins',
        'gevent.signal',

        'gevent.libev',
        'gevent.libev.corecext',
        'gevent.libev.corecffi',
        'gevent.libev.watcher',

        'gevent.libuv',
        'gevent.libuv._corecffi',
        'gevent.libuv.loop',
        'gevent.libuv.watcher',
    ]
```

```
(venv37) D:\AndCycle\Documents\work\corelight\proj\sview-agent>pyinstaller --clean --noupx --version-file=tmp\build_dist_ver.txt --additional-hooks-dir=. --add-data bin;bin --onefile ykaccess.py
84 INFO: PyInstaller: 3.6
84 INFO: Python: 3.7.4
84 INFO: Platform: Windows-10-10.0.18362-SP0
84 INFO: wrote D:\AndCycle\Documents\work\corelight\proj\sview-agent\ykaccess.spec
100 INFO: UPX is not available.
100 INFO: Removing temporary files and cleaning cache in C:\Users\AndCycle\AppData\Roaming\pyinstaller
100 INFO: Extending PYTHONPATH with paths
['D:\\AndCycle\\Documents\\work\\corelight\\proj\\sview-agent',
 'D:\\AndCycle\\Documents\\work\\corelight\\proj\\sview-agent']
100 INFO: checking Analysis
100 INFO: Building Analysis because Analysis-00.toc is non existent
100 INFO: Initializing module dependency graph...
100 INFO: Caching module graph hooks...
Traceback (most recent call last):
  File "C:\Python37\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "D:\AndCycle\Documents\work\corelight\proj\sview-agent\venv37\Scripts\pyinstaller.exe\__main__.py", line 7, in <module>
  File "d:\andcycle\documents\work\corelight\proj\sview-agent\venv37\lib\site-packages\PyInstaller\__main__.py", line 114, in run
    run_build(pyi_config, spec_file, **vars(args))
  File "d:\andcycle\documents\work\corelight\proj\sview-agent\venv37\lib\site-packages\PyInstaller\__main__.py", line 65, in run_build
    PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
  File "d:\andcycle\documents\work\corelight\proj\sview-agent\venv37\lib\site-packages\PyInstaller\building\build_main.py", line 734, in main
    build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))
  File "d:\andcycle\documents\work\corelight\proj\sview-agent\venv37\lib\site-packages\PyInstaller\building\build_main.py", line 681, in build
    exec(code, spec_namespace)
  File "D:\AndCycle\Documents\work\corelight\proj\sview-agent\ykaccess.spec", line 17, in <module>
    noarchive=False)
  File "d:\andcycle\documents\work\corelight\proj\sview-agent\venv37\lib\site-packages\PyInstaller\building\build_main.py", line 244, in __init__
    self.__postinit__()
  File "d:\andcycle\documents\work\corelight\proj\sview-agent\venv37\lib\site-packages\PyInstaller\building\datastruct.py", line 160, in __postinit__
    self.assemble()
  File "d:\andcycle\documents\work\corelight\proj\sview-agent\venv37\lib\site-packages\PyInstaller\building\build_main.py", line 343, in assemble
    excludes=self.excludes, user_hook_dirs=self.hookspath)
  File "d:\andcycle\documents\work\corelight\proj\sview-agent\venv37\lib\site-packages\PyInstaller\depend\analysis.py", line 734, in initialize_modgraph
    user_hook_dirs=user_hook_dirs,
  File "d:\andcycle\documents\work\corelight\proj\sview-agent\venv37\lib\site-packages\PyInstaller\depend\analysis.py", line 116, in __init__
    self._reset(user_hook_dirs)
  File "d:\andcycle\documents\work\corelight\proj\sview-agent\venv37\lib\site-packages\PyInstaller\depend\analysis.py", line 134, in _reset
    self._hooks = self._cache_hooks("")
  File "d:\andcycle\documents\work\corelight\proj\sview-agent\venv37\lib\site-packages\PyInstaller\depend\analysis.py", line 215, in _cache_hooks
    return ModuleHookCache(self, hook_dirs)
  File "d:\andcycle\documents\work\corelight\proj\sview-agent\venv37\lib\site-packages\PyInstaller\depend\imphook.py", line 100, in __init__
    self._cache_hook_dirs(hook_dirs)
  File "d:\andcycle\documents\work\corelight\proj\sview-agent\venv37\lib\site-packages\PyInstaller\depend\imphook.py", line 133, in _cache_hook_dirs
    hook_module_name_prefix=self._hook_module_name_prefix,
  File "d:\andcycle\documents\work\corelight\proj\sview-agent\venv37\lib\site-packages\PyInstaller\depend\imphook.py", line 289, in __init__
    assert self.hook_module_name not in HOOKS_MODULE_NAMES
AssertionError

```

and this is not the only place that failed, still digging.

@AndCycle Since you already have a clue and an environment which triggers this isse: Please track sit down, and send me an email afterwards. Many thanks.
@htgoebel 
https://github.com/pyinstaller/pyinstaller/commit/04d63e1d0acd003e5dcac71f04b940856e7e4f79

the safety check triggered when there is a custom hook conflict with builtin one.
an empty hook file can trigger it.

@AndCycle Thanks for the investigation, it was indeed a hook that I had in my code and the same hook was represented in the new version (from my PR 😃).
Even when the hooks are exactly the same it is failing.  
I was also getting this error, but after I rolled back to 3.5, I started getting this error when I launched my `--windowed` app:

<img width="292" alt="Screen Shot 2020-01-19 at 3 34 07 PM" src="https://user-images.githubusercontent.com/39964481/72688862-244fb200-3ad1-11ea-9ceb-cce4283866f1.png">

I wasn't getting this error before the upgrade to 3.6. If I build the app without the `--windowed` option, the app crashes and closes the command window before I can copy the traceback error.

Using Windows 10, Python 3.7.6.

Any help you guys can give?
>I was also getting this error, but after I rolled back to 3.5, I started getting this error

Please open another issue for this, as it's off-topic of this issue.

>the app crashes and closes the command window before I can copy the traceback error.

Try running the app from an existing cmd terminal, then the traceback will remain written on that terminal.
I have the same issue.

    Windows 8.1 Pro 64 bit
    Python 3.6.8 (tags/v3.6.8:3c6b436a57, Dec 23 2018, 23:31:17) [MSC v.1916 32 bit (Intel)] on win32

```
Traceback (most recent call last):
  File "C:\Python36_x86\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Python36_x86\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Python36_x86\lib\site-packages\PyInstaller\__main__.py", line 121, in
 <module>
    run()
  File "C:\Python36_x86\lib\site-packages\PyInstaller\__main__.py", line 114, in
 run
    run_build(pyi_config, spec_file, **vars(args))
  File "C:\Python36_x86\lib\site-packages\PyInstaller\__main__.py", line 65, in
run_build
    PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
  File "C:\Python36_x86\lib\site-packages\PyInstaller\building\build_main.py", l
ine 734, in main
    build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'
))
  File "C:\Python36_x86\lib\site-packages\PyInstaller\building\build_main.py", l
ine 681, in build
    exec(code, spec_namespace)
  File "pyfa.spec", line 78, in <module>
    cipher=block_cipher)
  File "C:\Python36_x86\lib\site-packages\PyInstaller\building\build_main.py", l
ine 244, in __init__
    self.__postinit__()
  File "C:\Python36_x86\lib\site-packages\PyInstaller\building\datastruct.py", l
ine 160, in __postinit__
    self.assemble()
  File "C:\Python36_x86\lib\site-packages\PyInstaller\building\build_main.py", l
ine 343, in assemble
    excludes=self.excludes, user_hook_dirs=self.hookspath)
  File "C:\Python36_x86\lib\site-packages\PyInstaller\depend\analysis.py", line
734, in initialize_modgraph
    user_hook_dirs=user_hook_dirs,
  File "C:\Python36_x86\lib\site-packages\PyInstaller\depend\analysis.py", line
116, in __init__
    self._reset(user_hook_dirs)
  File "C:\Python36_x86\lib\site-packages\PyInstaller\depend\analysis.py", line
134, in _reset
    self._hooks = self._cache_hooks("")
  File "C:\Python36_x86\lib\site-packages\PyInstaller\depend\analysis.py", line
215, in _cache_hooks
    return ModuleHookCache(self, hook_dirs)
  File "C:\Python36_x86\lib\site-packages\PyInstaller\depend\imphook.py", line 1
00, in __init__
    self._cache_hook_dirs(hook_dirs)
  File "C:\Python36_x86\lib\site-packages\PyInstaller\depend\imphook.py", line 1
33, in _cache_hook_dirs
    hook_module_name_prefix=self._hook_module_name_prefix,
  File "C:\Python36_x86\lib\site-packages\PyInstaller\depend\imphook.py", line 2
90, in __init__
    assert self.hook_module_name not in HOOKS_MODULE_NAMES
AssertionError
```

Pyinstaller fails on `__PyInstaller_hooks_0_matplotlib_backends` (verified by printing before the assertion).

Edit:
As comment above mentioned a conflict between hooks, i checked configuration:
```
             hookspath=['dist_assets/pyinstaller_hooks'],
```
I had custom hook for matplotlib there indeed, so I just removed it to give it a try, and so far it works.
@robertobrien130 run the exe from the command prompt (built without the windowed command).
@DarkFenX So you're saying its fixed when you remove the `hookspath` line from the spec?
@Legorooj I did not remove hookspath specification. I just removed offending file (had matplotlib hooks file in the specified path).

Commit message: Added changelog entry

Patch to review:
diff --git a/PyInstaller/depend/imphook.py b/PyInstaller/depend/imphook.py
index 41604789b21..8991eaa1060 100644
--- a/PyInstaller/depend/imphook.py
+++ b/PyInstaller/depend/imphook.py
@@ -285,7 +285,9 @@ def __init__(self, module_graph, module_name, hook_filename,
 
         # Safety check, see above
         global HOOKS_MODULE_NAMES
-        assert self.hook_module_name not in HOOKS_MODULE_NAMES
+        assert self.hook_module_name not in HOOKS_MODULE_NAMES, \
+            'Hook Conflict. Please remove the following custom hook: ' \
+            '{hook}'.format(hook=self._hook_name)
         HOOKS_MODULE_NAMES.add(self.hook_module_name)
 
         # Attributes subsequently defined by the _load_hook_module() method.
diff --git a/news/4626.hooks.rst b/news/4626.hooks.rst
new file mode 100644
index 00000000000..bc135b644c8
--- /dev/null
+++ b/news/4626.hooks.rst
@@ -0,0 +1 @@
+Add an AssertionError message so that users who get this error due to Hook conflicts can resolve it.
\ No newline at end of file


Code review comments:
diff_hunk: @@ -285,7 +285,9 @@ def __init__(self, module_graph, module_name, hook_filename,
 
         # Safety check, see above
         global HOOKS_MODULE_NAMES
-        assert self.hook_module_name not in HOOKS_MODULE_NAMES
+        assert self.hook_module_name not in HOOKS_MODULE_NAMES, \
+            'Hook Conflict. Please remove the following custom hook: ' \
+            '{hook}'.format(hook=self._hook_name)
path: PyInstaller/depend/imphook.py
text: @user1:
A quick question -- it `self._hook_name` correct? I can't find this being set anywhere in the code. Have you tested that this produces the correct error message?

@author:
@user1 Umm? I meant to type `self.hook_module_name`. Sometimes autocomplete isn't helpful.