Repository: pyinstaller/pyinstaller

Issue:
A way to completely override a default hook
I'm trying to override hook_matplotlib.py and hook_kivy.py (stop it from being executed) and use my own hook instead.

My use case for matplotlib is to not collect `mpl-data`.  I was trying to override `datas` in the custom hook, but it seems the values are being added to the global list, rather than being able to manipulate the list directly.

Having direct access to the globals would be nice (from a hook), but the best way would be to not execute the default hook at all, when needed. Otherwise we would have to write boilerplate just to undo what has been done in the default hook.

The comment below mentions overriding the globals completely, but i couldn't find a way in the docs or forums, sorry if this has been addressed already.
https://github.com/pyinstaller/pyinstaller/pull/1678#issuecomment-158761602



Discussion:
Thanks for the detailed feature request, dessant. :construction_worker: 

I'm currently refactoring our hook infrastructure to support [lazy hook loading](https://github.com/pyinstaller/pyinstaller/pull/1651). While my refactoring doesn't relate _directly_ to your feature request, I'm hip-deep in the PyInstaller codebase. This seems like an appropriate time to brainstorm possible solutions for hook overriding before I inevitably suffer a concussion and forget everything... _again._

In order of decreasing preference, we could:
1. Add support for a new hook attribute, function, or callback permitting custom hooks to inform PyInstaller of their desire to override the corresponding official hook. This effectively requires lazy hook loading, which should be good to go... _any minute now._ Given that, we could add support for a new optional:
   - `is_override` hook boolean, defaulting to `False` when undefined. If explicitly set to `True` by a custom hook, the corresponding official hook would be removed from the cache of hooks to be run.
   - `modify_hooks()` hook function, passed the list of all hooks to be run for this module – including both the corresponding official hook (if any) and the current custom hook. This hook function could then be defined by the current custom hook to:
     - Reorder the order of hooks in this list. Currently, official hooks are _always_ run before custom hooks.
     - Remove one or more hooks from this list – in this case, the first element of this list (i.e., the corresponding official hook).
     - Modify any attributes or callables defined by one or more hooks in this list – in this case, removing the `datas` attribute from the first element of this list.
2. Add a new CLI option `--exclude-hook={hook_name}`. PyInstaller would accept multiple such options and, for each, remove the corresponding official hook from the cache of hooks to be run. For the following reasons, I'm not terribly fond of this approach:
   - Any custom hook intended to override an official hook should be required to explicitly inform PyInstaller of that fact at runtime via some hook attribute, function, or callback embedded directly in that custom hook. Externalizing this relationship into a CLI option makes it all-too easy to accidentally forget to pass that option.
   - Adding yet another CLI option is feasible, but _not_ necessarily desirable.

## What Should We Do, leycec?

**I have a strong preference for the `modify_hooks()` approach.** It's the maximally general-purpose solution, ensuring that we don't have to continually add new hook attributes or CLI options whenever a user requests low-level support for manipulating yet another hook interaction (e.g., reordering hooks, removing hooks, modifying hooks).

A few examples in some hypothetical hook:
1. Prevent the corresponding official hook from being run:
   
   ```
   def modify_hooks(hooks):
       del hooks[0]   # It's really that simple, folks.
   ```
2. Remove the `datas` attribute from the corresponding official hook – but still run that hook:
   
   ```
   def modify_hooks(hooks):
       del hooks[0].code.datas   # I'm not kidding here, people.
   ```
3. Force the current custom hook to be run before the corresponding official hook – but still run that hook:
   
   ```
   def modify_hooks(hooks):
       official_hook = hooks[0]
       hooks = [hooks[1], official_hook]
   ```

What say you, glorious code compatriots?

I suggest that pyinstaller could just overriide default hooks when a hook with the same name would be present in the directory specified with option `--additional-hooks-dir`

What @matysek said also popped into my mind. Find any local hooks first (they are uncommon I believe); then do not load an official hook having the same name. Is there any presumption in current code that user-supplied hooks are loaded last?

@tallforasmurf I checked the source code and the implementation works this way:
- first process hooks bundled in pyinstaller
- second use user hooks if they have the same module name

In principal, I agree with the "just make local hooks _always_ override default hooks." [KISS](https://en.wikipedia.org/wiki/KISS_principle), amirite?

In practice, I've actually leveraged the fact that PyInstaller currently runs both default and local hooks for the same module in a shipped application. Since I didn't have time to submit my changes upstream, I simply defined a local hook "monkey patching" the default hook for a problematic module with additional functionality specific to that application. It is a thing of beauty. :mount_fuji: 

If we refactor PyInstaller to unconditionally override default hooks with local hooks, that sweet hackery breaks. I'd either need to **(A)** actually submit my changes upstream (..._which I really should do anyway_) or **(B)** copy-and-paste the contents of the default hook into the local hook. The issue with **(B)**, of course, is synchronization; upstream changes to the default hook would be ignored by my local hook. The issue with **(A)** is laziness in my case (..._it's a known problem_) and licensing concerns in the case of proprietary software.

Admittedly, this is probably an edge case. As [tallforasmurf](https://github.com/tallforasmurf) suggests, the number of applications actually using `--additional-hooks-dir` (excluding those authored by present company) might be negligibly small. Like, zero. But `--additional-hooks-dir` _does_ publicly exist and _is_ well-documented – so it would be nice to preserve backward compatibility here.

@leycec To be frank, the "hook patching" mechanisms is over-engineered IMHO. Much too much code for rare cases.

As The Zen of Python says:

> Simple is better than complex.
> Special cases aren't special enough to break the rules.

I'm not a so prolific Python programmer, but I use the "--additional-hooks-dir" and I don't understand how you may use pyInstaller without this option. 
To me, Numpy needed it absolutely (the optimization DLLs were not loaded by the standard hook) and my icons/splashscreen are also loaded through an hook.

The "user hook always overrides other hooks" is dead simple, but may not be as evolutive as the leycec solution. If the default hook is upgraded, the modification could be taken into account if the custom hook just modifies it, not if it completely overrides it.

I felt on this page because I would love to override the "hook-matplotlib.backends.py".
I'm not sure if I would like to completely override it.

> In principal, I agree with the "just make local hooks always override default hooks." KISS, amirite?

Yes.

> In practice, I've actually leveraged the fact that PyInstaller currently runs both default and local hooks for the same module in a shipped application. Since I didn't have time to submit my changes upstream, I simply defined a local hook "monkey patching" the default hook for a problematic module with additional functionality specific to that application. It is a thing of beauty. 🗻

> If we refactor PyInstaller to unconditionally override default hooks with local hooks, that sweet hackery breaks. I'd either need to (A) actually submit my changes upstream (...which I really should do anyway) or (B) copy-and-paste the contents of the default hook into the local hook.

Or (C) programmatically get the path to the default hook, and then import and run that hook as part of the local hook. Right?  Surely that's only a couple of lines of code today. I imagine it could be made into a one-liner if pyinstaller exposed some additional API function such as `myhookmodule = get_default_hook('mypackage.submodule')`.

I've run into this problem with pyinstaller and an application that imports `django` but is not itself a Django application: the default `hook-django.py` is inappropriate, but I can't quite figure out how to tell pyinstaller "This program needs you to package up `django`, but please don't run the default `hook-django.py`; I've got my own hook for that."
What are my options today, if I want to override twisted hook instead of monkey patching it myself.
Try this:
- mkdir my-hooks
- create an empty file my-hooks/hook_XXXX.py
- pass `--additional-hooks-dir=my-hooks` when running pyinstaller (or add it n the spec-file).

Please report back it this works.
> Try this:
> 
>     mkdir my-hooks
>     create an empty file my-hooks/hook_XXXX.py
>     pass --additional-hooks-dir=my-hooks when running pyinstaller (or add it n the spec-file).
> 
> Please report back it this works.

Thanks, I just tried and it did not help. I actually named the hook `py_rth_twisted.py` and added below code to it
```
from twisted.internet.iocpreactor import reactor

reactor.install()
```
but the environment actually had the default reactor, the one that PyInstaller installs. Though if I monkey patch the default twisted hook, it fixes my issue.
This issue is about hooks, not about run-time hooks. If your problem is about a run-time hook, please add open a new ticket. If you have problems with the existing twisted hook, please provide a pull-request.
I'm also interested in solving this (or maybe learning more about pyinstaller by explaining my issue). In my case, I do not need additional modules, but want to _exclude_ some. I have a script that imports QApplication from PyQt5.Qt, but the hook for PyQt5.Qt then includes everything under the sun in Qt as seen [here](https://github.com/pyinstaller/pyinstaller/blob/develop/PyInstaller/hooks/hook-PyQt5.Qt.py):
```python
hiddenimports = []
for f in os.listdir(os.path.dirname(get_module_file_attribute('PyQt5'))):
    root, ext = os.path.splitext(os.path.basename(f))
    if root.startswith('Qt') and root != 'Qt':
        print ("Adding this root" + str(root))
        hiddenimports.append('PyQt5.' + root)
```

Though I may be bumping into a Qt limitation that might bite me, I wanted to see if I could trim some of the unneeded libs (e.g. the huge QtWebEngineCore). Even if my specfile lists certain modules to exclude, the hook will still override my request. I see the xref html file produced indicates those modules were excluded, even though they are ultimately included. _Interesting aside: even if I modify the hook file to not include the QtWeb* files, they are still coming in somehow even though they are not listed in the xrefs html_

It makes sense to me that either a local, total replacement for the hook file behavior or my own explicit exclusions should be available to me. Per the discussion above, it allows me to take total responsibility for what's going on even if I may be reproducing boilerplate. If my explicit exclusions aren't working as intended, I can file/search a new issue for that.
You contributions would be welcome. I'd be in favor of having local hooks replace system hooks.
I found a way to replace system hooks (hook-gi.repository.Gtk.py) by my own version since the hook includes themes/ and icons/ folders who take Gigabytes...

So i have created a virtualenv, pip installed pyinstaller and all i need, after that i have replace the hook in my virtualenv by my own version and start pyinstaller with <ENV>/bin/pyinstaller <MYFILE.py>


We definitely need a way to have a local hook replace a default one, for broken hooks for instance (see Shapely's hook for instance: https://github.com/pyinstaller/pyinstaller/issues/2834). Would this be a lot of modifications to the codebase ?
How can I use a custom hook to override which DLLs are included by an imported library? Similar to  @pebbywebby's comment https://github.com/pyinstaller/pyinstaller/issues/1835#issuecomment-428015480, I want to prevent certain Qt5 DLLs from being included in the built executable, e.g. `opengl32sw.dll` which needlessly bloat the file size of the executable.
Well to answer my own [question](https://github.com/pyinstaller/pyinstaller/issues/1835#issuecomment-590483347), here's an example of using a "pre-find-module-path" hook to override a default hook (`hook-PyQt5` in my case): https://gist.github.com/logan-pugh/c37b359bc65a365480310c157b1d284e

It's not pretty, but at least it is better than editing the PyInstaller source.

Resolving patch:
diff --git a/PyInstaller/depend/analysis.py b/PyInstaller/depend/analysis.py
--- a/PyInstaller/depend/analysis.py
+++ b/PyInstaller/depend/analysis.py
@@ -124,10 +124,9 @@ def _reset(self, user_hook_dirs):
         """
         self._top_script_node = None
         self._additional_files_cache = AdditionalFilesCache()
-        # Prepend PyInstaller hook dir hook directories from entry points and
-        # user hook dirs. Thus PyInstaller hook will have lowest priority.q
+        # Command line, Entry Point, and then builtin hook dirs.
         self._user_hook_dirs = (
-            [os.path.join(PACKAGEPATH, 'hooks')] + list(user_hook_dirs)
+            list(user_hook_dirs) + [os.path.join(PACKAGEPATH, 'hooks')]
         )
         # Hook-specific lookup tables.
         # These need to reset when reusing cached PyiModuleGraph to avoid
@@ -179,9 +178,13 @@ def _merge_rthooks(self, rthooks, uhd, uhd_path):
                 'The value of %s key %s must be a list.' %
                 (uhd_path, module_name))
             if module_name in self._available_rthooks:
-                logger.warning("Several run-time hooks defined for module %r."
-                               "Please take care they do not conflict.",
-                               module_name)
+                logger.warning(
+                    'Runtime hooks for %s have already been defined. Skipping '
+                    'the runtime hooks for %s that are defined in %s.',
+                    module_name, module_name, os.path.join(uhd, 'rthooks')
+                )
+                # Skip this module
+                continue
             # Merge this with existing run-time hooks.
             for python_file_name in python_file_name_list:
                 # Ensure each item in the list is a string.
@@ -295,12 +298,13 @@ def run_script(self, pathname, caller=None):
         if self._top_script_node is None:
             # Remember the node for the first script.
             try:
-                self._top_script_node = super(PyiModuleGraph, self).run_script(pathname)
+                self._top_script_node = super(PyiModuleGraph, self).run_script(
+                    pathname)
             except SyntaxError:
                 print("\nSyntax error in", pathname, file=sys.stderr)
                 formatted_lines = traceback.format_exc().splitlines(True)
                 print(*formatted_lines[-4:], file=sys.stderr)
-                raise SystemExit(1)
+                sys.exit(1)
             # Create references from the top script to the base_modules in graph.
             for node in self._base_modules:
                 self.createReference(self._top_script_node, node)
@@ -308,10 +312,11 @@ def run_script(self, pathname, caller=None):
             return self._top_script_node
         else:
             if not caller:
-                # Defaults to as any additional script is called from the top-level
-                # script.
+                # Defaults to as any additional script is called from the
+                # top-level script.
                 caller = self._top_script_node
-            return super(PyiModuleGraph, self).run_script(pathname, caller=caller)
+            return super(PyiModuleGraph, self).run_script(
+                pathname, caller=caller)
 
 
     def process_post_graph_hooks(self):
diff --git a/PyInstaller/depend/imphook.py b/PyInstaller/depend/imphook.py
--- a/PyInstaller/depend/imphook.py
+++ b/PyInstaller/depend/imphook.py
@@ -290,10 +290,13 @@ def __init__(self, module_graph, module_name, hook_filename,
 
         # Safety check, see above
         global HOOKS_MODULE_NAMES
-        assert self.hook_module_name not in HOOKS_MODULE_NAMES, \
-            'Hook Conflict. Please remove the following custom hook: ' \
-            '{hook}'.format(hook=self.hook_module_name)
-        HOOKS_MODULE_NAMES.add(self.hook_module_name)
+        if self.hook_module_name in HOOKS_MODULE_NAMES:
+            # When self._shallow is true, this class never loads the hook and
+            # sets the attributes to empty values
+            self._shallow = True
+        else:
+            self._shallow = False
+            HOOKS_MODULE_NAMES.add(self.hook_module_name)
 
         # Attributes subsequently defined by the _load_hook_module() method.
         self._hook_module = None
@@ -378,8 +381,21 @@ def _load_hook_module(self):
         Class docstring for supported attributes.
         """
 
-        # If this hook script module has already been loaded, noop.
-        if self._hook_module is not None:
+        # If this hook script module has already been loaded,
+        # or we are _shallow, noop.
+        if self._hook_module is not None or self._shallow:
+            if self._shallow:
+                self._hook_module = True  # Not None
+                # Inform the user
+                logger.debug(
+                    'Skipping module hook %r from %r because a hook for %s has'
+                    ' already been loaded.',
+                    *os.path.split(self.hook_filename)[::-1], self.module_name
+                )
+                # Set the default attributes to empty instances of the type.
+                for attr_name, \
+                        (attr_type, _) in _MAGIC_MODULE_HOOK_ATTRS.items():
+                    super(ModuleHook, self).__setattr__(attr_name, attr_type())
             return
 
         # Load and execute the hook script. Even if mechanisms from the import
@@ -578,8 +594,11 @@ def __init__(self):
         self._datas = {}
 
     def add(self, modname, binaries, datas):
-        self._binaries[modname] = binaries or []
-        self._datas[modname] = datas or []
+
+        self._binaries.setdefault(modname, [])
+        self._binaries[modname].extend(binaries or [])
+        self._datas.setdefault(modname, [])
+        self._datas[modname].extend(datas or [])
 
     def __contains__(self, name):
         return name in self._binaries or name in self._datas


Test patch:
diff --git a/tests/unit/conftest.py b/tests/unit/conftest.py
--- a/tests/unit/conftest.py
+++ b/tests/unit/conftest.py
@@ -18,3 +18,5 @@
 _ROOT_DIR = os.path.normpath(os.path.join(os.path.dirname(os.path.abspath(__file__)), '..', '..'))
 sys.path.append(_ROOT_DIR)
 
+# Bring all fixtures into this file.
+from PyInstaller.utils.conftest import *  # noqa
diff --git a/tests/unit/hook_order_hooks/MANIFEST.in b/tests/unit/hook_order_hooks/MANIFEST.in
new file mode 100644
diff --git a/tests/unit/hook_order_hooks/pyi_example_package/__init__.py b/tests/unit/hook_order_hooks/pyi_example_package/__init__.py
new file mode 100644
--- /dev/null
+++ b/tests/unit/hook_order_hooks/pyi_example_package/__init__.py
@@ -0,0 +1,5 @@
+import os
+
+
+def get_hook_dirs():
+    return [os.path.abspath(os.path.dirname(__file__))]
diff --git a/tests/unit/hook_order_hooks/pyi_example_package/hook-pyi_example_package.py b/tests/unit/hook_order_hooks/pyi_example_package/hook-pyi_example_package.py
new file mode 100644
--- /dev/null
+++ b/tests/unit/hook_order_hooks/pyi_example_package/hook-pyi_example_package.py
@@ -0,0 +1 @@
+raise RuntimeError('I shouldn\'t be running')
diff --git a/tests/unit/hook_order_hooks/pyi_example_package/rthooks.dat b/tests/unit/hook_order_hooks/pyi_example_package/rthooks.dat
new file mode 100644
--- /dev/null
+++ b/tests/unit/hook_order_hooks/pyi_example_package/rthooks.dat
@@ -0,0 +1,3 @@
+{
+    'pyi_example_package': ['pyi_rth_naughty.py']
+}
\ No newline at end of file
diff --git a/tests/unit/hook_order_hooks/pyi_example_package/rthooks/pyi_rth_naughty.py b/tests/unit/hook_order_hooks/pyi_example_package/rthooks/pyi_rth_naughty.py
new file mode 100644
--- /dev/null
+++ b/tests/unit/hook_order_hooks/pyi_example_package/rthooks/pyi_rth_naughty.py
@@ -0,0 +1 @@
+raise RuntimeError('I should not be running')
diff --git a/tests/unit/hook_order_hooks/rthooks.dat b/tests/unit/hook_order_hooks/rthooks.dat
new file mode 100644
--- /dev/null
+++ b/tests/unit/hook_order_hooks/rthooks.dat
@@ -0,0 +1,3 @@
+{
+    'pyi_example_package': ['pyi_rth_good.py']
+}
\ No newline at end of file
diff --git a/tests/unit/hook_order_hooks/rthooks/pyi_rth_good.py b/tests/unit/hook_order_hooks/rthooks/pyi_rth_good.py
new file mode 100644
--- /dev/null
+++ b/tests/unit/hook_order_hooks/rthooks/pyi_rth_good.py
@@ -0,0 +1,10 @@
+#-----------------------------------------------------------------------------
+# Copyright (c) 2005-2020, PyInstaller Development Team.
+#
+# Distributed under the terms of the GNU General Public License (version 2
+# or later) with exception for distributing the bootloader.
+#
+# The full license is in the file COPYING.txt, distributed with this software.
+#
+# SPDX-License-Identifier: (GPL-2.0-or-later WITH Bootloader-exception)
+#-----------------------------------------------------------------------------
diff --git a/tests/unit/hook_order_hooks/setup.py b/tests/unit/hook_order_hooks/setup.py
new file mode 100644
--- /dev/null
+++ b/tests/unit/hook_order_hooks/setup.py
@@ -0,0 +1,13 @@
+from setuptools import setup
+
+setup(
+    name='pyi_example_package',
+    setup_requires="setuptools >= 40.0.0",
+    packages=['pyi_example_package'],
+    author='PyInstaller development team',
+    entry_points={
+        'pyinstaller40': [
+            'hooks = pyi_example_package.__init__:get_hook_dirs'
+        ]
+    }
+)
diff --git a/tests/unit/test_additionalfilescache.py b/tests/unit/test_additionalfilescache.py
new file mode 100644
--- /dev/null
+++ b/tests/unit/test_additionalfilescache.py
@@ -0,0 +1,33 @@
+#-----------------------------------------------------------------------------
+# Copyright (c) 2005-2020, PyInstaller Development Team.
+#
+# Distributed under the terms of the GNU General Public License (version 2
+# or later) with exception for distributing the bootloader.
+#
+# The full license is in the file COPYING.txt, distributed with this software.
+#
+# SPDX-License-Identifier: (GPL-2.0-or-later WITH Bootloader-exception)
+#-----------------------------------------------------------------------------
+
+from PyInstaller.depend.imphook import AdditionalFilesCache
+
+
+def test_binaries_and_datas():
+    datas, binaries, modnames = [
+        [('source', 'dest'), ('src', 'dst')],
+        [('abc', 'def'), ('ghi', 'jkl')],
+        ['testmodule1', 'testmodule2']
+    ]
+
+    cache = AdditionalFilesCache()
+    for modname in modnames:
+        cache.add(
+            modname, binaries, datas
+        )
+        assert cache.datas(modname) == datas
+        cache.add(
+            modname, binaries, datas
+        )
+        # This should be extended. Therefore it should be binaries*2
+        assert cache.binaries(modname) != binaries
+        assert cache.binaries(modname) == binaries * 2
diff --git a/tests/unit/test_hook_order.py b/tests/unit/test_hook_order.py
new file mode 100644
--- /dev/null
+++ b/tests/unit/test_hook_order.py
@@ -0,0 +1,44 @@
+#-----------------------------------------------------------------------------
+# Copyright (c) 2005-2020, PyInstaller Development Team.
+#
+# Distributed under the terms of the GNU General Public License (version 2
+# or later) with exception for distributing the bootloader.
+#
+# The full license is in the file COPYING.txt, distributed with this software.
+#
+# SPDX-License-Identifier: (GPL-2.0-or-later WITH Bootloader-exception)
+#-----------------------------------------------------------------------------
+import os
+import sys
+import subprocess
+import atexit
+
+
+def test_hook_order(pyi_builder):
+
+    subprocess.run(
+        [
+            sys.executable, '-m', 'pip', 'install', '-e',
+            os.path.join(os.path.dirname(__file__), 'hook_order_hooks')
+        ]
+    )
+
+    atexit.register(lambda: subprocess.run(
+        [
+            sys.executable, '-m', 'pip', 'uninstall', 'pyi_example_package',
+            '--yes', '-q', '-q', '-q'
+        ]
+    ))
+
+    pyi_builder.test_source(
+        '''
+        try:
+            import pyi_example_package
+        except:
+            pass
+        ''',
+        pyi_args=[
+            '--additional-hooks-dir={}'.format(
+                os.path.join(os.path.dirname(__file__), 'hook_order_hooks')
+            )
+        ])
diff --git a/tests/unit/test_pyimodulegraph.py b/tests/unit/test_pyimodulegraph.py
--- a/tests/unit/test_pyimodulegraph.py
+++ b/tests/unit/test_pyimodulegraph.py
@@ -169,7 +169,7 @@ def test_collect_rthooks_3(tmpdir, monkeypatch):
     hd1 = _gen_pseudo_rthooks("h1", rh1, tmpdir)
     hd2 = _gen_pseudo_rthooks("h2", rh2, tmpdir)
     mg = FakePyiModuleGraph(HOMEPATH, user_hook_dirs=[str(hd1), str(hd2)])
-    assert len(mg._available_rthooks["test_pyimodulegraph_mymod1"]) == 3
+    assert len(mg._available_rthooks["test_pyimodulegraph_mymod1"]) == 1
 
 
 def test_collect_rthooks_fail_1(tmpdir, monkeypatch):