Repository: pyinstaller/pyinstaller

Issue:
Add `exclude` to utils.hooks.collect_all()
hook-gevent includes the complete test suite: `gevent.testing` and `gevent.tests`. These are collected by `utils.hooks.collect_all()`. While the modules can be excluded using `excludedimports = ["gevent.testing", "gevent.tests"]`, the datafile will still be collected and added to the bundle.

Please add an option to `utils.hooks.collect_all()` to allow excluding directories or files (based on glob). Some of the hook utils (e.g. `collect_data_files`) already can exclude files, these functions can be sued as a blueprint.


Discussion:
Hi @htgoebel, I took a look at this. 

The issue I can see is, `gevent.testing` and `gevent.tests` are collected in `collect_all` by [collect_submodules](https://github.com/pyinstaller/pyinstaller/blob/develop/PyInstaller/utils/hooks/__init__.py#L1043), and `collect_submodules` [returns](https://github.com/pyinstaller/pyinstaller/blob/develop/PyInstaller/utils/hooks/__init__.py#L529)  a list that contains the file names only - not their paths. 

As a result, I can't see why we need to use glob to filter out modules from `hiddenimports` (which is returned by `collect_modules` [here](https://github.com/pyinstaller/pyinstaller/blob/develop/PyInstaller/utils/hooks/__init__.py#L1043)). 

We can filter them out on the basis of filename alone, without using a path (which would mean not needing to use glob).  

Or am I not seeing something here? 


@bwoodsend do you have any thoughts regarding my last comment? 
Hmm, I'm guessing what @htgoebel means is add and propagate the `excludes` and `includes` parameters from `collect_data_files` and the `filter` parameter from `collect_submodules`. (Maybe rename them to `exclude_datas` or something to make it clear what goes where.)

Then we could fix the `gevent` hook using something like:
```python
collect_all("gevent", filter_submodules=lambda x: "test" not in x)
```

Resolving patch:
diff --git a/PyInstaller/hooks/hook-gevent.py b/PyInstaller/hooks/hook-gevent.py
--- a/PyInstaller/hooks/hook-gevent.py
+++ b/PyInstaller/hooks/hook-gevent.py
@@ -13,4 +13,7 @@
 
 excludedimports = ["gevent.testing", "gevent.tests"]
 
-datas, binaries, hiddenimports = collect_all('gevent')
+datas, binaries, hiddenimports = collect_all(
+    'gevent',
+    filter_submodules=lambda name: (
+        "gevent.testing" not in name or "gevent.tests" not in name))
diff --git a/PyInstaller/utils/hooks/__init__.py b/PyInstaller/utils/hooks/__init__.py
--- a/PyInstaller/utils/hooks/__init__.py
+++ b/PyInstaller/utils/hooks/__init__.py
@@ -1032,15 +1032,22 @@ def requirements_for_package(package_name):
 # ``collect_data_files``.
 #
 # Typical use: ``datas, binaries, hiddenimports = collect_all('my_module_name')``.
-def collect_all(package_name, include_py_files=True):
+def collect_all(
+        package_name, include_py_files=True, filter_submodules=None,
+        exclude_datas=None, include_datas=None):
     datas = []
     try:
         datas += copy_metadata(package_name)
     except Exception as e:
         logger.warning('Unable to copy metadata for %s: %s', package_name, e)
-    datas += collect_data_files(package_name, include_py_files)
+    datas += collect_data_files(package_name, include_py_files,
+                                excludes=exclude_datas, includes=include_datas)
     binaries = collect_dynamic_libs(package_name)
-    hiddenimports = collect_submodules(package_name)
+    if filter_submodules:
+        hiddenimports = collect_submodules(package_name,
+                                           filter=filter_submodules)
+    else:
+        hiddenimports = collect_submodules(package_name)
     try:
         hiddenimports += requirements_for_package(package_name)
     except Exception as e:


Test patch: