Repository: pyinstaller/pyinstaller
Pull request: hooks: Add `exclude_datas` to utils.hooks.collect_all().
utils.hooks.collect_all() now contains an `exclude_datas` parameter
that is passed to util.hooks.collect_data_files(). This means that
when calling utils.hooks.collect_all() the caller can exclude
datafiles they select. Fixes #5113

Problem statement:
Add `exclude` to utils.hooks.collect_all()
hook-gevent includes the complete test suite: `gevent.testing` and `gevent.tests`. These are collected by `utils.hooks.collect_all()`. While the modules can be excluded using `excludedimports = ["gevent.testing", "gevent.tests"]`, the datafile will still be collected and added to the bundle.

Please add an option to `utils.hooks.collect_all()` to allow excluding directories or files (based on glob). Some of the hook utils (e.g. `collect_data_files`) already can exclude files, these functions can be sued as a blueprint.

Hints from issue discussion:
Hi @htgoebel, I took a look at this. 

The issue I can see is, `gevent.testing` and `gevent.tests` are collected in `collect_all` by [collect_submodules](https://github.com/pyinstaller/pyinstaller/blob/develop/PyInstaller/utils/hooks/__init__.py#L1043), and `collect_submodules` [returns](https://github.com/pyinstaller/pyinstaller/blob/develop/PyInstaller/utils/hooks/__init__.py#L529)  a list that contains the file names only - not their paths. 

As a result, I can't see why we need to use glob to filter out modules from `hiddenimports` (which is returned by `collect_modules` [here](https://github.com/pyinstaller/pyinstaller/blob/develop/PyInstaller/utils/hooks/__init__.py#L1043)). 

We can filter them out on the basis of filename alone, without using a path (which would mean not needing to use glob).  

Or am I not seeing something here? 


@bwoodsend do you have any thoughts regarding my last comment? 
Hmm, I'm guessing what @htgoebel means is add and propagate the `excludes` and `includes` parameters from `collect_data_files` and the `filter` parameter from `collect_submodules`. (Maybe rename them to `exclude_datas` or something to make it clear what goes where.)

Then we could fix the `gevent` hook using something like:
```python
collect_all("gevent", filter_submodules=lambda x: "test" not in x)
```

Commit message: hooks: hook-gevent passes filter_submodules to collect_all().

Hook-gevent passes filter_submodules to utils.hooks.collect_all()
so that utils.hooks.collect_submodules() can filter out those
submodules.

Patch to review:
diff --git a/PyInstaller/hooks/hook-gevent.py b/PyInstaller/hooks/hook-gevent.py
index cb6e90d9e6f..ebd7f62468b 100644
--- a/PyInstaller/hooks/hook-gevent.py
+++ b/PyInstaller/hooks/hook-gevent.py
@@ -13,4 +13,7 @@
 
 excludedimports = ["gevent.testing", "gevent.tests"]
 
-datas, binaries, hiddenimports = collect_all('gevent')
+datas, binaries, hiddenimports = collect_all(
+    'gevent',
+    filter_submodules=lambda name: (
+        "gevent.testing" not in name or "gevent.tests" not in name))
diff --git a/PyInstaller/utils/hooks/__init__.py b/PyInstaller/utils/hooks/__init__.py
index b9f5a8a0d8f..e42001485d7 100755
--- a/PyInstaller/utils/hooks/__init__.py
+++ b/PyInstaller/utils/hooks/__init__.py
@@ -1032,15 +1032,22 @@ def requirements_for_package(package_name):
 # ``collect_data_files``.
 #
 # Typical use: ``datas, binaries, hiddenimports = collect_all('my_module_name')``.
-def collect_all(package_name, include_py_files=True):
+def collect_all(
+        package_name, include_py_files=True, filter_submodules=None,
+        exclude_datas=None, include_datas=None):
     datas = []
     try:
         datas += copy_metadata(package_name)
     except Exception as e:
         logger.warning('Unable to copy metadata for %s: %s', package_name, e)
-    datas += collect_data_files(package_name, include_py_files)
+    datas += collect_data_files(package_name, include_py_files,
+                                excludes=exclude_datas, includes=include_datas)
     binaries = collect_dynamic_libs(package_name)
-    hiddenimports = collect_submodules(package_name)
+    if filter_submodules:
+        hiddenimports = collect_submodules(package_name,
+                                           filter=filter_submodules)
+    else:
+        hiddenimports = collect_submodules(package_name)
     try:
         hiddenimports += requirements_for_package(package_name)
     except Exception as e:
diff --git a/news/5113.hooks.rst b/news/5113.hooks.rst
new file mode 100644
index 00000000000..8bc0357cd82
--- /dev/null
+++ b/news/5113.hooks.rst
@@ -0,0 +1 @@
+Add `exclude_datas` to utils.hooks.collect_all() to allow caller to exclude datafiles.
diff --git a/news/5201.hooks.rst b/news/5201.hooks.rst
new file mode 100644
index 00000000000..9a693369d55
--- /dev/null
+++ b/news/5201.hooks.rst
@@ -0,0 +1 @@
+Modify hook for ``gevent`` to exclude test submodules.


Code review comments:
diff_hunk: @@ -0,0 +1 @@
+Add `exclude_datas` to utils.hooks.collect_all() to allow caller to exclude datafiles.
path: news/5113.hooks.rst
text: @user1:
```suggestion
Add ``exclude_datas``, ``include_datas``, and ``filter_submodules`` to ``collect_all()``. These arguments map to the ``excludes`` and ``includes`` arguments of ``collect_data_files``, and to the `filter` argument of ``collect_submodules``.
```