Repository: pyinstaller/pyinstaller

Issue:
utils.hooks.collect_all() collects __pycache__ and .py[co] files
How to reproduce:
- Install gevent
- In `tests/functional/test_libraries.py` add some line to the script in `test_gevent` to make the script fail. E.g. add "raise SystemExit(1)"
    This is only to keep the build data after the test run.  Our test-suite removes data on successful runs. (There might ne another way to keep the data, but I can't remember and have no time for searching for it.)
- Run `pytest tests/functional/test_libraries.py -k _gevent[onedir]`
- In (path might vary) `$TMP/pytest-of-*/pytest-current/test_gevent_onedir_current/dist/test_source/gevent` you will find a __pycache_ directory containing .pyc files.

Related to #5113.


Discussion:
Just to simplify this a bit: You can see and reproduce the issue without having to actually build anything.

The following:
```python
from PyInstaller.utils.hooks import collect_data_files
from pprint import pprint
pprint(collect_data_files("email"))
```
Should give me:

```python
 ('E:\\pythons\\winpython3.8\\WPy64-3800\\python-3.8.0.amd64\\lib\\email\\architecture.rst',
  'email')]
```

<details><summary>But also includes a whole load of `.pyo` files too.</summary>

```python
[('E:\\pythons\\winpython3.8\\WPy64-3800\\python-3.8.0.amd64\\lib\\email\\iterators.pyo',
  'email'),
 ('E:\\pythons\\winpython3.8\\WPy64-3800\\python-3.8.0.amd64\\lib\\email\\utils.pyo',
  'email'),
 ('E:\\pythons\\winpython3.8\\WPy64-3800\\python-3.8.0.amd64\\lib\\email\\_header_value_parser.pyo',
  'email'),
 ('E:\\pythons\\winpython3.8\\WPy64-3800\\python-3.8.0.amd64\\lib\\email\\base64mime.pyo',
  'email'),
 ('E:\\pythons\\winpython3.8\\WPy64-3800\\python-3.8.0.amd64\\lib\\email\\feedparser.pyo',
  'email'),
 ('E:\\pythons\\winpython3.8\\WPy64-3800\\python-3.8.0.amd64\\lib\\email\\_parseaddr.pyo',
  'email'),
 ('E:\\pythons\\winpython3.8\\WPy64-3800\\python-3.8.0.amd64\\lib\\email\\_encoded_words.pyo',
  'email'),
 ('E:\\pythons\\winpython3.8\\WPy64-3800\\python-3.8.0.amd64\\lib\\email\\errors.pyo',
  'email'),
 ('E:\\pythons\\winpython3.8\\WPy64-3800\\python-3.8.0.amd64\\lib\\email\\encoders.pyo',
  'email'),
 ('E:\\pythons\\winpython3.8\\WPy64-3800\\python-3.8.0.amd64\\lib\\email\\parser.pyo',
  'email'),
 ('E:\\pythons\\winpython3.8\\WPy64-3800\\python-3.8.0.amd64\\lib\\email\\charset.pyo',
  'email'),
 ('E:\\pythons\\winpython3.8\\WPy64-3800\\python-3.8.0.amd64\\lib\\email\\quoprimime.pyo',
  'email'),
 ('E:\\pythons\\winpython3.8\\WPy64-3800\\python-3.8.0.amd64\\lib\\email\\_policybase.pyo',
  'email'),
 ('E:\\pythons\\winpython3.8\\WPy64-3800\\python-3.8.0.amd64\\lib\\email\\generator.pyo',
  'email'),
 ('E:\\pythons\\winpython3.8\\WPy64-3800\\python-3.8.0.amd64\\lib\\email\\__init__.pyo',
  'email'),
 ('E:\\pythons\\winpython3.8\\WPy64-3800\\python-3.8.0.amd64\\lib\\email\\headerregistry.pyo',
  'email'),
 ('E:\\pythons\\winpython3.8\\WPy64-3800\\python-3.8.0.amd64\\lib\\email\\message.pyo',
  'email'),
 ('E:\\pythons\\winpython3.8\\WPy64-3800\\python-3.8.0.amd64\\lib\\email\\policy.pyo',
  'email'),
 ('E:\\pythons\\winpython3.8\\WPy64-3800\\python-3.8.0.amd64\\lib\\email\\header.pyo',
  'email'),
 ('E:\\pythons\\winpython3.8\\WPy64-3800\\python-3.8.0.amd64\\lib\\email\\contentmanager.pyo',
  'email'),
 ('E:\\pythons\\winpython3.8\\WPy64-3800\\python-3.8.0.amd64\\lib\\email\\architecture.rst',
  'email')]
```

</details>

And with `collect_data_files("email", include_py_file=True)` it should also include `.py` files but not `__pycache__` directories or `.pyc` files. Currently it includes everything.

@RS357 Could you have a look at this one whilst I work out how to tackle #5104?

Hi @bwoodsend - perhaps instead of adding a `.pyo` filter ourselves, we should file an issue with the CPython project, asking them to add the `.pyo` suffix to the [`ALL_SUFFIXES`](https://github.com/pyinstaller/pyinstaller/blob/master/PyInstaller/utils/hooks/__init__.py#L730) variable returned by `importlib.machinery` [here](Lib/importlib/machinery.py)? 

For some reason they include `.pyc` but not `.pyo` - even though `.pyo` is also a suffix for Python bytecode. 
@RS357 They are a Python suffix and probably should be listed in `ALL_SUFFIXES` but that doesn't actually affect this issue. As far as PyInstaller is concerned `.pyo`s are junk files and should be excluded **with or without** `include_py_file=True` set. Adding `.pyo` to `ALL_SUFFIXES` would change `collect_data_files` so that it now excludes `.pyo`s with `include_py_file=False` but still includes them with `include_py_file=True`.
> perhaps instead of adding a .pyo filter ourselves, we should

AFAIR the .pyo files are to be replaced be a different suffix, so they'll end with `.pyc`, too. On my system `pydoc3.7 importlib.machinery` already says. So I'm curious why bwoodsend has these files ate all (ans why he does not have a `__pycahce__`directory)

```
    OPTIMIZED_BYTECODE_SUFFIXES = ['.pyc']
```

> as PyInstaller is concerned .pyos are junk files and should be excluded with or without include_py_file=True set

Whether these are "junk" files, depends on whether PyInstaller is run with `-O` (and on whether the resp. version of Python uses .pyo files, of course). When running e.g. `python -O -m pyinstaller …` Pyinstaller is expected to pick up the optimized files. (Though I can not tell without testing what happens with the collected `.pyo` files)

Resolving patch:
diff --git a/PyInstaller/utils/hooks/__init__.py b/PyInstaller/utils/hooks/__init__.py
--- a/PyInstaller/utils/hooks/__init__.py
+++ b/PyInstaller/utils/hooks/__init__.py
@@ -729,6 +729,10 @@ def collect_data_files(package, include_py_files=False, subdir=None,
     if not include_py_files:
         excludes += ['**/*' + s for s in ALL_SUFFIXES]
 
+    # Exclude .pyo files if include_py_files is False.
+    if not include_py_files and ".pyo" not in ALL_SUFFIXES:
+        excludes.append('**/*.pyo')
+
     # If not specified, include all files. Follow the same process as the
     # excludes.
     includes = list(includes) if includes else ["**/*"]


Test patch: