Repository: pyinstaller/pyinstaller

Issue:
distutils not included with latest virtualenv (16.4.0)
This was already reported in #4031. The issue was closed without a fix so I'm creating this one.

**With virtualenv 16.4.0, pyinstaller reports :**

```
3583 INFO: distutils: retargeting to non-venv dir '/usr/lib64/python3.6/distutils/__init__.py'
```
and then during "Loading module hook" sequence, the `hook-distutils.py` is missing and distutils modules are not included into the final executable binary.

When executing the binary the error is:

```
ModuleNotFoundError: No module named 'distutils'
[10373] Failed to execute script <name here>
```

**With virtualenv 16.1.0, pyinstaller reports :**

```
3157 INFO: Processing pre-find module path hook   distutils
5053 INFO: Loading module hook "hook-distutils.py"...
```

and distutils modules are included into the final executable binary.



Discussion:
I also have a pbl with disutils on 16.3. Pyinstaller does not says anything:
```
5055 INFO: Processing pre-find module path hook   distutils
5055 INFO: distutils: retargeting to non-venv dir '/usr/local/lib/python3.6/distutils/__init__.py'
```

But later on 
```
16793 WARNING: Hidden import "distutils.command.build_ext" not found!
...
17570 DEBUG: Collecting submodules for distutils
17676 DEBUG: collect_submodules - Found submodules: {'distutils'}
```

But at execution I still get:
```
ModuleNotFoundError: No module named 'distutils'
```

Worked fine on 16.1

Running `pyi-archive_viewer` on the executable does not list distutils package.
On an executable that worked, pyi-archive_viewer shows the following packages:
```
 'distutils': (1, 1487595, 267),
 'distutils._msvccompiler': (0, 1487862, 7159),
 'distutils.archive_util': (0, 1495021, 3352),
 'distutils.ccompiler': (0, 1498373, 12557),
 'distutils.cmd': (0, 1510930, 6328),
 'distutils.command': (1, 1517258, 326),
 'distutils.command.bdist': (0, 1517584, 2030),
 'distutils.command.build_ext': (0, 1519614, 8083),
 'distutils.command.build_scripts': (0, 1527697, 2375),
 'distutils.command.install': (0, 1530072, 6322),
 'distutils.command.install_lib': (0, 1536394, 2602),
 'distutils.command.install_scripts': (0, 1538996, 1128),
 'distutils.command.sdist': (0, 1540124, 6183),
 'distutils.config': (0, 1546307, 1895),
 'distutils.core': (0, 1548202, 3246),
 'distutils.debug': (0, 1551448, 137),
 'distutils.dep_util': (0, 1551585, 1430),
 'distutils.dir_util': (0, 1553015, 3048),
 'distutils.dist': (0, 1556063, 13966),
 'distutils.errors': (0, 1570029, 1966),
 'distutils.extension': (0, 1571995, 3282),
 'distutils.fancy_getopt': (0, 1575277, 5261),
 'distutils.file_util': (0, 1580538, 2989),
 'distutils.filelist': (0, 1583527, 4493),
 'distutils.log': (0, 1588020, 1080),
 'distutils.msvc9compiler': (0, 1589100, 8918),
 'distutils.spawn': (0, 1598018, 2749),
 'distutils.sysconfig': (0, 1600767, 6128),
 'distutils.text_file': (0, 1606895, 3793),
 'distutils.util': (0, 1610688, 7966),
 'distutils.version': (0, 1618654, 3110),
 'distutils.versionpredicate': (0, 1621764, 2346),
```
Solution for the moment: force the system's virtualenv to be 16.1 (freeze the virtualenv package version inside the virtualenvironment is not enough)
Just to pinpoint the problem a bit further: `<virtual_env>/Lib/distutils/__init__.py` attribute `distutils_path` returns a path to a file `<system_interpreter>/Lib/distutils/__init__.py` from virtualenv 16.4.0 on, but previously returned a path to the directory `<system_interpreter>/Lib/distutils/`.

PyInstaller hook [hook-distutils](https://github.com/pyinstaller/pyinstaller/blob/develop/PyInstaller/hooks/pre_find_module_path/hook-distutils.py) uses `os.path.dirname` to expand the search directorys which will result in `<system_interpreter>/Lib/distutils/` with virtualenv 16.4.0 which would result with `<system_interpreter>/Lib/` with previous version of virtualenv.
Downgrading my system virtualenv to 16.1 did not seem to help.

However, based on @tappi287 's great analysis, I came up with the following work-around. Close to the start of your spec file, add the following:

```python
# work-around for https://github.com/pyinstaller/pyinstaller/issues/4064
import distutils
if distutils.distutils_path.endswith('__init__.py'):
    distutils.distutils_path = os.path.dirname(distutils.distutils_path)
```

This will temporarily override the distutils_path to what pyinstaller expects, resulting in working builds.
I also had this problem with `virtualenv`. So i switch to use python's built in `venv`.

Resolving patch:
diff --git a/PyInstaller/hooks/pre_find_module_path/hook-distutils.py b/PyInstaller/hooks/pre_find_module_path/hook-distutils.py
--- a/PyInstaller/hooks/pre_find_module_path/hook-distutils.py
+++ b/PyInstaller/hooks/pre_find_module_path/hook-distutils.py
@@ -1,11 +1,11 @@
-#-----------------------------------------------------------------------------
+# -----------------------------------------------------------------------------
 # Copyright (c) 2005-2019, PyInstaller Development Team.
 #
 # Distributed under the terms of the GNU General Public License with exception
 # for distributing bootloader.
 #
 # The full license is in the file COPYING.txt, distributed with this software.
-#-----------------------------------------------------------------------------
+# -----------------------------------------------------------------------------
 
 """
 `distutils`-specific pre-find module path hook.
@@ -26,8 +26,16 @@
 def pre_find_module_path(api):
     # Absolute path of the system-wide "distutils" package when run from within
     # a venv or None otherwise.
-    distutils_dir = getattr(distutils, 'distutils_path', None)
-    if distutils_dir is not None:
+
+    # opcode is not a virtualenv module, so we can use it to find the stdlib.
+    # Technique taken from virtualenv's "distutils" package detection at
+    # https://github.com/pypa/virtualenv/blob/16.3.0/virtualenv_embedded/distutils-init.py#L5
+    import opcode
+
+    system_module_path = os.path.normpath(os.path.dirname(opcode.__file__))
+    loaded_module_path = os.path.normpath(os.path.dirname(distutils.__file__))
+    if system_module_path != loaded_module_path:
         # Find this package in its parent directory.
-        api.search_dirs = [os.path.dirname(distutils_dir)]
-        logger.info('distutils: retargeting to non-venv dir %r' % distutils_dir)
+        api.search_dirs = [system_module_path]
+        logger.info('distutils: retargeting to non-venv dir %r',
+                    system_module_path)


Test patch: