Repository: pyinstaller/pyinstaller

Issue:
Altered `sys.path` is not respected in `collect_data_files` call to `isolated.call`
## Description of the issue

In our old spec files we used to locally add the current directory to `sys.path`, in order to make sure that the development (not installed) version of the python code could be packaged.  This used to work fine probably version with 4.8.

We added the spec file with a very early line of:
```
sys.path.append(os.path.dirname(os.path.abspath(SPEC)))
```
and later called an `Analysis` containing `data = collect_data_files('package.thing')` where under the current directory `package.thing` existed and was a valid package.

```
140 WARNING: collect_data_files - skipping data collection for module 'package.thing' as it is not a package.
338 INFO: Extending PYTHONPATH with paths
['/path/to/project', '/path/to/project', '/value/in/pathex']
```

Upon debugging it appears that the calls to `isolated.call` ignore the earlier call to `sys.path.append`, reverting to the initial pyinstaller's `sys.path` value.  The calls to "Extending PYTHONPATH" appear to occur after the data collection has occurred and it's not clear whether they would be taken into account or not.

A work around is to call `PYTHONPATH=/path/to/project pyinstaller file.spec`, which does then find package.thing and all other relevant files.

Please let me know if you need further information.

### Context information (for bug reports)

* Output of `pyinstaller --version`: ```5.8.0```
* Version of Python: 3.10
* Platform: Gentoo Linux
* How you installed Python: Gentoo package
* Did you also try this on another platform:  Also tested on Windows, also does not work
* Try the latest development version: Same issue

* follow *all* the instructions in our "If Things Go Wrong" Guide
  (https://github.com/pyinstaller/pyinstaller/wiki/If-Things-Go-Wrong) and

### Make sure [everything is packaged correctly](https://github.com/pyinstaller/pyinstaller/wiki/How-to-Report-Bugs#make-sure-everything-is-packaged-correctly)

  * [x] start with clean installation
  * [x] use the latest development version
  * [x] Run your frozen program **from a command window (shell)** — instead of double-clicking on it
  * [ ] Package your program in **--onedir mode** 
  * [ ] Package **without UPX**, say: use the option `--noupx` or set `upx=False` in your .spec-file
  * [ ] Repackage you application in **verbose/debug mode**. For this, pass the option `--debug` to `pyi-makespec` or `pyinstaller` or use `EXE(..., debug=1, ...)` in your .spec file.


### A minimal example program which shows the error

```
import os
import sys

from PyInstaller.building.api import PYZ, EXE
from PyInstaller.building.build_main import Analysis
from PyInstaller.utils.hooks import collect_submodules, collect_data_files, collect_dynamic_libs

sys.path.append(os.path.dirname(os.path.abspath(SPEC)))

analysis = Analysis(['scipt.py'],
                        pathex = [os.path.dirname(os.path.abspath(SPEC)), '/value/in/pathex'],
                        datas = collect_data_files('package.thing'))

pyz = PYZ(analysis.pure, analysis.zipped_data)
exe = EXE(pyz,
              analysis.scripts,
              analysis.binaries,
              analysis.zipfiles,
              analysis.datas,
              [('u', None, 'OPTION')],
              name = 'thing',
              debug = False,
              bootloader_ignore_signals = False,
              strip = False,
              upx = False,
              runtime_tmpdir = None,
              console = True)
```

### Stacktrace / full error message


```
140 WARNING: collect_data_files - skipping data collection for module 'package.thing' as it is not a package.
```

Please also see <https://github.com/pyinstaller/pyinstaller/wiki/How-to-Report-Bugs>
for more about what would use to solve the issue.


Discussion:
Is `package.thing` a package? I.e., is `thing` a subpackage of `package`, or is it actually a submodule?
It passes the [`is_package` check](https://github.com/pyinstaller/pyinstaller/blob/c2011f1fee56fab611070e365fb26edd4a2c2fff/PyInstaller/utils/hooks/__init__.py#L457) as indicated when PYTHONPATH is appropriately set.  It's just that when it's set dynamically within the python spec itself, it vanishes in the isolated container.
 #6294 may be related.
Hmm, `sys.path` indeed does not seem to be propagated from parent process to the child. Marking this as a bug for now, because perhaps it should be.



> #6294 may be related.

No, that was the old behavior as well. If you want(ed) to use `collect_*` in spec for packages in non-default paths, you always had to either set `PYTHONPATH` accordingly or modify `sys.path` (which does not work anymore, as you noted). That's because additional paths passed via `--paths ` are applied when `Analysis` is instantiated, which is *after* `collect_*` functions in the spec file are called.

I guess a `python_path += sys.path` somewhere in [here](https://github.com/pyinstaller/pyinstaller/blob/55eb4764f6c09894cd29180d5128476b8212fd21/PyInstaller/isolated/_parent.py#L171-L172) would do. My head hurts trying to figure out the order in the items should go in though.
I suppose the parent's `PYTHONPATH` will already be in `sys.path` so it should be the `pathex` entries first, then `sys.path` then ignore `PYTHONPATH`.
What if we serialized parent's `sys.path` as part of args passed to the child, and overwrote it at the start of child process?
That sounds like a better idea. Although make it *serialize parent's `pathex + sys.path`*. 
Wouldn't `pathex` already be in `sys.path` once we pass [this point](https://github.com/pyinstaller/pyinstaller/blob/55eb4764f6c09894cd29180d5128476b8212fd21/PyInstaller/building/build_main.py#L349-L355), i.e., after `Analysis` is instantiated? 

Prior to that, `pathex` paths (as well as entry-point script's parent directory) have no effect on main process - and I don't see a straightforward way of dealing with that (which is what #6294 is about).
Huh, why on earth do we go to all the trouble of keeping track of a separate `self.pathex` variable if we then go ahead and merge it with `sys.path` anyway...
> Huh, why on earth do we go to all the trouble of keeping track of a separate `self.pathex` variable if we then go ahead and merge it with `sys.path` anyway...

One use is [the guts check](https://github.com/pyinstaller/pyinstaller/blob/55eb4764f6c09894cd29180d5128476b8212fd21/PyInstaller/building/build_main.py#L427) to trigger re-analysis on change. Although now that I see it, I think it should be using full `sys.path` instead, because external changes induced via `PYTHONPATH` environment variable should probably also trigger re-analysis.

Resolving patch:
diff --git a/PyInstaller/isolated/_child.py b/PyInstaller/isolated/_child.py
--- a/PyInstaller/isolated/_child.py
+++ b/PyInstaller/isolated/_child.py
@@ -90,6 +90,8 @@ def run_next_command(read_fh, write_fh):
 
     with _open(read_from_parent, "rb") as read_fh:
         with _open(write_to_parent, "wb") as write_fh:
+            sys.path = loads(b64decode(read_fh.readline()))
+
             # Keep receiving and running instructions until the parent sends the signal to stop.
             while run_next_command(read_fh, write_fh):
                 pass
diff --git a/PyInstaller/isolated/_parent.py b/PyInstaller/isolated/_parent.py
--- a/PyInstaller/isolated/_parent.py
+++ b/PyInstaller/isolated/_parent.py
@@ -15,6 +15,7 @@
 from base64 import b64encode, b64decode
 import functools
 import subprocess
+import sys
 
 from PyInstaller import compat
 from PyInstaller import log as logging
@@ -236,6 +237,8 @@ def __enter__(self):
             self._write_handle = os.fdopen(write_to_child, "wb")
             self._read_handle = os.fdopen(read_from_child, "rb")
 
+        self._send(sys.path)
+
         return self
 
     def __exit__(self, exc_type, exc_val, exc_tb):
@@ -283,19 +286,7 @@ def call(self, function, *args, **kwargs):
         if self._child is None:
             raise RuntimeError("An isolated.Python object must be used in a 'with' clause.")
 
-        # Send 5 lines to the child process: The function's code attribute, its default args and kwargs, and its actual
-        # args and kwargs, all serialised.
-        self._write_handle.writelines([
-            b64encode(dumps(function.__code__)), b"\n",
-            b64encode(dumps(function.__defaults__)), b"\n",
-            b64encode(dumps(function.__kwdefaults__)), b"\n",
-            b64encode(dumps(args)), b"\n",
-            b64encode(dumps(kwargs)), b"\n",
-        ])  # yapf: disable
-
-        # Flushing is very important. Without it, the data is not sent but forever sits in a buffer so that the child is
-        # forever waiting for its data and the parent in turn is forever waiting for the child's response.
-        self._write_handle.flush()
+        self._send(function.__code__, function.__defaults__, function.__kwdefaults__, args, kwargs)
 
         # Read a single line of output back from the child. This contains if the function worked and either its return
         # value or a traceback. This will block indefinitely until it receives a '\n' byte.
@@ -310,6 +301,14 @@ def call(self, function, *args, **kwargs):
         # trace.
         raise RuntimeError(f"Child process call to {function.__name__}() failed with:\n" + output)
 
+    def _send(self, *objects):
+        for object in objects:
+            self._write_handle.write(b64encode(dumps(object)))
+            self._write_handle.write(b"\n")
+        # Flushing is very important. Without it, the data is not sent but forever sits in a buffer so that the child is
+        # forever waiting for its data and the parent in turn is forever waiting for the child's response.
+        self._write_handle.flush()
+
 
 def call(function, *args, **kwargs):
     r"""


Test patch:
diff --git a/tests/unit/test_hookutils.py b/tests/unit/test_hookutils.py
--- a/tests/unit/test_hookutils.py
+++ b/tests/unit/test_hookutils.py
@@ -108,9 +108,6 @@ def test_win32style_path(self):
 
 @pytest.fixture
 def mod_list(monkeypatch):
-    # Add 'hookutils_files' to sys.path (so ``is_package`` can find it) and to
-    # ``pathex`` (so code run in a subprocess can find it).
-    monkeypatch.setattr('PyInstaller.config.CONF', {'pathex': [TEST_MOD_PATH]})
     monkeypatch.syspath_prepend(TEST_MOD_PATH)
     # Use the hookutils_test_files package for testing.
     return collect_submodules(TEST_MOD)