Repository: pyinstaller/pyinstaller

Issue:
multiprocessing start blocked if start() is in subprocess
## Description of the issue

Python multiprocessing buildin module.
By starting a new proccess (subprocess) in a process by proc.start() script is blocked when executed in pyinstaller 5.9. Last working pyinstaller version supports sub-processes seems to be v.5.6.

### Context information (for bug reports)

* Output of `pyinstaller --version`: ```5.9.0)```
* Version of Python: 3.11.2
* Platform: GNU/Linux Debian
* How you installed Python:  apt
* Did you also try this on another platform? Yes Windows, Does it work there? No


* try the latest development version, using the following command:

```shell
pip install https://github.com/pyinstaller/pyinstaller/archive/develop.zip
```

The same

### A minimal example program which shows the error

```
from multiprocessing import Process

def runner2():
    print('runner2 was executed')

def runner1():
    print('runner1 in proc')
    proc2 = Process(target=runner2)
    print('try to start subproc')

    proc2.start() # BLOCKED HERE
    
    print('start executed')

proc = Process(target=runner1)
proc.start()
```

### Execute

```
$ dist/run 
runner1 in proc
try to start subpr
```

### Stacktrace / full error message

CONTROL+C

```
^CException ignored in sys.unraisablehook: <built-in function unraisablehook>
KeyboardInterrupt
Process Process-1:
Traceback (most recent call last):
  File "multiprocessing/process.py", line 314, in _bootstrap
  File "multiprocessing/process.py", line 108, in run
  File "client/run.py", line 10, in runner1
    proc2.start() # BLOCK HERE
    ^^^^^^^^^^^^^
  File "multiprocessing/process.py", line 121, in start
  File "multiprocessing/context.py", line 224, in _Popen
  File "multiprocessing/context.py", line 281, in _Popen
  File "PyInstaller/hooks/rthooks/pyi_rth_multiprocessing.py", line 73, in __init__
KeyboardInterrupt
```

Any ideas, fixes, workarounds?



Discussion:
At first glance, this looks like regression caused by https://github.com/pyinstaller/pyinstaller/pull/7411.
And it likely happens because the default (`fork`) start method is used, under which the child process inherits the copy of the (now locked) lock, and ends up dead-locked.

But since `fork` does not in fact start the new process from scratch, I wonder if we even need to override the `forking.Popen` to restore `_MEIPASS2` in the first place...

Until this is resolved, you'll need to either use older version of PyInstaller, or manually undo the changes made in 617be5df5fbfc30bfcce3901083d567c1fd6892.

Resolving patch:
diff --git a/PyInstaller/hooks/rthooks/pyi_rth_multiprocessing.py b/PyInstaller/hooks/rthooks/pyi_rth_multiprocessing.py
--- a/PyInstaller/hooks/rthooks/pyi_rth_multiprocessing.py
+++ b/PyInstaller/hooks/rthooks/pyi_rth_multiprocessing.py
@@ -9,91 +9,103 @@
 # SPDX-License-Identifier: Apache-2.0
 #-----------------------------------------------------------------------------
 
-import threading
-import multiprocessing
-import multiprocessing.spawn as spawn
-# 'spawn' multiprocessing needs some adjustments on osx
-import os
-import sys
-from subprocess import _args_from_interpreter_flags
-
-# prevent spawn from trying to read __main__ in from the main script
-multiprocessing.process.ORIGINAL_DIR = None
-
-
-def _freeze_support():
-    # We want to catch the two processes that are spawned by the multiprocessing code:
-    # - the semaphore tracker, which cleans up named semaphores in the spawn multiprocessing mode
-    # - the fork server, which keeps track of worker processes in forkserver mode.
-    # both of these processes are started by spawning a new copy of the running executable, passing it the flags from
-    # _args_from_interpreter_flags and then "-c" and an import statement.
-    # Look for those flags and the import statement, then exec() the code ourselves.
-
-    if (
-        len(sys.argv) >= 2 and sys.argv[-2] == '-c' and sys.argv[-1].startswith((
-            'from multiprocessing.semaphore_tracker import main',  # Py<3.8
-            'from multiprocessing.resource_tracker import main',  # Py>=3.8
-            'from multiprocessing.forkserver import main'
-        )) and set(sys.argv[1:-2]) == set(_args_from_interpreter_flags())
-    ):
-        exec(sys.argv[-1])
-        sys.exit()
-
-    if spawn.is_forking(sys.argv):
-        kwds = {}
-        for arg in sys.argv[2:]:
-            name, value = arg.split('=')
-            if value == 'None':
-                kwds[name] = None
-            else:
-                kwds[name] = int(value)
-        spawn.spawn_main(**kwds)
-        sys.exit()
-
-
-multiprocessing.freeze_support = spawn.freeze_support = _freeze_support
-
-# Bootloader unsets _MEIPASS2 for child processes to allow running PyInstaller binaries inside pyinstaller binaries.
-# This is ok for mac or unix with fork() system call. But on Windows we need to overcome missing fork() function.
-
-if sys.platform.startswith('win'):
-    import multiprocessing.popen_spawn_win32 as forking
-else:
-    import multiprocessing.popen_fork as forking
-    import multiprocessing.popen_spawn_posix as spawning
-
-
-# Mix-in to re-set _MEIPASS2 from sys._MEIPASS.
-class FrozenSupportMixIn:
-    _lock = threading.Lock()
-
-    def __init__(self, *args, **kw):
-        # The whole code block needs be executed under a lock to prevent race conditions between `os.putenv` and
-        # `os.unsetenv` calls when processes are spawned concurrently from multiple threads. See #7410.
-        with self._lock:
-            # We have to set original _MEIPASS2 value from sys._MEIPASS to get --onefile mode working.
-            os.putenv('_MEIPASS2', sys._MEIPASS)  # @UndefinedVariable
-            try:
-                super().__init__(*args, **kw)
-            finally:
-                # On some platforms (e.g. AIX) 'os.unsetenv()' is not available. In those cases we cannot delete the
-                # variable but only set it to the empty string. The bootloader can handle this case.
-                if hasattr(os, 'unsetenv'):
-                    os.unsetenv('_MEIPASS2')
-                else:
-                    os.putenv('_MEIPASS2', '')
-
-
-# Patch forking.Popen to re-set _MEIPASS2 from sys._MEIPASS.
-class _Popen(FrozenSupportMixIn, forking.Popen):
-    pass
-
 
-forking.Popen = _Popen
-
-if not sys.platform.startswith('win'):
-    # Patch spawning.Popen to re-set _MEIPASS2 from sys._MEIPASS.
-    class _Spawning_Popen(FrozenSupportMixIn, spawning.Popen):
-        pass
-
-    spawning.Popen = _Spawning_Popen
+def _pyi_rth_multiprocessing():
+    import os
+    import sys
+
+    import threading
+    import multiprocessing
+    import multiprocessing.spawn
+
+    from subprocess import _args_from_interpreter_flags
+
+    # Prevent `spawn` from trying to read `__main__` in from the main script
+    multiprocessing.process.ORIGINAL_DIR = None
+
+    def _freeze_support():
+        # We want to catch the two processes that are spawned by the multiprocessing code:
+        # - the semaphore tracker, which cleans up named semaphores in the `spawn` multiprocessing mode
+        # - the fork server, which keeps track of worker processes in the `forkserver` mode.
+        # Both of these processes are started by spawning a new copy of the running executable, passing it the flags
+        # from `_args_from_interpreter_flags` and then "-c" and an import statement.
+        # Look for those flags and the import statement, then `exec()` the code ourselves.
+
+        if (
+            len(sys.argv) >= 2 and sys.argv[-2] == '-c' and sys.argv[-1].startswith((
+                'from multiprocessing.semaphore_tracker import main',  # Py<3.8
+                'from multiprocessing.resource_tracker import main',  # Py>=3.8
+                'from multiprocessing.forkserver import main'
+            )) and set(sys.argv[1:-2]) == set(_args_from_interpreter_flags())
+        ):
+            exec(sys.argv[-1])
+            sys.exit()
+
+        if multiprocessing.spawn.is_forking(sys.argv):
+            kwds = {}
+            for arg in sys.argv[2:]:
+                name, value = arg.split('=')
+                if value == 'None':
+                    kwds[name] = None
+                else:
+                    kwds[name] = int(value)
+            multiprocessing.spawn.spawn_main(**kwds)
+            sys.exit()
+
+    multiprocessing.freeze_support = multiprocessing.spawn.freeze_support = _freeze_support
+
+    # Bootloader clears the `_MEIPASS2` environment variable, which allows a PyInstaller-frozen executable to run a
+    # different PyInstaller-frozen executable. However, in the case of `multiprocessing`, we are actually trying
+    # to run the same executable, so we need to restore `_MEIPASS2` to prevent onefile executable from unpacking
+    # again in a different directory.
+    #
+    # This is needed for `spawn` start method (default on Windows and macOS) and also with `forkserver` start method
+    # (available on Linux and macOS). It is not needed for `fork` start method (default on Linux and other Unix OSes),
+    # because fork copies the parent process instead of starting it from scratch.
+
+    # Mix-in to re-set _MEIPASS2 from sys._MEIPASS.
+    class FrozenSupportMixIn:
+        _lock = threading.Lock()
+
+        def __init__(self, *args, **kw):
+            # The whole code block needs be executed under a lock to prevent race conditions between `os.putenv` and
+            # `os.unsetenv` calls when processes are spawned concurrently from multiple threads. See #7410.
+            with self._lock:
+                # We have to set original _MEIPASS2 value from sys._MEIPASS to get --onefile mode working.
+                os.putenv('_MEIPASS2', sys._MEIPASS)  # @UndefinedVariable
+                try:
+                    super().__init__(*args, **kw)
+                finally:
+                    # On some platforms (e.g. AIX) 'os.unsetenv()' is not available. In those cases we cannot delete the
+                    # variable but only set it to the empty string. The bootloader can handle this case.
+                    if hasattr(os, 'unsetenv'):
+                        os.unsetenv('_MEIPASS2')
+                    else:
+                        os.putenv('_MEIPASS2', '')
+
+    if sys.platform.startswith('win'):
+        # Windows; patch `Popen` for `spawn` start method
+        from multiprocessing import popen_spawn_win32
+
+        class _SpawnPopen(FrozenSupportMixIn, popen_spawn_win32.Popen):
+            pass
+
+        popen_spawn_win32.Popen = _SpawnPopen
+    else:
+        # UNIX OSes; patch `Popen` for `spawn` and `forkserver` start methods
+        from multiprocessing import popen_spawn_posix
+        from multiprocessing import popen_forkserver
+
+        class _SpawnPopen(FrozenSupportMixIn, popen_spawn_posix.Popen):
+            pass
+
+        class _ForkserverPopen(FrozenSupportMixIn, popen_forkserver.Popen):
+            pass
+
+        popen_spawn_posix.Popen = _SpawnPopen
+        popen_forkserver.Popen = _ForkserverPopen
+
+
+# Run the hook function, then delete it. This prevents unnecessary pollution of the global namespace.
+_pyi_rth_multiprocessing()
+del _pyi_rth_multiprocessing


Test patch: