Repository: pyinstaller/pyinstaller

Issue:
Multiprocessing "spawn" not thread-safe on Linux.
<!--
Welcome to the PyInstaller issue tracker! Before creating an issue, please heed the following:

1. This tracker should only be used to report bugs and request features / enhancements to PyInstaller
    - For questions and general support, use the discussions forum.
2. Use the search function before creating a new issue. Duplicates will be closed and directed to
   the original discussion.
3. When making a bug report, make sure you provide all required information. The easier it is for
   maintainers to reproduce, the faster it'll be fixed.
-->

<!-- +++ ONLY TEXT +++ DO NOT POST IMAGES +++ -->

## Description of the issue

When using multiprocessing with the "spawn" method on Linux, processes sometimes fail to start with the message: `FileNotFoundError: [Errno 2] No such file or directory: '/tmp/_MEIOchafX/multiprocessing_bug.py'` This happens if different threads try to launch processes concurrently. It would appear that the "spawn" method is not thread-safe when used with freeze support.

As mentioned below, this bug does not manifest when built with `--onedir`.

[debug.log](https://github.com/pyinstaller/pyinstaller/files/10560051/debug.log) contains import and bootloader logging for a failure case. 


Q: Does this happen with "fork"? 
A: No

Q: Does this happen when running from source?
A: No, only when packaged as a pyinstaller executable with `--onefile`.

Q: Does this happen on Windows?
A: Unknown

### Context information (for bug reports)

* Output of `pyinstaller --version`: ```5.7.0```
* Version of Python: Python 3.10.6
* Platform: Ubuntu 22.04.1 LTS
* How you installed Python: apt
* Did you also try this on another platform?
    * Ubuntu 18.04.6 LTS, pyinstaller 4.7, Python 3.7 - Bug is present
    * WSL2 pyinstaller 4.7, Python 3.7 - Bug is present


* try the latest development version, using the following command:

```shell
pip install https://github.com/pyinstaller/pyinstaller/archive/develop.zip
```

* follow *all* the instructions in our "If Things Go Wrong" Guide
  (https://github.com/pyinstaller/pyinstaller/wiki/If-Things-Go-Wrong) and

### Make sure [everything is packaged correctly](https://github.com/pyinstaller/pyinstaller/wiki/How-to-Report-Bugs#make-sure-everything-is-packaged-correctly)

  * [x] start with clean installation
  * [x] use the latest development version
  * [x] Run your frozen program **from a command window (shell)** — instead of double-clicking on it
  * [x] Package your program in **--onedir mode** - **BUG DOES NOT MANIFEST**
  * [x] Package **without UPX**, say: use the option `--noupx` or set `upx=False` in your .spec-file -  - **BUG DOES NOT MANIFEST**
  * [x] Repackage you application in **verbose/debug mode**. For this, pass the option `--debug` to `pyi-makespec` or `pyinstaller` or use `EXE(..., debug=1, ...)` in your .spec file.


### A minimal example program which shows the error

```python
import multiprocessing
import sys
from threading import Thread

DEFAULT_N = 3

def main():
    try:
        n = int(sys.argv[1])
    except IndexError:
        n=DEFAULT_N

    threads = []
    for i in range(n):
        threads.append(Thread(target=foo, args=(i, )))
    for i in range(n):
        threads[i].start()
    for i in range(n):
        threads[i].join()

def foo(i):
    multiprocessing_context = multiprocessing.get_context(method="spawn")
    q = multiprocessing_context.Queue()
    p = multiprocessing_context.Process(target=bar, args=(q, i), daemon=True)
    p.start()
    p.join()


def bar(q, i):
    q.put('hello')
    print(f"{i} Added to queue")

if __name__ == "__main__":
    multiprocessing.freeze_support()
    main()

```

### Stacktrace / full error message

Note: If you can't reproduce the bug, try increasing the parameter from 2 to 5 (or higher).

```
$> dist/multiprocessing_bug 2
0 Added to queue
Traceback (most recent call last):
  File "multiprocessing_bug.py", line 34, in <module>
    multiprocessing.freeze_support()
  File "PyInstaller/hooks/rthooks/pyi_rth_multiprocessing.py", line 49, in _freeze_support
  File "multiprocessing/spawn.py", line 116, in spawn_main
  File "multiprocessing/spawn.py", line 125, in _main
  File "multiprocessing/spawn.py", line 236, in prepare
  File "multiprocessing/spawn.py", line 287, in _fixup_main_from_path
  File "runpy.py", line 288, in run_path
  File "runpy.py", line 252, in _get_code_from_file
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/_MEIOchafX/multiprocessing_bug.py'
[8216] Failed to execute script 'multiprocessing_bug' due to unhandled exception!

```

### Workaround

As shown below, adding a lock around the call to `process.start()` seems to resolve the issue.

```python
import multiprocessing
import sys
from threading import Thread, Lock

DEFAULT_N = 3

def main():
    try:
        n = int(sys.argv[1])
    except IndexError:
        n=DEFAULT_N

    threads = []
    for i in range(n):
        threads.append(Thread(target=foo, args=(i, )))
    for i in range(n):
        threads[i].start()
    for i in range(n):
        threads[i].join()

lock = Lock()
def foo(i):
    multiprocessing_context = multiprocessing.get_context(method="spawn")
    q = multiprocessing_context.Queue()
    p = multiprocessing_context.Process(target=bar, args=(q, i), daemon=True)

    with lock:
        p.start()

    p.join()


def bar(q, i):
    q.put('hello')
    print(f"{i} Added to queue")

if __name__ == "__main__":
    multiprocessing.freeze_support()
    main()

```



Discussion:
Thanks for preliminary analysis and reproducer! 

The problem is indeed lack of thread-safety in our `multiprocess` support. Specifically, `putenv`/`unsetenv` calls from `FrozenSupportMixIn`,
https://github.com/pyinstaller/pyinstaller/blob/d6ff0802dd6f2fbeccb4a6c47e1cf24c61c31205/PyInstaller/hooks/rthooks/pyi_rth_multiprocessing.py#L66-L80
affect the global state and when multiple instances are created (multiple processes are spawned from different threads), they end up racing each other. E.g., the 2nd process' instance sets the environment variable, only for it to be cleared by the 1st process' instance - therefore, the 2nd process ends up unpacking itself into different temporary location, leading to filename/path mismatch during lookup.

I don't think we have any other way of passing custom environment to `multiprocessing.popen_*.Popen`, so we'll probably need to resort to a class-wide `Lock` to force serialization of those environment modifications.

Resolving patch:
diff --git a/PyInstaller/hooks/rthooks/pyi_rth_multiprocessing.py b/PyInstaller/hooks/rthooks/pyi_rth_multiprocessing.py
--- a/PyInstaller/hooks/rthooks/pyi_rth_multiprocessing.py
+++ b/PyInstaller/hooks/rthooks/pyi_rth_multiprocessing.py
@@ -9,6 +9,7 @@
 # SPDX-License-Identifier: Apache-2.0
 #-----------------------------------------------------------------------------
 
+import threading
 import multiprocessing
 import multiprocessing.spawn as spawn
 # 'spawn' multiprocessing needs some adjustments on osx
@@ -64,14 +65,17 @@ def _freeze_support():
 
 # Mix-in to re-set _MEIPASS2 from sys._MEIPASS.
 class FrozenSupportMixIn:
+    _lock = threading.Lock()
+
     def __init__(self, *args, **kw):
-        if hasattr(sys, 'frozen'):
+        # The whole code block needs be executed under a lock to prevent race conditions between `os.putenv` and
+        # `os.unsetenv` calls when processes are spawned concurrently from multiple threads. See #7410.
+        with self._lock:
             # We have to set original _MEIPASS2 value from sys._MEIPASS to get --onefile mode working.
             os.putenv('_MEIPASS2', sys._MEIPASS)  # @UndefinedVariable
-        try:
-            super().__init__(*args, **kw)
-        finally:
-            if hasattr(sys, 'frozen'):
+            try:
+                super().__init__(*args, **kw)
+            finally:
                 # On some platforms (e.g. AIX) 'os.unsetenv()' is not available. In those cases we cannot delete the
                 # variable but only set it to the empty string. The bootloader can handle this case.
                 if hasattr(os, 'unsetenv'):


Test patch: