Repository: pyqtgraph/pyqtgraph

Issue:
Multiprocess does not work in a venv
<!-- In the following, please describe your issue in detail! -->
<!-- If some of the sections do not apply, just remove them. -->

### Short description
The multiprocessing module does not work in a python3 venv on windows. This is related to the use of Popen to create a new python instance, and incorrect detection of PID

### Code to reproduce
1. Create a new venv:
```bash
$ mkdir tempvenv
$ cd tempvenv
$ python3 -m venv .
```
2. Try to spawn a multiprocess version of pyqtgraph
```python
import pyqtgraph as pg
import pyqtgraph.multiprocess as mp
app = pg.mkQApp()
rp = mp.QtProcess()
ros = rp._import("os")
```

### Expected behavior
The last line should return a remote proxy to the os module on the remote python instance.

### Real behavior
We get a key error:
```pytb
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
c:\users\x\documents\pyqtgraph\pyqtgraph\multiprocess\remoteproxy.py in handleRequest(self)
    210             print(optStr)
    211             print(os.getpid())
--> 212             opts = pickle.loads(optStr)
    213             self.debugMsg("    handleRequest: id=%s opts=%s", reqId, opts)
    214             #print os.getpid(), "received request:", cmd, reqId, opts

c:\users\x\documents\pyqtgraph\pyqtgraph\multiprocess\remoteproxy.py in unpickleObjectProxy(processId, proxyId, typeStr, attributes, opts)
    735         return obj
    736     else:
--> 737         proxy = ObjectProxy(processId, proxyId=proxyId, typeStr=typeStr)
    738         if opts is not None:
    739             proxy._setProxyOptions(**opts)

c:\users\x\documents\pyqtgraph\pyqtgraph\multiprocess\remoteproxy.py in __init__(self, processId, proxyId, typeStr, parent)
    805         }
    806
--> 807         self.__dict__['_handler'] = RemoteEventHandler.getHandler(processId)
    808         self.__dict__['_handler'].registerProxy(self)  ## handler will watch proxy; inform remote process when the proxy is deleted.
    809

c:\users\x\documents\pyqtgraph\pyqtgraph\multiprocess\remoteproxy.py in getHandler(cls, pid)
     93     def getHandler(cls, pid):
     94         try:
---> 95             return cls.handlers[pid]
     96         except:
     97             print(pid, cls.handlers)

KeyError: 12608
```

### Tested environment(s)

 * PyQtGraph version: 0.11.0.dev0 (ed6586c7)
 * Qt Python binding: PyQt5 5.12.3 Qt 5.12.5
 * Python version: Python 3.7.4
 * Operating system: Windows 10

### Additional context
The reason this occurs is because in a venv on windows, python creates a stub executable in the venv, which then spawns a real instance of python.

This causes the detected PID of the child process to not match with the child process stored in `RemoteEventHandler.handlers`.

We can see this with a minimal example on windows. Running inside the venv:
```python
import os, sys, time
from subprocess import Popen

def info(title):
    print(title)
    print('module name:', __name__)
    print('parent process:', os.getppid())
    print('process id:', os.getpid())

if __name__ == '__main__':
    if len(sys.argv) == 1:
        info("Parent Process")
        p = Popen((sys.executable, __file__, "proc"))
        print("Detected Child PID:", p.pid)
        while p.poll() is None:
            time.sleep(1)
    else:
        info("Child Process")
```
I get the output:
```
Parent Process
module name: __main__
parent process: 10220
process id: 17508
Detected Child PID: 10100
Child Process
module name: __main__
parent process: 10100
process id: 17480
```
Note that the detected child PID is 10100, while the child itself thinks it has the PID 17480. The child's parent process is 10100, which is the PID of the stub `python.exe` that venv creates.

### Potential Solutions
- Using `multiprocessing.Process` does not have this issue.
- Send the child PID back once the child has spawned.


Discussion:
@spauka Any idea on how to solve this? With recent installations using `virtualenv` (even with the option `--copies`) results in the same error
@eendebakpt As mentioned above the easiest solution is to use `multiprocessing.Process` to spawn the child process. I've implemented this as a wrapper in my personal scripts here: https://github.com/QNLSydney/qcodes_measurements/blob/master/qcodes_measurements/plot/multiprocess.py

I'll prepare a pull request in the next few days that uses this solution, but I will need to test further to ensure that this doesn't break backwards compatibility.
@spauka Sounds good. Let me know when a PR is available so I can test as well.
@spauka I tested the remote plotting demo app on windows venv and got the same issue.
@spauka if you think you have a fix and would be up for submitting a PR, that would be greatly appreciated!

Some links that were posted on a duplicate issue I opened:

https://stackoverflow.com/questions/61290972/popen-creates-one-additional-process-in-venv
https://bugs.python.org/issue38905

Resolving patch:
diff --git a/pyqtgraph/multiprocess/processes.py b/pyqtgraph/multiprocess/processes.py
--- a/pyqtgraph/multiprocess/processes.py
+++ b/pyqtgraph/multiprocess/processes.py
@@ -122,13 +122,24 @@ def __init__(self, name=None, target=None, executable=None, copySysPath=True, de
         targetStr = pickle.dumps(target)  ## double-pickle target so that child has a chance to 
                                           ## set its sys.path properly before unpickling the target
         pid = os.getpid() # we must send pid to child because windows does not have getppid
-        
+
+        # When running in a venv on Windows platform, since Python >= 3.7.3, the launched
+        # subprocess is a grandchild instead of a child, leading to self.proc.pid not being
+        # the pid of the launched subprocess.
+        # https://bugs.python.org/issue38905
+        #
+        # As a workaround, when we detect such a situation, we perform exchange of pids via
+        # the multiprocessing connection. Technically, only the launched subprocess needs to
+        # send its pid back. Practically, we hijack the ppid parameter to indicate to the
+        # subprocess that pid exchange is needed.
+        xchg_pids = sys.platform == 'win32' and os.getenv('VIRTUAL_ENV') is not None
+
         ## Send everything the remote process needs to start correctly
         data = dict(
             name=name+'_child', 
             port=port, 
             authkey=authkey, 
-            ppid=pid, 
+            ppid=pid if not xchg_pids else None,
             targetStr=targetStr, 
             path=sysPath, 
             qt_lib=QT_LIB,
@@ -150,7 +161,14 @@ def __init__(self, name=None, target=None, executable=None, copySysPath=True, de
                 else:
                     raise
 
-        RemoteEventHandler.__init__(self, conn, name+'_parent', pid=self.proc.pid, debug=self.debug)
+        child_pid = self.proc.pid
+        if xchg_pids:
+            # corresponding code is in:
+            #   remoteproxy.py::RemoteEventHandler.__init__()
+            conn.send(pid)
+            child_pid = conn.recv()
+
+        RemoteEventHandler.__init__(self, conn, name+'_parent', pid=child_pid, debug=self.debug)
         self.debugMsg('Connected to child process.')
         
         atexit.register(self.join)
diff --git a/pyqtgraph/multiprocess/remoteproxy.py b/pyqtgraph/multiprocess/remoteproxy.py
--- a/pyqtgraph/multiprocess/remoteproxy.py
+++ b/pyqtgraph/multiprocess/remoteproxy.py
@@ -86,6 +86,13 @@ def __init__(self, connection, name, pid, debug=False):
         # Mutexes to help prevent issues when multiple threads access the same RemoteEventHandler
         self.processLock = threading.RLock()
         self.sendLock = threading.RLock()
+
+        # parent sent us None as its pid, wants us to exchange pids
+        # corresponding code is in:
+        #   processes.py::Process.__init__()
+        if pid is None:
+            connection.send(os.getpid())
+            pid = connection.recv()
         
         RemoteEventHandler.handlers[pid] = self  ## register this handler as the one communicating with pid
     


Test patch: