Repository: spyder-ide/spyder

Issue:
multiprocessing failing when function contains a class on the same file
## Problem Description
The following code executes properly from the command line but not within spyder. The program hangs and the command window show the  traceback pasted below. Placing the class code (everything before ``if __name__ == "__main__")`` in a separate file and importing the class solves the problem - but it is not clear why this code does not work as it does when executing from the command line. 

### What steps reproduce the problem?

```
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
from multiprocessing import Pool
class myClass():
    
    def __init__(self,q_seed=0,*args,**kwargs,) :
        self.q_seed = q_seed
        print(self.q_seed)

def simulateLocPool(kwargs) : 

    m = myClass(**kwargs)  
    return m   

#%%
if __name__ == "__main__":
    
    nboots = 10
    nprocs = 10
    
    # construct list to pass to processes
    iterlist = []
    kwargs = {'q_seed' : 0}
    
    # change seeds
    for bootn in range(nboots):
        newkw = kwargs.copy()
        newkw['q_seed'] = kwargs['q_seed'] + bootn
        iterlist.append((newkw)) 
    
    #spawn nboots processes, nprocs at once    
    pool = Pool(processes=nprocs)
    allmodels = pool.map(simulateLocPool,(iterlist))
    pool.close()
    pool.join()
```
### Output

The code hangs with the following traceback

```python-traceback

Exception in thread Thread-23:
Traceback (most recent call last):
  File "/Users/moroa/opt/miniconda3/lib/python3.9/threading.py", line 954, in _bootstrap_inner
    self.run()
  File "/Users/moroa/opt/miniconda3/lib/python3.9/threading.py", line 892, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/moroa/opt/miniconda3/lib/python3.9/multiprocessing/pool.py", line 576, in _handle_results
    task = get()
  File "/Users/moroa/opt/miniconda3/lib/python3.9/multiprocessing/connection.py", line 256, in recv
    return _ForkingPickler.loads(buf.getbuffer())
AttributeError: Can't get attribute 'myClass' on <module 'spyder_kernels.console.__main__' from '/Users/moroa/opt/miniconda3/lib/python3.9/site-packages/spyder_kernels/console/__main__.py'>

```
### Expected output
A printout of digits from 0 to 9

## Versions
* Spyder 5.1.5|Python 3.9.5 64-bit | Qt 5.12.10 | PyQt5 5.12.3 | Darwin 20.6.0 
* Operating System name/version: MacOS 11.5.2




Discussion:
Hi @andreamoro-git,

Thanks for opening this issue, I could reproduce the problem with the snippet you provided. @jitseniesen, @ccordoba12 do you know why this is happening?
fwiw, I have always directed people towards using "run in external console" when using multiprocessing due to a number of issues. Chiefly among them is that stdout is not always correctly handled (no child print statements in ipython console). The way to debug things like this is then to start spyder like so: `python -m spyder.app.start`, and all child output will go to the terminal, while main process output goes to the ipython console.

Doing this shows that the code is being run in the child, so \_\_main__ is clearly importable in the child, and confirms what the error message suggests (go figure....) that the error occurs when un-pickling the`myClass` instances sent back from the children. I also took a look at `sys.modules` in various scenarios, and while I don't necessarily understand the difference... `__main__` shows up in two different ways: 

```
'__main__': <module '__main__' (built-in)>
or
'__main__': <module '__main__' from 'C:\\Users\\aaron\\Documents\\scripts\\python\\untitled0.py'>
```
So, @andreamoro-git and @athompson673, if you try to run the snippet of code in an ipython console, the error appears?
I think @athompson673 meant "Execute in an external system terminal". 
I don't know why this is not working, but

* If you're running your code in an unsaved file (as @athompson673 seems to be doing), it could be related to this because `multiprocessing` could have problems pickling code in that situation.
* The work @impact27 is doing in PR #14025 could help at least to debug these problems.
The file is saved
So I checked this issue. I can reproduce with iPython. If you:
 - `import multiprocessing.spawn`
 - then run the file
You will get: `AttributeError: Can't get attribute 'myClass' on <module '__main__'>`

This might be a python error?


```
Desktop % ipython3
Python 3.8.6 (default, Oct  8 2020, 14:06:32) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.29.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import multiprocessing.spawn

In [2]: %run untitled1.py
0
1
Exception in thread Thread-3:
2
Traceback (most recent call last):
  File "/usr/local/Cellar/python@3.8/3.8.6/Frameworks/Python.framework/Versions/3.8/lib/python3.8/threading.py", line 932, in _bootstrap_inner
3
4
5
6
7
8
9
    self.run()
  File "/usr/local/Cellar/python@3.8/3.8.6/Frameworks/Python.framework/Versions/3.8/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/Cellar/python@3.8/3.8.6/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/pool.py", line 576, in _handle_results
    task = get()
  File "/usr/local/Cellar/python@3.8/3.8.6/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/connection.py", line 251, in recv
    return _ForkingPickler.loads(buf.getbuffer())
AttributeError: Can't get attribute 'myClass' on <module '__main__'>
```
The problem here is that spyder-kernels imports `multiprocessing.spawn`. But this should not fail just because a module was imported!
Thanks @impact27 for your help with this!

This will be fixed in our next version (5.2.2), to be released next week.

Resolving patch:
diff --git a/external-deps/spyder-kernels/spyder_kernels/customize/spydercustomize.py b/external-deps/spyder-kernels/spyder_kernels/customize/spydercustomize.py
--- a/external-deps/spyder-kernels/spyder_kernels/customize/spydercustomize.py
+++ b/external-deps/spyder-kernels/spyder_kernels/customize/spydercustomize.py
@@ -549,6 +549,14 @@ def runfile(filename=None, args=None, wdir=None, namespace=None,
         if args is not None:
             for arg in shlex.split(args):
                 sys.argv.append(arg)
+
+        if "multiprocessing" in sys.modules:
+            # See https://github.com/spyder-ide/spyder/issues/16696
+            try:
+                sys.modules['__mp_main__'] = sys.modules['__main__']
+            except Exception:
+                pass
+
         if wdir is not None:
             if PY2:
                 try:


Test patch:
diff --git a/external-deps/spyder-kernels/spyder_kernels/console/tests/test_console_kernel.py b/external-deps/spyder-kernels/spyder_kernels/console/tests/test_console_kernel.py
--- a/external-deps/spyder-kernels/spyder_kernels/console/tests/test_console_kernel.py
+++ b/external-deps/spyder-kernels/spyder_kernels/console/tests/test_console_kernel.py
@@ -479,6 +479,54 @@ def f(x):
         assert content['found']
 
 
+@flaky(max_runs=3)
+@pytest.mark.skipif(not PY3,
+                    reason="Only meant for Python 3")
+def test_multiprocessing_2(tmpdir):
+    """
+    Test that multiprocessing works on Python 3.
+    """
+    # Command to start the kernel
+    cmd = "from spyder_kernels.console import start; start.main()"
+
+    with setup_kernel(cmd) as client:
+        # Remove all variables
+        client.execute("%reset -f")
+        client.get_shell_msg(timeout=TIMEOUT)
+
+        # Write multiprocessing code to a file
+        code = """
+from multiprocessing import Pool
+
+class myClass():
+    def __init__(self, i):
+        self.i = i + 10
+
+def myFunc(i):
+    return myClass(i)
+
+if __name__ == '__main__':
+    with Pool(5) as p:
+        result = p.map(myFunc, [1, 2, 3])
+    result = [r.i for r in result]
+"""
+        p = tmpdir.join("mp-test.py")
+        p.write(code)
+
+        # Run code
+        client.execute("runfile(r'{}')".format(to_text_string(p)))
+        client.get_shell_msg(timeout=TIMEOUT)
+
+        # Verify that the `result` variable is defined
+        client.inspect('result')
+        msg = client.get_shell_msg(timeout=TIMEOUT)
+        while "found" not in msg['content']:
+            msg = client.get_shell_msg(timeout=TIMEOUT)
+        content = msg['content']
+        assert content['found']
+        assert "[11, 12, 13]" in content['data']['text/plain']
+
+
 @flaky(max_runs=3)
 @pytest.mark.skipif(not PY3,
                     reason="Only meant for Python 3")