Repository: spyder-ide/spyder

Issue:
Cannot use imported modules with joblib
## Description

### What steps will reproduce the problem?

<!--- You can use Markdown here --->

Run joblib with an imported local function. This will generate an error "ModuleNotFoundError: No module named 'MyOwnModule' ending up with 'BrokenProcessPool: A task has failed to un-serialize. Please ensure that the arguments of the function are all picklable.'

For example, save:

    import time
    
    def slow_mean(data, sl):
        """Simulate a time consuming processing."""
        time.sleep(0.01)
        return data[sl].mean()

to slowmean.py and then try to use it with joblib:

    # from https://joblib.readthedocs.io/en/latest/auto_examples/parallel_memmap.html
    import numpy as np
    import time
    from joblib import Parallel, delayed
    import slowmean as sm

    data = np.random.random((int(1e7),))
    window_size = int(5e5)
    slices = [slice(start, start + window_size) for start in range(0, data.size - window_size, int(1e5))]

    tic = time.time()
    results = Parallel(n_jobs=2)(delayed(sm.slow_mean)(data, sl) for sl in slices)
    toc = time.time()
    print('\nElapsed time computing the average of couple of slices {:.2f} s'.format(toc - tic))

The error does not appear if the above is run from terminal or if the function (in this example 'slow_mean()') is defined directly instead of being imported. The problem does not occur with Spyder v3.3.6 but does occur in v4.0.0 and v4.1.4 (I haven't tested other versions in between though).

However, the above example will run without error if you first manually restart the kernel from the IPython console before doing anything else. So I therefore think this is probably a Spyder problem rather than a joblib problem.

I am running Linux and have seen this on various Python versions from 3.6 to 3.8.5 too.

### Traceback

<details>

```python-traceback
  File "/home/dave/envCAML_py385/lib/python3.8/site-packages/qtconsole/comms.py", line 177, in comm_msg
    comm.handle_msg(msg)
  File "/home/dave/envCAML_py385/lib/python3.8/site-packages/qtconsole/comms.py", line 265, in handle_msg
    return self._msg_callback(msg)
  File "/home/dave/envCAML_py385/lib/python3.8/site-packages/spyder_kernels/comms/commbase.py", line 330, in _comm_message
    self._message_handlers[spyder_msg_type](
TypeError: _handle_remote_call_reply() missing 1 required positional argument: 'load_exception'
ERROR:traitlets:Exception in comm_msg for c64971a0e78c11ea814f83083ff53143
Traceback (most recent call last):
  File "/home/dave/envCAML_py385/lib/python3.8/site-packages/qtconsole/comms.py", line 177, in comm_msg
    comm.handle_msg(msg)
  File "/home/dave/envCAML_py385/lib/python3.8/site-packages/qtconsole/comms.py", line 265, in handle_msg
    return self._msg_callback(msg)
  File "/home/dave/envCAML_py385/lib/python3.8/site-packages/spyder_kernels/comms/commbase.py", line 330, in _comm_message
    self._message_handlers[spyder_msg_type](
TypeError: _handle_remote_call_reply() missing 1 required positional argument: 'load_exception'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/dave/envCAML_py385/lib/python3.8/site-packages/qtconsole/comms.py", line 177, in comm_msg
    comm.handle_msg(msg)
  File "/home/dave/envCAML_py385/lib/python3.8/site-packages/qtconsole/comms.py", line 265, in handle_msg
    return self._msg_callback(msg)
  File "/home/dave/envCAML_py385/lib/python3.8/site-packages/spyder_kernels/comms/commbase.py", line 330, in _comm_message
    self._message_handlers[spyder_msg_type](
TypeError: _handle_remote_call_reply() missing 1 required positional argument: 'load_exception'
ERROR:traitlets:Exception in comm_msg for c64971a0e78c11ea814f83083ff53143
Traceback (most recent call last):
  File "/home/dave/envCAML_py385/lib/python3.8/site-packages/qtconsole/comms.py", line 177, in comm_msg
    comm.handle_msg(msg)
  File "/home/dave/envCAML_py385/lib/python3.8/site-packages/qtconsole/comms.py", line 265, in handle_msg
    return self._msg_callback(msg)
  File "/home/dave/envCAML_py385/lib/python3.8/site-packages/spyder_kernels/comms/commbase.py", line 330, in _comm_message
    self._message_handlers[spyder_msg_type](
TypeError: _handle_remote_call_reply() missing 1 required positional argument: 'load_exception'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/dave/envCAML_py385/lib/python3.8/site-packages/qtconsole/comms.py", line 177, in comm_msg
    comm.handle_msg(msg)
  File "/home/dave/envCAML_py385/lib/python3.8/site-packages/qtconsole/comms.py", line 265, in handle_msg
    return self._msg_callback(msg)
  File "/home/dave/envCAML_py385/lib/python3.8/site-packages/spyder_kernels/comms/commbase.py", line 330, in _comm_message
    self._message_handlers[spyder_msg_type](
TypeError: _handle_remote_call_reply() missing 1 required positional argument: 'load_exception'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/dave/envCAML_py385/lib/python3.8/site-packages/qtconsole/comms.py", line 177, in comm_msg
    comm.handle_msg(msg)
  File "/home/dave/envCAML_py385/lib/python3.8/site-packages/qtconsole/comms.py", line 265, in handle_msg
    return self._msg_callback(msg)
  File "/home/dave/envCAML_py385/lib/python3.8/site-packages/spyder_kernels/comms/commbase.py", line 330, in _comm_message
    self._message_handlers[spyder_msg_type](
TypeError: _handle_remote_call_reply() missing 1 required positional argument: 'load_exception'
```

</details>

## Versions

* Spyder version: 4.0.0 
* Python version: 3.8.5
* Qt version: 5.12.9
* PyQt5 version: 5.12.3
* Operating System: Linux 5.4.0-42-generic

### Dependencies

```
cloudpickle >=0.5.0          :  1.5.0 (OK)
pygments >=2.0               :  2.6.1 (OK)
qtconsole >=4.6.0            :  4.7.6 (OK)
nbconvert >=4.0              :  5.6.1 (OK)
sphinx >=0.6.6               :  3.2.1 (OK)
pylint >=0.25                :  2.6.0 (OK)
psutil >=0.3                 :  5.7.2 (OK)
qtawesome >=0.5.7            :  0.7.2 (OK)
qtpy >=1.5.0                 :  1.9.0 (OK)
pickleshare >=0.4            :  0.7.5 (OK)
zmq >=17                     :  19.0.2 (OK)
chardet >=2.0.0              :  3.0.4 (OK)
numpydoc >=0.6.0             :  1.1.0 (OK)
spyder_kernels >=1.8.1;<2.0.0:  1.9.3 (OK)
qdarkstyle >=2.7             :  2.8.1 (OK)
atomicwrites >=1.2.0         :  1.4.0 (OK)
diff_match_patch >=20181111  :  20200713 (OK)
intervaltree                 :  None (OK)
watchdog                     :  None (OK)
keyring                      :  None (OK)
pexpect >=4.4.0              :  4.8.0 (OK)
pympler                      :  None (OK)
sympy >=0.7.3                :  1.6.2 (OK)
cython >=0.21                :  0.29.21 (OK)
IPython >=4.0                :  7.17.0 (OK)
matplotlib >=2.0.0           :  3.3.1 (OK)
pandas >=0.13.1              :  1.1.1 (OK)
numpy >=1.7                  :  1.18.5 (OK)
scipy >=0.17.0               :  1.4.1 (OK)
pyls >=0.31.2;<0.32.0        :  0.31.10 (OK)
xdg >=0.26                   :  0.26 (OK)
```



Discussion:
Thanks for reporting. This issue is fixed in our latest Spyder 4 version. Please update by opening the Anaconda Prompt and running there:<br><br>`conda install spyder=4.1.*`
As I said, I still get this error in v4.1.4, which is the latest release. It also occurs in 5.0.0a1. I have installed these using pip; I am not using Anaconda.
@quokka79, thanks for reporting and the reproducible example as well. I managed to run your main script by going to the menu `Run > Configuration per file` and selecting the option called `Run in console's namespace instead of an empty one`:

![Selección_053](https://user-images.githubusercontent.com/365293/91329902-570c0800-e78e-11ea-9439-29b058d864bd.png)

That was added in Spyder 4 to prevent picking up already defined variables in the console when running your files.

Please try that and let us know if it works for you.
@ccordoba12 thanks for looking into this!

I have just tried the 'Configuration per file' option using the console's namespace (in v4.1.4) but it still gives the error as before. 

If I restart 'Console 1/A' kernel or if I choose 'Execute in a dedicated console' then it runs without any problems.
Yes, it's very strange. Things don't work for the first console created by Spyder, but they do for any other console or after restarting the kernel.

@impact27, any ideas about this?
I have the same issue/observation. Is it possible that the current working directory (initially set to the user's home folder) or some search path is not properly updated to the script/project directory when the script is first run on the first console?
This issue is not easily resolved. I think `multiprocessing` doesn't conserve the current working directory.
It looks like the cwd of the process is actually the initial cwd where python was launched.
You can recreate the issure by:
 - launch spyder, the console cwd is home 
 - test joblib -> doesn't work, but change the CWD to the current folder
 - restart the console. The console CWD is now the folder you expect
 - test joblib -> works
 - change CWD
 - restart the console. The console CWD is now the another folder
 - test joblib -> doesn't work
I wouldn't rely of the CWD for multiprocessing. Try to either:
 - set the search path manually with os
 - create a package that you can install with `pip install -e .`  <- best way
 - put everything in the same file
To fix it, add this to your code:

```
import multiprocessing.process
multiprocessing.process.ORIGINAL_DIR = os.path.abspath(os.getcwd())
```
Thanks for your help @impact27! Any chance to make this fix work automatically in `spyder-kernels`?
https://github.com/spyder-ide/spyder-kernels/pull/244

Thanks @impact27 for your help!

Resolving patch:
diff --git a/external-deps/spyder-kernels/spyder_kernels/customize/spydercustomize.py b/external-deps/spyder-kernels/spyder_kernels/customize/spydercustomize.py
--- a/external-deps/spyder-kernels/spyder_kernels/customize/spydercustomize.py
+++ b/external-deps/spyder-kernels/spyder_kernels/customize/spydercustomize.py
@@ -522,14 +522,22 @@ def runfile(filename=None, args=None, wdir=None, namespace=None,
             for arg in shlex.split(args):
                 sys.argv.append(arg)
         if wdir is not None:
-            try:
-                wdir = wdir.decode('utf-8')
-            except (UnicodeError, TypeError, AttributeError):
-                # UnicodeError, TypeError --> eventually raised in Python 2
-                # AttributeError --> systematically raised in Python 3
-                pass
+            if PY2:
+                try:
+                    wdir = wdir.decode('utf-8')
+                except (UnicodeError, TypeError):
+                    # UnicodeError, TypeError --> eventually raised in Python 2
+                    pass
             if os.path.isdir(wdir):
                 os.chdir(wdir)
+                # See https://github.com/spyder-ide/spyder/issues/13632
+                if "multiprocessing.process" in sys.modules:
+                    try:
+                        import multiprocessing.process
+                        multiprocessing.process.ORIGINAL_DIR = os.path.abspath(
+                            wdir)
+                    except Exception:
+                        pass
             else:
                 _print("Working directory {} doesn't exist.\n".format(wdir))
 


Test patch: