Repository: spyder-ide/spyder

Issue:
Code completion breaks with  Spyder 3 for modules added through the Python path manager
## Description

**What steps will reproduce the problem?**
1.  Jedi cannot starts auto-complete on custom module, some modules (numpy, pandas can auto-complete).
2. 
3. 

**What is the expected output? What do you see instead?**
Spyder 2.3  and Python 2.7 work well and Jedi auto-complete does not break.

**Please provide any additional information below**


## Version and main components
* Spyder Version: 3.0.0 
* Python Version: 2.7.12
* Qt Versions: 5.6.0, PyQt5 5.6 on Windows

## Dependencies
```
jedi >=0.8.1     :  0.8.1 (OK)
matplotlib >=1.0 :  1.5.3 (OK)
nbconvert >=4.0  :  4.2.0 (OK)
numpy >=1.7      :  1.11.3 (OK)
pandas >=0.13.1  :  0.19.2 (OK)
pep8 >=0.6       :  1.7.0 (OK)
pyflakes >=0.5.0 :  1.3.0 (OK)
pygments >=2.0   :  2.1.3 (OK)
pylint >=0.25    :  1.5.4 (OK)
qtconsole >=4.2.0:  4.2.1 (OK)
rope >=0.9.4     :  0.9.4 (OK)
sphinx >=0.6.6   :  1.4.6 (OK)
sympy >=0.7.3    :  1.0 (OK)


```



Discussion:
Please update Jedi to 0.9.0 and try again (Spyder doesn't work with Jedi 0.10.0 yet).
Jedi 0.9.0 seems not working at all.

Jedi 0.8.1 works in this case:
    module is inside sites-packages  (like numpy)

If I add my own custom directory in the Python Path manager
  and do import myModule as mm
it does not work....

It used to work with Python 2.7 and Spyder 2.3.1.....

How to investigate the break ?






> Jedi 0.9.0 seems not working at all.

That's very strange. All our tests are passing with Jedi 0.9.

> If I add my own custom directory in the Python Path manager
and do import myModule as mm

Thanks for the example. We'll take a look at it.

> It used to work with Python 2.7 and Spyder 2.3.1

Jedi support was introduced in Spyder 2.3.3, so completions were provided by Rope, not Jedi.
In this configuration, Jedi works well with custom packages from PythonPathManager:


```
Spyder 2.3.9 
The Scientific PYthon Development EnviRonment 
Win7 64bits - Anaconda
Python 2.7.12 64bits, Qt 4.8.7, PyQt4 (API v2) 4.11.4 on Windows

IPython >=1.0      :  4.2.0 (OK)
jedi >=0.8.1;<0.9.0:  0.8.1 (OK)
matplotlib >=1.0   :  1.5.1 (OK)
pandas >=0.13.1    :  0.19.2 (OK)
pep8 >=0.6         :  1.7.0 (OK)
pyflakes >=0.5.0   :  1.2.3 (OK)
pygments >=1.6     :  2.1.3 (OK)
pylint >=0.25      :  None (NOK)
qtconsole >=4.0    :  4.2.1 (OK)
rope >=0.9.2       :  0.9.4 (OK)
sphinx >=0.6.6     :  1.4.1 (OK)
sympy >=0.7.3      :  None (NOK)
zmq >=2.1.11       :  15.2.0 (OK)
```

@rlaverde, please take a look at this one.
It seems like PythonCustomPath are not loaded before the start of Jedi....
So Jedi cannot know the PythonCustompath to load the packages....


Test is simple:
   numpy package or anypackages in sites-package works well.

if you add a folder in Python Custom Folder and import a module from this folder:
   does not work.

---->  It seems module were not loaded in Jedi.....




We're probably not adding Spyder's PYTHONPATH to the list of paths used by Jedi.
I found the jedi plugin : 
https://github.com/spyder-ide/spyder/blob/539cb5ca584b75f0c21333de9fa7909729b423bc/spyder/utils/introspection/jedi_plugin.py

Could not find where Spyder JEDI  is loading the Spyder's PYTHONPATH
Probably JEDI is launched before Spyder load custome folder.

Do you have an idea where it can be located in Spyder code (I can do a small workaround).






In Jedi plugin, 

Can we add PYTHONPath here in that part ?

```
    def load_plugin(self):
        """Load the Jedi introspection plugin"""
        if not programs.is_module_installed('jedi', JEDI_REQVER):
            raise ImportError('Requires Jedi %s' % JEDI_REQVER)
        jedi.settings.case_insensitive_completion = False
        for lib in ['numpy', 'matplotlib', 'pandas', 'mypackage']:
            jedi.preload_module(lib)
```
That's not enough if changes are made to Spyder's PYTHONPATH while it's running. @rlaverde is working now on a better solution.

Resolving patch:
diff --git a/spyder/plugins/editor.py b/spyder/plugins/editor.py
--- a/spyder/plugins/editor.py
+++ b/spyder/plugins/editor.py
@@ -426,7 +426,8 @@ def __init__(self, parent, ignore_last_opened_files=False):
         self.toolbar_list = None
         self.menu_list = None
 
-        self.introspector = IntrospectionManager()
+        self.introspector = IntrospectionManager(
+                extra_path=self.main.get_spyder_pythonpath())
 
         # Setup new windows:
         self.main.all_actions_defined.connect(self.setup_other_windows)
@@ -1462,6 +1463,9 @@ def set_current_filename(self, filename, editorwindow=None):
     def set_path(self):
         for finfo in self.editorstacks[0].data:
             finfo.path = self.main.get_spyder_pythonpath()
+        if self.introspector:
+            self.introspector.change_extra_path(
+                    self.main.get_spyder_pythonpath())
     
     #------ Refresh methods
     def refresh_file_dependent_actions(self):
diff --git a/spyder/utils/introspection/manager.py b/spyder/utils/introspection/manager.py
--- a/spyder/utils/introspection/manager.py
+++ b/spyder/utils/introspection/manager.py
@@ -43,12 +43,13 @@ class PluginManager(QObject):
 
     introspection_complete = Signal(object)
 
-    def __init__(self, executable):
+    def __init__(self, executable, extra_path=None):
+
         super(PluginManager, self).__init__()
         plugins = OrderedDict()
         for name in PLUGINS:
             try:
-                plugin = PluginClient(name, executable)
+                plugin = PluginClient(name, executable, extra_path=extra_path)
                 plugin.run()
             except Exception as e:
                 debug_print('Introspection Plugin Failed: %s' % name)
@@ -129,7 +130,9 @@ def handle_response(self, response):
             self.pending = response
 
     def close(self):
-        [plugin.close() for plugin in self.plugins]
+        for name, plugin in self.plugins.items():
+            plugin.close()
+            debug_print("Introspection Plugin Closed: {}".format(name))
 
     def _finalize(self, response):
         self.waiting = False
@@ -160,17 +163,29 @@ class IntrospectionManager(QObject):
     send_to_help = Signal(str, str, str, str, bool)
     edit_goto = Signal(str, int, str)
 
-    def __init__(self, executable=None):
+    def __init__(self, executable=None, extra_path=None):
         super(IntrospectionManager, self).__init__()
         self.editor_widget = None
         self.pending = None
-        self.plugin_manager = PluginManager(executable)
+        self.extra_path = extra_path
+        self.executable = executable
+        self.plugin_manager = PluginManager(executable, extra_path)
         self.plugin_manager.introspection_complete.connect(
             self._introspection_complete)
 
     def change_executable(self, executable):
+        self.executable = executable
+        self._restart_plugin()
+
+    def change_extra_path(self, extra_path):
+        if extra_path != self.extra_path:
+            self.extra_path = extra_path
+            self._restart_plugin()
+
+    def _restart_plugin(self):
         self.plugin_manager.close()
-        self.plugin_manager = PluginManager(executable)
+        self.plugin_manager = PluginManager(self.executable,
+                                            extra_path=self.extra_path)
         self.plugin_manager.introspection_complete.connect(
             self._introspection_complete)
 
diff --git a/spyder/utils/introspection/plugin_client.py b/spyder/utils/introspection/plugin_client.py
--- a/spyder/utils/introspection/plugin_client.py
+++ b/spyder/utils/introspection/plugin_client.py
@@ -18,7 +18,7 @@
 import zmq
 
 # Local imports
-from spyder.config.base import debug_print, DEV, get_module_path
+from spyder.config.base import debug_print, get_module_path
 
 
 # Heartbeat timer in milliseconds
@@ -41,7 +41,8 @@ class AsyncClient(QObject):
     received = Signal(object)
 
     def __init__(self, target, executable=None, name=None,
-                 extra_args=None, libs=None, cwd=None, env=None):
+                 extra_args=None, libs=None, cwd=None, env=None,
+                 extra_path=None):
         super(AsyncClient, self).__init__()
         self.executable = executable or sys.executable
         self.extra_args = extra_args
@@ -50,8 +51,11 @@ def __init__(self, target, executable=None, name=None,
         self.libs = libs
         self.cwd = cwd
         self.env = env
+        self.extra_path = extra_path
         self.is_initialized = False
         self.closing = False
+        self.notifier = None
+        self.process = None
         self.context = zmq.Context()
         QApplication.instance().aboutToQuit.connect(self.close)
 
@@ -77,7 +81,7 @@ def run(self):
         # Set up environment variables.
         processEnvironment = QProcessEnvironment()
         env = self.process.systemEnvironment()
-        if (self.env and 'PYTHONPATH' not in self.env) or DEV:
+        if (self.env and 'PYTHONPATH' not in self.env) or self.env is None:
             python_path = osp.dirname(get_module_path('spyder'))
             # Add the libs to the python path.
             for lib in self.libs:
@@ -86,6 +90,13 @@ def run(self):
                     python_path = osp.pathsep.join([python_path, path])
                 except ImportError:
                     pass
+            if self.extra_path:
+                try:
+                    python_path = osp.pathsep.join([python_path] +
+                                                   self.extra_path)
+                except Exception as e:
+                    debug_print("Error when adding extra_path to plugin env")
+                    debug_print(e)
             env.append("PYTHONPATH=%s" % python_path)
         if self.env:
             env.update(self.env)
@@ -129,12 +140,17 @@ def close(self):
         self.closing = True
         self.is_initialized = False
         self.timer.stop()
-        self.notifier.activated.disconnect(self._on_msg_received)
-        self.notifier.setEnabled(False)
-        del self.notifier
+
+        if self.notifier is not None:
+            self.notifier.activated.disconnect(self._on_msg_received)
+            self.notifier.setEnabled(False)
+            self.notifier = None
+
         self.request('server_quit')
-        self.process.waitForFinished(1000)
-        self.process.close()
+
+        if self.process is not None:
+            self.process.waitForFinished(1000)
+            self.process.close()
         self.context.destroy()
 
     def _on_finished(self):
@@ -192,11 +208,13 @@ def _send(self, obj):
 
 class PluginClient(AsyncClient):
 
-    def __init__(self, plugin_name, executable=None, env=None):
+    def __init__(self, plugin_name, executable=None, env=None,
+                 extra_path=None):
         cwd = os.path.dirname(__file__)
         super(PluginClient, self).__init__('plugin_server.py',
             executable=executable, cwd=cwd, env=env,
-            extra_args=[plugin_name], libs=[plugin_name])
+            extra_args=[plugin_name], libs=[plugin_name],
+            extra_path=extra_path)
         self.name = plugin_name
 
 


Test patch:
diff --git a/spyder/utils/introspection/tests/test_plugin_client.py b/spyder/utils/introspection/tests/test_plugin_client.py
new file mode 100644
--- /dev/null
+++ b/spyder/utils/introspection/tests/test_plugin_client.py
@@ -0,0 +1,42 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright © Spyder Project Contributors
+# Licensed under the terms of the MIT License
+#
+"""Tests for plugin_client.py."""
+
+# Standard library imports
+import os.path as osp
+
+# Test library imports
+import pytest
+
+# Local imports
+from spyder.utils.introspection.plugin_client import PluginClient
+from spyder.utils.introspection.manager import PLUGINS
+
+
+@pytest.mark.parametrize("plugin_name", PLUGINS)
+def test_plugin_client(qtbot, plugin_name):
+    """Test creation of the diferent plugin clients."""
+    plugin = PluginClient(plugin_name=plugin_name)
+
+    assert plugin
+
+
+@pytest.mark.parametrize("plugin_name", PLUGINS)
+def test_plugin_client_extra_path(qtbot, plugin_name):
+    """Test adding of extra path.
+
+    Extra path is used for adding spyder_path to plugin clients.
+    """
+    extra_path = '/some/dummy/path'
+
+    plugin = PluginClient(plugin_name=plugin_name, extra_path=[extra_path])
+    plugin.run()
+    python_path = plugin.process.processEnvironment().value('PYTHONPATH')
+    assert extra_path in python_path.split(osp.pathsep)
+
+
+if __name__ == "__main__":
+    pytest.main()