Repository: spyder-ide/spyder

Issue:
Kite completions not being shown
<!--- **PLEASE READ:** When submitting here, please ensure you've completed the following checklist and checked the boxes to confirm. Issue reports without it may be closed. Thanks! --->


## Problem Description
Sometimes Kite completions aren't shown even if they are available and if Kite has returned faster than LSP.


### What steps reproduce the problem?

1. Enable Kite
2. Type the following in a file `import matplotlib.py`
3. Request completions manually if necessary

### What is the expected output? What do you see instead?
Based on the logs, I would expect to see completions from Kite since Kite returned faster than LSP:

```
2019-10-17 15:27:25,235 [DEBUG] [spyder.plugins.completion.languageserver.client] -> python request: textDocument/completion
2019-10-17 15:27:25,237 [DEBUG] [spyder.plugins.completion.fallback.actor] -> Got request id 74: textDocument/completion for file /Users/dhung/src/tests/spyder.py
2019-10-17 15:27:25,240 [DEBUG] [spyder.plugins.completion.plugin] -> Completion plugin: Request 74 Got response from fallback
2019-10-17 15:27:25,242 [DEBUG] [urllib3.connectionpool] -> http://localhost:46624 "POST /clientapi/editor/event HTTP/1.1" 200 0
2019-10-17 15:27:25,243 [DEBUG] [spyder.plugins.completion.plugin] -> Completion plugin: Request 73 Got response from kite
2019-10-17 15:27:25,243 [DEBUG] [spyder.plugins.completion.kite.client] -> Perform textDocument/completion request with id 74
2019-10-17 15:27:25,245 [DEBUG] [spyder.plugins.completion.plugin] -> Completion plugin: Request 74 Got response from lsp
2019-10-17 15:27:25,245 [DEBUG] [spyder.plugins.completion.plugin] -> Gather responses for textDocument/completion
2019-10-17 15:27:25,246 [DEBUG] [spyder.plugins.completion.plugin] -> Responses statistics: {'fallback': 281, 'lsp': 0}
2019-10-17 15:27:25,265 [DEBUG] [urllib3.connectionpool] -> http://localhost:46624 "POST /clientapi/editor/complete HTTP/1.1" 200 1358
2019-10-17 15:27:25,266 [DEBUG] [spyder.plugins.completion.kite.providers.document] -> {'completions': [{'snippet': {'text': 'pyplot as plt', 'placeholders': []}, 'replace': {'begin': 37, 'end': 39}, 'display': 'pyplot as plt', 'documentation': {'text': 'Provides a MATLAB-like plotting framework.\n\npylab ( #matplotlib.pylab ) combines pyplot with numpy into a single namespace. This is convenient for interactive work, but for programming it is recommended that the namespaces be kept separate, e.g.:\n\nimport numpy as np import matplotlib.pyplot as plt x = np.arange(0, 5, 0.1); y = np.sin(x) plt.plot(x, y)'}, 'web_id': 'matplotlib.pyplot', 'local_id': 'python;;;;matplotlib.pyplot', 'hint': 'module', 'smart': False}, {'snippet': {'text': 'pylab as plt', 'placeholders': []}, 'replace': {'begin': 37, 'end': 39}, 'display': 'pylab as plt', 'documentation': {'text': 'This is a procedural interface to the matplotlib object-oriented plotting library.\n\nThe following plotting commands are provided; the majority have MATLAB ® [*] ( ##id2 ) analogs and similar arguments.\n\n_Plotting commands\n\nacorr - plot the autocorrelation function annotate - annotate something in the figure arrow - add an arrow to the axes axes - Create a new axes axhline - draw a horizontal line across axes axvline - draw a vertical line across axes axhspan - draw a horizontal bar across axes '}, 'web_id': 'matplotlib.pylab', 'local_id': 'python;;;;matplotlib.pylab', 'hint': 'module', 'smart': False}]}
2019-10-17 15:27:25,267 [DEBUG] [spyder.plugins.completion.plugin] -> Completion plugin: Request 74 Got response from kite
2019-10-17 15:27:25,745 [DEBUG] [spyder.plugins.completion.languageserver.client] -> python response: textDocument/publishDiagnostics
```

However, no completions are shown.


## Versions
<!--- You can get this information from Help > About Spyder...
or (if Spyder won't launch) the "conda list" command
from the Anaconda Prompt/Terminal/command line. --->

* Spyder version: 4.0.0.dev0 (Commit: c2b7673b1)
* Python version: 3.7.4 64-bit
* Qt version: Qt 5.9.6
* PyQt version: PyQt5 5.9.2
* Operating System name/version: Darwin 18.7.0

### Dependencies
<!--- Please go to the menu entry Help > Dependencies,
press the Copy to clipboard button and paste below --->

```
cloudpickle >=0.5.0          :  1.2.2 (OK)
pygments >=2.0               :  2.4.2 (OK)
qtconsole >=4.5.5            :  4.5.5 (OK)
nbconvert >=4.0              :  5.6.0 (OK)
sphinx >=0.6.6               :  2.2.0 (OK)
pylint >=0.25                :  2.4.2 (OK)
psutil >=0.3                 :  5.6.3 (OK)
qtawesome >=0.5.7            :  0.6.0 (OK)
qtpy >=1.5.0                 :  1.9.0 (OK)
pickleshare >=0.4            :  0.7.5 (OK)
zmq >=17                     :  18.1.0 (OK)
chardet >=2.0.0              :  3.0.4 (OK)
numpydoc >=0.6.0             :  0.9.1 (OK)
spyder_kernels >=1.5.0;<2.0.0:  1.6.0 (OK)
qdarkstyle >=2.7             :  2.7 (OK)
atomicwrites >=1.2.0         :  1.3.0 (OK)
diff_match_patch >=20181111  :  20181111 (OK)
watchdog                     :  None (OK)
keyring                      :  None (OK)
pexpect >=4.4.0              :  4.7.0 (OK)
pympler                      :  None (OK)
sympy >=0.7.3                :  None (NOK)
cython >=0.21                :  None (NOK)
IPython >=4.0                :  7.8.0 (OK)
matplotlib >=2.0.0           :  None (NOK)
pandas >=0.13.1              :  None (NOK)
numpy >=1.7                  :  None (NOK)
scipy >=0.17.0               :  None (NOK)
pyls >=0.28.2;<0.29.0        :  0.28.3 (OK)
rtree >=0.8.3                :  0.8.3 (OK)
```



Discussion:
@metalogical was the one who worked the most on fixing this problem for beta6. @metalogical, could you take a look at this one?

Resolving patch:
diff --git a/spyder/config/main.py b/spyder/config/main.py
--- a/spyder/config/main.py
+++ b/spyder/config/main.py
@@ -207,6 +207,7 @@
               'automatic_completions': True,
               'automatic_completions_after_chars': 3,
               'automatic_completions_after_ms': 300,
+              'completions_wait_for_ms': 200,
               'completions_hint': True,
               'underline_errors': False,
               'highlight_current_line': True,
diff --git a/spyder/plugins/completion/languageserver/confpage.py b/spyder/plugins/completion/languageserver/confpage.py
--- a/spyder/plugins/completion/languageserver/confpage.py
+++ b/spyder/plugins/completion/languageserver/confpage.py
@@ -994,10 +994,17 @@ def setup_page(self):
         self.fallback_enabled = newcb(_("Enable fallback completions"),
                                       'enable',
                                       section='fallback-completions')
+        self.completions_wait_for_ms = self.create_spinbox(
+            _("Time to wait for all providers to return (ms):"), None,
+            'completions_wait_for_ms', min_=0, max_=5000, step=10,
+            tip=_("Beyond this timeout, "
+                  "the first available provider will be returned"),
+            section='editor')
 
         clients_layout = QVBoxLayout()
         clients_layout.addWidget(self.kite_enabled)
         clients_layout.addWidget(self.fallback_enabled)
+        clients_layout.addWidget(self.completions_wait_for_ms)
         clients_group.setLayout(clients_layout)
 
         kite_layout = QVBoxLayout()
diff --git a/spyder/plugins/completion/plugin.py b/spyder/plugins/completion/plugin.py
--- a/spyder/plugins/completion/plugin.py
+++ b/spyder/plugins/completion/plugin.py
@@ -16,7 +16,7 @@
 import functools
 
 # Third-party imports
-from qtpy.QtCore import QObject, Slot, QMutex, QMutexLocker
+from qtpy.QtCore import QObject, Slot, QMutex, QMutexLocker, QTimer
 from qtpy.QtWidgets import QMessageBox
 
 # Local imports
@@ -83,6 +83,8 @@ def __init__(self, parent, plugins=['lsp', 'kite', 'fallback']):
         self.req_id = 0
         self.collection_mutex = QMutex(QMutex.Recursive)
 
+        self.update_configuration()
+
         for plugin in plugins:
             if plugin in self.BASE_PLUGINS:
                 Plugin = self.BASE_PLUGINS[plugin]
@@ -115,28 +117,49 @@ def receive_response(self, completion_source, req_id, resp):
             request_responses = self.requests[req_id]
             request_responses['sources'][completion_source] = resp
 
-            wait_for = self.WAIT_FOR_SOURCE[request_responses['req_type']]
-            if not resp:
-                # Any response is better than no response
-                keep_waiting = any(
-                    self._is_client_running(source) and
-                    source not in request_responses['sources']
-                    for source in self.clients)
-            elif completion_source not in wait_for:
-                # A wait_for response is better than non-wait_for
-                keep_waiting = any(
-                    self._is_client_running(source) and
-                    source not in request_responses['sources']
-                    for source in wait_for)
-            else:
-                keep_waiting = False
-
-            if keep_waiting:
-                return
+            self._maybe_send_locked(req_id)
+
+    @Slot(int)
+    def receive_timeout(self, req_id):
+        # On timeout, collect all completions and return to the user
+        if req_id not in self.requests:
+            return
+
+        logger.debug("Completion plugin: Request {} timed out".format(req_id))
+
+        with QMutexLocker(self.collection_mutex):
+            request_responses = self.requests[req_id]
+            request_responses['timed_out'] = True
+
+            self._maybe_send_locked(req_id)
 
+    def _maybe_send_locked(self, req_id):
+        if req_id not in self.requests:
+            return
+        request_responses = self.requests[req_id]
+
+        def send():
             del self.requests[req_id]
             self.gather_and_send(request_responses)
 
+        wait_for = set(source for source
+                       in self.WAIT_FOR_SOURCE[request_responses['req_type']]
+                       if self._is_client_running(source))
+        timed_out = request_responses['timed_out']
+
+        all_returned = all(source in request_responses['sources']
+                           for source in wait_for)
+        if not timed_out:
+            # Before the timeout
+            if all_returned:
+                send()
+        else:
+            # After the timeout
+            any_nonempty = any(request_responses['sources'].get(source)
+                               for source in wait_for)
+            if all_returned or any_nonempty:
+                send()
+
     @Slot(str)
     def client_available(self, client_name):
         client_info = self.clients[client_name]
@@ -209,7 +232,16 @@ def send_request(self, language, req_type, req):
             'req_type': req_type,
             'response_instance': req['response_instance'],
             'sources': {},
+            'timed_out': False,
         }
+
+        # Start the timer on this request
+        if self.wait_for_ms > 0:
+            QTimer.singleShot(self.wait_for_ms,
+                              lambda: self.receive_timeout(req_id))
+        else:
+            self.requests[req_id]['timed_out'] = True
+
         for client_name in self.clients:
             client_info = self.clients[client_name]
             client_info['plugin'].send_request(
@@ -246,6 +278,8 @@ def register_file(self, language, filename, codeeditor):
                 )
 
     def update_configuration(self):
+        self.wait_for_ms = self.get_option('completions_wait_for_ms',
+                                           section='editor')
         for client_name in self.clients:
             client_info = self.clients[client_name]
             if client_info['status'] == self.RUNNING:


Test patch:
diff --git a/spyder/plugins/editor/widgets/tests/test_introspection.py b/spyder/plugins/editor/widgets/tests/test_introspection.py
--- a/spyder/plugins/editor/widgets/tests/test_introspection.py
+++ b/spyder/plugins/editor/widgets/tests/test_introspection.py
@@ -43,6 +43,7 @@ def test_space_completion(lsp_codeeditor, qtbot):
     code_editor, _ = lsp_codeeditor
     code_editor.toggle_automatic_completions(False)
     code_editor.toggle_code_snippets(False)
+    CONF.set('editor', 'completions_wait_for_ms', 0)
 
     completion = code_editor.completion_widget