Repository: spyder-ide/spyder

Issue:
Autocomplete doesn't replace text sometimes



<!--- **PLEASE READ:** When submitting here, please ensure you've completed the following checklist and checked the boxes to confirm. Issue reports without it may be closed. Thanks! --->

### Issue Report Checklist

* [x] Searched the [issues page](https://github.com/spyder-ide/spyder/issues?q=is%3Aissue) for similar reports
* [x] Read the relevant sections of the [Spyder Troubleshooting Guide](https://github.com/spyder-ide/spyder/wiki/Troubleshooting-Guide-and-FAQ) and followed its advice
* [x] Reproduced the issue after updating with ``conda update spyder`` (or ``pip``, if not using Anaconda)
* [x] Could not reproduce inside ``jupyter qtconsole`` (if console-related)
* [x] Tried basic troubleshooting (if a bug/error)
    * [x] Restarted Spyder
    * [x] Reset preferences with ``spyder --reset``
    * [ ] Reinstalled the latest version of [Anaconda](https://www.anaconda.com/download/)
    * [ ] Tried the other applicable steps from the Troubleshooting Guide
* [x] Completed the **Problem Description**, **Steps to Reproduce** and **Version** sections below


## Problem Description
Sometimes the autocomplete will not work as expected. Usually after using Spyder for a while


### What steps reproduce the problem?

1. type "np.newa"
2. press "tab" to get an autocompletion 


### What is the expected output? What do you see instead?
I would expect "np.newaxis"

Instead I get "np.newanewaxis"


## Versions
<!--- You can get this information from Help > About Spyder...
or (if Spyder won't launch) the "conda list" command
from the Anaconda Prompt/Terminal/command line. --->

* Spyder version: Spyder 4.0.0.dev0
* Python version: Python 3.7.0 64-bit | Qt 5.9.3 | PyQt5 5.9.2 | Darwin 18.2.0 
* Qt version: 
* PyQt version: 
* Operating System name/version: Mojave 10.14.1

### Dependencies
<!--- Please go to the menu entry Help > Dependencies,
press the Copy to clipboard button and paste below --->
pygments >=2.0    :  2.2.0 (OK)
sphinx >=0.6.6    :  1.8.1 (OK)
psutil >=0.3      :  5.4.7 (OK)
pyls >=0.19.0     :  0.21.2 (OK)
nbconvert >=4.0   :  5.4.0 (OK)
pandas >=0.13.1   :  0.23.4 (OK)
numpy >=1.7       :  1.15.2 (OK)
sympy >=0.7.3     :  1.3 (OK)
cython >=0.21     :  0.28.5 (OK)
qtconsole >=4.2.0 :  4.4.1 (OK)
IPython >=4.0     :  7.0.1 (OK)
matplotlib >=2.0.0:  3.0.0 (OK)
pylint >=0.25     :  2.1.1 (OK)



Discussion:
I can reproduce this on Windows 10 too.
Yes, I noticed that too. It should be an easy fix.
@dalthviz, please work on this one.
![testautocomplete](https://user-images.githubusercontent.com/22970578/49809011-3248e480-fda1-11e8-80ea-45effc163b9c.gif)

I used the latest git clone. If there is the parentheses at the end of the statement, autocomplete does not seem to work well.
@ccordoba12 I do not see how PR #8387 addresses this issue since only the tests were changed.
This issue was already fixed in PR #8257. I only asked @dalthviz to improve a little bit the completion test.

@ok97465, are you using the latest master?
I just updated to the newest master and can reproduce @ok97465
> This issue was already fixed in PR #8257. I only asked @dalthviz to improve a little bit the completion test.

Ah ok thanks, that makes sense. For the record, I can reproduce this also on Windows.
@dalthviz, I can reproduce @ok97465 example. You only need to type until line 7 to get the error.
I don't even need any of that: 
In an otherwise empty file:
`(sli<tab>)` gives `(slislice)`
but:
`(sli<tab>` gives `(slice`
Ok, so it's the presence of braces which generates that! Thanks @impact27!
More generally this seems to be reproducible whenever there isn't white-space after the word that is being `<tab>` completed.

`np.ze<tab>,` gives `np.zezeros,`

Interestingly if there are non punctuation characters after the cursor the completion is even more strange:  
`np.ze<tab>m` gives `np.zeosm`

I'm not sure what to do in this case but I think `np.zerosm` would be best so the user can continue with a space or brackets and complete the line.

The problem, from what I checked so far, is in the logic of the `TextEditBaseWidget.insert_completion` and `BaseEditMixin.get_current_word` since it only makes a guess of what is the current word under the cursor and uses the diff in length between the completion and the current word to insert a certain amount of characters
So the "completion" from pls is is the whole word `zeros` and not just the actual completion of `np.ze` in this case `ros`?
Yep, I think so, and then there is where it comes the logic of the diff
According to my reading of this:
https://microsoft.github.io/language-server-protocol/specification#textDocument_completion

It seems like `textEdit?: TextEdit;` parameter should give us only the string that is needed for completion from the server.
```
/**
	 * A string that should be inserted into a document when selecting
	 * this completion. When `falsy` the label is used.
	 *
	 * The `insertText` is subject to interpretation by the client side.
	 * Some tools might not take the string literally. For example
	 * VS Code when code complete is requested in this example `con<cursor position>`
	 * and a completion item with an `insertText` of `console` is provided it
	 * will only insert `sole`. Therefore it is recommended to use `textEdit` instead
	 * since it avoids additional client side interpretation.
	 *
	 * @deprecated Use textEdit instead.
	 */
	insertText?: string;
```
Checking the current reponse of `pyls` [at this line](https://github.com/spyder-ide/spyder/blob/4f976e81d8c1161302b98fd9d02465e81780e7ee/spyder/plugins/editor/lsp/providers/document.py#L104), seems like the `textEdit` value is not returned. So probably that's why we made the current logic and then `textEdit` value needs to be added over `pyls` side.

So we could handle this client side (basically fixing the `insert_completion` and using the whole word) or appending the `textEdit` value to the responses of the completions of `pyls` (probably [here](https://github.com/palantir/python-language-server/blob/000993507a1bee471f4ab74f652e11875faed9a4/pyls/plugins/rope_completion.py#L51) and [here](https://github.com/palantir/python-language-server/blob/000993507a1bee471f4ab74f652e11875faed9a4/pyls/plugins/jedi_completion.py#L17)) and managing this value in the `QListWidgetItem` (probably as data)

@spyder-ide/core-developers what do you guys think?
Another idea would be to compute the starting prefix between the existing string and the ``textEdit`` value. 

Resolving patch:
diff --git a/spyder/plugins/editor/widgets/base.py b/spyder/plugins/editor/widgets/base.py
--- a/spyder/plugins/editor/widgets/base.py
+++ b/spyder/plugins/editor/widgets/base.py
@@ -13,6 +13,7 @@
 
 # Standard library imports
 import os
+import os.path as osp
 import re
 import sys
 
@@ -1136,14 +1137,19 @@ def select_completion_list(self):
 
     def insert_completion(self, text):
         if text:
-            cursor = self.textCursor()
-            word = self.get_current_word()
-            start = len(word) if word is not None else 0
-            text = text[start:]
-            # cursor.movePosition(QTextCursor.PreviousCharacter,
-            #                     QTextCursor.KeepAnchor,
-            #                     len(self.completion_text))
-            # cursor.removeSelectedText()
+            word = self.get_current_word(completion=True) or ""
+            common_prefix = osp.commonprefix([text, word])
+            start = len(common_prefix) if common_prefix is not None else 0
+            if start == len(word):
+                text = text[start:]
+            else:
+                # Remove current word since the completion is not
+                # a perfect match
+                cursor = self.textCursor()
+                cursor.movePosition(QTextCursor.PreviousCharacter,
+                                    QTextCursor.KeepAnchor,
+                                    len(word))
+                cursor.removeSelectedText()
             self.insert_text(text)
             self.document_did_change()
 
diff --git a/spyder/widgets/mixins.py b/spyder/widgets/mixins.py
--- a/spyder/widgets/mixins.py
+++ b/spyder/widgets/mixins.py
@@ -340,7 +340,7 @@ def remove_text(self, position_from, position_to):
         cursor = self.__select_text(position_from, position_to)
         cursor.removeSelectedText()
 
-    def get_current_word(self):
+    def get_current_word(self, completion=False):
         """Return current word, i.e. word at cursor position"""
         cursor = self.textCursor()
 
@@ -358,14 +358,24 @@ def get_current_word(self):
             # (see below), the word to the left of the cursor won't be
             # selected), but only if the first character to the left is not a
             # white space too.
-            def is_space(move):
-                curs = self.textCursor()
-                curs.movePosition(move, QTextCursor.KeepAnchor)
-                return not to_text_string(curs.selectedText()).strip()
-            if is_space(QTextCursor.NextCharacter):
-                if is_space(QTextCursor.PreviousCharacter):
-                    return
-                cursor.movePosition(QTextCursor.WordLeft)
+            if not completion:
+                def is_space(move):
+                    curs = self.textCursor()
+                    curs.movePosition(move, QTextCursor.KeepAnchor)
+                    return not to_text_string(curs.selectedText()).strip()
+                if is_space(QTextCursor.NextCharacter):
+                    if is_space(QTextCursor.PreviousCharacter):
+                        return
+                    cursor.movePosition(QTextCursor.WordLeft)
+            else:
+                def is_special_character(move):
+                    curs = self.textCursor()
+                    curs.movePosition(move, QTextCursor.KeepAnchor)
+                    text_cursor = to_text_string(curs.selectedText()).strip()
+                    return len(re.findall(r'([^\d\W]\w*)',
+                                          text_cursor, re.UNICODE)) == 0
+                if is_special_character(QTextCursor.NextCharacter):
+                    cursor.movePosition(QTextCursor.WordLeft)
 
         cursor.select(QTextCursor.WordUnderCursor)
         text = to_text_string(cursor.selectedText())


Test patch:
diff --git a/spyder/plugins/editor/tests/test_introspection.py b/spyder/plugins/editor/tests/test_introspection.py
--- a/spyder/plugins/editor/tests/test_introspection.py
+++ b/spyder/plugins/editor/tests/test_introspection.py
@@ -72,7 +72,6 @@ def get_spyder_pythonpath(*args):
     os.environ['SPY_TEST_USE_INTROSPECTION'] = 'False'
     editor.main.lspmanager.closing_plugin()
 
-
 @pytest.mark.slow
 @pytest.mark.skipif(PY2, reason="Segfaults with other tests on Py2.")
 @pytest.mark.skipif(os.name == 'nt' and not PY2,
@@ -100,6 +99,39 @@ def test_introspection(setup_editor):
     qtbot.keyPress(completion, Qt.Key_Enter, delay=1000)
     assert code_editor.toPlainText() == 'import math\n'
 
+    # enter for new line
+    qtbot.keyPress(code_editor, Qt.Key_Enter, delay=1000)
+
+    # Complete math.d() -> math.degrees()
+    qtbot.keyClicks(code_editor, 'math.d')
+    qtbot.wait(20000)
+
+    with qtbot.waitSignal(completion.sig_show_completions,
+                          timeout=10000) as sig:
+        qtbot.keyPress(code_editor, Qt.Key_Tab)
+    assert "degrees(x)" in [x['label'] for x in sig.args[0]]
+
+    qtbot.keyPress(completion, Qt.Key_Enter, delay=1000)
+    assert code_editor.toPlainText() == 'import math\nmath.degrees\n'
+
+    # enter for new line
+    qtbot.keyPress(code_editor, Qt.Key_Enter, delay=1000)
+
+    # Complete math.d() -> math.degrees()
+    qtbot.keyClicks(code_editor, 'math.d(')
+    qtbot.keyPress(code_editor, Qt.Key_Left, delay=1000)
+    qtbot.keyClicks(code_editor, 'e')
+    qtbot.wait(20000)
+
+    with qtbot.waitSignal(completion.sig_show_completions,
+                          timeout=10000) as sig:
+        qtbot.keyPress(code_editor, Qt.Key_Tab)
+    assert "degrees(x)" in [x['label'] for x in sig.args[0]]
+
+    qtbot.keyPress(completion, Qt.Key_Enter, delay=1000)
+    assert code_editor.toPlainText() == 'import math\nmath.degrees' \
+                                        '\nmath.degrees()\n'
+
     # Modify PYTHONPATH
     # editor.introspector.change_extra_path([LOCATION])
     # qtbot.wait(10000)