Repository: spyder-ide/spyder

Issue:
Make fallback completions respect prefix
<!--- **PLEASE READ:** When submitting here, please ensure you've completed the following checklist and checked the boxes to confirm. Issue reports without it may be closed. Thanks! --->

## Problem Description
It appears that fallback completions do not always respect the prefix underneath the user's cursor.

### What steps reproduce the problem?

1. Have fallback completions enabled.
2. Have the following buffer state:
```python
x = 1$
```
3. Manually request completions.

### What is the expected output? What do you see instead?

In this case, I would expect no completions to be returned. Instead, fallback completions seem to show builtin symbols:

![image](https://user-images.githubusercontent.com/3752225/74003773-a12fb800-4928-11ea-92a2-91b4164fa35f.png)

This is especially a problem when on-the-fly completions are enabled. The user ends up seeing wrong completions in cases like this.

## Versions
<!--- You can get this information from Help > About Spyder...
or (if Spyder won't launch) the "conda list" command
from the Anaconda Prompt/Terminal/command line. --->

Spyder 4.0.1 | Python 3.7.0 64-bit | Qt 5.9.6 | PyQt5 5.9.2 | Darwin 19.3.0 

### Dependencies
<!--- Please go to the menu entry Help > Dependencies,
press the Copy to clipboard button and paste below --->

```
applaunchservices >=0.1.7    :  0.2.1 (OK)
atomicwrites >=1.2.0         :  1.3.0 (OK)
chardet >=2.0.0              :  3.0.4 (OK)
cloudpickle >=0.5.0          :  1.2.2 (OK)
diff_match_patch >=20181111  :  20181111 (OK)
intervaltree                 :  None (OK)
IPython >=4.0                :  7.12.0 (OK)
jedi =0.14.1                 :  0.14.1 (OK)
nbconvert >=4.0              :  5.6.1 (OK)
numpydoc >=0.6.0             :  0.9.2 (OK)
pexpect >=4.4.0              :  4.8.0 (OK)
pickleshare >=0.4            :  0.7.5 (OK)
psutil >=0.3                 :  5.6.7 (OK)
pygments >=2.0               :  2.5.2 (OK)
pylint >=0.25                :  2.4.4 (OK)
pyls >=0.31.2;<0.32.0        :  0.31.7 (OK)
zmq >=17                     :  18.1.1 (OK)
qdarkstyle >=2.7             :  2.8 (OK)
qtawesome >=0.5.7            :  0.6.1 (OK)
qtconsole >=4.6.0            :  4.6.0 (OK)
qtpy >=1.5.0                 :  1.9.0 (OK)
rtree >=0.8.3                :  0.9.3 (OK)
sphinx >=0.6.6               :  2.3.1 (OK)
spyder_kernels >=1.8.1;<2.0.0:  1.8.1 (OK)
watchdog                     :  None (OK)
cython >=0.21                :  0.29.14 (OK)
matplotlib >=2.0.0           :  3.1.3 (OK)
numpy >=1.7                  :  1.18.1 (OK)
pandas >=0.13.1              :  1.0.0 (OK)
scipy >=0.17.0               :  1.4.1 (OK)
sympy >=0.7.3                :  1.5.1 (OK)
```



Discussion:
@andfoy, please take a look at this one.

Resolving patch:
diff --git a/spyder/plugins/completion/fallback/actor.py b/spyder/plugins/completion/fallback/actor.py
--- a/spyder/plugins/completion/fallback/actor.py
+++ b/spyder/plugins/completion/fallback/actor.py
@@ -26,7 +26,8 @@
 # Local imports
 from spyder.plugins.completion.languageserver import CompletionItemKind
 from spyder.plugins.completion.languageserver import LSPRequestTypes
-from spyder.plugins.completion.fallback.utils import get_keywords, get_words
+from spyder.plugins.completion.fallback.utils import (
+    get_keywords, get_words, is_prefix_valid)
 
 
 FALLBACK_COMPLETION = "Fallback"
@@ -58,6 +59,9 @@ def tokenize(self, text, offset, language):
         Return all tokens in `text` and all keywords associated by
         Pygments to `language`.
         """
+        valid = is_prefix_valid(text, offset, language)
+        if not valid:
+            return []
         try:
             lexer = get_lexer_by_name(language)
             keywords = get_keywords(lexer)
diff --git a/spyder/plugins/completion/fallback/utils.py b/spyder/plugins/completion/fallback/utils.py
--- a/spyder/plugins/completion/fallback/utils.py
+++ b/spyder/plugins/completion/fallback/utils.py
@@ -26,6 +26,8 @@
 from pygments.lexers import (get_lexer_for_filename, get_lexer_by_name,
                              TextLexer)
 
+letter_regex = re.compile(r'\w')
+empty_regex = re.compile(r'\s')
 
 # CamelCase and snake_case regex:
 # Get all valid tokens that start by a letter (Unicode) and are
@@ -113,6 +115,25 @@ def get_words(text, exclude_offset=None, language=None):
     return tokens
 
 
+def is_prefix_valid(text, offset, language):
+    """Check if current offset prefix ."""
+    regex = LANGUAGE_REGEX.get(language.lower(), all_regex)
+    prefix = ''
+    current_pos_text = text[offset - 1]
+    empty_start = empty_regex.match(current_pos_text) is not None
+    max_end = -1
+    for match in regex.finditer(text):
+        start, end = match.span()
+        max_end = max(end, max_end)
+        if offset >= start and offset <= end:
+            prefix = match.group()
+    if offset > max_end:
+        if letter_regex.match(current_pos_text):
+            prefix = current_pos_text
+    valid = prefix != '' or (prefix == '' and empty_start)
+    return valid
+
+
 @memoize
 def get_parent_until(path):
     """


Test patch:
diff --git a/spyder/plugins/completion/fallback/tests/data/example.py b/spyder/plugins/completion/fallback/tests/data/example.py
--- a/spyder/plugins/completion/fallback/tests/data/example.py
+++ b/spyder/plugins/completion/fallback/tests/data/example.py
@@ -63,4 +63,4 @@ def iterate_2(Z):
 #print(Z)
 for i in range(4):
     iterate_2(Z)
-    #print (Z)
\ No newline at end of file
+    #print (Z)
diff --git a/spyder/plugins/completion/fallback/tests/test_fallback.py b/spyder/plugins/completion/fallback/tests/test_fallback.py
--- a/spyder/plugins/completion/fallback/tests/test_fallback.py
+++ b/spyder/plugins/completion/fallback/tests/test_fallback.py
@@ -102,7 +102,7 @@ def test_tokenize(qtbot_module, fallback_fixture, file_fixture):
     open_request = {
         'file': filename,
         'text': contents,
-        'offset': -1,
+        'offset': len(contents),
     }
     fallback.send_request(
         language, LSPRequestTypes.DOCUMENT_DID_OPEN, open_request)
@@ -128,7 +128,7 @@ def test_token_update(qtbot_module, fallback_fixture):
     open_request = {
         'file': 'test.py',
         'text': TEST_FILE,
-        'offset': -1,
+        'offset': len(TEST_FILE),
     }
     fallback.send_request(
         'python', LSPRequestTypes.DOCUMENT_DID_OPEN, open_request)
@@ -149,7 +149,7 @@ def test_token_update(qtbot_module, fallback_fixture):
     update_request = {
         'file': 'test.py',
         'diff': diff,
-        'offset': -1,
+        'offset': len(diff),
     }
     fallback.send_request(
         'python', LSPRequestTypes.DOCUMENT_DID_CHANGE, update_request)