Repository: spyder-ide/spyder

Issue:
Crash in fallback plugin when typing character
## Description

### What steps will reproduce the problem?

<!--- You can use Markdown here --->

I have an animated gif available. I have many things loaded into memory and just type a character and Spyder crashes.

### Traceback
```python-traceback
  File "C:\Users\suzbm\Anaconda3\lib\site-packages\spyder\plugins\completion\fallback\actor.py", line 155, in handle_msg
    msg['current_word'])
  File "C:\Users\suzbm\Anaconda3\lib\site-packages\spyder\plugins\completion\fallback\actor.py", line 62, in tokenize
    valid = is_prefix_valid(text, offset, language)
  File "C:\Users\suzbm\Anaconda3\lib\site-packages\spyder\plugins\completion\fallback\utils.py", line 125, in is_prefix_valid
    current_pos_text = text[offset - utf16_diff - 1]
IndexError: string index out of range
```

## Versions

* Spyder version: 4.1.2 
* Python version: 3.7.4
* Qt version: 5.12.5
* PyQt5 version: 5.12.3
* Operating System: Windows 10

### Dependencies

```

# Mandatory:
atomicwrites >=1.2.0           :  1.3.0 (OK)
chardet >=2.0.0                :  3.0.4 (OK)
cloudpickle >=0.5.0            :  1.3.0 (OK)
diff_match_patch >=20181111    :  20181111 (OK)
intervaltree                   :  None (OK)
IPython >=4.0                  :  7.12.0 (OK)
jedi =0.15.2                   :  0.15.2 (OK)
nbconvert >=4.0                :  5.6.1 (OK)
numpydoc >=0.6.0               :  0.9.2 (OK)
paramiko >=2.4.0               :  2.7.1 (OK)
parso =0.5.2                   :  0.5.2 (OK)
pexpect >=4.4.0                :  4.8.0 (OK)
pickleshare >=0.4              :  0.7.5 (OK)
psutil >=5.3                   :  5.7.0 (OK)
pygments >=2.0                 :  2.5.2 (OK)
pylint >=0.25                  :  2.4.4 (OK)
pyls >=0.31.9;<0.32.0          :  0.31.9 (OK)
qdarkstyle >=2.8               :  2.8 (OK)
qtawesome >=0.5.7              :  0.7.0 (OK)
qtconsole >=4.6.0              :  4.6.0 (OK)
qtpy >=1.5.0                   :  1.9.0 (OK)
rtree >=0.8.3                  :  0.9.4 (OK)
sphinx >=0.6.6                 :  2.4.3 (OK)
spyder_kernels >=1.9.0;<1.10.0 :  1.9.0 (OK)
watchdog                       :  None (OK)
zmq >=17                       :  19.0.0 (OK)

# Optional:
cython >=0.21                  :  0.29.15 (OK)
matplotlib >=2.0.0             :  3.1.3 (OK)
numpy >=1.7                    :  1.18.1 (OK)
pandas >=0.13.1                :  1.0.1 (OK)
scipy >=0.17.0                 :  1.4.1 (OK)
sympy >=0.7.3                  :  1.5.1 (OK)
```



Discussion:
![spydercrash](https://user-images.githubusercontent.com/7544707/78997244-2ba4bb80-7b14-11ea-85e1-bdbe738d0a42.gif)

Thanks a lot the detailed report @suzannejb! Could you upload the `preprocessing.py` file that you're showing above (not its contents) here, so we can try to reproduce the bug in our side? Thanks!
Pinging @suzannejb again. Could you help us here by providing `preprocessing.py`?
I have the exact same issue arising when typing any character in a ~400ish line python script:

>   File "/Users/dmattox/anaconda3/lib/python3.7/site-packages/spyder/plugins/completion/fallback/actor.py", line 155, in handle_msg
>     msg['current_word'])
>   File "/Users/dmattox/anaconda3/lib/python3.7/site-packages/spyder/plugins/completion/fallback/actor.py", line 62, in tokenize
>     valid = is_prefix_valid(text, offset, language)
>   File "/Users/dmattox/anaconda3/lib/python3.7/site-packages/spyder/plugins/completion/fallback/utils.py", line 125, in is_prefix_valid
>     current_pos_text = text[offset - utf16_diff - 1]
> IndexError: string index out of range

In probing a little more, this occurs when I type anything into any of the three scripts I have open as tabs. Tried closing spyder and error occurs again on reopening.

- Spyder version 4.1.3 [conda installed]
- Python version 3.7.0
- qt 5.12.5
- pyqt 5.12.3 
- Mac OS Catalina 10.15.3

@demattox, please upload one of your scripts here to see if we can reproduce this problem in our side.
@ccordoba12  Sorry, where should I upload it? I tried dragging and dropping it here and it says it is a unsupported file type. It is available on github already if that works: https://github.com/demattox/plip/blob/stable/plip/modules/plipxml.py

> It is available on github already if that works

Thanks, that works great!

Could you also create an animated gif (for instance with the [Licecap](https://www.cockos.com/licecap/) program) and upload it here (there shouldn't be issues with that)? That way I could see at what exact point (line and column) you're getting the error.

Finally, please let us know what operating system, Python, Qt and PyQt versions you're using. Thanks!
> Spyder version 4.1.3 [conda installed]
> Python version 3.7.0
> qt 5.12.5
> pyqt 5.12.3
> Mac OS Catalina 10.15.3

![spyder_bug_demo](https://user-images.githubusercontent.com/26228450/82601546-b8b04b00-9b7d-11ea-9c37-eff120d457d6.gif)

@ccordoba12  Sorry, that GIF got sped up from the screen capture, not sure how. But hopefully it shows that typing a comment did not produce an error, but then typing any random characters on the same line gives the error. And then picking another line at random and typing gives the error, as well as backspacing.
No worries, i think it's fine.
@ccordoba12 So I made a potentially interesting and hopefully helpful observation. It seems this error does not arise when I've split horizontally, but after closing the split panel, the error occurs again. And I noticed the tab selector button options (not sure the name for them, the square buttons to the left and right of the tab names) are highlighted when working in a single pane, but look normal in the split pane. Might be unrelated though.

Please let me know if there's anything else I can do.

![spyder_demo2](https://user-images.githubusercontent.com/26228450/82603230-49882600-9b80-11ea-94cf-eeaa96505ab4.gif)

@demattox  I have same issue here in spyder
@ccordoba12 Please take note of the issue. It started happening just an hour ago after a restart. As per my knowledge I have not updated anything since my last start.

Spyder version 4.1.2 
Python 3.7.7 64-bit | Qt 5.9.6 | PyQt5 5.9.2 | Windows 10 
Using Kite for indexing

Here is the gif showing the error.
![1](https://user-images.githubusercontent.com/63765230/82671080-df44b380-9c5b-11ea-970f-5fa4cc10491e.gif)

Also i tried it with horizontal splitting and the problem goes away.

Resolving patch:
diff --git a/spyder/plugins/completion/fallback/utils.py b/spyder/plugins/completion/fallback/utils.py
--- a/spyder/plugins/completion/fallback/utils.py
+++ b/spyder/plugins/completion/fallback/utils.py
@@ -122,7 +122,12 @@ def is_prefix_valid(text, offset, language):
     # such as emojis in the editor.
     # Fixes spyder-ide/spyder#11862
     utf16_diff = qstring_length(text) - len(text)
-    current_pos_text = text[offset - utf16_diff - 1]
+
+    new_offset = offset - utf16_diff - 1
+    if new_offset >= len(text) or new_offset < 0:
+        return False
+
+    current_pos_text = text[new_offset]
 
     empty_start = empty_regex.match(current_pos_text) is not None
     max_end = -1


Test patch: