Repository: spyder-ide/spyder

Issue:
Error when deleting UTF character
## Description

### What steps will reproduce the problem?

<!--- You can use Markdown here --->

1. Enter "🠔" (`U+1F814 LEFTWARDS ARROW WITH EQUILATERAL ARROWHEAD`) in a docstring.
2. Position the cursor before the "🠔".
3. Press the DEL key.

It works, if you position the cursor behind the "🠔" and press the backspace.

### Traceback
```python-traceback
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/spyder/utils/syntaxhighlighters.py", line 320, in highlightBlock
    self.highlight_block(text)
  File "/usr/local/lib/python3.8/dist-packages/spyder/utils/syntaxhighlighters.py", line 646, in highlight_block
    self.setFormat(0, qstring_length(text), self.formats["normal"])
  File "/usr/local/lib/python3.8/dist-packages/spyder/utils/qstringhelpers.py", line 19, in qstring_length
    utf16_text = text.encode('utf16')
UnicodeEncodeError: 'utf-16' codec can't encode character '\udc14' in position 14: surrogates not allowed
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/spyder/plugins/editor/utils/autosave.py", line 135, in do_autosave
    stack.autosave.autosave_all()
  File "/usr/local/lib/python3.8/dist-packages/spyder/plugins/editor/utils/autosave.py", line 417, in autosave_all
    self.maybe_autosave(index)
  File "/usr/local/lib/python3.8/dist-packages/spyder/plugins/editor/utils/autosave.py", line 386, in maybe_autosave
    self.autosave(finfo)
  File "/usr/local/lib/python3.8/dist-packages/spyder/plugins/editor/utils/autosave.py", line 405, in autosave
    self.stack._write_to_file(finfo, autosave_filename)
  File "/usr/local/lib/python3.8/dist-packages/spyder/plugins/editor/widgets/editor.py", line 1808, in _write_to_file
    fileinfo.encoding = encoding.write(txt, filename, fileinfo.encoding)
  File "/usr/local/lib/python3.8/dist-packages/spyder/utils/encoding.py", line 236, in write
    text, encoding = encode(text, encoding)
  File "/usr/local/lib/python3.8/dist-packages/spyder/utils/encoding.py", line 214, in encode
    return text.encode('utf-8'), 'utf-8'
UnicodeEncodeError: 'utf-8' codec can't encode character '\udc14' in position 15397: surrogates not allowed
Failed to send bug report on Github. response={'code': 401, 'json': {'message': 'Bad credentials', 'documentation_url': 'https://docs.github.com/rest'}}
```

## Versions

* Spyder version: 5.0.2 
* Python version: 3.8.5
* Qt version: 5.12.9
* PyQt5 version: 5.12.3
* Operating System: Linux 5.12.2-surface

### Dependencies

```

# Mandatory:
atomicwrites >=1.2.0          :  1.4.0 (OK)
chardet >=2.0.0               :  4.0.0 (OK)
cloudpickle >=0.5.0           :  1.6.0 (OK)
cookiecutter >=1.6.0          :  1.7.2 (OK)
diff_match_patch >=20181111   :  20200713 (OK)
intervaltree >=3.0.2          :  3.1.0 (OK)
IPython >=7.6.0               :  7.23.1 (OK)
jedi =0.17.2                  :  0.17.2 (OK)
jsonschema >=3.2.0            :  3.2.0 (OK)
keyring >=17.0.0              :  23.0.1 (OK)
nbconvert >=4.0               :  6.0.7 (OK)
numpydoc >=0.6.0              :  1.1.0 (OK)
parso =0.7.0                  :  0.7.0 (OK)
pexpect >=4.4.0               :  4.6.0 (OK)
pickleshare >=0.4             :  0.7.5 (OK)
psutil >=5.3                  :  5.5.1 (OK)
pygments >=2.0                :  2.9.0 (OK)
pylint >=1.0                  :  2.8.2 (OK)
pyls >=0.36.2;<1.0.0          :  0.36.2 (OK)
pyls_black >=0.4.6            :  0.4.6 (OK)
pyls_spyder >=0.3.2;<0.4.0    :  0.3.2 (OK)
qdarkstyle =3.0.2             :  3.0.2 (OK)
qstylizer >=0.1.10            :  0.2.0 (OK)
qtawesome >=1.0.2             :  1.0.2 (OK)
qtconsole >=5.1.0             :  5.1.0 (OK)
qtpy >=1.5.0                  :  1.9.0 (OK)
setuptools >=39.0.0           :  56.2.0 (OK)
sphinx >=0.6.6                :  4.0.1 (OK)
spyder_kernels >=2.0.2;<2.1.0 :  2.0.2 (OK)
textdistance >=4.2.0          :  4.2.1 (OK)
three_merge >=0.1.1           :  0.1.1 (OK)
watchdog >=0.10.3;<2.0.0      :  1.0.2 (OK)
xdg >=0.26                    :  0.26 (OK)
zmq >=17                      :  22.0.3 (OK)

# Optional:
cython >=0.21                 :  None (NOK)
matplotlib >=2.0.0            :  3.4.2 (OK)
numpy >=1.7                   :  1.20.3 (OK)
pandas >=1.1.1                :  1.2.4 (OK)
scipy >=0.17.0                :  1.6.3 (OK)
sympy >=0.7.3                 :  1.8 (OK)
```



Discussion:
Hey @sphh, thanks for reporting. I could reproduce this bug and we'll try to fix it in our next version.
@impact27, could you give us a hand with this one?
Looks like a qt bug, I am not sure we can do much. `qstring_length` tries to return the length of a string as seen by qt, but if the string is corrupted, the length has no meaning.
We can fix the problem in `qstring_length` but there is still corrupted text in the editor that will crash something else. This is a qt bug and not a spyder bug. I would recommand closing this issue as I can't think of anything reasonable we could do in this situation.
@impact27: If that is a qt bug, would you mind to file a bug report with them and talk to them from developer to developer?
@sphh, I plan to catch the error and pass for 5.0.4, to be released shortly.
I mean I pushed my trial here: https://github.com/spyder-ide/spyder/pull/15804/files but as soon as you try to run the code you will get:
`UnicodeEncodeError: 'utf-8' codec can't encode character '\udc14' in position 93: surrogates not allowed` errors
Sorry I was wrong, Qt is fine, we actually make math on position in spyder here:
https://github.com/spyder-ide/spyder/blob/39c29319fbdd406669334f5a1c307c93c2f5ea9d/spyder/plugins/editor/widgets/codeeditor.py#L2247
Which is not correct, We can not assume we can delete just one character without leaving behind corrupted code.
Possibly problematic as well:
https://github.com/spyder-ide/spyder/blob/39c29319fbdd406669334f5a1c307c93c2f5ea9d/spyder/plugins/editor/widgets/codeeditor.py#L3821
https://github.com/spyder-ide/spyder/blob/39c29319fbdd406669334f5a1c307c93c2f5ea9d/spyder/plugins/editor/widgets/codeeditor.py#L3836
https://github.com/spyder-ide/spyder/blob/39c29319fbdd406669334f5a1c307c93c2f5ea9d/spyder/plugins/editor/widgets/codeeditor.py#L3839

Resolving patch:
diff --git a/spyder/plugins/editor/widgets/codeeditor.py b/spyder/plugins/editor/widgets/codeeditor.py
--- a/spyder/plugins/editor/widgets/codeeditor.py
+++ b/spyder/plugins/editor/widgets/codeeditor.py
@@ -35,7 +35,7 @@
 from qtpy.QtGui import (QColor, QCursor, QFont, QKeySequence, QPaintEvent,
                         QPainter, QMouseEvent, QTextCursor, QDesktopServices,
                         QKeyEvent, QTextDocument, QTextFormat, QTextOption,
-                        QTextCharFormat)
+                        QTextCharFormat, QTextLayout)
 from qtpy.QtWidgets import (QApplication, QMenu, QMessageBox, QSplitter,
                             QToolTip, QScrollBar)
 from spyder_kernels.utils.dochelpers import getobj
@@ -2237,14 +2237,36 @@ def get_current_object(self):
         offset = self.get_position('cursor')
         return sourcecode.get_primary_at(source_code, offset)
 
+    def next_cursor_position(self, position=None,
+                             mode=QTextLayout.SkipCharacters):
+        """
+        Get next valid cursor position.
+
+        Adapted from:
+        https://github.com/qt/qtbase/blob/5.15.2/src/gui/text/qtextdocument_p.cpp#L1361
+        """
+        cursor = self.textCursor()
+        if cursor.atEnd():
+            return position
+        if position is None:
+            position = cursor.position()
+        else:
+            cursor.setPosition(position)
+        it = cursor.block()
+        start = it.position()
+        end = start + it.length() - 1
+        if (position == end):
+            return end + 1
+        return it.layout().nextCursorPosition(position - start, mode) + start
+
     @Slot()
     def delete(self):
         """Remove selected text or next character."""
         if not self.has_selected_text():
             cursor = self.textCursor()
-            position = cursor.position()
             if not cursor.atEnd():
-                cursor.setPosition(position + 1, QTextCursor.KeepAnchor)
+                cursor.setPosition(
+                    self.next_cursor_position(), QTextCursor.KeepAnchor)
             self.setTextCursor(cursor)
         self.remove_selected_text()
 
@@ -3818,7 +3840,7 @@ def _get_word_start_cursor(self, position):
                 document.characterAt(position)):
             position -= 1
         cursor = self.textCursor()
-        cursor.setPosition(position + 1)
+        cursor.setPosition(self.next_cursor_position())
         return cursor
 
     def _get_word_end_cursor(self, position):
@@ -3833,10 +3855,10 @@ def _get_word_end_cursor(self, position):
         end = cursor.position()
         while (position < end and
                not self.is_letter_or_number(document.characterAt(position))):
-            position += 1
+            position = self.next_cursor_position(position)
         while (position < end and
                self.is_letter_or_number(document.characterAt(position))):
-            position += 1
+            position = self.next_cursor_position(position)
         cursor.setPosition(position)
         return cursor
 


Test patch: