Repository: spyder-ide/spyder

Issue:
Response of LSP Requests results in wrong position in editor
<!--- **PLEASE READ:** When submitting here, please ensure you've completed the following checklist and checked the boxes to confirm. Issue reports without it may be closed. Thanks! --->

### Issue Report Checklist

* [x] Searched the [issues page](https://github.com/spyder-ide/spyder/issues?q=is%3Aissue) for similar reports
* [x] Read the relevant sections of the [Spyder Troubleshooting Guide](https://github.com/spyder-ide/spyder/wiki/Troubleshooting-Guide-and-FAQ) and followed its advice
* [x] Reproduced the issue after updating with ``conda update spyder`` (or ``pip``, if not using Anaconda)
* [ ] Could not reproduce inside ``jupyter qtconsole`` (if console-related)
* [ ] Tried basic troubleshooting (if a bug/error)
    * [x] Restarted Spyder
    * [x] Reset preferences with ``spyder --reset``
    * [x] Reinstalled the latest version of [Anaconda](https://www.anaconda.com/download/)
    * [ ] Tried the other applicable steps from the Troubleshooting Guide
* [x] Completed the **Problem Description**, **Steps to Reproduce** and **Version** sections below


## Problem Description

Spyder always sends transformed document text EOL to LF in LSP requests. Therefore, when the server calculates an object position, it's considering LF as EOL. However, this results in a position mismatch if the document EOL is CRLF or CR as Spyder expects the position accordingly.

This could be server-sided solved, as the server could identify the document EOL and use that to convert the position accordingly, but I don't know if this approach breaks the (Language Server Protocol)[https://microsoft.github.io/language-server-protocol/]. Besides, the Server Implementation that was tested does not result in this bug with VSCode, so it probably should be solved client-sided.

### What steps reproduce the problem?

1. Open a document with CRLF as EOL
2. Use the go-to reference (can be any LSP request that's position sensitive)
3. Results in the wrong position.

### What is the expected output? What do you see instead?

It is expected the correct object's reference in the editor, but it leads to a wrong position instead.

### Paste Traceback/Error Below (if applicable)
<!--- Copy from error dialog or View > Panes > Internal Console --->

No traceback is applicable.

## Versions
<!--- You can get this information from Help > About Spyder...
or (if Spyder won't launch) the "conda list" command
from the Anaconda Prompt/Terminal/command line. --->

* Spyder version: 5.0.4
* Python version: 3.8.5
* Qt version: 5.12.10
* PyQt version: 5.12.3
* Operating System name/version: Windows-10-10.0.21390-SP0

### Dependencies
<!--- Please go to the menu entry Help > Dependencies,
press the Copy to clipboard button and paste below --->

```
# Mandatory:
atomicwrites >=1.2.0          :  1.4.0 (OK)
chardet >=2.0.0               :  4.0.0 (OK)
cloudpickle >=0.5.0           :  1.6.0 (OK)
cookiecutter >=1.6.0          :  1.7.2 (OK)
diff_match_patch >=20181111   :  20200713 (OK)
intervaltree >=3.0.2          :  3.1.0 (OK)
IPython >=7.6.0               :  7.19.0 (OK)
jedi =0.17.2                  :  0.17.2 (OK)
jsonschema >=3.2.0            :  3.2.0 (OK)
keyring >=17.0.0              :  21.8.0 (OK)
nbconvert >=4.0               :  6.0.7 (OK)
numpydoc >=0.6.0              :  1.1.0 (OK)
paramiko >=2.4.0              :  2.7.2 (OK)
parso =0.7.0                  :  0.7.0 (OK)
pexpect >=4.4.0               :  4.8.0 (OK)
pickleshare >=0.4             :  0.7.5 (OK)
psutil >=5.3                  :  5.8.0 (OK)
pygments >=2.0                :  2.8.1 (OK)
pylint >=1.0                  :  2.6.0 (OK)
pyls >=0.36.2;<1.0.0          :  0.36.2 (OK)
pyls_black >=0.4.6            :  0.4.6 (OK)
pyls_spyder >=0.3.2;<0.4.0    :  0.3.2 (OK)
qdarkstyle =3.0.2             :  3.0.2 (OK)
qstylizer >=0.1.10            :  0.1.10 (OK)
qtawesome >=1.0.2             :  1.0.2 (OK)
qtconsole >=5.1.0             :  5.1.0 (OK)
qtpy >=1.5.0                  :  1.9.0 (OK)
rtree >=0.9.7                 :  0.9.7 (OK)
setuptools >=39.0.0           :  56.0.0 (OK)
sphinx >=0.6.6                :  3.4.3 (OK)
spyder_kernels >=2.0.4;<2.1.0 :  2.0.4 (OK)
textdistance >=4.2.0          :  4.2.0 (OK)
three_merge >=0.1.1           :  0.1.1 (OK)
watchdog >=0.10.3             :  1.0.2 (OK)
zmq >=17                      :  22.0.3 (OK)

# Optional:
cython >=0.21                 :  None (NOK)
matplotlib >=2.0.0            :  3.3.3 (OK)
numpy >=1.7                   :  1.20.3 (OK)
pandas >=1.1.1                :  1.2.1 (OK)
scipy >=0.17.0                :  1.6.0 (OK)
sympy >=0.7.3                 :  1.7.1 (OK)

# Spyder plugins:
spyder_f_language             :  1.3.dev (OK)
```



Discussion:
I can help with this problem if needed as it breaks compatibility with my language server project and Spyder.
It seems that ``spyder.plugins.editor.widgets.codeeditor`` methods ``document_did_open``, ``document_did_change`` and ``document_did_save`` uses ``toPlainText`` method to send the document text to the language server:
https://github.com/spyder-ide/spyder/blob/7a4103d97df612947f6916a59f9ac560bd98e0d2/spyder/plugins/editor/widgets/codeeditor.py#L1088
https://github.com/spyder-ide/spyder/blob/7a4103d97df612947f6916a59f9ac560bd98e0d2/spyder/plugins/editor/widgets/codeeditor.py#L1137
https://github.com/spyder-ide/spyder/blob/7a4103d97df612947f6916a59f9ac560bd98e0d2/spyder/plugins/editor/widgets/codeeditor.py#L1836

And this method always returns the text with LF as EOL. A possible solution would be using the ``get_text_with_eol`` method:
```py
text = self.get_text_with_eol()
```

Although this method defined in ``widgets.mixins`` could be improved to ignore when EOL is already LF:
```py
    def get_text_with_eol(self):
        """Same as 'toPlainText', replace '\n'
        by correct end-of-line characters"""
        utext = to_text_string(self.toPlainText())
        linesep = self.get_line_separator()
        if linesep != "\n":
            lines = utext.splitlines()
            txt = linesep.join(lines)
            if utext.endswith('\n'):
                txt += linesep
            return txt
        else:
            return utext
```
Hey @hlouzada, thanks a lot for reporting and investigating this problem. Please submit a pull request with the fix you consider necessary for it. That way it would be easier for us to understand what you have in mind and give you feedback about it.

Resolving patch:
diff --git a/spyder/plugins/editor/widgets/codeeditor.py b/spyder/plugins/editor/widgets/codeeditor.py
--- a/spyder/plugins/editor/widgets/codeeditor.py
+++ b/spyder/plugins/editor/widgets/codeeditor.py
@@ -1086,7 +1086,7 @@ def stop_completion_services(self):
     def document_did_open(self):
         """Send textDocument/didOpen request to the server."""
         cursor = self.textCursor()
-        text = self.toPlainText()
+        text = self.get_text_with_eol()
         if self.is_ipython():
             # Send valid python text to LSP as it doesn't support IPython
             text = self.ipython_to_python(text)
@@ -1135,7 +1135,7 @@ def process_symbols(self, params):
     def document_did_change(self, text=None):
         """Send textDocument/didChange request to the server."""
         self.text_version += 1
-        text = self.toPlainText()
+        text = self.get_text_with_eol()
         if self.is_ipython():
             # Send valid python text to LSP
             text = self.ipython_to_python(text)
@@ -1478,7 +1478,7 @@ def process_signatures(self, params):
     # ------------- LSP: Hover/Mouse ---------------------------------------
     @request(method=CompletionRequestTypes.DOCUMENT_CURSOR_EVENT)
     def request_cursor_event(self):
-        text = self.toPlainText()
+        text = self.get_text_with_eol()
         cursor = self.textCursor()
         params = {
             'file': self.filename,
@@ -1710,7 +1710,7 @@ def _apply_document_edits(self, edits):
         if edits is None:
             return
 
-        text = self.toPlainText()
+        text = self.get_text_with_eol()
         text_tokens = list(text)
         merged_text = None
         for edit in edits:
@@ -1851,7 +1851,7 @@ def notify_save(self):
         """Send save request."""
         params = {'file': self.filename}
         if self.save_include_text:
-            params['text'] = self.toPlainText()
+            params['text'] = self.get_text_with_eol()
         return params
 
     @request(method=CompletionRequestTypes.DOCUMENT_DID_CLOSE,
diff --git a/spyder/plugins/editor/widgets/editor.py b/spyder/plugins/editor/widgets/editor.py
--- a/spyder/plugins/editor/widgets/editor.py
+++ b/spyder/plugins/editor/widgets/editor.py
@@ -1799,7 +1799,7 @@ def compute_hash(self, fileinfo):
         Returns:
             int: computed hash.
         """
-        txt = fileinfo.editor.get_text_with_eol()
+        txt = to_text_string(fileinfo.editor.get_text_with_eol())
         return hash(txt)
 
     def _write_to_file(self, fileinfo, filename):
@@ -1812,7 +1812,7 @@ def _write_to_file(self, fileinfo, filename):
         This is a low-level function that only saves the text to file in the
         correct encoding without doing any error handling.
         """
-        txt = fileinfo.editor.get_text_with_eol()
+        txt = to_text_string(fileinfo.editor.get_text_with_eol())
         fileinfo.encoding = encoding.write(txt, filename, fileinfo.encoding)
 
     def save(self, index=None, force=False, save_new_files=True):
diff --git a/spyder/widgets/mixins.py b/spyder/widgets/mixins.py
--- a/spyder/widgets/mixins.py
+++ b/spyder/widgets/mixins.py
@@ -692,15 +692,14 @@ def get_line_separator(self):
             return os.linesep
 
     def get_text_with_eol(self):
-        """Same as 'toPlainText', replace '\n'
-        by correct end-of-line characters"""
-        utext = to_text_string(self.toPlainText())
-        lines = utext.splitlines()
+        """
+        Same as 'toPlainText', replacing '\n' by correct end-of-line
+        characters.
+        """
+        text = self.toPlainText()
         linesep = self.get_line_separator()
-        txt = linesep.join(lines)
-        if utext.endswith('\n'):
-            txt += linesep
-        return txt
+        text.replace('\n', linesep)
+        return text
 
     #------Positions, coordinates (cursor, EOF, ...)
     def get_position(self, subject):


Test patch: