Repository: sublimelsp/LSP

Issue:
Formatting adding trailing newline clears last line
* OS and language server
Linux + Gopls
* How you installed LSP (Package Control or from git?)
Package Control
* Minimal reproduction steps
```go
package main

import (
	"fmt"
)

func main() {
	fmt.Println("Hello, world")
} // No newline!
```

Format

```go
package main

import (
	"fmt"
)

func main() {
	fmt.Println("Hello, world")
```
* Log
No diagnostic output. 

Initially reported to [gopls](https://github.com/golang/go/issues/33717), but they pointed out that the gopls commandline does the right thing.

Is this a LSP issue or Sublime itself?
Let me know if I can provide any other helpful information!


Discussion:
Managed to reproduce this and get the payload so we can see what is happening.
```
[
    {
        "range": {
            "start": {
                "line": 8,
                "character": 0
            },
            "end": {
                "line": 9,
                "character": 0
            }
        },
        "newText": ""
    },
    {
        "range": {
            "start": {
                "line": 9,
                "character": 0
            },
            "end": {
                "line": 9,
                "character": 0
            }
        },
        "newText": "} // No newline!\n"
    }
]
```
I've added a debug print to see how the edits are applied.
This seems to trip up our "do the edits in reverse-order" trick:
```
LSP: applying change: (92,92) } // No newline!
LSP: applying change: (76,109) 
```
cc @haferburg

Let's bring in the spec :)

> Text edits ranges must never overlap, that means no part of the original document must be manipulated by more than one edit. However, it is possible that multiple edits have the same start position: multiple inserts, or any number of inserts followed by a single remove or replace edit. If multiple inserts have the same position, the order in the array defines the order in which the inserted strings appear in the resulting text.

In this case it looks like this is breaking spec. We first remove the line `} // No newline!` and then insert at that position.

However we could also fix this by changing [this line](https://github.com/tomv564/LSP/blob/61fe0bc46f8a3730906712049cd42514381b9a4b/plugin/core/edit.py#L38) to
```
        return change[1][0], change[1][1], -index
```
I'd say the two edits are adjacent, but not overlapping. So I don't think this is breaking the spec.

But I also don't see how this input would trip up the algorithm. If you first insert the "} // No newline!\n" at 9:0, then delete 8:0 to 9:0.

But it looks like this might be some corner case because it is editing the last line of the file. Maybe the Sublime API is weird in this regard?
Wait a second, so the original file has no newline, but the deletion is still from 8:0 to 9:0, even though there is no line 9. How come?
I would expect the deletion to be 8:0 to 8:16.
Just a note that coc.nvim uses this library

https://github.com/microsoft/vscode-languageserver-node

to defer its sorting logic of the text edits. I'm not an expert at TypeScript but I believe the relevant logic for it is here:

https://github.com/microsoft/vscode-languageserver-node/blob/9175d3894ed952b0ffd416e55c03c877b475b51e/types/src/main.ts#L2523-L2580
Nice find @rwols, I like their logic but they can return the entire text of the edited document _and_ refer to the unmodified original for each of the edits. Perhaps there's an approach that lets us return to this simplicity while avoiding the slowness of the initial approach we had?
> Perhaps there's an approach that lets us return to this simplicity while avoiding the slowness of the initial approach we had?

We would have to implement our own "convert row-col to offset", that would be painful.

There's this doc comment about what happens to invalid points:

https://github.com/microsoft/vscode-languageserver-node/blob/9175d3894ed952b0ffd416e55c03c877b475b51e/types/src/main.ts#L25

To reiterate what's said there: the line number is clamped between 0 and #lines (inclusive).

It then depends on what the `offsetAt` function does with such a point, and what `substring` does when its arguments are (or may be) OOB. Does `substring` create newlines?
I want to mark this as a language server bug. We have thought about it for a while. We verified our unit tests. And we discovered an out-of-bounds position provided by the server. It depends on the implementation of the text editor client what will happen with OOB positions, and language servers shouldn't depend on implemention-defined behavior.
> If a line number is greater than the number of lines in a document, it defaults back to the number of lines in the document.

What if we apply this to the incoming positions? Will it fix the issue?
> What if we apply this to the incoming positions? Will it fix the issue?

After screwing around in the console we could perhaps use this info to our advantage:
```
>>> view.rowcol(view.size())[0]
49
>>> view.rowcol(view.size() - 1)
(48, 69)
```

Resolving patch:
diff --git a/plugin/core/edit.py b/plugin/core/edit.py
--- a/plugin/core/edit.py
+++ b/plugin/core/edit.py
@@ -1,4 +1,5 @@
 from .url import uri_to_filename
+import operator
 try:
     from typing import List, Dict, Optional, Any, Iterable, Tuple
     TextEdit = Tuple[Tuple[int, int], Tuple[int, int], str]
@@ -32,11 +33,6 @@ def parse_text_edit(text_edit: 'Dict[str, Any]') -> 'TextEdit':
 
 
 def sort_by_application_order(changes: 'Iterable[TextEdit]') -> 'List[TextEdit]':
-
-    def get_start_position(pair: 'Tuple[int, TextEdit]'):
-        index, change = pair
-        return change[0][0], change[0][1], index
-
     # The spec reads:
     # > However, it is possible that multiple edits have the same start position: multiple
     # > inserts, or any number of inserts followed by a single remove or replace edit. If
@@ -45,4 +41,4 @@ def get_start_position(pair: 'Tuple[int, TextEdit]'):
     # So we sort by start position. But if multiple text edits start at the same position,
     # we use the index in the array as the key.
 
-    return list(map(lambda pair: pair[1], sorted(enumerate(changes), key=get_start_position, reverse=True)))
+    return list(sorted(changes, key=operator.itemgetter(0)))
diff --git a/plugin/edit.py b/plugin/edit.py
--- a/plugin/edit.py
+++ b/plugin/edit.py
@@ -41,14 +41,24 @@ def open_and_apply_edits(self, path, file_changes):
 
 
 class LspApplyDocumentEditCommand(sublime_plugin.TextCommand):
+
     def run(self, edit, changes: 'Optional[List[TextEdit]]' = None):
         # Apply the changes in reverse, so that we don't invalidate the range
         # of any change that we haven't applied yet.
         if changes:
-            for change in sort_by_application_order(changes):
+            last_row, last_col = self.view.rowcol(self.view.size())
+            for change in reversed(sort_by_application_order(changes)):
                 start, end, newText = change
                 region = sublime.Region(self.view.text_point(*start), self.view.text_point(*end))
-                self.apply_change(region, newText, edit)
+
+                if start[0] > last_row and newText[0] != '\n':
+                    # Handle when a language server (eg gopls) inserts at a row beyond the document
+                    # some editors create the line automatically, sublime needs to have the newline prepended.
+                    debug('adding new line for edit at line {}, document ended at line {}'.format(start[0], last_row))
+                    self.apply_change(region, '\n' + newText, edit)
+                    last_row, last_col = self.view.rowcol(self.view.size())
+                else:
+                    self.apply_change(region, newText, edit)
 
     def apply_change(self, region: 'sublime.Region', newText: str, edit):
         if region.empty():


Test patch:
diff --git a/plugin/core/test_edit.py b/plugin/core/test_edit.py
--- a/plugin/core/test_edit.py
+++ b/plugin/core/test_edit.py
@@ -58,14 +58,14 @@ class SortByApplicationOrderTests(unittest.TestCase):
     def test_empty_sort(self):
         self.assertEqual(sort_by_application_order([]), [])
 
-    def test_sorts_backwards(self):
+    def test_sorts_in_application_order(self):
         edits = [
             ((0, 0), (0, 0), 'b'),
             ((0, 0), (0, 0), 'a'),
             ((0, 2), (0, 2), 'c')
         ]
         # expect 'c' (higher start), 'a' now reverse order before 'b'
-        sorted = sort_by_application_order(edits)
-        self.assertEqual(sorted[0][2], 'c')
-        self.assertEqual(sorted[1][2], 'a')
-        self.assertEqual(sorted[2][2], 'b')
+        sorted_edits = sort_by_application_order(edits)
+        self.assertEqual(sorted_edits[0][2], 'b')
+        self.assertEqual(sorted_edits[1][2], 'a')
+        self.assertEqual(sorted_edits[2][2], 'c')
diff --git a/tests/test_edit.py b/tests/test_edit.py
--- a/tests/test_edit.py
+++ b/tests/test_edit.py
@@ -13,6 +13,29 @@ class ApplyDocumentEditTests(DeferrableTestCase):
     def setUp(self):
         self.view = sublime.active_window().new_file()
 
+    def test_remove_line_and_then_insert_at_that_line_at_end(self):
+        original = (
+            'a\n'
+            'b\n'
+            'c'
+        )
+        file_changes = [
+            ((2, 0), (3, 0), ''),  # out-of-bounds end position, but this is fine
+            ((3, 0), (3, 0), 'c\n')  # out-of-bounds start and end, this line doesn't exist
+        ]
+        expected = (
+            'a\n'
+            'b\n'
+            'c\n'
+        )
+        # Old behavior:
+        # 1) first we end up with ('a\n', 'b\n', 'cc\n')
+        # 2) then we end up with ('a\n', 'b\n', '')
+        # New behavior:
+        # 1) line index 3 is "created" ('a\n', 'b\n', 'c\n', c\n'))
+        # 2) deletes line index 2.
+        self.run_test(original, expected, file_changes)
+
     def test_apply(self):
         original = (
             '<dom-module id="some-thing">\n'