Repository: vscode-neovim/vscode-neovim (TypeScript)

Issue:
Lower-case marks lost after changes to line of mark in insert mode
**Summary**

It appears changing the buffer (e.g. `cw`) can mess up local marks.

**Steps**

Given following buffer (with cursor at `|`)

```
|Lorem ipsum dolor sit amet.
```

`ma`, `w`, `cw`, `foo`, `<esc>`, `'a`

**Actual**

```
Lorem foo| dolor sit amet.
```

(E20: Mark not set)

**Expected** 

```
|Lorem foo dolor sit amet.
```

(No error)

**Notes**

Upper-case marks (global) are not affected.


Discussion: I did some further debugging:
* Only marks of changed lines are affected (i.e. marks in other lines are not lost)
* Marks are only lost if entering/exiting insert mode (e.g. `cw` not `D`)

Reading debug logs and looking through the code, my best guess so far is that lines [changed](https://github.com/asvetliakov/vscode-neovim/blob/master/src/document_change_manager.ts#L208) in Neovim buffer (after exiting insert mode) with API [method](https://neovim.io/doc/user/api.html#nvim_buf_set_lines()) `nvim_buf_set_lines` loose their marks.

What are your thoughts?
>Reading debug logs and looking through the code, my best guess so far is that lines changed in Neovim buffer (after exiting insert mode) with API method nvim_buf_set_lines loose their marks.

Yes, you're right. `nvim_buf_set_lines` is not preserving marks. Recent neovim nightly builds have new `nvim_buf_set_character` method (if i recall correctly) which doesn't suffer from this, but i haven't got time to look into this yet
Reading through related issues (https://github.com/neovim/neovim/issues/11368, https://github.com/neovim/neovim/pull/11833), it seems this is now called `nvim_buf_set_text`. 

```
nvim_buf_set_text({buffer}, {start_row}, {start_col}, {end_row}, {end_col},{replacement})
                Sets (replaces) a range in the buffer

                This is recommended over nvim_buf_set_lines when only
                modifying parts of a line, as extmarks will be preserved on
                non-modified parts of the touched lines.
```

[prepareEditRangesFromDiff](https://github.com/asvetliakov/vscode-neovim/blob/master/src/utils.ts#L151) does not appear to  report start/end columns yet. Is that feasible with diffs generated by [fast-diff](https://github.com/jhchen/fast-diff)?
> Is that feasible with diffs generated by fast-diff

For reference, I believe it is feasible but not exactly obvious.

```
diff('Lorem ipsum dolores', 'Lorem foo dolores')
[ [ 0, 'Lorem ' ], [ -1, 'ipsum' ], [ 1, 'foo' ], [ 0, ' dolores' ] ]
```
Nah, i believe fast-diff with character diff mode would be kinda slow for typing needs (especially with 1000+ lines), but I think it's possible to send changes immediately from `onDocumentChange` event, if `nvim_buf_set_text` doesn't create unnecessary undo point and handles newline character. May be it would eliminate the need of dot-repeat workaround also.
I see. Sounds like `nvim_buf_set_text` could even simplify the implementation. This would be ideal, of course.

> Nah, i believe fast-diff with character diff mode would be kinda slow for typing needs (especially with 1000+ lines)

I'm probably missing something, but I would assume the diff operation only takes place when leaving insert mode and detecting very few (relatively speaking) changes may take an acceptable time?

Here is a rough [benchmark](https://gist.github.com/trkoch/66933818d965069b0f8dccce19690460) of reading/diffing a paragraph added in the `sqlite3.c` [amalgamation](https://www.sqlite.org/amalgamation.html). It seems `fast-diff` bears its name for valid reasons.

```
Read took 53 milliseconds
[
  1,
  '  if( pPager->nSavepoint>0 ){\n    rc = subjournalPageIfRequired(pPg);\n  }\n'
]
Diff took 3 milliseconds
Benchmark took 61 milliseconds
```
> It seems fast-diff bears its name for valid reasons.

That said editing `sqlite3.c` in Vscode with neovim extension enabled clearly shows performance bottlenecks (as reported in https://github.com/asvetliakov/vscode-neovim/issues/538). 🙃 Though maybe it is not `fast-diff` itself that is slow? 🏃 
I experimented a bit with `nvim_buf_set_text` using `NVIM v0.5.0-dev+1055-g8950f4e94` and [node-client](https://github.com/neovim/node-client).

`nvim` running with socket `/tmp/nvim`

```
$ nvim --listen /tmp/nvim
```

A buffer

```
|one, two
```

Mark `a` at 0/0.

`ma`

```
$ NVIM_LISTEN_ADDRESS=/tmp/vim NVIM_NODE_LOG_FILE=nvim.log node --experimental-repl-await
> const nvim = await require('neovim/scripts/nvim')
> await nvim.request('nvim_buf_set_text', [0, 0, 5, 0, 8, ['TWO']])
```

Results in buffer

```
|one, TWO
```

But mark `a`  is lost.

```
:marks a
E283: No marks matching "a"
```

Seems like [extmarks](https://github.com/neovim/neovim/pull/5031) are not the same as marks.

> but I think it's possible to send changes immediately from onDocumentChange event

Does this event include start/end row/column?

> (…) if nvim_buf_set_text doesn't create unnecessary undo point and handles newline character

In the example above, one undo point was created (i.e. I was able to undo API changes with a single `u`).
> Does this event include start/end row/column?

If I'm reading [TextDocumentChangeEvent](https://code.visualstudio.com/api/references/vscode-api#TextDocumentChangeEvent) correctly it does.
> (...) and handles newline character

Not looking good here.

```
{"message":"Error making request to nvim_buf_set_textnvim_buf_set_text: String cannot contain newlines"}
```
While you're here and have ext local build, can you make another test - disable this line
https://github.com/asvetliakov/vscode-neovim/blob/6469c216f30c4bf29d5718f6d3fe99d517a33db0/src/typing_manager.ts#L128

and check if dot repeat works with `buf_set_text`. If so, we can probably ask neovim team why there is such restriction and is it possible to lift it out
I don't have a local build of the extension running right now, sorry. Only experimenting with a visible instance of `nvim`.

Nevertheless, maybe this helps answering your question?

```
one, |two
```

`gUw`

```
one, |TWO
```

```
> await nvim.request('nvim_buf_set_text', [0, 0, 5, 0, 8, ['xxx']])
```

```
one, |xxx
```

`.`

```
one, |XXX
```

Seems fine to me, at least it's the behavior I would expect.
>> (...) and handles newline character

> Not looking good here.

I should add that the last parameter `replacement` is an array of lines to use as replacement. In my tests, it behaved strangely though: I wasn't able to replace the current line and add new ones.
@theol0403 Now that 0.5 is out, perhaps this can be revisited.
Definitely worth revisiting this. Should improve reliability and allow deleting a chunk of code.
I started working on this at https://github.com/vscode-neovim/vscode-neovim/tree/set-text.
So far it seems to work well, though a couple tests fail.

Unfortunately, it does not seem to fix either dot-repeat or marks. Marks are lost like before, and dot-repeat does not replay any of the changes sent using set_text. 

I tried sending the changes to nvim in real-time, which showed promise. With this, when you type, it sends individual characters to nvim asynchronously (so no typing lag). However, this also didn't fix dot-repeat, and had cursor pos syncronization issues. 
Upstream: https://github.com/neovim/neovim/pull/14630



Fix (patch):
diff --git a/runtime/lua/vscode-neovim/api.lua b/runtime/lua/vscode-neovim/api.lua
index acf77b5b4..4c072e227 100644
--- a/runtime/lua/vscode-neovim/api.lua
+++ b/runtime/lua/vscode-neovim/api.lua
@@ -1,3 +1,5 @@
+local api = vim.api
+local fn = vim.fn
 -- used to execute vscode command
 local command_event_name = "vscode-command"
 -- used for extension communications
@@ -77,4 +79,41 @@ function M.delete_buffers(bufs)
   end
 end
 
+---Handle document changes
+---@param bufnr number
+---@param changes (string | integer)[][]
+---@return number: changed tick of the buffer
+function M.handle_changes(bufnr, changes)
+  -- Save and restore local marks
+  -- Code modified from https://github.com/neovim/neovim/pull/14630
+  local marks = {}
+  for _, m in pairs(fn.getmarklist(bufnr or api.nvim_get_current_buf())) do
+    if m.mark:match("^'[a-z]$") then
+      marks[m.mark:sub(2, 2)] = { m.pos[2], m.pos[3] - 1 } -- api-indexed
+    end
+  end
+
+  for _, change in ipairs(changes) do
+    api.nvim_buf_set_text(bufnr, unpack(change))
+  end
+
+  local max = api.nvim_buf_line_count(bufnr)
+  -- no need to restore marks that still exist
+  for _, m in pairs(fn.getmarklist(bufnr or api.nvim_get_current_buf())) do
+    marks[m.mark:sub(2, 2)] = nil
+  end
+  -- restore marks
+  for mark, pos in pairs(marks) do
+    if pos then
+      -- make sure we don't go out of bounds
+      local line = (api.nvim_buf_get_lines(bufnr, pos[1] - 1, pos[1], false))[1] or ""
+      pos[1] = math.min(pos[1], max)
+      pos[2] = math.min(pos[2], #line)
+      api.nvim_buf_set_mark(bufnr or 0, mark, pos[1], pos[2], {})
+    end
+  end
+
+  return api.nvim_buf_get_changedtick(bufnr)
+end
+
 return M
diff --git a/src/document_change_manager.ts b/src/document_change_manager.ts
index ce8f1ea2b..dfb94f5a0 100644
--- a/src/document_change_manager.ts
+++ b/src/document_change_manager.ts
@@ -493,15 +493,15 @@ export class DocumentChangeManager implements Disposable, NeovimExtensionRequest
             }
         }
 
-        const requests: [string, unknown[]][] = [];
-
-        for (const c of contentChanges) {
-            const start = c.range.start;
-            const end = c.range.end;
-            const text = c.text;
+        const changeArgs = [];
+        for (const change of contentChanges) {
+            const {
+                text,
+                range: { start, end },
+            } = change;
             const startBytes = convertCharNumToByteNum(origText.split(eol)[start.line], start.character);
             const endBytes = convertCharNumToByteNum(origText.split(eol)[end.line], end.character);
-            requests.push(["nvim_buf_set_text", [bufId, start.line, startBytes, end.line, endBytes, text.split(eol)]]);
+            changeArgs.push([start.line, startBytes, end.line, endBytes, text.split(eol)]);
         }
 
         const bufTick: number = await this.client.request("nvim_buf_get_changedtick", [bufId]);
@@ -509,15 +509,13 @@ export class DocumentChangeManager implements Disposable, NeovimExtensionRequest
             this.logger.warn(`${LOG_PREFIX}: Can't get changed tick for bufId: ${bufId}, deleted?`);
             return;
         }
-        this.logger.debug(
-            `${LOG_PREFIX}: BufId: ${bufId}, lineChanges: ${requests.length}, tick: ${bufTick}, skipTick: ${
-                bufTick + requests.length
-            }`,
-        );
-        this.bufferSkipTicks.set(bufId, bufTick + contentChanges.length);
+
+        this.bufferSkipTicks.set(bufId, bufTick + changeArgs.length);
 
         this.logger.debug(`${LOG_PREFIX}: Setting wantInsertCursorUpdate to false`);
         this.main.cursorManager.wantInsertCursorUpdate = false;
-        if (requests.length) await callAtomic(this.client, requests, this.logger, LOG_PREFIX);
+
+        const code = "return require('vscode-neovim.api').handle_changes(...)";
+        await this.client.executeLua(code, [bufId, changeArgs]);
     };
 }