Repository: microsoft/vscode (TypeScript)
Task: Bug Fix

Issue:
Completions which use spaces are broken after updating from July to August version
<!-- ⚠️⚠️ Do Not Delete This! bug_report_template ⚠️⚠️ -->
<!-- Please read our Rules of Conduct: https://opensource.microsoft.com/codeofconduct/ -->
<!-- Please search existing issues to avoid creating duplicates. -->
<!-- Also please test using the latest insiders build to make sure your issue has not already been fixed: https://code.visualstudio.com/insiders/ -->

<!-- Use Help > Report Issue to prefill these. -->
- VSCode Version: August 2020
- OS Version: Windows 10

Updating to August from July version breaks completions which use spaces.  When I start typing the first word, everything seems fine:

![image](https://user-images.githubusercontent.com/2399754/93011894-8058d080-f5a3-11ea-8a09-06c0c1e9c531.png)

However, when I get to the second word, the completion starts anew from the second word:

![image](https://user-images.githubusercontent.com/2399754/93011901-96ff2780-f5a3-11ea-91a6-f6c253aadd88.png)

Contrast this with how the completions work in July version:

![image](https://user-images.githubusercontent.com/2399754/93011908-b1d19c00-f5a3-11ea-800c-e92b4d0a01b8.png)

<!-- Launch with `code --disable-extensions` to check. -->

These completions are generated by the Latex Workshop extension. All other extensions are disabled. However, Workshop developers say that they have no control over the completions.



Discussion:
I am a maintainer of LaTeX-Workshop. I confirm that the same version of the extension is used with vscode `1.49` and `1.48.2`.

The list of suggestions are `completionItem`s with undefined `filterText ` and `range` attributes. In both version, invoking `document.getWordRangeAtPosition(vscode.window.activeTextEditor.selection.active)` returns the current white space delimited word. 

Let me know if I can provide more information/context.
likely due to https://github.com/microsoft/vscode/commit/37ebb445e250275cf794de93a37533f7c65f7616

> n both version, invoking document.getWordRangeAtPosition(vscode.window.activeTextEditor.selection.active) returns the current white space delimited word.

With the change mentioned above we will invoke completions again when entering the next word. Can you confirm that in both invocation the correct, full,  word is returned? I would be confused because the word definition that defines the range also defines that start of the new word. 

Also, with 1.48, what happens when you invoke IntelliSense manually (F1 > Trigger Suggest). Does the word start after the curly `{` or after the space?


fyi @esbenk
We are having a complete backend language server written in C#. We rely on LSP protocol. The protocol message flow have changed with the version 1.49 and now a new "textDocument/completion" message with a different position is sent after the "special" character. We rely on the LSP message and their position to provide completionItems.

btw we supports quoted identifier like "Sales Cr. Memo-Printed". This have work perfectly for the last 3 years.

It seems like you now do the filtering based on what is typed after the second message.

1.48 works also as expected with the manual invocation.

> The protocol message flow have changed with the version 1.49 and now a new "textDocument/completion" message with a different position is sent after the "special" character.

Yes - see the bug fix mentioned above. I still think that was a valid fix

> 1.48 works also as expected with the manual invocation.

So, you say that manual invoking intellisene after the special character does the right thing?
I have the provider below and things work for me. It is important to define a correct range because the default range goes for words (which by default are broken up at whitespaces)

```ts
vscode.languages.registerCompletionItemProvider('fooLang', new class implements vscode.CompletionItemProvider {
    provideCompletionItems(document: vscode.TextDocument, position: vscode.Position, token: vscode.CancellationToken, context: vscode.CompletionContext): vscode.ProviderResult<vscode.CompletionItem[] | vscode.CompletionList<vscode.CompletionItem>> {

        const startPos = document.lineAt(position).text.lastIndexOf('{', position.character);
        if (startPos < 0) {
            return;
        }
        const range = new vscode.Range(position.line, startPos + 1, position.line, position.character);
        return [{
            label: 'This is a suggestion',
            range
        }, {
            label: 'These suggestion have a range',
            range
        }, {
            label: 'because they span multiple words',
            range
        }]
    }
})
```
I don't think that your definition of new words holds true for all different languages out there. At least not for ours. 

1.48 works with a manual "Trigger Suggest".

![wQHFRlXUoH](https://user-images.githubusercontent.com/24257835/93177471-26871080-f733-11ea-9b08-c5517453b622.gif)

What range(s) for completion item are you returning then? Can you add a sample that is copy-pastable so that I can repo?
What do mean by ranges for completion items? 

We are using the LanguageClient with our own backend language server. 

I can't provide a copy-pastable sample. I can provide you with a zipped folder structure your can open and try with.
> What do mean by ranges for completion items?

I mean the [range of the completion item](https://github.com/microsoft/vscode/blob/e02afd19048014222134762f5b606e30eb9d95df/src/vs/vscode.d.ts#L3874), as it is shown in my snippet above. I believe in LSP land that's defined via the textEdit of a completion.

> I can't provide a copy-pastable sample. I can provide you with a zipped folder structure your can open and try with.

Whatever allows me to reproduce/debug this. Animated gifs don't.
Install the AL Language extension from marketplace.
If you unzip the attached folder and open it with VsCode.
Open HelloWorld.al
Start typing `Crazy  meth` after the `c.` in line 8

Try with both 1.49 and 1.48

[IntellisenseTest.zip](https://github.com/microsoft/vscode/files/5223195/IntellisenseTest.zip)

We are not sending ranges back. We get this request over the LSP to our backend

```
{
  "jsonrpc": "2.0",
  "id": 13,
  "method": "textDocument/completion",
  "params": {
    "textDocument": {
      "uri": "file:///c%3A/Users/esbenk/Documents/AL/IntellisenseTest/HelloWorld.al"
    },
    "position": {
      "line": 7,
      "character": 10
    },
    "context": {
      "triggerKind": 1
    }
  }
}

```

and returns this back to the LanguageClient.

```
{
  "jsonrpc": "2.0",
  "id": 13,
  "result": [
    {
      "label": "\"Crazy method name\"",
      "kind": 2,
      "detail": "procedure \"Crazy method name\"()",
      "documentation": null,
      "sortText": "1006Crazy method name",
      "filterText": "\"Crazy method name\"",
      "insertText": null,
      "insertTextFormat": 1,
      "data": null,
      "range": {
        "start": null,
        "end": null
      },
      "textEdit": {
        "range": {
          "start": {
            "line": 7,
            "character": 10
          },
          "end": {
            "line": 7,
            "character": 10
          }
        },
        "newText": "\"Crazy method name\"();"
      },
      "tags": null
    },
    {
      "label": "\"Some-other-name\"",
      "kind": 2,
      "detail": "procedure \"Some-other-name\"()",
      "documentation": null,
      "sortText": "1006Some-other-name",
      "filterText": "\"Some-other-name\"",
      "insertText": null,
      "insertTextFormat": 1,
      "data": null,
      "range": {
        "start": null,
        "end": null
      },
      "textEdit": {
        "range": {
          "start": {
            "line": 7,
            "character": 10
          },
          "end": {
            "line": 7,
            "character": 10
          }
        },
        "newText": "\"Some-other-name\"();"
      },
      "tags": null
    },
    {
      "label": "Run",
      "kind": 2,
      "detail": "procedure Run(): Boolean",
      "documentation": null,
      "sortText": "1019Run",
      "filterText": "\"Run\"",
      "insertText": null,
      "insertTextFormat": 1,
      "data": null,
      "range": {
        "start": null,
        "end": null
      },
      "textEdit": {
        "range": {
          "start": {
            "line": 7,
            "character": 10
          },
          "end": {
            "line": 7,
            "character": 10
          }
        },
        "newText": "Run()"
      },
      "tags": null
    },
    {
      "label": "Test",
      "kind": 2,
      "detail": "procedure Test()",
      "documentation": null,
      "sortText": "1006Test",
      "filterText": "\"Test\"",
      "insertText": null,
      "insertTextFormat": 1,
      "data": null,
      "range": {
        "start": null,
        "end": null
      },
      "textEdit": {
        "range": {
          "start": {
            "line": 7,
            "character": 10
          },
          "end": {
            "line": 7,
            "character": 10
          }
        },
        "newText": "Test();"
      },
      "tags": null
    }
  ]
}

```


Not sending back a range is the problem here and now just more visible but also present with 1.48. Do the following

* use 1.48
* in your sample, open `HelloWorld.al`
* on line 8 after `c.` type `crazy m`
* hit ESC to cancel quick suggestion (or disable quick suggestions via `editor.quickSuggestions`)
* manually trigger suggestion via `Ctrl+Space`
* :bug: no suggestion 

it seems that the completion provider is unaware of the anchor of the current completion, e.g the `.`-character of the `c` and therefore only works when triggered at the word following it


![Sep-15-2020 09-58-16](https://user-images.githubusercontent.com/1794099/93182533-13c40a00-f73a-11ea-8f1f-788f580be754.gif)


I do acknowledge the issue but it I am unsure how to proceed. Yes, the fix for https://github.com/microsoft/vscode/issues/99504 does have a negative impact on your extension but in the end it just shows an existing bug in your extension more prominently. 
@jrieken thanks for your detailed explanations. It is now clear how we can fix this within the extension.
IMO invoking completion at after `c.cr m n` should not give any results. The code is broken at that point. 

After the `.` we provide a list of valid CompletionItems based on the position. After the first special character + letter you send another textDocument/completion request from a new position where the syntax is now broken.

Can this changed behavior (the re-triggering) be made configurable. I know that our notion of identifier are a bit different than most languages but we are probably not alone. This has been working like this for many years now.

I have a hard time seeing how we can fix this on our end. We don't have the original invoked position anymore. Am I missing something?
> IMO invoking completion at after c.cr m n should not give any results. The code is broken at that point.

Yeah - depending on the language that makes sense

> Can this changed behavior (the re-triggering) be made configurable. I know that our notion of identifier are a bit different than most languages but we are probably not alone. 

No, we cannot make this configurable as that would allow you suppress other providers from working. What I will be looking into is to skip those providers that already have results when re-triggering on entering a new word.

Patch:
diff --git a/src/vs/editor/contrib/suggest/completionModel.ts b/src/vs/editor/contrib/suggest/completionModel.ts
--- a/src/vs/editor/contrib/suggest/completionModel.ts
+++ b/src/vs/editor/contrib/suggest/completionModel.ts
@@ -56,6 +56,7 @@ export class CompletionModel {
 	private _refilterKind: Refilter;
 	private _filteredItems?: StrictCompletionItem[];
 	private _isIncomplete?: Set<CompletionItemProvider>;
+	private _allProvider?: Set<CompletionItemProvider>; // TODO@jrieken merge incomplete and all provider info
 	private _stats?: ICompletionStats;
 
 	constructor(
@@ -99,6 +100,11 @@ export class CompletionModel {
 		return this._filteredItems!;
 	}
 
+	get allProvider(): Set<CompletionItemProvider> {
+		this._ensureCachedState();
+		return this._allProvider!;
+	}
+
 	get incomplete(): Set<CompletionItemProvider> {
 		this._ensureCachedState();
 		return this._isIncomplete!;
@@ -136,6 +142,7 @@ export class CompletionModel {
 	private _createCachedState(): void {
 
 		this._isIncomplete = new Set();
+		this._allProvider = new Set();
 		this._stats = { suggestionCount: 0, snippetCount: 0, textCount: 0 };
 
 		const { leadingLineContent, characterCountDelta } = this._lineContext;
@@ -164,6 +171,7 @@ export class CompletionModel {
 			if (item.container.incomplete) {
 				this._isIncomplete.add(item.provider);
 			}
+			this._allProvider.add(item.provider);
 
 			// 'word' is that remainder of the current line that we
 			// filter and score against. In theory each suggestion uses a
diff --git a/src/vs/editor/contrib/suggest/suggestModel.ts b/src/vs/editor/contrib/suggest/suggestModel.ts
--- a/src/vs/editor/contrib/suggest/suggestModel.ts
+++ b/src/vs/editor/contrib/suggest/suggestModel.ts
@@ -44,6 +44,7 @@ export interface ISuggestEvent {
 export interface SuggestTriggerContext {
 	readonly auto: boolean;
 	readonly shy: boolean;
+	readonly triggerKind?: CompletionTriggerKind;
 	readonly triggerCharacter?: string;
 }
 
@@ -393,16 +394,12 @@ export class SuggestModel implements IDisposable {
 		this._context = ctx;
 
 		// Build context for request
-		let suggestCtx: CompletionContext;
+		let suggestCtx: CompletionContext = { triggerKind: context.triggerKind ?? CompletionTriggerKind.Invoke };
 		if (context.triggerCharacter) {
 			suggestCtx = {
 				triggerKind: CompletionTriggerKind.TriggerCharacter,
 				triggerCharacter: context.triggerCharacter
 			};
-		} else if (onlyFrom && onlyFrom.size > 0) {
-			suggestCtx = { triggerKind: CompletionTriggerKind.TriggerForIncompleteCompletions };
-		} else {
-			suggestCtx = { triggerKind: CompletionTriggerKind.Invoke };
 		}
 
 		this._requestToken = new CancellationTokenSource();
@@ -558,7 +555,13 @@ export class SuggestModel implements IDisposable {
 
 		if (ctx.leadingWord.word.length !== 0 && ctx.leadingWord.startColumn > this._context.leadingWord.startColumn) {
 			// started a new word while IntelliSense shows -> retrigger
-			this.trigger({ auto: this._context.auto, shy: false }, true);
+
+			// Select those providers have not contributed to this completion model and re-trigger completions for
+			// them. Also adopt the existing items and merge them into the new completion model
+			const inactiveProvider = new Set(CompletionProviderRegistry.all(this._editor.getModel()!));
+			this._completionModel.allProvider.forEach(provider => inactiveProvider.delete(provider));
+			const items = this._completionModel.adopt(new Set());
+			this.trigger({ auto: this._context.auto, shy: false }, true, inactiveProvider, items);
 			return;
 		}
 
@@ -566,7 +569,7 @@ export class SuggestModel implements IDisposable {
 			// typed -> moved cursor RIGHT & incomple model & still on a word -> retrigger
 			const { incomplete } = this._completionModel;
 			const adopted = this._completionModel.adopt(incomplete);
-			this.trigger({ auto: this._state === State.Auto, shy: false }, true, incomplete, adopted);
+			this.trigger({ auto: this._state === State.Auto, shy: false, triggerKind: CompletionTriggerKind.TriggerForIncompleteCompletions }, true, incomplete, adopted);
 
 		} else {
 			// typed -> moved cursor RIGHT -> update UI


Test patch:
diff --git a/src/vs/editor/contrib/suggest/test/suggestModel.test.ts b/src/vs/editor/contrib/suggest/test/suggestModel.test.ts
--- a/src/vs/editor/contrib/suggest/test/suggestModel.test.ts
+++ b/src/vs/editor/contrib/suggest/test/suggestModel.test.ts
@@ -815,6 +815,9 @@ suite('SuggestModel - TriggerAndCancelOracle', function () {
 		disposables.push(CompletionProviderRegistry.register({ scheme: 'test' }, {
 			provideCompletionItems(doc, pos) {
 				countB += 1;
+				if (!doc.getWordUntilPosition(pos).word.startsWith('a')) {
+					return;
+				}
 				return {
 					incomplete: false,
 					suggestions: [{
@@ -850,7 +853,7 @@ suite('SuggestModel - TriggerAndCancelOracle', function () {
 				assert.equal(event.completionModel.items[0].textLabel, 'Z aaa');
 				assert.equal(event.completionModel.items[1].textLabel, 'aaa');
 
-				assert.equal(countA, 2); // should we keep the suggestions from the "active" provider?
+				assert.equal(countA, 1); // should we keep the suggestions from the "active" provider?, Yes! See: #106573
 				assert.equal(countB, 2);
 			});
 		});