Repository: microsoft/vscode (TypeScript)
Task: Bug Fix

Issue:
onDidChangeSemanticTokens doesn't consistently trigger provideDocumentSemanticTokens
<!-- ⚠️⚠️ Do Not Delete This! bug_report_template ⚠️⚠️ -->
<!-- Please read our Rules of Conduct: https://opensource.microsoft.com/codeofconduct/ -->
<!-- 🕮 Read our guide about submitting issues: https://github.com/microsoft/vscode/wiki/Submitting-Bugs-and-Suggestions -->
<!-- 🔎 Search existing issues to avoid creating duplicates. -->
<!-- 🧪 Test using the latest Insiders build to see if your issue has already been fixed: https://code.visualstudio.com/insiders/ -->
<!-- 💡 Instead of creating your report here, use 'Report Issue' from the 'Help' menu in VS Code to pre-fill useful information. -->
<!-- 🔧 Launch with `code --disable-extensions` to check. -->
Does this issue occur when all extensions are disabled?: Extension Development

<!-- 🪓 If you answered No above, use 'Help: Start Extension Bisect' from Command Palette to try to identify the cause. -->
<!-- 📣 Issues caused by an extension need to be reported directly to the extension publisher. The 'Help > Report Issue' dialog can assist with this. -->
- VS Code Version: 1.71.2
- OS Version: Windows 10

For my extension I'm implementing the SemanticTokenProvider. This works great when edits are being parsed by my server, and everything looks good.

However, I've created functionality regarding TextEdits so they are bundled together if typed within a short period of each other. Every new edit will cancel the message to the server, and bundle the old and new edits and add it to the queue once more.

Due to how the SemanticTokenProvider works, it will fire itself after every edit, but sometimes the actual edit is removed from the queue, and the provider will act on data which hasn't changed.

To fix this, I created an event which is fired when TextEdits are actually sent to the server. When it is fired, I get the vscode.EventEmitter that is connected to the onDidChangeSemanticTokens Event, and fire it. Unfortunately, this doesn't actually seem to trigger the Provider, which seems like a bug.

I created a dummy command, which also asks the Provider for new tokens, and when this command is executed, it does ask for new tokens. If I execute the command from my event, it does not work.

See the basics of my implementation here:

Here the vscode.Event and vscode.EventEmitter

```
private mOnSentTextEditsEventEmitter: vscode.EventEmitter<void> = new vscode.EventEmitter<void>();
private mOnSentTextEditsEvent: vscode.Event<void> = this.mOnSentTextEditsEventEmitter.event;
```
My callback implementation. The then() is called once the queued message has been sent.

```
vscode.workspace.onDidChangeTextDocument( (event: vscode.TextDocumentChangeEvent) => {

        // Executing code to queue a textedit message
                    
        // If succesfully sent, clear the text edit array.
        this.mQueuedNotification.then( () => {
            this.mTextEdits = [];
            this.mQueuedNotification = null;

            this.mOnSentTextEditsEventEmitter.fire();
        });

)})
```
The function to register a callback

```
RegisterOnSentTextEdits(inThis: unknown, inCallback: () => void)
{
    this.mOnSentTextEditsEvent.call(inThis, inCallback);
}
```
Here I register the callback by calling the above function

```
file_watcher.RegisterOnSentTextEdits(semantic_highlighter_provider_instance, () => {
        semantic_highlighter_provider_instance.mOnDidchangeSemanticTokens.fire();
})
```
I do reach this last part, so I know my callback fires correctly. However, after this fire(), I never hit the breakpoint in provideDocumentSemanticTokens()

I also created a command for this, which does trigger correctly and causes the provider to ask for new tokens, which looks like follows:

```
let update_semantic_tokens = vscode.commands.registerCommand('update_semantic_tokens', () => {
        semantic_highlighter_provider_instance.mOnDidchangeSemanticTokens.fire();
    })
```

As the onDidChangeSemanticTokens event is handled within VSCode, I am unable to properly debug what is going on here.



Discussion:
The logic on our side is to [listen to change events coming in from semantic tokens providers](https://github.com/microsoft/vscode/blob/8e1235ee25e3aad3598ab58016c071b5596b826a/src/vs/editor/common/services/modelService.ts#L817) and then we schedule a request for semantic tokens [**if one is not currently running**](https://github.com/microsoft/vscode/blob/8e1235ee25e3aad3598ab58016c071b5596b826a/src/vs/editor/common/services/modelService.ts#L852-L855). That means that you first need to resolve the current provider request and only afterwards trigger the emitter. 

The flow could be something like this: In case the server is not ready to provide semantic tokens, from the provider you can throw a [`CancellationError`](https://github.com/microsoft/vscode/blob/8e1235ee25e3aad3598ab58016c071b5596b826a/src/vscode-dts/vscode.d.ts#L1534). We will not complain in logs and we will not throw away the previous result id, as we use this error kind exactly for this situation Then, when the server is ready to provide semantic tokens, you can fire the change event and we will request again.


I see. That makes sense. It would be nice to have this rule be documented for the event. Or have some logging that the event is denied due to having an already active one. But I'd understand if that could clutter consoles too much. 

Thanks for the explanation!

Patch:
diff --git a/src/vs/editor/common/services/modelService.ts b/src/vs/editor/common/services/modelService.ts
--- a/src/vs/editor/common/services/modelService.ts
+++ b/src/vs/editor/common/services/modelService.ts
@@ -770,6 +770,7 @@ export class ModelSemanticColoring extends Disposable {
 	private _currentDocumentResponse: SemanticTokensResponse | null;
 	private _currentDocumentRequestCancellationTokenSource: CancellationTokenSource | null;
 	private _documentProvidersChangeListeners: IDisposable[];
+	private _providersChangedDuringRequest: boolean;
 
 	constructor(
 		model: ITextModel,
@@ -789,6 +790,7 @@ export class ModelSemanticColoring extends Disposable {
 		this._currentDocumentResponse = null;
 		this._currentDocumentRequestCancellationTokenSource = null;
 		this._documentProvidersChangeListeners = [];
+		this._providersChangedDuringRequest = false;
 
 		this._register(this._model.onDidChangeContent(() => {
 			if (!this._fetchDocumentSemanticTokens.isScheduled()) {
@@ -814,7 +816,14 @@ export class ModelSemanticColoring extends Disposable {
 			this._documentProvidersChangeListeners = [];
 			for (const provider of this._provider.all(model)) {
 				if (typeof provider.onDidChange === 'function') {
-					this._documentProvidersChangeListeners.push(provider.onDidChange(() => this._fetchDocumentSemanticTokens.schedule(0)));
+					this._documentProvidersChangeListeners.push(provider.onDidChange(() => {
+						if (this._currentDocumentRequestCancellationTokenSource) {
+							// there is already a request running,
+							this._providersChangedDuringRequest = true;
+							return;
+						}
+						this._fetchDocumentSemanticTokens.schedule(0);
+					}));
 				}
 			}
 		};
@@ -868,6 +877,7 @@ export class ModelSemanticColoring extends Disposable {
 		const lastResultId = this._currentDocumentResponse ? this._currentDocumentResponse.resultId || null : null;
 		const request = getDocumentSemanticTokens(this._provider, this._model, lastProvider, lastResultId, cancellationTokenSource.token);
 		this._currentDocumentRequestCancellationTokenSource = cancellationTokenSource;
+		this._providersChangedDuringRequest = false;
 
 		const pendingChanges: IModelContentChangedEvent[] = [];
 		const contentChangeListener = this._model.onDidChangeContent((e) => {
@@ -898,7 +908,7 @@ export class ModelSemanticColoring extends Disposable {
 			this._currentDocumentRequestCancellationTokenSource = null;
 			contentChangeListener.dispose();
 
-			if (pendingChanges.length > 0) {
+			if (pendingChanges.length > 0 || this._providersChangedDuringRequest) {
 				// More changes occurred while the request was running
 				if (!this._fetchDocumentSemanticTokens.isScheduled()) {
 					this._fetchDocumentSemanticTokens.schedule(this._debounceInformation.get(this._model));
@@ -918,7 +928,7 @@ export class ModelSemanticColoring extends Disposable {
 	private _setDocumentSemanticTokens(provider: DocumentSemanticTokensProvider | null, tokens: SemanticTokens | SemanticTokensEdits | null, styling: SemanticTokensProviderStyling | null, pendingChanges: IModelContentChangedEvent[]): void {
 		const currentResponse = this._currentDocumentResponse;
 		const rescheduleIfNeeded = () => {
-			if (pendingChanges.length > 0 && !this._fetchDocumentSemanticTokens.isScheduled()) {
+			if ((pendingChanges.length > 0 || this._providersChangedDuringRequest) && !this._fetchDocumentSemanticTokens.isScheduled()) {
 				this._fetchDocumentSemanticTokens.schedule(this._debounceInformation.get(this._model));
 			}
 		};


Test patch:
diff --git a/src/vs/editor/test/common/services/modelService.test.ts b/src/vs/editor/test/common/services/modelService.test.ts
--- a/src/vs/editor/test/common/services/modelService.test.ts
+++ b/src/vs/editor/test/common/services/modelService.test.ts
@@ -5,7 +5,7 @@
 
 import * as assert from 'assert';
 import { CharCode } from 'vs/base/common/charCode';
-import { Event } from 'vs/base/common/event';
+import { Emitter, Event } from 'vs/base/common/event';
 import * as platform from 'vs/base/common/platform';
 import { URI } from 'vs/base/common/uri';
 import { EditOperation } from 'vs/editor/common/core/editOperation';
@@ -543,6 +543,40 @@ suite('ModelSemanticColoring', () => {
 		});
 	});
 
+	test('issue #161573: onDidChangeSemanticTokens doesn\'t consistently trigger provideDocumentSemanticTokens', async () => {
+		await runWithFakedTimers({}, async () => {
+
+			disposables.add(languageService.registerLanguage({ id: 'testMode' }));
+
+			const emitter = new Emitter<void>();
+			let requestCount = 0;
+			disposables.add(languageFeaturesService.documentSemanticTokensProvider.register('testMode', new class implements DocumentSemanticTokensProvider {
+				onDidChange = emitter.event;
+				getLegend(): SemanticTokensLegend {
+					return { tokenTypes: ['class'], tokenModifiers: [] };
+				}
+				async provideDocumentSemanticTokens(model: ITextModel, lastResultId: string | null, token: CancellationToken): Promise<SemanticTokens | SemanticTokensEdits | null> {
+					requestCount++;
+					if (requestCount === 1) {
+						await timeout(1000);
+						// send a change event
+						emitter.fire();
+						await timeout(1000);
+						return null;
+					}
+					return null;
+				}
+				releaseDocumentSemanticTokens(resultId: string | undefined): void {
+				}
+			}));
+
+			disposables.add(modelService.createModel('', languageService.createById('testMode')));
+
+			await timeout(5000);
+			assert.deepStrictEqual(requestCount, 2);
+		});
+	});
+
 	test('DocumentSemanticTokens should be pick the token provider with actual items', async () => {
 		await runWithFakedTimers({}, async () => {