Repository: microsoft/vscode (TypeScript)
Date: 2021-11-11 08:45:03+00:00
Task type: Bug Fix

Issue:
Semantic highlighting flickers on 1.62.0 and later
<!-- ⚠️⚠️ Do Not Delete This! bug_report_template ⚠️⚠️ -->
<!-- Please read our Rules of Conduct: https://opensource.microsoft.com/codeofconduct/ -->
<!-- 🕮 Read our guide about submitting issues: https://github.com/microsoft/vscode/wiki/Submitting-Bugs-and-Suggestions -->
<!-- 🔎 Search existing issues to avoid creating duplicates. -->
<!-- 🧪 Test using the latest Insiders build to see if your issue has already been fixed: https://code.visualstudio.com/insiders/ -->
<!-- 💡 Instead of creating your report here, use 'Report Issue' from the 'Help' menu in VS Code to pre-fill useful information. -->
<!-- 🔧 Launch with `code --disable-extensions` to check. -->
Does this issue occur when all extensions are disabled?: Yes(except for rust-analyzer as that provides the highlighting)

<!-- 🪓 If you answered No above, use 'Help: Start Extension Bisect' from Command Palette to try to identify the cause. -->
<!-- 📣 Issues caused by an extension need to be reported directly to the extension publisher. The 'Help > Report Issue' dialog can assist with this. -->
- VS Code Version: 1.62.0-insider b3318bc0524af3d74034b8bb8a64df0ccf35549a
- OS Version: Windows 10

Steps to Reproduce:

1. Open a rust project with the rust-analyzer installed
2. Type in a rust file where semantic highlighted tokens can be observed.

All the semantic tokens will lose their coloring until the server resends the highlighting response that has been requested by the client. This does not occur on VSCode 1.61.0 it does occur on 1.62.0(non-insider) though.

cc https://github.com/rust-analyzer/rust-analyzer/issues/10690
> 
> 
> I'm not sure when this started since I only started to notice it and get annoyed by it earlier yesterday but as of recently the syntax highlighting that underlines mutable bindings in VSCode has been persistently annoying. Everytime I start and stop typing, every mutable binding on my screen briefly loses its underlining. This occurs even if I'm not changing any actual code
> 
> Even weirder, it only really happens when I type two characters consecutively. Typing one, pausing, then typing another doesn't trigger it. See below:
> 
> ![bfa7682f-e613-4871-bdaf-00d0edd77481](https://user-images.githubusercontent.com/20936452/140256338-a8cd6fa2-0c79-4894-9749-24073bd79549.gif)
> 
> Let me know if there's anything else I can provide, but here are my relevant versions:
> 
> rust-analyzer version: [a824768](https://github.com/rust-analyzer/rust-analyzer/commit/a8247685cfa09084bd620c0877ea1eb3d605d8a2) 2021-11-04 nightly vscode-insiders version: 1.62.0-insider b3318bc0524af3d74034b8bb8a64df0ccf35549a OS: Win 10


Issue discussion:
@Veykril I cannot reproduce the flicker on Linux. I believe this is not a problem with semantic tokens, but a problem with the rendering of underlines. To rule out a problem related with underlines, could you please configure the following:

```
  "editor.semanticTokenColorCustomizations": {
    "rules": {
      "parameter": "#ff0000"
    }
  }
```


https://user-images.githubusercontent.com/5047891/141199601-417054c1-36d7-4a39-b7b0-1427a81e6d7c.mp4


To give you an idea, this is the kind of issue we have seen in the past: https://github.com/microsoft/vscode/issues/76642
This is unfortunately not just an issue with underlines as can be seen here, the issue becomes more apparent in bigger rust files(when the server takes longer to calculate the highlighting again) but is also visible for me in very small files already ![MIF0HEaPFI](https://user-images.githubusercontent.com/3757771/141201005-01086aa6-82e3-4487-993d-2cc5303510ca.gif)

Version: 1.62.0 (user setup)
Commit: b3318bc0524af3d74034b8bb8a64df0ccf35549a
Date: 2021-11-03T15:23:01.379Z (1 wk ago)
Electron: 13.5.1
Chrome: 91.0.4472.164
Node.js: 14.16.0
V8: 9.1.269.39-electron.0
OS: Windows_NT x64 10.0.19042

Solution patch:
diff --git a/src/vs/editor/common/services/getSemanticTokens.ts b/src/vs/editor/common/services/getSemanticTokens.ts
--- a/src/vs/editor/common/services/getSemanticTokens.ts
+++ b/src/vs/editor/common/services/getSemanticTokens.ts
@@ -27,6 +27,7 @@ export class DocumentSemanticTokensResult {
 	constructor(
 		public readonly provider: DocumentSemanticTokensProvider,
 		public readonly tokens: SemanticTokens | SemanticTokensEdits | null,
+		public readonly error: any
 	) { }
 }
 
@@ -45,10 +46,11 @@ export async function getDocumentSemanticTokens(model: ITextModel, lastProvider:
 	// Get tokens from all providers at the same time.
 	const results = await Promise.all(providers.map(async (provider) => {
 		let result: SemanticTokens | SemanticTokensEdits | null | undefined;
+		let error: any = null;
 		try {
 			result = await provider.provideDocumentSemanticTokens(model, (provider === lastProvider ? lastResultId : null), token);
 		} catch (err) {
-			onUnexpectedExternalError(err);
+			error = err;
 			result = null;
 		}
 
@@ -56,11 +58,15 @@ export async function getDocumentSemanticTokens(model: ITextModel, lastProvider:
 			result = null;
 		}
 
-		return new DocumentSemanticTokensResult(provider, result);
+		return new DocumentSemanticTokensResult(provider, result, error);
 	}));
 
-	// Try to return the first result with actual tokens
+	// Try to return the first result with actual tokens or
+	// the first result which threw an error (!!)
 	for (const result of results) {
+		if (result.error) {
+			throw result.error;
+		}
 		if (result.tokens) {
 			return result;
 		}


Test patch:
diff --git a/src/vs/editor/test/common/services/getSemanticTokens.test.ts b/src/vs/editor/test/common/services/getSemanticTokens.test.ts
new file mode 100644
--- /dev/null
+++ b/src/vs/editor/test/common/services/getSemanticTokens.test.ts
@@ -0,0 +1,44 @@
+/*---------------------------------------------------------------------------------------------
+ *  Copyright (c) Microsoft Corporation. All rights reserved.
+ *  Licensed under the MIT License. See License.txt in the project root for license information.
+ *--------------------------------------------------------------------------------------------*/
+
+import * as assert from 'assert';
+import { CancellationToken } from 'vs/base/common/cancellation';
+import { canceled } from 'vs/base/common/errors';
+import { DisposableStore } from 'vs/base/common/lifecycle';
+import { ITextModel } from 'vs/editor/common/model';
+import { DocumentSemanticTokensProvider, DocumentSemanticTokensProviderRegistry, ProviderResult, SemanticTokens, SemanticTokensEdits, SemanticTokensLegend } from 'vs/editor/common/modes';
+import { getDocumentSemanticTokens } from 'vs/editor/common/services/getSemanticTokens';
+import { createTextModel } from 'vs/editor/test/common/editorTestUtils';
+
+suite('getSemanticTokens', () => {
+
+	test('issue #136540: semantic highlighting flickers', async () => {
+		const disposables = new DisposableStore();
+
+		const provider = new class implements DocumentSemanticTokensProvider {
+			getLegend(): SemanticTokensLegend {
+				return { tokenTypes: ['test'], tokenModifiers: [] };
+			}
+			provideDocumentSemanticTokens(model: ITextModel, lastResultId: string | null, token: CancellationToken): ProviderResult<SemanticTokens | SemanticTokensEdits> {
+				throw canceled();
+			}
+			releaseDocumentSemanticTokens(resultId: string | undefined): void {
+			}
+		};
+
+		disposables.add(DocumentSemanticTokensProviderRegistry.register('testLang', provider));
+
+		const textModel = disposables.add(createTextModel('example', undefined, 'testLang'));
+
+		await getDocumentSemanticTokens(textModel, null, null, CancellationToken.None).then((res) => {
+			assert.fail();
+		}, (err) => {
+			assert.ok(!!err);
+		});
+
+		disposables.dispose();
+	});
+
+});