Repository: microsoft/vscode (TypeScript)
Date: 2021-11-10 14:38:17+00:00
Task type: Bug Fix

Issue:
setDecorations using `contentText` do not render on non ASCII lines
<!-- ⚠️⚠️ Do Not Delete This! bug_report_template ⚠️⚠️ -->![image](https://user-images.githubusercontent.com/9395594/140651835-19152086-1ddd-4f69-b3e2-51c230bdb0cc.png)<!-- Please read our Rules of Conduct: https://opensource.microsoft.com/codeofconduct/ -->
<!-- 🕮 Read our guide about submitting issues: https://github.com/microsoft/vscode/wiki/Submitting-Bugs-and-Suggestions -->
<!-- 🔎 Search existing issues to avoid creating duplicates. -->
<!-- 🧪 Test using the latest Insiders build to see if your issue has already been fixed: https://code.visualstudio.com/insiders/ -->
<!-- 💡 Instead of creating your report here, use 'Report Issue' from the 'Help' menu in VS Code to pre-fill useful information. -->
<!-- 🔧 Launch with `code --disable-extensions` to check. -->
Does this issue occur when all extensions are disabled?: Yes/No

<!-- 🪓 If you answered No above, use 'Help: Start Extension Bisect' from Command Palette to try to identify the cause. -->
<!-- 📣 Issues caused by an extension need to be reported directly to the extension publisher. The 'Help > Report Issue' dialog can assist with this. -->
- VS Code Version: 1.62
- OS Version: mac 10.14.6

Steps to Reproduce:

1. install `markless_sean10` from marketplace.
2.  open a markdown file. 
3.  write like following:

``` markdown
* 1
* 2
* 我
```

Expected result:  (In 1.61.2, no such problem)
![image](https://user-images.githubusercontent.com/9395594/140651835-19152086-1ddd-4f69-b3e2-51c230bdb0cc.png)

Actual result:
![image](https://user-images.githubusercontent.com/9395594/140651813-70b80c42-713e-4acf-9ada-de1b9bdef3aa.png)





Issue discussion:
@Sean10 I am having trouble reproducing. Do you have any more information? Here is what I have tried:

* `package.json`:
```json
{
    "name": "136622",
    "publisher": "alex",
    "version": "0.0.1",
    "engines": {
        "vscode": "^1.22.0"
    },
    "activationEvents": ["*"],
    "main": "main.js"
}
```
* `main.js`:
```js
const vscode = require('vscode');

exports.activate = function() {
    const dec = vscode.window.createTextEditorDecorationType({});
    vscode.window.activeTextEditor.setDecorations(dec, [
        {
            range: new vscode.Range(2, 0, 2, 1),
            renderOptions: {
                after: {
                    contentText: '😀'
                }
            }
        },
        {
            range: new vscode.Range(1, 0, 1, 1),
            renderOptions: {
                after: {
                    contentText: '😀'
                }
            }
        }
    ]);
};
```
* `file.txt`:
```
* 1
* 2
* 我
```

And here is the outcome:
![image](https://user-images.githubusercontent.com/5047891/140887808-7109754a-b1e2-48ea-a6f5-80c512dc16ea.png)

> * 1
> * 2
> * 我

@alexdima  Are your vscode version is `1.62`? 

This problem in my device only occur when i upgrade vscode to `1.62`

Here is the outcome with your code in vscode `1.62`
![image](https://user-images.githubusercontent.com/9395594/140949424-f1385749-5008-426a-b10a-770f43720a4a.png)


From what I can tell there are currently 4 open issues caused by the same problem introduced in 1.62.0:

- https://github.com/microsoft/vscode/issues/136622
- https://github.com/microsoft/vscode/issues/136813
- https://github.com/microsoft/vscode/issues/136676
- https://github.com/microsoft/vscode/issues/136690

The issue seems to be that decorations don't work if the line contains non-ASCII characters.  This issue appeared in 1.62.0

Solution patch:
diff --git a/src/vs/editor/common/viewLayout/viewLineRenderer.ts b/src/vs/editor/common/viewLayout/viewLineRenderer.ts
--- a/src/vs/editor/common/viewLayout/viewLineRenderer.ts
+++ b/src/vs/editor/common/viewLayout/viewLineRenderer.ts
@@ -474,6 +474,11 @@ function resolveRenderLineInput(input: RenderLineInput): ResolvedRenderLineInput
 	}
 
 	let tokens = transformAndRemoveOverflowing(input.lineTokens, input.fauxIndentLength, len);
+	if (input.renderControlCharacters && !input.isBasicASCII) {
+		// Calling `extractControlCharacters` before adding (possibly empty) line parts
+		// for inline decorations. `extractControlCharacters` removes empty line parts.
+		tokens = extractControlCharacters(lineContent, tokens);
+	}
 	if (input.renderWhitespace === RenderWhitespace.All ||
 		input.renderWhitespace === RenderWhitespace.Boundary ||
 		(input.renderWhitespace === RenderWhitespace.Selection && !!input.selectionsOnLine) ||
@@ -500,9 +505,6 @@ function resolveRenderLineInput(input: RenderLineInput): ResolvedRenderLineInput
 		// We can never split RTL text, as it ruins the rendering
 		tokens = splitLargeTokens(lineContent, tokens, !input.isBasicASCII || input.fontLigatures);
 	}
-	if (input.renderControlCharacters && !input.isBasicASCII) {
-		tokens = extractControlCharacters(lineContent, tokens);
-	}
 
 	return new ResolvedRenderLineInput(
 		input.useMonospaceOptimizations,


Test patch:
diff --git a/src/vs/editor/test/common/viewLayout/viewLineRenderer.test.ts b/src/vs/editor/test/common/viewLayout/viewLineRenderer.test.ts
--- a/src/vs/editor/test/common/viewLayout/viewLineRenderer.test.ts
+++ b/src/vs/editor/test/common/viewLayout/viewLineRenderer.test.ts
@@ -1722,6 +1722,46 @@ suite('viewLineRenderer.renderLine 2', () => {
 		assert.deepStrictEqual(actual.html, expected);
 	});
 
+	test('issue #136622: Inline decorations are not rendering on non-ASCII lines when renderControlCharacters is on', () => {
+
+		let actual = renderViewLine(new RenderLineInput(
+			true,
+			true,
+			'some text £',
+			false,
+			false,
+			false,
+			0,
+			createViewLineTokens([createPart(11, 3)]),
+			[
+				new LineDecoration(5, 5, 'inlineDec1', InlineDecorationType.After),
+				new LineDecoration(6, 6, 'inlineDec2', InlineDecorationType.Before),
+			],
+			4,
+			0,
+			10,
+			10,
+			10,
+			10000,
+			'none',
+			true,
+			false,
+			null
+		));
+
+		let expected = [
+			'<span>',
+			'<span class="mtk3">some</span>',
+			'<span class="mtk3 inlineDec1"></span>',
+			'<span class="mtk3">\u00a0</span>',
+			'<span class="mtk3 inlineDec2"></span>',
+			'<span class="mtk3">text\u00a0£</span>',
+			'</span>'
+		].join('');
+
+		assert.deepStrictEqual(actual.html, expected);
+	});
+
 	test('issue #22832: Consider fullwidth characters when rendering tabs', () => {
 
 		let actual = renderViewLine(new RenderLineInput(