Repository: microsoft/vscode (TypeScript)
Task: Bug Fix

Issue:
Wrong highlight if there are duplicate strings in completion item
<!-- ⚠️⚠️ Do Not Delete This! bug_report_template ⚠️⚠️ -->
<!-- Please read our Rules of Conduct: https://opensource.microsoft.com/codeofconduct/ -->
<!-- Please search existing issues to avoid creating duplicates. -->
<!-- Also please test using the latest insiders build to make sure your issue has not already been fixed: https://code.visualstudio.com/insiders/ -->

<!-- Use Help > Report Issue to prefill these. -->
- VSCode Version: 1.52.1
- OS Version: MacOS Big Sur 11.1

If a suggestion contains two "div", the second "div" is wrongly highlighted after "di" is entered

![div](https://user-images.githubusercontent.com/8640918/103114926-9e05fd80-469b-11eb-8211-81d53189c122.gif)



<!-- Launch with `code --disable-extensions` to check. -->
Does this issue occur when all extensions are disabled?: No, with my customed completion extension

Wrong highlight if there are duplicate strings in completion item
<!-- ⚠️⚠️ Do Not Delete This! bug_report_template ⚠️⚠️ -->
<!-- Please read our Rules of Conduct: https://opensource.microsoft.com/codeofconduct/ -->
<!-- Please search existing issues to avoid creating duplicates. -->
<!-- Also please test using the latest insiders build to make sure your issue has not already been fixed: https://code.visualstudio.com/insiders/ -->

<!-- Use Help > Report Issue to prefill these. -->
- VSCode Version: 1.52.1
- OS Version: MacOS Big Sur 11.1

If a suggestion contains two "div", the second "div" is wrongly highlighted after "di" is entered

![div](https://user-images.githubusercontent.com/8640918/103114926-9e05fd80-469b-11eb-8211-81d53189c122.gif)



<!-- Launch with `code --disable-extensions` to check. -->
Does this issue occur when all extensions are disabled?: No, with my customed completion extension



Discussion:
(Experimental duplicate detection)
Thanks for submitting this issue. Please also check if it is already covered by an existing one, like:
- [Onclick div it will show outer div boundary instead of current div (#70316)](https://www.github.com/microsoft/vscode/issues/70316) <!-- score: 0.544 -->
- [Emmet expression with attributes not expanding (#92231)](https://www.github.com/microsoft/vscode/issues/92231) <!-- score: 0.505 -->
- [\[folding\] Folding HTML tags should hide the closing tag.  (#24515)](https://www.github.com/microsoft/vscode/issues/24515) <!-- score: 0.502 -->
<!-- potential_duplicates_comment -->
@jrieken I'd like to open a PR to fix it, would you please give me some hints about where can I start?
Steps to reproduce it:

1. Create an extension, use the following `extension.ts`
```typescript
import * as vscode from 'vscode';

export function activate(context: vscode.ExtensionContext) {
	const provider1 = vscode.languages.registerCompletionItemProvider('plaintext', {
		provideCompletionItems(document: vscode.TextDocument, position: vscode.Position, token: vscode.CancellationToken, context: vscode.CompletionContext) {
			const simpleCompletion = new vscode.CompletionItem('⭐div classname=""></div>');
			return [simpleCompletion];
		}
	}, ' ', '.', '<');
	context.subscriptions.push(provider1);
}

export function deactivate() {}
```
2. Set activateEvent in `package.json`:
```json
"activationEvents": [
		"*"
	],
```
3. open a txt and enter character "d"
![image](https://user-images.githubusercontent.com/8640918/103371327-248e6500-4b0a-11eb-8dca-4632225de15f.png)

only the later "d" is highlighted
Understood. Our fuzzy score function treats certain characters as separators, like `_` and `.`. However, emojis and such aren't on that list. The code is here: https://github.com/microsoft/vscode/blob/696dca786ac73ca0779a65e18f4bdb613d66b26f/src/vs/base/common/filters.ts#L466
> Understood. Our fuzzy score function treats certain characters as separators, like `_` and `.`. However, emojis and such aren't on that list.

@jrieken Thanks for the hint! Here is another case, the later match of `di` is highlighted even though the first character is not emoji nor symbol.
![image](https://user-images.githubusercontent.com/8640918/103535623-08a10f80-4ecc-11eb-8e2a-c40e3eb2f58e.png)

It seems that just adding emoji or symbols to `isSeparatorAtPos` couldn't solve the entire problem. 

That is a different case. Having `adiv` and `di` isn't match for us because the first match (`d` in `ad`) is considered weak. A strong match is one after a separator or a the word start and matches much start with a strong match.
So the `adiv` case is by design, right? 
yes, see here https://github.com/microsoft/vscode/issues/53715
Thanks, I'll open a PR to fix the emoji case ;D
(Experimental duplicate detection)
Thanks for submitting this issue. Please also check if it is already covered by an existing one, like:
- [Onclick div it will show outer div boundary instead of current div (#70316)](https://www.github.com/microsoft/vscode/issues/70316) <!-- score: 0.544 -->
- [Emmet expression with attributes not expanding (#92231)](https://www.github.com/microsoft/vscode/issues/92231) <!-- score: 0.505 -->
- [\[folding\] Folding HTML tags should hide the closing tag.  (#24515)](https://www.github.com/microsoft/vscode/issues/24515) <!-- score: 0.502 -->
<!-- potential_duplicates_comment -->
@jrieken I'd like to open a PR to fix it, would you please give me some hints about where can I start?
Steps to reproduce it:

1. Create an extension, use the following `extension.ts`
```typescript
import * as vscode from 'vscode';

export function activate(context: vscode.ExtensionContext) {
	const provider1 = vscode.languages.registerCompletionItemProvider('plaintext', {
		provideCompletionItems(document: vscode.TextDocument, position: vscode.Position, token: vscode.CancellationToken, context: vscode.CompletionContext) {
			const simpleCompletion = new vscode.CompletionItem('⭐div classname=""></div>');
			return [simpleCompletion];
		}
	}, ' ', '.', '<');
	context.subscriptions.push(provider1);
}

export function deactivate() {}
```
2. Set activateEvent in `package.json`:
```json
"activationEvents": [
		"*"
	],
```
3. open a txt and enter character "d"
![image](https://user-images.githubusercontent.com/8640918/103371327-248e6500-4b0a-11eb-8dca-4632225de15f.png)

only the later "d" is highlighted
Understood. Our fuzzy score function treats certain characters as separators, like `_` and `.`. However, emojis and such aren't on that list. The code is here: https://github.com/microsoft/vscode/blob/696dca786ac73ca0779a65e18f4bdb613d66b26f/src/vs/base/common/filters.ts#L466
> Understood. Our fuzzy score function treats certain characters as separators, like `_` and `.`. However, emojis and such aren't on that list.

@jrieken Thanks for the hint! Here is another case, the later match of `di` is highlighted even though the first character is not emoji nor symbol.
![image](https://user-images.githubusercontent.com/8640918/103535623-08a10f80-4ecc-11eb-8e2a-c40e3eb2f58e.png)

It seems that just adding emoji or symbols to `isSeparatorAtPos` couldn't solve the entire problem. 

That is a different case. Having `adiv` and `di` isn't match for us because the first match (`d` in `ad`) is considered weak. A strong match is one after a separator or a the word start and matches much start with a strong match.
So the `adiv` case is by design, right? 
yes, see here https://github.com/microsoft/vscode/issues/53715
Thanks, I'll open a PR to fix the emoji case ;D

Patch:
diff --git a/src/vs/base/common/filters.ts b/src/vs/base/common/filters.ts
--- a/src/vs/base/common/filters.ts
+++ b/src/vs/base/common/filters.ts
@@ -467,7 +467,7 @@ function isSeparatorAtPos(value: string, index: number): boolean {
 	if (index < 0 || index >= value.length) {
 		return false;
 	}
-	const code = value.charCodeAt(index);
+	const code = value.codePointAt(index);
 	switch (code) {
 		case CharCode.Underline:
 		case CharCode.Dash:
@@ -479,8 +479,16 @@ function isSeparatorAtPos(value: string, index: number): boolean {
 		case CharCode.DoubleQuote:
 		case CharCode.Colon:
 		case CharCode.DollarSign:
+		case CharCode.LessThan:
+		case CharCode.OpenParen:
+		case CharCode.OpenSquareBracket:
 			return true;
+		case undefined:
+			return false;
 		default:
+			if (strings.isEmojiImprecise(code)) {
+				return true;
+			}
 			return false;
 	}
 }


Test patch:
diff --git a/src/vs/base/test/common/filters.test.ts b/src/vs/base/test/common/filters.test.ts
--- a/src/vs/base/test/common/filters.test.ts
+++ b/src/vs/base/test/common/filters.test.ts
@@ -534,6 +534,11 @@ suite('Filters', () => {
 		assert.ok(Boolean(match));
 	});
 
+	test('Wrong highlight after emoji #113404', function () {
+		assertMatches('di', '✨div classname=""></div>', '✨^d^iv classname=""></div>', fuzzyScore);
+		assertMatches('di', 'adiv classname=""></div>', 'adiv classname=""></^d^iv>', fuzzyScore);
+	});
+
 	test('Suggestion is not highlighted #85826', function () {
 		assertMatches('SemanticTokens', 'SemanticTokensEdits', '^S^e^m^a^n^t^i^c^T^o^k^e^n^sEdits', fuzzyScore);
 		assertMatches('SemanticTokens', 'SemanticTokensEdits', '^S^e^m^a^n^t^i^c^T^o^k^e^n^sEdits', fuzzyScoreGracefulAggressive);