Repository: sublimelsp/LSP

Issue:
Error diagnostic highlight doesn't appear with semantic highlighting enabled
**Describe the bug**

With LSP-typescript and semantic highlighting enabled, I often notice that the error highlighting don't show up in the document if a token is also highlighted by semantic highlighting.

**To Reproduce**
Steps to reproduce the behavior:
1. Check this line with LSP-typescript for example: https://github.com/microsoft/vscode/blob/4be6df74d4ea0c6511c42d5951df0d4f78f1cf83/extensions/typescript-language-features/src/languageFeatures/inlayHints.ts#L72-L72

**Screenshots**

The `hint` word is not marked with a squiggly line:

![Screenshot 2022-01-16 at 21 52 17](https://user-images.githubusercontent.com/153197/149677707-4035c586-6161-42a6-8fe2-6e701e4cf805.png)


**Environment (please complete the following information):**
- OS: macOS
- Sublime Text version: 4125
- LSP version: 1.15.0
- Language servers used: LSP-typescript



Discussion:
I believe that is a consequence from appending the modifier to the semantic token scope in https://github.com/sublimelsp/LSP/blob/de0ef67befd7d2e813ebe7fcabacdb143ff9e535/plugin/core/sessions.py#L117 (for the purpose of being able to define a color scheme rule for tokens with modifiers). In this case the region key for this token, which is derived from the scope, will be `lsp_variable.parameter.lsp meta.semantic-token.parameter.declaration.lsp`.

But when we initialize the region keys, we use only the scopes defined in our predefined mapping: https://github.com/sublimelsp/LSP/blob/de0ef67befd7d2e813ebe7fcabacdb143ff9e535/plugin/session_view.py#L118

For this case we don't have a special scope defined, only for "parameter" token without any modifier: https://github.com/sublimelsp/LSP/blob/de0ef67befd7d2e813ebe7fcabacdb143ff9e535/plugin/core/views.py#L131

I'm not sure how this can be solved in a general way, because we don't know beforehand which of the possible combinations of token types & modifiers the server will use. It might work to add all possible combinations of token types & modifiers which were announced by the server, but that would also mean a lot of unnecessary region keys.

As a quick solution, you can explicitly declare this token with a scope in LSP-typescript.sublime-settings, for example
```json
{
    "semantic_tokens": {
        "parameter.declaration": "variable.parameter.lsp"
    }
}
```

Then the corresponding region key will also be initialized. I've tested it a few times with this adjustment, and it seems to work most of the time - but for some reason still not everytime reliably. Hm, maybe there is still some unknown behavior of the region draw order by ST, that we didn't consider yet.

As a sidenote, I found an unrelated bug that we forgot to lowercase the scope name when initializing the region keys, so the don't necessarily match in a few cases.
Thanks for the explanation.

I would maybe use the workaround for now but I guess the same issue could apply to other types of tokens.
![image](https://user-images.githubusercontent.com/36449676/167441962-eaa99e90-5b8a-48e6-9fce-c3d6618953d8.png)

It seems to me that the region draw always draws the highlight lines first then draws the semantic highlight over it hence the issue,
dun know if its possible to fix their order tho

Resolving patch:
diff --git a/plugin/core/sessions.py b/plugin/core/sessions.py
--- a/plugin/core/sessions.py
+++ b/plugin/core/sessions.py
@@ -1083,7 +1083,7 @@ def __init__(self, manager: Manager, logger: Logger, workspace_folders: List[Wor
         self._plugin = None  # type: Optional[AbstractPlugin]
         self._status_messages = {}  # type: Dict[str, str]
         self.diagnostics_manager = DiagnosticsManager()
-        self.semantic_tokens_map = get_semantic_tokens_map(config.semantic_tokens)
+        self._semantic_tokens_map = get_semantic_tokens_map(config.semantic_tokens)
 
     def __getattr__(self, name: str) -> Any:
         """
@@ -1532,7 +1532,7 @@ def decode_semantic_token(
         types_legend = tuple(cast(List[str], self.get_capability('semanticTokensProvider.legend.tokenTypes')))
         modifiers_legend = tuple(cast(List[str], self.get_capability('semanticTokensProvider.legend.tokenModifiers')))
         return decode_semantic_token(
-            types_legend, modifiers_legend, self.semantic_tokens_map, token_type_encoded, token_modifiers_encoded)
+            types_legend, modifiers_legend, self._semantic_tokens_map, token_type_encoded, token_modifiers_encoded)
 
     # --- server request handlers --------------------------------------------------------------------------------------
 
diff --git a/plugin/session_buffer.py b/plugin/session_buffer.py
--- a/plugin/session_buffer.py
+++ b/plugin/session_buffer.py
@@ -12,7 +12,7 @@
 from .core.types import debounced
 from .core.types import Debouncer
 from .core.types import FEATURES_TIMEOUT
-from .core.typing import Any, Callable, Iterable, Optional, List, Dict, Tuple
+from .core.typing import Any, Callable, Iterable, Optional, List, Set, Dict, Tuple
 from .core.views import DIAGNOSTIC_SEVERITY
 from .core.views import diagnostic_severity
 from .core.views import did_change
@@ -61,12 +61,12 @@ def __init__(self, severity: int) -> None:
 class SemanticTokensData:
 
     __slots__ = (
-        'data', 'result_id', 'active_scopes', 'tokens', 'view_change_count', 'needs_refresh', 'pending_response')
+        'data', 'result_id', 'active_region_keys', 'tokens', 'view_change_count', 'needs_refresh', 'pending_response')
 
     def __init__(self) -> None:
         self.data = []  # type: List[int]
         self.result_id = None  # type: Optional[str]
-        self.active_scopes = []  # type: List[str]
+        self.active_region_keys = set()  # type: Set[int]
         self.tokens = []  # type: List[SemanticToken]
         self.view_change_count = 0
         self.needs_refresh = False
@@ -105,6 +105,8 @@ def __init__(self, session_view: SessionViewProtocol, buffer_id: int, uri: Docum
         self.diagnostics_debouncer = Debouncer()
         self.color_phantoms = sublime.PhantomSet(view, "lsp_color")
         self.semantic_tokens = SemanticTokensData()
+        self._semantic_region_keys = {}  # type: Dict[str, int]
+        self._last_semantic_region_key = 0
         self._check_did_open(view)
         self._session.register_session_buffer_async(self)
 
@@ -502,7 +504,7 @@ def _draw_semantic_tokens_async(self) -> None:
         if view is None:
             return
         self.semantic_tokens.tokens.clear()
-        scope_regions = dict()  # type: Dict[str, List[sublime.Region]]
+        scope_regions = dict()  # type: Dict[int, Tuple[str, List[sublime.Region]]]
         prev_line = 0
         prev_col_utf16 = 0
         for idx in range(0, len(self.semantic_tokens.data), 5):
@@ -526,7 +528,6 @@ def _draw_semantic_tokens_async(self) -> None:
                 # This logic should not be cached (in the decode_semantic_token method) because otherwise new user
                 # customizations in the color scheme for the scopes of custom token types would require a restart of
                 # Sublime Text to take effect.
-                # Note that the region keys for these scopes are not initialized in SessionView._initialize_region_keys.
                 token_general_style = view.style_for_scope("meta.semantic-token")
                 token_type_style = view.style_for_scope("meta.semantic-token.{}".format(token_type.lower()))
                 if token_general_style["source_line"] != token_type_style["source_line"] or \
@@ -537,24 +538,30 @@ def _draw_semantic_tokens_async(self) -> None:
                         scope = "meta.semantic-token.{}.lsp".format(token_type.lower())
             self.semantic_tokens.tokens.append(SemanticToken(r, token_type, token_modifiers))
             if scope:
-                scope_regions.setdefault(scope, []).append(r)
+                scope_regions.setdefault(self._get_semantic_region_key_for_scope(scope), (scope, []))[1].append(r)
         # don't update regions if there were additional changes to the buffer in the meantime
         if self.semantic_tokens.view_change_count != view.change_count():
             return
-        for scope in self.semantic_tokens.active_scopes.copy():
-            if scope not in scope_regions.keys():
-                self.semantic_tokens.active_scopes.remove(scope)
+        for region_key in self.semantic_tokens.active_region_keys.copy():
+            if region_key not in scope_regions.keys():
+                self.semantic_tokens.active_region_keys.remove(region_key)
                 for sv in self.session_views:
-                    sv.view.erase_regions("lsp_{}".format(scope))
-        for scope, regions in scope_regions.items():
-            if scope not in self.semantic_tokens.active_scopes:
-                self.semantic_tokens.active_scopes.append(scope)
+                    sv.view.erase_regions("lsp_semantic_{}".format(region_key))
+        for region_key, (scope, regions) in scope_regions.items():
+            if region_key not in self.semantic_tokens.active_region_keys:
+                self.semantic_tokens.active_region_keys.add(region_key)
             for sv in self.session_views:
-                sv.view.add_regions("lsp_{}".format(scope), regions, scope, flags=sublime.DRAW_NO_OUTLINE)
+                sv.view.add_regions("lsp_semantic_{}".format(region_key), regions, scope, flags=sublime.DRAW_NO_OUTLINE)
+
+    def _get_semantic_region_key_for_scope(self, scope: str) -> int:
+        if scope not in self._semantic_region_keys:
+            self._last_semantic_region_key += 1
+            self._semantic_region_keys[scope] = self._last_semantic_region_key
+        return self._semantic_region_keys[scope]
 
     def _clear_semantic_token_regions(self, view: sublime.View) -> None:
-        for scope in self.semantic_tokens.active_scopes:
-            view.erase_regions("lsp_{}".format(scope))
+        for region_key in self.semantic_tokens.active_region_keys:
+            view.erase_regions("lsp_semantic_{}".format(region_key))
 
     def set_semantic_tokens_pending_refresh(self, needs_refresh: bool = True) -> None:
         self.semantic_tokens.needs_refresh = needs_refresh
diff --git a/plugin/session_view.py b/plugin/session_view.py
--- a/plugin/session_view.py
+++ b/plugin/session_view.py
@@ -114,8 +114,8 @@ def _initialize_region_keys(self) -> None:
         document_highlight_kinds = ["text", "read", "write"]
         line_modes = ["m", "s"]
         self.view.add_regions(self.CODE_ACTIONS_KEY, r)  # code actions lightbulb icon should always be on top
-        for key, scope in self.session.semantic_tokens_map:
-            self.view.add_regions("lsp_{} meta.semantic-token.{}.lsp".format(scope, key.lower()), r)
+        for key in range(1, 100):
+            self.view.add_regions("lsp_semantic_{}".format(key), r)
         if document_highlight_style == "fill":
             for kind in document_highlight_kinds:
                 for mode in line_modes:


Test patch: