Repository: scalameta/scalameta (scala)

Issue:
Memory overflow occurs when some special symbols are encountered.
**Describe the bug**

When encountering `。` outside the string, the memory and cpu usage of the language server rises rapidly and eventually crashes due to OutOfMemory.

**To Reproduce** Steps to reproduce the behavior:

Add a `。` in a scala file.

**Installation:**

- Operating system: Windows
- VSCode version: 1.93.1
- VSCode extension version: v1.39.6 (pre-release) (aslo happen in v1.39.0)
- Metals version: 1.3.5


**Search terms**

OutOfMemoryError

Pull request description:
Fixes https://github.com/scalameta/metals-vscode/issues/1540

Patch:
diff --git a/scalameta/tokenizers/shared/src/main/scala/scala/meta/internal/tokenizers/LegacyScanner.scala b/scalameta/tokenizers/shared/src/main/scala/scala/meta/internal/tokenizers/LegacyScanner.scala
index 835ae68d29..352a71ca1b 100644
--- a/scalameta/tokenizers/shared/src/main/scala/scala/meta/internal/tokenizers/LegacyScanner.scala
+++ b/scalameta/tokenizers/shared/src/main/scala/scala/meta/internal/tokenizers/LegacyScanner.scala
@@ -448,7 +448,10 @@ class LegacyScanner(input: Input, dialect: Dialect) {
           } else if (isSpecial(ch)) {
             putCharAndNext()
             getOperatorRest()
-          } else reportIllegalCharacter()
+          } else {
+            reportIllegalCharacter()
+            nextChar()
+          }
         fetchOther()
     }
   }


Test patch:
diff --git a/tests/shared/src/test/scala/scala/meta/tests/tokenizers/TokenizerSuite.scala b/tests/shared/src/test/scala/scala/meta/tests/tokenizers/TokenizerSuite.scala
index 6da932fedc..1c3c7bbc98 100644
--- a/tests/shared/src/test/scala/scala/meta/tests/tokenizers/TokenizerSuite.scala
+++ b/tests/shared/src/test/scala/scala/meta/tests/tokenizers/TokenizerSuite.scala
@@ -2136,5 +2136,24 @@ class TokenizerSuite extends BaseTokenizerSuite {
          |""".stripMargin.replace("'''", "\"\"\"").nl2lf
     assertTokenizedAsStructureLines(code, struct)
   }
+  test("unexpected-character") {
+    val code = """|
+                  |val 。 = 123
+                  |
+                  |""".stripMargin.replace("'''", "\"\"\"")
+
+    val struct = s"""|BOF [0..0)
+                     |LF [0..1)
+                     |KwVal [1..4)
+                     |Invalid(illegal character '\\u3002') [5..6)
+                     |MultiHS(3) [4..7)
+                     |Equals [7..8)
+                     |Space [8..9)
+                     |Constant.Int(123) [9..12)
+                     |MultiNL(2) [12..14)
+                     |EOF [14..14)
+                     |""".stripMargin.nl2lf
+    assertTokenizedAsStructureLines(code, struct)
+  }
 
 }