Repository: flix/flix
Pull request #10160: feat: introduce TestCompletionProvider
Author: login: stormckey
Created: 2025-03-18T14:51:31Z | Merged: 2025-03-24T12:41:06Z

fixes #10136

first invariant for #10138

Here we introduce TestCompletionProvider to check invariants over Completion Provider.

This file could be placed somewhere else if there is a convention.

The first invariant is that no completion should be provided after a complete keyword.

Questions:
- Should we hard code the programs to be tests or we can read them from example files?
- Should I place the helper functions in an other file?
- Any suggestions about the api?

Linked issue #10136: Add `TestCompletionProvider`
We want to build some infrastructure for automatic testing. 

I would propose functions like:

- A function that returns the list of source programs want to test on (e.g. could just start with one element, `examples/effects-and-handlers/running...flix`)
- A function to compile a Flix program into a `TypedAst.Root`
- A function to extract all tokens of specific kinds, e.g. keyword tokens.
- A function to take a program and slightly modify it (for example replacing a keyword token "def" by just "de") 
- A function to trigger autocomplete at given source locations and compare them to expected results.

For example, we discussed that triggering auto-complete on keywords should not offer suggestions. 

Diff:
diff --git a/main/src/ca/uwaterloo/flix/language/ast/TokenKind.scala b/main/src/ca/uwaterloo/flix/language/ast/TokenKind.scala
index 53b5cf41f39..0e16cc840ad 100644
--- a/main/src/ca/uwaterloo/flix/language/ast/TokenKind.scala
+++ b/main/src/ca/uwaterloo/flix/language/ast/TokenKind.scala
@@ -705,6 +705,28 @@ sealed trait TokenKind {
     case _ => false
   }
 
+  /**
+    * Checks if this token is a literal.
+    */
+  def isLiteral: Boolean = this match {
+    case TokenKind.LiteralBigDecimal
+         | TokenKind.LiteralBigInt
+         | TokenKind.LiteralChar
+         | TokenKind.LiteralDebugStringL
+         | TokenKind.LiteralDebugStringR
+         | TokenKind.LiteralFloat32
+         | TokenKind.LiteralFloat64
+         | TokenKind.LiteralInt8
+         | TokenKind.LiteralInt16
+         | TokenKind.LiteralInt32
+         | TokenKind.LiteralInt64
+         | TokenKind.LiteralRegex
+         | TokenKind.LiteralString
+         | TokenKind.LiteralStringInterpolationL
+         | TokenKind.LiteralStringInterpolationR => true
+    case _ => false
+  }
+
   /*
     * The following isRecover* functions represent sets of TokenKinds used for error-recovery in the parser.
     * When parsing sequences of expressions in a loop and an unexpected token is found,
diff --git a/main/test/ca/uwaterloo/flix/api/lsp/LspSuite.scala b/main/test/ca/uwaterloo/flix/api/lsp/LspSuite.scala
index 96ef48fc004..e27b67491d7 100644
--- a/main/test/ca/uwaterloo/flix/api/lsp/LspSuite.scala
+++ b/main/test/ca/uwaterloo/flix/api/lsp/LspSuite.scala
@@ -18,5 +18,5 @@ package ca.uwaterloo.flix.api.lsp
 import org.scalatest.Suites
 
 class LspSuite extends Suites(
-  new VisitorSuite
+  new VisitorSuite,
 )
diff --git a/main/test/ca/uwaterloo/flix/api/lsp/TestCompletionProvider.scala b/main/test/ca/uwaterloo/flix/api/lsp/TestCompletionProvider.scala
new file mode 100644
index 00000000000..cc64a9cbd42
--- /dev/null
+++ b/main/test/ca/uwaterloo/flix/api/lsp/TestCompletionProvider.scala
@@ -0,0 +1,171 @@
+/*
+ * Copyright 2025 Chenhao Gao
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package ca.uwaterloo.flix.api.lsp
+
+import ca.uwaterloo.flix.api.Flix
+import ca.uwaterloo.flix.api.lsp.provider.CompletionProvider
+import ca.uwaterloo.flix.language.CompilationMessage
+import ca.uwaterloo.flix.language.ast.Token
+import ca.uwaterloo.flix.language.ast.TypedAst.Root
+import ca.uwaterloo.flix.language.ast.shared.{Input, SecurityContext, Source}
+import ca.uwaterloo.flix.util.Options
+import org.scalatest.funsuite.AnyFunSuite
+
+class TestCompletionProvider extends AnyFunSuite  {
+
+  test("No completions after complete keyword"){
+    checkInvariant(
+      token => token.kind.isKeyword,
+      identity,
+      assertEmptyCompletions
+    )
+  }
+
+  test("No completions after complete literal"){
+    checkInvariant(
+      token => token.kind.isLiteral,
+      identity,
+      assertEmptyCompletions
+    )
+  }
+
+  /**
+    * A list of programs to test invariants on.
+    */
+  private val Programs = List(
+    s"""
+       |def main(): Unit \\ IO =
+       |    run {
+       |        Console.println("Please enter your name: ");
+       |        let name = Console.readln();
+       |        Console.println("Hello $${name}")
+       |    } with Console.runWithIO
+       |
+       |""".stripMargin,
+    s"""
+       |def main(): Unit \\ IO =
+       |    run {
+       |        let timestamp = Clock.currentTime(TimeUnit.Milliseconds);
+       |        println("$${timestamp} ms since the epoc")
+       |    } with Clock.runWithIO
+       |
+       |""".stripMargin,
+    s"""
+       |def main(): Unit \\ {Net, IO} =
+       |    run {
+       |        let url = "http://example.com/";
+       |        Logger.info("Downloading URL: '$${url}'");
+       |        match HttpWithResult.get(url, Map.empty()) {
+       |            case Result.Ok(response) =>
+       |                let file = "data.txt";
+       |                Logger.info("Saving response to file: '$${file}'");
+       |                let body = Http.Response.body(response);
+       |                match FileWriteWithResult.write(str = body, file) {
+       |                    case Result.Ok(_) =>
+       |                        Logger.info("Response saved to file: '$${file}'")
+       |                    case Result.Err(err) =>
+       |                        Logger.fatal("Unable to write file: '$${err}'")
+       |                }
+       |            case Result.Err(err) =>
+       |                Logger.fatal("Unable to download URL: '$${err}'")
+       |        }
+       |    } with FileWriteWithResult.runWithIO
+       |      with HttpWithResult.runWithIO
+       |      with Logger.runWithIO
+       |
+       |""".stripMargin,
+    s"""
+       |def parMap(f: a -> b, l: List[a]): List[b] = match l {
+       |    case Nil     => Nil
+       |    case x :: xs =>
+       |        par (r <- f(x); rs <- parMap(f, xs))
+       |            yield r :: rs
+       |}
+       |
+       |def main(): Unit \\ IO =
+       |    let l = List.range(1, 100);
+       |    println(parMap(x -> x + 1, l))
+       |
+       |""".stripMargin
+  )
+
+  /**
+    * The uri of the test source.
+    */
+  private val Uri = "<test>"
+
+  /**
+    * Compiles the given input string `s` with the given compilation options `o`.
+    */
+  private def compile(s: String, o: Options): (Root, Flix, List[CompilationMessage]) = {
+    implicit val sctx: SecurityContext = SecurityContext.AllPermissions
+    val flix = new Flix().setOptions(o).addSourceCode(Uri, s)
+    flix.check() match {
+      case (Some(root), errors) => (root, flix, errors)
+      case (None, _) => fail("Compilation failed: a root is expected.")
+    }
+  }
+
+  /**
+    * Creates a source object from the given string `content`.
+    */
+  private def mkSource(content: String): Source = {
+    val sctx = SecurityContext.AllPermissions
+    val input = Input.Text(Uri, content, sctx)
+    Source(input, content.toCharArray)
+  }
+
+  /**
+    * Checks the completion provider invariant for the given token filter, string transformer and completion assertion.
+    * For all the Programs to be tested, we use the filter to select tokens to transform, apply the transformer to the selected tokens,
+    * call the completion provider with the transformed program and check the completions against the assertion.
+    *
+    * @param tokenFilter          the filter to select tokens to transform.
+    * @param stringTransformer    the transformer to apply to the selected tokens.
+    * @param completionAssertion  the assertion to check the completions against.
+    */
+  private def checkInvariant(tokenFilter: Token => Boolean, stringTransformer: String => String, completionAssertion: CompletionList => Unit ): Unit = {
+    Programs.foreach { program =>
+      // Compile the original program so that we can locate all target tokens.
+      val (root, flix, errors) = compile(program, Options.Default)
+      val source = mkSource(program)
+      val tokensToTransform = root.tokens(source).toList.filter(tokenFilter)
+      tokensToTransform.foreach { token =>
+        val replacingString = stringTransformer(token.text)
+        val newProgram = program.substring(0, token.start) + replacingString + program.substring(token.end)
+        val (newRoot, newFlix, newErrors) = if (stringTransformer != identity)
+          compile(newProgram, Options.Default)
+        else (root, flix, errors)
+        val newPos = Position(token.sp1.line, token.sp1.col + replacingString.length) // position after the replacement
+        val completions = CompletionProvider.autoComplete(Uri, newPos, newErrors)(newRoot, newFlix)
+        completionAssertion(completions)
+      }
+    }
+  }
+
+  /**
+    * Asserts that the given completions are empty.
+    */
+  private def assertEmptyCompletions(completions: CompletionList): Unit = {
+    assert(completions.items.isEmpty)
+  }
+
+  /**
+    * A function that returns the input string as is.
+    */
+  private def identity(s: String): String = s
+}


Reviews:
user: login: magnus-madsen
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-20T07:05:26Z
user: login: stormckey
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-20T10:54:51Z
user: login: magnus-madsen
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-21T07:32:21Z
user: login: magnus-madsen
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-22T11:31:52Z
user: login: stormckey
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T12:15:22Z
user: login: magnus-madsen
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-24T12:32:07Z
user: login: magnus-madsen
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-24T12:32:41Z

Review comments:
diff_hunk: @@ -0,0 +1,120 @@
+/*
+ * Copyright 2025 Chenhao Gao
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package ca.uwaterloo.flix.api.lsp
+
+import ca.uwaterloo.flix.api.Flix
+import ca.uwaterloo.flix.api.lsp.provider.CompletionProvider
+import ca.uwaterloo.flix.language.CompilationMessage
+import ca.uwaterloo.flix.language.ast.Token
+import ca.uwaterloo.flix.language.ast.TypedAst.Root
+import ca.uwaterloo.flix.language.ast.shared.{Input, SecurityContext, Source}
+import ca.uwaterloo.flix.util.Options
+import org.scalatest.funsuite.AnyFunSuite
+
+class TestCompletionProvider extends AnyFunSuite  {
+  /**
+    * A list of programs to test invariants on.
+    */
+  private val programs = List(
path: main/test/ca/uwaterloo/flix/api/lsp/TestCompletionProvider.scala
user: login: magnus-madsen
body: programs -> Programs
created_at: 2025-03-20T06:58:36Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,120 @@
+/*
+ * Copyright 2025 Chenhao Gao
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package ca.uwaterloo.flix.api.lsp
+
+import ca.uwaterloo.flix.api.Flix
+import ca.uwaterloo.flix.api.lsp.provider.CompletionProvider
+import ca.uwaterloo.flix.language.CompilationMessage
+import ca.uwaterloo.flix.language.ast.Token
+import ca.uwaterloo.flix.language.ast.TypedAst.Root
+import ca.uwaterloo.flix.language.ast.shared.{Input, SecurityContext, Source}
+import ca.uwaterloo.flix.util.Options
+import org.scalatest.funsuite.AnyFunSuite
+
+class TestCompletionProvider extends AnyFunSuite  {
+  /**
+    * A list of programs to test invariants on.
+    */
+  private val programs = List(
+    s"""
+       |def main(): Unit \\ IO =
+       |    run {
+       |        Console.println("Please enter your name: ");
+       |        let name = Console.readln();
+       |        Console.println("Hello $${name}")
+       |    } with Console.runWithIO
+       |""".stripMargin,
+    s"""
+       |def main(): Unit \\ IO =
+       |    run {
+       |        let timestamp = Clock.currentTime(TimeUnit.Milliseconds);
+       |        println("$${timestamp} ms since the epoc")
+       |    } with Clock.runWithIO
+       |""".stripMargin
path: main/test/ca/uwaterloo/flix/api/lsp/TestCompletionProvider.scala
user: login: magnus-madsen
body: I would add a few more complex/longer examples, but the inline list is fine for now.
created_at: 2025-03-20T06:58:56Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,120 @@
+/*
+ * Copyright 2025 Chenhao Gao
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package ca.uwaterloo.flix.api.lsp
+
+import ca.uwaterloo.flix.api.Flix
+import ca.uwaterloo.flix.api.lsp.provider.CompletionProvider
+import ca.uwaterloo.flix.language.CompilationMessage
+import ca.uwaterloo.flix.language.ast.Token
+import ca.uwaterloo.flix.language.ast.TypedAst.Root
+import ca.uwaterloo.flix.language.ast.shared.{Input, SecurityContext, Source}
+import ca.uwaterloo.flix.util.Options
+import org.scalatest.funsuite.AnyFunSuite
+
+class TestCompletionProvider extends AnyFunSuite  {
+  /**
+    * A list of programs to test invariants on.
+    */
+  private val programs = List(
+    s"""
+       |def main(): Unit \\ IO =
+       |    run {
+       |        Console.println("Please enter your name: ");
+       |        let name = Console.readln();
+       |        Console.println("Hello $${name}")
+       |    } with Console.runWithIO
+       |""".stripMargin,
+    s"""
+       |def main(): Unit \\ IO =
+       |    run {
+       |        let timestamp = Clock.currentTime(TimeUnit.Milliseconds);
+       |        println("$${timestamp} ms since the epoc")
+       |    } with Clock.runWithIO
+       |""".stripMargin
+  )
+
+  /**
+    * The uri of the test source.
+    */
+  private val uri = "<test>"
path: main/test/ca/uwaterloo/flix/api/lsp/TestCompletionProvider.scala
user: login: magnus-madsen
body: -> Uri
created_at: 2025-03-20T06:59:34Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,120 @@
+/*
+ * Copyright 2025 Chenhao Gao
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package ca.uwaterloo.flix.api.lsp
+
+import ca.uwaterloo.flix.api.Flix
+import ca.uwaterloo.flix.api.lsp.provider.CompletionProvider
+import ca.uwaterloo.flix.language.CompilationMessage
+import ca.uwaterloo.flix.language.ast.Token
+import ca.uwaterloo.flix.language.ast.TypedAst.Root
+import ca.uwaterloo.flix.language.ast.shared.{Input, SecurityContext, Source}
+import ca.uwaterloo.flix.util.Options
+import org.scalatest.funsuite.AnyFunSuite
+
+class TestCompletionProvider extends AnyFunSuite  {
+  /**
+    * A list of programs to test invariants on.
+    */
+  private val programs = List(
+    s"""
+       |def main(): Unit \\ IO =
+       |    run {
+       |        Console.println("Please enter your name: ");
+       |        let name = Console.readln();
+       |        Console.println("Hello $${name}")
+       |    } with Console.runWithIO
+       |""".stripMargin,
+    s"""
+       |def main(): Unit \\ IO =
+       |    run {
+       |        let timestamp = Clock.currentTime(TimeUnit.Milliseconds);
+       |        println("$${timestamp} ms since the epoc")
+       |    } with Clock.runWithIO
+       |""".stripMargin
+  )
+
+  /**
+    * The uri of the test source.
+    */
+  private val uri = "<test>"
+
+  /**
+    * Compiles the given input string `s` with the given compilation options `o`.
+    */
+  private def compile(s: String, o: Options): (Root, Flix, List[CompilationMessage]) = {
path: main/test/ca/uwaterloo/flix/api/lsp/TestCompletionProvider.scala
user: login: magnus-madsen
body: This is fine
created_at: 2025-03-20T06:59:42Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,120 @@
+/*
+ * Copyright 2025 Chenhao Gao
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package ca.uwaterloo.flix.api.lsp
+
+import ca.uwaterloo.flix.api.Flix
+import ca.uwaterloo.flix.api.lsp.provider.CompletionProvider
+import ca.uwaterloo.flix.language.CompilationMessage
+import ca.uwaterloo.flix.language.ast.Token
+import ca.uwaterloo.flix.language.ast.TypedAst.Root
+import ca.uwaterloo.flix.language.ast.shared.{Input, SecurityContext, Source}
+import ca.uwaterloo.flix.util.Options
+import org.scalatest.funsuite.AnyFunSuite
+
+class TestCompletionProvider extends AnyFunSuite  {
+  /**
+    * A list of programs to test invariants on.
+    */
+  private val programs = List(
+    s"""
+       |def main(): Unit \\ IO =
+       |    run {
+       |        Console.println("Please enter your name: ");
+       |        let name = Console.readln();
+       |        Console.println("Hello $${name}")
+       |    } with Console.runWithIO
+       |""".stripMargin,
+    s"""
+       |def main(): Unit \\ IO =
+       |    run {
+       |        let timestamp = Clock.currentTime(TimeUnit.Milliseconds);
+       |        println("$${timestamp} ms since the epoc")
+       |    } with Clock.runWithIO
+       |""".stripMargin
+  )
+
+  /**
+    * The uri of the test source.
+    */
+  private val uri = "<test>"
+
+  /**
+    * Compiles the given input string `s` with the given compilation options `o`.
+    */
+  private def compile(s: String, o: Options): (Root, Flix, List[CompilationMessage]) = {
path: main/test/ca/uwaterloo/flix/api/lsp/TestCompletionProvider.scala
user: login: magnus-madsen
body: But all private helper methods should go below the tests, near the bottom of the file.
created_at: 2025-03-20T06:59:58Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,120 @@
+/*
+ * Copyright 2025 Chenhao Gao
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package ca.uwaterloo.flix.api.lsp
+
+import ca.uwaterloo.flix.api.Flix
+import ca.uwaterloo.flix.api.lsp.provider.CompletionProvider
+import ca.uwaterloo.flix.language.CompilationMessage
+import ca.uwaterloo.flix.language.ast.Token
+import ca.uwaterloo.flix.language.ast.TypedAst.Root
+import ca.uwaterloo.flix.language.ast.shared.{Input, SecurityContext, Source}
+import ca.uwaterloo.flix.util.Options
+import org.scalatest.funsuite.AnyFunSuite
+
+class TestCompletionProvider extends AnyFunSuite  {
+  /**
+    * A list of programs to test invariants on.
+    */
+  private val programs = List(
+    s"""
+       |def main(): Unit \\ IO =
+       |    run {
+       |        Console.println("Please enter your name: ");
+       |        let name = Console.readln();
+       |        Console.println("Hello $${name}")
+       |    } with Console.runWithIO
+       |""".stripMargin,
+    s"""
+       |def main(): Unit \\ IO =
+       |    run {
+       |        let timestamp = Clock.currentTime(TimeUnit.Milliseconds);
+       |        println("$${timestamp} ms since the epoc")
+       |    } with Clock.runWithIO
+       |""".stripMargin
+  )
+
+  /**
+    * The uri of the test source.
+    */
+  private val uri = "<test>"
+
+  /**
+    * Compiles the given input string `s` with the given compilation options `o`.
+    */
+  private def compile(s: String, o: Options): (Root, Flix, List[CompilationMessage]) = {
+    implicit val sctx: SecurityContext = SecurityContext.AllPermissions
+    val flix = new Flix().setOptions(o).addSourceCode(uri, s)
+    flix.check() match {
+      case (Some(root), errors) => (root, flix, errors)
+      case (None, _) => fail("Compilation failed: a root is expected.")
+    }
+  }
+
+  /**
+    * Creates a source object from the given string `content`.
+    */
+  private def mkSource(content: String): Source = {
+    val sctx = SecurityContext.AllPermissions
+    val input = Input.Text(uri, content, sctx)
+    Source(input, content.toCharArray)
+  }
+
+  /**
+    * Checks the completion provider invariant for the given token filter, string transformer and completion assertion.
+    * For all the programs to be tested, we use the filter to select tokens to transform, apply the transformer to the selected tokens,
+    * call the completion provider with the transformed program and check the completions against the assertion.
+    *
+    * @param tokenFilter          the filter to select tokens to transform.
+    * @param stringTransformer    the transformer to apply to the selected tokens.
+    * @param completionAssertion  the assertion to check the completions against.
+    */
+  private def checkInvariant(tokenFilter: Token => Boolean, stringTransformer: String => String, completionAssertion: CompletionList => Unit ): Unit = {
path: main/test/ca/uwaterloo/flix/api/lsp/TestCompletionProvider.scala
user: login: magnus-madsen
body: This is like a "god function". In OO one would say that it has too much responsibility. 
created_at: 2025-03-20T07:00:29Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,120 @@
+/*
+ * Copyright 2025 Chenhao Gao
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package ca.uwaterloo.flix.api.lsp
+
+import ca.uwaterloo.flix.api.Flix
+import ca.uwaterloo.flix.api.lsp.provider.CompletionProvider
+import ca.uwaterloo.flix.language.CompilationMessage
+import ca.uwaterloo.flix.language.ast.Token
+import ca.uwaterloo.flix.language.ast.TypedAst.Root
+import ca.uwaterloo.flix.language.ast.shared.{Input, SecurityContext, Source}
+import ca.uwaterloo.flix.util.Options
+import org.scalatest.funsuite.AnyFunSuite
+
+class TestCompletionProvider extends AnyFunSuite  {
+  /**
+    * A list of programs to test invariants on.
+    */
+  private val programs = List(
+    s"""
+       |def main(): Unit \\ IO =
+       |    run {
+       |        Console.println("Please enter your name: ");
+       |        let name = Console.readln();
+       |        Console.println("Hello $${name}")
+       |    } with Console.runWithIO
+       |""".stripMargin,
+    s"""
+       |def main(): Unit \\ IO =
+       |    run {
+       |        let timestamp = Clock.currentTime(TimeUnit.Milliseconds);
+       |        println("$${timestamp} ms since the epoc")
+       |    } with Clock.runWithIO
+       |""".stripMargin
+  )
+
+  /**
+    * The uri of the test source.
+    */
+  private val uri = "<test>"
+
+  /**
+    * Compiles the given input string `s` with the given compilation options `o`.
+    */
+  private def compile(s: String, o: Options): (Root, Flix, List[CompilationMessage]) = {
+    implicit val sctx: SecurityContext = SecurityContext.AllPermissions
+    val flix = new Flix().setOptions(o).addSourceCode(uri, s)
+    flix.check() match {
+      case (Some(root), errors) => (root, flix, errors)
+      case (None, _) => fail("Compilation failed: a root is expected.")
+    }
+  }
+
+  /**
+    * Creates a source object from the given string `content`.
+    */
+  private def mkSource(content: String): Source = {
+    val sctx = SecurityContext.AllPermissions
+    val input = Input.Text(uri, content, sctx)
+    Source(input, content.toCharArray)
+  }
+
+  /**
+    * Checks the completion provider invariant for the given token filter, string transformer and completion assertion.
+    * For all the programs to be tested, we use the filter to select tokens to transform, apply the transformer to the selected tokens,
+    * call the completion provider with the transformed program and check the completions against the assertion.
+    *
+    * @param tokenFilter          the filter to select tokens to transform.
+    * @param stringTransformer    the transformer to apply to the selected tokens.
+    * @param completionAssertion  the assertion to check the completions against.
+    */
+  private def checkInvariant(tokenFilter: Token => Boolean, stringTransformer: String => String, completionAssertion: CompletionList => Unit ): Unit = {
path: main/test/ca/uwaterloo/flix/api/lsp/TestCompletionProvider.scala
user: login: magnus-madsen
body: We need to break it down into smaller reusable parts. 
created_at: 2025-03-20T07:00:48Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,120 @@
+/*
+ * Copyright 2025 Chenhao Gao
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package ca.uwaterloo.flix.api.lsp
+
+import ca.uwaterloo.flix.api.Flix
+import ca.uwaterloo.flix.api.lsp.provider.CompletionProvider
+import ca.uwaterloo.flix.language.CompilationMessage
+import ca.uwaterloo.flix.language.ast.Token
+import ca.uwaterloo.flix.language.ast.TypedAst.Root
+import ca.uwaterloo.flix.language.ast.shared.{Input, SecurityContext, Source}
+import ca.uwaterloo.flix.util.Options
+import org.scalatest.funsuite.AnyFunSuite
+
+class TestCompletionProvider extends AnyFunSuite  {
+  /**
+    * A list of programs to test invariants on.
+    */
+  private val programs = List(
+    s"""
+       |def main(): Unit \\ IO =
+       |    run {
+       |        Console.println("Please enter your name: ");
+       |        let name = Console.readln();
+       |        Console.println("Hello $${name}")
+       |    } with Console.runWithIO
+       |""".stripMargin,
+    s"""
+       |def main(): Unit \\ IO =
+       |    run {
+       |        let timestamp = Clock.currentTime(TimeUnit.Milliseconds);
+       |        println("$${timestamp} ms since the epoc")
+       |    } with Clock.runWithIO
+       |""".stripMargin
+  )
+
+  /**
+    * The uri of the test source.
+    */
+  private val uri = "<test>"
+
+  /**
+    * Compiles the given input string `s` with the given compilation options `o`.
+    */
+  private def compile(s: String, o: Options): (Root, Flix, List[CompilationMessage]) = {
+    implicit val sctx: SecurityContext = SecurityContext.AllPermissions
+    val flix = new Flix().setOptions(o).addSourceCode(uri, s)
+    flix.check() match {
+      case (Some(root), errors) => (root, flix, errors)
+      case (None, _) => fail("Compilation failed: a root is expected.")
+    }
+  }
+
+  /**
+    * Creates a source object from the given string `content`.
+    */
+  private def mkSource(content: String): Source = {
+    val sctx = SecurityContext.AllPermissions
+    val input = Input.Text(uri, content, sctx)
+    Source(input, content.toCharArray)
+  }
+
+  /**
+    * Checks the completion provider invariant for the given token filter, string transformer and completion assertion.
+    * For all the programs to be tested, we use the filter to select tokens to transform, apply the transformer to the selected tokens,
+    * call the completion provider with the transformed program and check the completions against the assertion.
+    *
+    * @param tokenFilter          the filter to select tokens to transform.
+    * @param stringTransformer    the transformer to apply to the selected tokens.
+    * @param completionAssertion  the assertion to check the completions against.
+    */
+  private def checkInvariant(tokenFilter: Token => Boolean, stringTransformer: String => String, completionAssertion: CompletionList => Unit ): Unit = {
+    programs.foreach { program =>
+      val (root, _, _) = compile(program, Options.Default)
+      val source = mkSource(program)
+      val tokensToTransform = root.tokens(source).toList.filter(tokenFilter)
+      tokensToTransform.foreach { token =>
+        val replacingString = stringTransformer(token.text)
+        val newProgram = program.substring(0, token.start) + replacingString + program.substring(token.end)
+        val (newRoot, newFlix, newErrors) = compile(newProgram, Options.Default)
+        val newPos = Position(token.sp1.line, token.sp1.col + replacingString.length) // position after the replacement
+        val completions = CompletionProvider.autoComplete(uri, newPos, newProgram, newErrors)(newRoot, newFlix)
+        completionAssertion(completions)
+      }
+    }
+  }
+
+  /**
+    * Asserts that the given completions are empty.
+    */
+  private def assertEmptyCompletions(completions: CompletionList): Unit = {
+    assert(completions.items.isEmpty)
+  }
+
+  /**
+    * A function that returns the input string as is.
+    */
+  private def identity(s: String): String = s
+
+  test("No completions after complete keyword"){
path: main/test/ca/uwaterloo/flix/api/lsp/TestCompletionProvider.scala
user: login: magnus-madsen
body: If we look at the test, what we would like to say is something like

\forall C[keyword], pos \in KeywordPos(C[keyword]). autoComplete(pos) = Nil

So I think we want building like:

- getKeywordTokens
- getPositionsInToken

but we don't need `assertEmptyCompletions`. That is just == Nil.
created_at: 2025-03-20T07:05:03Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,120 @@
+/*
+ * Copyright 2025 Chenhao Gao
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package ca.uwaterloo.flix.api.lsp
+
+import ca.uwaterloo.flix.api.Flix
+import ca.uwaterloo.flix.api.lsp.provider.CompletionProvider
+import ca.uwaterloo.flix.language.CompilationMessage
+import ca.uwaterloo.flix.language.ast.Token
+import ca.uwaterloo.flix.language.ast.TypedAst.Root
+import ca.uwaterloo.flix.language.ast.shared.{Input, SecurityContext, Source}
+import ca.uwaterloo.flix.util.Options
+import org.scalatest.funsuite.AnyFunSuite
+
+class TestCompletionProvider extends AnyFunSuite  {
+  /**
+    * A list of programs to test invariants on.
+    */
+  private val programs = List(
+    s"""
+       |def main(): Unit \\ IO =
+       |    run {
+       |        Console.println("Please enter your name: ");
+       |        let name = Console.readln();
+       |        Console.println("Hello $${name}")
+       |    } with Console.runWithIO
+       |""".stripMargin,
+    s"""
+       |def main(): Unit \\ IO =
+       |    run {
+       |        let timestamp = Clock.currentTime(TimeUnit.Milliseconds);
+       |        println("$${timestamp} ms since the epoc")
+       |    } with Clock.runWithIO
+       |""".stripMargin
+  )
+
+  /**
+    * The uri of the test source.
+    */
+  private val uri = "<test>"
+
+  /**
+    * Compiles the given input string `s` with the given compilation options `o`.
+    */
+  private def compile(s: String, o: Options): (Root, Flix, List[CompilationMessage]) = {
+    implicit val sctx: SecurityContext = SecurityContext.AllPermissions
+    val flix = new Flix().setOptions(o).addSourceCode(uri, s)
+    flix.check() match {
+      case (Some(root), errors) => (root, flix, errors)
+      case (None, _) => fail("Compilation failed: a root is expected.")
+    }
+  }
+
+  /**
+    * Creates a source object from the given string `content`.
+    */
+  private def mkSource(content: String): Source = {
+    val sctx = SecurityContext.AllPermissions
+    val input = Input.Text(uri, content, sctx)
+    Source(input, content.toCharArray)
+  }
+
+  /**
+    * Checks the completion provider invariant for the given token filter, string transformer and completion assertion.
+    * For all the programs to be tested, we use the filter to select tokens to transform, apply the transformer to the selected tokens,
+    * call the completion provider with the transformed program and check the completions against the assertion.
+    *
+    * @param tokenFilter          the filter to select tokens to transform.
+    * @param stringTransformer    the transformer to apply to the selected tokens.
+    * @param completionAssertion  the assertion to check the completions against.
+    */
+  private def checkInvariant(tokenFilter: Token => Boolean, stringTransformer: String => String, completionAssertion: CompletionList => Unit ): Unit = {
+    programs.foreach { program =>
+      val (root, _, _) = compile(program, Options.Default)
+      val source = mkSource(program)
+      val tokensToTransform = root.tokens(source).toList.filter(tokenFilter)
+      tokensToTransform.foreach { token =>
+        val replacingString = stringTransformer(token.text)
+        val newProgram = program.substring(0, token.start) + replacingString + program.substring(token.end)
+        val (newRoot, newFlix, newErrors) = compile(newProgram, Options.Default)
+        val newPos = Position(token.sp1.line, token.sp1.col + replacingString.length) // position after the replacement
+        val completions = CompletionProvider.autoComplete(uri, newPos, newProgram, newErrors)(newRoot, newFlix)
+        completionAssertion(completions)
+      }
+    }
+  }
+
+  /**
+    * Asserts that the given completions are empty.
+    */
+  private def assertEmptyCompletions(completions: CompletionList): Unit = {
+    assert(completions.items.isEmpty)
+  }
+
+  /**
+    * A function that returns the input string as is.
+    */
+  private def identity(s: String): String = s
+
+  test("No completions after complete keyword"){
path: main/test/ca/uwaterloo/flix/api/lsp/TestCompletionProvider.scala
user: login: magnus-madsen
body: There is also no need to recompile the program or anything.
created_at: 2025-03-20T07:05:23Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,120 @@
+/*
+ * Copyright 2025 Chenhao Gao
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package ca.uwaterloo.flix.api.lsp
+
+import ca.uwaterloo.flix.api.Flix
+import ca.uwaterloo.flix.api.lsp.provider.CompletionProvider
+import ca.uwaterloo.flix.language.CompilationMessage
+import ca.uwaterloo.flix.language.ast.Token
+import ca.uwaterloo.flix.language.ast.TypedAst.Root
+import ca.uwaterloo.flix.language.ast.shared.{Input, SecurityContext, Source}
+import ca.uwaterloo.flix.util.Options
+import org.scalatest.funsuite.AnyFunSuite
+
+class TestCompletionProvider extends AnyFunSuite  {
+  /**
+    * A list of programs to test invariants on.
+    */
+  private val programs = List(
+    s"""
+       |def main(): Unit \\ IO =
+       |    run {
+       |        Console.println("Please enter your name: ");
+       |        let name = Console.readln();
+       |        Console.println("Hello $${name}")
+       |    } with Console.runWithIO
+       |""".stripMargin,
+    s"""
+       |def main(): Unit \\ IO =
+       |    run {
+       |        let timestamp = Clock.currentTime(TimeUnit.Milliseconds);
+       |        println("$${timestamp} ms since the epoc")
+       |    } with Clock.runWithIO
+       |""".stripMargin
+  )
+
+  /**
+    * The uri of the test source.
+    */
+  private val uri = "<test>"
+
+  /**
+    * Compiles the given input string `s` with the given compilation options `o`.
+    */
+  private def compile(s: String, o: Options): (Root, Flix, List[CompilationMessage]) = {
+    implicit val sctx: SecurityContext = SecurityContext.AllPermissions
+    val flix = new Flix().setOptions(o).addSourceCode(uri, s)
+    flix.check() match {
+      case (Some(root), errors) => (root, flix, errors)
+      case (None, _) => fail("Compilation failed: a root is expected.")
+    }
+  }
+
+  /**
+    * Creates a source object from the given string `content`.
+    */
+  private def mkSource(content: String): Source = {
+    val sctx = SecurityContext.AllPermissions
+    val input = Input.Text(uri, content, sctx)
+    Source(input, content.toCharArray)
+  }
+
+  /**
+    * Checks the completion provider invariant for the given token filter, string transformer and completion assertion.
+    * For all the programs to be tested, we use the filter to select tokens to transform, apply the transformer to the selected tokens,
+    * call the completion provider with the transformed program and check the completions against the assertion.
+    *
+    * @param tokenFilter          the filter to select tokens to transform.
+    * @param stringTransformer    the transformer to apply to the selected tokens.
+    * @param completionAssertion  the assertion to check the completions against.
+    */
+  private def checkInvariant(tokenFilter: Token => Boolean, stringTransformer: String => String, completionAssertion: CompletionList => Unit ): Unit = {
+    programs.foreach { program =>
+      val (root, _, _) = compile(program, Options.Default)
+      val source = mkSource(program)
+      val tokensToTransform = root.tokens(source).toList.filter(tokenFilter)
+      tokensToTransform.foreach { token =>
+        val replacingString = stringTransformer(token.text)
+        val newProgram = program.substring(0, token.start) + replacingString + program.substring(token.end)
+        val (newRoot, newFlix, newErrors) = compile(newProgram, Options.Default)
+        val newPos = Position(token.sp1.line, token.sp1.col + replacingString.length) // position after the replacement
+        val completions = CompletionProvider.autoComplete(uri, newPos, newProgram, newErrors)(newRoot, newFlix)
+        completionAssertion(completions)
+      }
+    }
+  }
+
+  /**
+    * Asserts that the given completions are empty.
+    */
+  private def assertEmptyCompletions(completions: CompletionList): Unit = {
+    assert(completions.items.isEmpty)
+  }
+
+  /**
+    * A function that returns the input string as is.
+    */
+  private def identity(s: String): String = s
+
+  test("No completions after complete keyword"){
path: main/test/ca/uwaterloo/flix/api/lsp/TestCompletionProvider.scala
user: login: stormckey
body: I think a general test procedure (also the one I impl here) is something like:
```
\forall program in programs
  \forall token in programs.token.filter(desired)
      new_program = replace(token, program)
      completion = autocomplete(new_program)
      assert(completion)
```
A few notes for why I design it this way:
- We need a replace and recompile since in some cases we could change the program for every target token, it just happens to be an identity replacement in these two cases. Thus we don't need to actually change it and recompile. We can either special case over identity or write a dedicated helper function for this class of invariants. But a general framework should keep this interface to change the program.
- getKeywordTokens is impl by passing a filter that will accept all keyword tokens. I think this is powerful enough?
- for the assertion, you can see that it's inside the inner most loop, if we write == Nil right in the inner most loop, we will have to write n functions for n invariants. I think passing a assertion is fine here?
- I didn't break the function into small pieces since the local variables are used across the function, it's a little bit hard to break it down.

The general idea to design this function is to provide a general entry point for most invariants. I think it would work for most invariants except the one that assert a subset, or the one that checks no shadowing?
created_at: 2025-03-20T10:54:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -18,5 +18,6 @@ package ca.uwaterloo.flix.api.lsp
 import org.scalatest.Suites
 
 class LspSuite extends Suites(
-  new VisitorSuite
+  new VisitorSuite,
+  new TestCompletionProvider
path: main/test/ca/uwaterloo/flix/api/lsp/LspSuite.scala
user: login: magnus-madsen
body: That is, leave it out here.
created_at: 2025-03-21T07:32:21Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,120 @@
+/*
+ * Copyright 2025 Chenhao Gao
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package ca.uwaterloo.flix.api.lsp
+
+import ca.uwaterloo.flix.api.Flix
+import ca.uwaterloo.flix.api.lsp.provider.CompletionProvider
+import ca.uwaterloo.flix.language.CompilationMessage
+import ca.uwaterloo.flix.language.ast.Token
+import ca.uwaterloo.flix.language.ast.TypedAst.Root
+import ca.uwaterloo.flix.language.ast.shared.{Input, SecurityContext, Source}
+import ca.uwaterloo.flix.util.Options
+import org.scalatest.funsuite.AnyFunSuite
+
+class TestCompletionProvider extends AnyFunSuite  {
+  /**
+    * A list of programs to test invariants on.
+    */
+  private val programs = List(
+    s"""
+       |def main(): Unit \\ IO =
+       |    run {
+       |        Console.println("Please enter your name: ");
+       |        let name = Console.readln();
+       |        Console.println("Hello $${name}")
+       |    } with Console.runWithIO
+       |""".stripMargin,
+    s"""
+       |def main(): Unit \\ IO =
+       |    run {
+       |        let timestamp = Clock.currentTime(TimeUnit.Milliseconds);
+       |        println("$${timestamp} ms since the epoc")
+       |    } with Clock.runWithIO
+       |""".stripMargin
+  )
+
+  /**
+    * The uri of the test source.
+    */
+  private val uri = "<test>"
+
+  /**
+    * Compiles the given input string `s` with the given compilation options `o`.
+    */
+  private def compile(s: String, o: Options): (Root, Flix, List[CompilationMessage]) = {
+    implicit val sctx: SecurityContext = SecurityContext.AllPermissions
+    val flix = new Flix().setOptions(o).addSourceCode(uri, s)
+    flix.check() match {
+      case (Some(root), errors) => (root, flix, errors)
+      case (None, _) => fail("Compilation failed: a root is expected.")
+    }
+  }
+
+  /**
+    * Creates a source object from the given string `content`.
+    */
+  private def mkSource(content: String): Source = {
+    val sctx = SecurityContext.AllPermissions
+    val input = Input.Text(uri, content, sctx)
+    Source(input, content.toCharArray)
+  }
+
+  /**
+    * Checks the completion provider invariant for the given token filter, string transformer and completion assertion.
+    * For all the programs to be tested, we use the filter to select tokens to transform, apply the transformer to the selected tokens,
+    * call the completion provider with the transformed program and check the completions against the assertion.
+    *
+    * @param tokenFilter          the filter to select tokens to transform.
+    * @param stringTransformer    the transformer to apply to the selected tokens.
+    * @param completionAssertion  the assertion to check the completions against.
+    */
+  private def checkInvariant(tokenFilter: Token => Boolean, stringTransformer: String => String, completionAssertion: CompletionList => Unit ): Unit = {
path: main/test/ca/uwaterloo/flix/api/lsp/TestCompletionProvider.scala
user: login: magnus-madsen
body: Will you break this down now or later?
created_at: 2025-03-22T11:31:52Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,120 @@
+/*
+ * Copyright 2025 Chenhao Gao
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package ca.uwaterloo.flix.api.lsp
+
+import ca.uwaterloo.flix.api.Flix
+import ca.uwaterloo.flix.api.lsp.provider.CompletionProvider
+import ca.uwaterloo.flix.language.CompilationMessage
+import ca.uwaterloo.flix.language.ast.Token
+import ca.uwaterloo.flix.language.ast.TypedAst.Root
+import ca.uwaterloo.flix.language.ast.shared.{Input, SecurityContext, Source}
+import ca.uwaterloo.flix.util.Options
+import org.scalatest.funsuite.AnyFunSuite
+
+class TestCompletionProvider extends AnyFunSuite  {
+  /**
+    * A list of programs to test invariants on.
+    */
+  private val programs = List(
+    s"""
+       |def main(): Unit \\ IO =
+       |    run {
+       |        Console.println("Please enter your name: ");
+       |        let name = Console.readln();
+       |        Console.println("Hello $${name}")
+       |    } with Console.runWithIO
+       |""".stripMargin,
+    s"""
+       |def main(): Unit \\ IO =
+       |    run {
+       |        let timestamp = Clock.currentTime(TimeUnit.Milliseconds);
+       |        println("$${timestamp} ms since the epoc")
+       |    } with Clock.runWithIO
+       |""".stripMargin
+  )
+
+  /**
+    * The uri of the test source.
+    */
+  private val uri = "<test>"
+
+  /**
+    * Compiles the given input string `s` with the given compilation options `o`.
+    */
+  private def compile(s: String, o: Options): (Root, Flix, List[CompilationMessage]) = {
+    implicit val sctx: SecurityContext = SecurityContext.AllPermissions
+    val flix = new Flix().setOptions(o).addSourceCode(uri, s)
+    flix.check() match {
+      case (Some(root), errors) => (root, flix, errors)
+      case (None, _) => fail("Compilation failed: a root is expected.")
+    }
+  }
+
+  /**
+    * Creates a source object from the given string `content`.
+    */
+  private def mkSource(content: String): Source = {
+    val sctx = SecurityContext.AllPermissions
+    val input = Input.Text(uri, content, sctx)
+    Source(input, content.toCharArray)
+  }
+
+  /**
+    * Checks the completion provider invariant for the given token filter, string transformer and completion assertion.
+    * For all the programs to be tested, we use the filter to select tokens to transform, apply the transformer to the selected tokens,
+    * call the completion provider with the transformed program and check the completions against the assertion.
+    *
+    * @param tokenFilter          the filter to select tokens to transform.
+    * @param stringTransformer    the transformer to apply to the selected tokens.
+    * @param completionAssertion  the assertion to check the completions against.
+    */
+  private def checkInvariant(tokenFilter: Token => Boolean, stringTransformer: String => String, completionAssertion: CompletionList => Unit ): Unit = {
path: main/test/ca/uwaterloo/flix/api/lsp/TestCompletionProvider.scala
user: login: stormckey
body: What kind of break down🤔

I mean, the `root, flix, errors` defined in the first line is referred nearly at the end of this function, since we don't want to recompile it when there is no actual change.

In that case, if we still want to break it down, we just pass everything in the first part(`root, flix, errors, toeknsToTransform, stringTransformer, completionAssertion`) to the second?
created_at: 2025-03-24T12:15:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,120 @@
+/*
+ * Copyright 2025 Chenhao Gao
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package ca.uwaterloo.flix.api.lsp
+
+import ca.uwaterloo.flix.api.Flix
+import ca.uwaterloo.flix.api.lsp.provider.CompletionProvider
+import ca.uwaterloo.flix.language.CompilationMessage
+import ca.uwaterloo.flix.language.ast.Token
+import ca.uwaterloo.flix.language.ast.TypedAst.Root
+import ca.uwaterloo.flix.language.ast.shared.{Input, SecurityContext, Source}
+import ca.uwaterloo.flix.util.Options
+import org.scalatest.funsuite.AnyFunSuite
+
+class TestCompletionProvider extends AnyFunSuite  {
+  /**
+    * A list of programs to test invariants on.
+    */
+  private val programs = List(
+    s"""
+       |def main(): Unit \\ IO =
+       |    run {
+       |        Console.println("Please enter your name: ");
+       |        let name = Console.readln();
+       |        Console.println("Hello $${name}")
+       |    } with Console.runWithIO
+       |""".stripMargin,
+    s"""
+       |def main(): Unit \\ IO =
+       |    run {
+       |        let timestamp = Clock.currentTime(TimeUnit.Milliseconds);
+       |        println("$${timestamp} ms since the epoc")
+       |    } with Clock.runWithIO
+       |""".stripMargin
+  )
+
+  /**
+    * The uri of the test source.
+    */
+  private val uri = "<test>"
+
+  /**
+    * Compiles the given input string `s` with the given compilation options `o`.
+    */
+  private def compile(s: String, o: Options): (Root, Flix, List[CompilationMessage]) = {
+    implicit val sctx: SecurityContext = SecurityContext.AllPermissions
+    val flix = new Flix().setOptions(o).addSourceCode(uri, s)
+    flix.check() match {
+      case (Some(root), errors) => (root, flix, errors)
+      case (None, _) => fail("Compilation failed: a root is expected.")
+    }
+  }
+
+  /**
+    * Creates a source object from the given string `content`.
+    */
+  private def mkSource(content: String): Source = {
+    val sctx = SecurityContext.AllPermissions
+    val input = Input.Text(uri, content, sctx)
+    Source(input, content.toCharArray)
+  }
+
+  /**
+    * Checks the completion provider invariant for the given token filter, string transformer and completion assertion.
+    * For all the programs to be tested, we use the filter to select tokens to transform, apply the transformer to the selected tokens,
+    * call the completion provider with the transformed program and check the completions against the assertion.
+    *
+    * @param tokenFilter          the filter to select tokens to transform.
+    * @param stringTransformer    the transformer to apply to the selected tokens.
+    * @param completionAssertion  the assertion to check the completions against.
+    */
+  private def checkInvariant(tokenFilter: Token => Boolean, stringTransformer: String => String, completionAssertion: CompletionList => Unit ): Unit = {
path: main/test/ca/uwaterloo/flix/api/lsp/TestCompletionProvider.scala
user: login: magnus-madsen
body: Don't add more higher order arguments. The opposite. You want to have small functions that each test case compose.
created_at: 2025-03-24T12:32:07Z
author_association: MEMBER

PR comments:
user: login: magnus-madsen
created_at: 2025-03-20T06:53:44Z
body: >     * Should we hard code the programs to be tests or we can read them from example files?

That is fine for now. I would suggest to add a few more examples.

Later we will restructure the code to use _all_ our examples (in the example folder).

>     * Should I place the helper functions in an other file?

I think it is good to keep everything in one file for now.

>     * Any suggestions about the api?

I am adding review comments.

author_association: MEMBER
user: login: magnus-madsen
created_at: 2025-03-20T06:58:08Z
body: What we are trying to do here is called "property-based testing" or "quick checking"- but we are not reusing any library or framework, instead we are building our own DSL.
author_association: MEMBER
user: login: magnus-madsen
created_at: 2025-03-20T06:59:08Z
body: How long does it take to run this test suite by the way?
author_association: MEMBER
user: login: magnus-madsen
created_at: 2025-03-20T07:06:07Z
body: To help you towards a modular design, I think you should consider two properties:

- Completion on keyword => empty
- Completion on literal => empty

author_association: MEMBER
user: login: stormckey
created_at: 2025-03-20T10:44:30Z
body: > To help you towards a modular design, I think you should consider two properties:
> 
> * Completion on keyword => empty
> * Completion on literal => empty

Done. We dont actually need any other helper function (except isLiteral from TokenKind) to add the invariant.
author_association: CONTRIBUTOR
user: login: stormckey
created_at: 2025-03-20T11:08:20Z
body: > How long does it take to run this test suite by the way?

After adding two other examples and another invariant, now it takes 24s on my computer
author_association: CONTRIBUTOR
user: login: magnus-madsen
created_at: 2025-03-21T07:32:07Z
body: > > How long does it take to run this test suite by the way?
> 
> After adding two other examples and another invariant, now it takes 24s on my computer

OK; and its only going to grow. So for now, we have to leave it out of the default test suite.
author_association: MEMBER
user: login: magnus-madsen
created_at: 2025-03-21T07:32:49Z
body: In a separate PR, we should add something like: https://github.com/flix/flix/blob/master/.github/workflows/fuzzing.yaml for this specific test class.
author_association: MEMBER
user: login: stormckey
created_at: 2025-03-21T11:01:27Z
body: > In a separate PR, we should add something like: https://github.com/flix/flix/blob/master/.github/workflows/fuzzing.yaml for this specific test class.

OK, in previous commit. I tried the workflow and it works fine. But now it's removed. I'll add it back in another PR.

It's also removed from the test suit.
author_association: CONTRIBUTOR
user: login: magnus-madsen
created_at: 2025-03-24T12:32:51Z
body: I will merge this now so you can iterate.
author_association: MEMBER