Repository: denoland/deno
Pull request #29221: feat(lsp): display JsErrors at their original line
Author: login: stefnotch
Created: 2025-05-08T21:45:11Z | Merged: 2025-05-18T16:29:00Z

<!--
Before submitting a PR, please read https://docs.deno.com/runtime/manual/references/contributing

1. Give the PR a descriptive title.

  Examples of good title:
    - fix(std/http): Fix race condition in server
    - docs(console): Update docstrings
    - feat(doc): Handle nested reexports

  Examples of bad title:
    - fix #7123
    - update docs
    - fix bugs

2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
7. Open as a draft PR if your work is still in progress. The CI won't run
   all steps, but you can add '[ci]' to a commit message to force it to.
8. If you would like to run the benchmarks on the CI, add the 'ci-bench' label.
-->

fix https://github.com/denoland/vscode_deno/issues/1297

This displays the LSP errors in the correct line instead of displaying them in the `Deno.test` line. This gives us a much nicer user experience when a test has multiple assertions.

![image](https://github.com/user-attachments/assets/327c5fea-2bee-43e2-936f-46917cd31476)

I originally wanted to hook up the `expected` and `actual` reporters of the LSP as well, but decided against it, since that functionality can also be mimiced by having a smarter `assertEquals` function in the Deno standard library https://github.com/denoland/std/issues/6645



Diff:
diff --git a/cli/lsp/testing/execution.rs b/cli/lsp/testing/execution.rs
index c36b34599fcb5a..7931e7506a15e1 100644
--- a/cli/lsp/testing/execution.rs
+++ b/cli/lsp/testing/execution.rs
@@ -42,6 +42,7 @@ use crate::lsp::urls::url_to_uri;
 use crate::tools::test;
 use crate::tools::test::create_test_event_channel;
 use crate::tools::test::FailFastTracker;
+use crate::tools::test::TestFailure;
 use crate::tools::test::TestFailureFormatOptions;
 
 /// Logic to convert a test request into a set of test modules to be tested and
@@ -102,24 +103,32 @@ fn as_queue_and_filters(
   (queue, filters)
 }
 
-fn as_test_messages<S: AsRef<str>>(
-  message: S,
-  is_markdown: bool,
-) -> Vec<lsp_custom::TestMessage> {
+fn failure_to_test_message(failure: &TestFailure) -> lsp_custom::TestMessage {
   let message = lsp::MarkupContent {
-    kind: if is_markdown {
-      lsp::MarkupKind::Markdown
-    } else {
-      lsp::MarkupKind::PlainText
-    },
-    value: message.as_ref().to_string(),
+    kind: lsp::MarkupKind::PlainText,
+    value: failure
+      .format(&TestFailureFormatOptions::default())
+      .to_string(),
   };
-  vec![lsp_custom::TestMessage {
+  let location = failure.error_location().and_then(|v| {
+    let pos = lsp::Position {
+      line: v.line_number,
+      character: v.column_number,
+    };
+    // Does not have to match the test URI
+    // since one can write `Deno.test(importedFunction)`
+    let uri = uri_parse_unencoded(&v.file_name).ok()?;
+    Some(lsp::Location {
+      uri,
+      range: lsp::Range::new(pos, pos),
+    })
+  });
+  lsp_custom::TestMessage {
     message,
     expected_output: None,
     actual_output: None,
-    location: None,
-  }]
+    location,
+  }
 }
 
 #[derive(Debug, Clone, Default, PartialEq)]
@@ -672,10 +681,7 @@ impl LspTestReporter {
         let desc = self.tests.get(&desc.id).unwrap();
         self.progress(lsp_custom::TestRunProgressMessage::Failed {
           test: desc.as_test_identifier(&self.tests),
-          messages: as_test_messages(
-            failure.format(&TestFailureFormatOptions::default()),
-            false,
-          ),
+          messages: vec![failure_to_test_message(failure)],
           duration: Some(elapsed as u32),
         })
       }
@@ -697,7 +703,15 @@ impl LspTestReporter {
       origin,
       test::fmt::format_test_error(js_error, &TestFailureFormatOptions::default())
     );
-    let messages = as_test_messages(err_string, false);
+    let messages = vec![lsp_custom::TestMessage {
+      message: lsp::MarkupContent {
+        kind: lsp::MarkupKind::PlainText,
+        value: err_string,
+      },
+      expected_output: None,
+      actual_output: None,
+      location: None,
+    }];
     for desc in self.tests.values().filter(|d| d.origin() == origin) {
       self.progress(lsp_custom::TestRunProgressMessage::Failed {
         test: desc.as_test_identifier(&self.tests),
@@ -772,10 +786,7 @@ impl LspTestReporter {
       test::TestStepResult::Failed(failure) => {
         self.progress(lsp_custom::TestRunProgressMessage::Failed {
           test: desc.as_test_identifier(&self.tests),
-          messages: as_test_messages(
-            failure.format(&TestFailureFormatOptions::default()),
-            false,
-          ),
+          messages: vec![failure_to_test_message(failure)],
           duration: Some(elapsed as u32),
         })
       }
diff --git a/cli/tools/test/mod.rs b/cli/tools/test/mod.rs
index d399db23522f75..224194f1175bad 100644
--- a/cli/tools/test/mod.rs
+++ b/cli/tools/test/mod.rs
@@ -399,6 +399,38 @@ impl TestFailure {
     }
   }
 
+  pub fn error_location(&self) -> Option<TestLocation> {
+    const TEST_RUNNER: &str = "ext:cli/40_test.js";
+    match self {
+      TestFailure::JsError(js_error) => js_error
+        .frames
+        .iter()
+        // The first line of user code comes above the test file.
+        // The call stack usually contains the top 10 frames, and cuts off after that.
+        // We need to explicitly check for the test runner here.
+        // - Checking for a `ext:` is not enough, since other Deno `ext:`s can appear in the call stack.
+        // - This check guarantees that the next frame is inside of the Deno.test(),
+        //   and not somewhere else.
+        .position(|v| v.file_name.as_deref() == Some(TEST_RUNNER))
+        // Go one up in the stack frame, this is where the user code was
+        .and_then(|index| index.checked_sub(1))
+        .and_then(|index| {
+          let user_frame = &js_error.frames[index];
+          let file_name = user_frame.file_name.as_ref()?.to_string();
+          // Turn into zero based indices
+          let line_number = user_frame.line_number.map(|v| v - 1)? as u32;
+          let column_number =
+            user_frame.column_number.map(|v| v - 1).unwrap_or(0) as u32;
+          Some(TestLocation {
+            file_name,
+            line_number,
+            column_number,
+          })
+        }),
+      _ => None,
+    }
+  }
+
   fn format_label(&self) -> String {
     match self {
       TestFailure::Incomplete => colors::gray("INCOMPLETE").to_string(),
diff --git a/tests/integration/lsp_tests.rs b/tests/integration/lsp_tests.rs
index 971513c5745e24..bb444a5163f6ab 100644
--- a/tests/integration/lsp_tests.rs
+++ b/tests/integration/lsp_tests.rs
@@ -5,6 +5,7 @@ use std::fs;
 use ntest_timeout::timeout;
 use pretty_assertions::assert_eq;
 use serde::Deserialize;
+use serde::Serialize;
 use serde_json::json;
 use serde_json::Value;
 use test_util::assert_starts_with;
@@ -14085,7 +14086,7 @@ export function B() {
   client.shutdown();
 }
 
-#[derive(Debug, Clone, Deserialize, PartialEq)]
+#[derive(Debug, Clone, Deserialize, Serialize, PartialEq)]
 #[serde(rename_all = "camelCase")]
 struct TestData {
   id: String,
@@ -14383,6 +14384,125 @@ Deno.test({
   client.shutdown();
 }
 
+#[test]
+#[timeout(300_000)]
+fn lsp_testing_api_failure() {
+  let context = TestContextBuilder::new().use_temp_cwd().build();
+  let temp_dir = context.temp_dir();
+  temp_dir.write("./deno.json", json!({}).to_string());
+  let file = temp_dir.source_file(
+    "test.ts",
+    r#"
+      Deno.test("test a", () => {
+        throw new Error("Some message.");
+      });
+    "#,
+  );
+  let mut client = context.new_lsp_command().build();
+  client.initialize_default();
+  client.did_open_file(&file);
+  let notification =
+    client.read_notification_with_method::<Value>("deno/testModule");
+  let params: TestModuleNotificationParams =
+    serde_json::from_value(notification.unwrap()).unwrap();
+  assert_eq!(params.text_document.uri.as_str(), file.uri().as_str());
+  assert_eq!(params.kind, TestModuleNotificationKind::Replace);
+  assert_eq!(params.label, "test.ts");
+  assert_eq!(params.tests.len(), 1);
+  let test = params.tests.into_iter().next().unwrap();
+  assert_eq!(test.label, "test a");
+  assert_eq!(
+    json!(test.range),
+    json!({
+      "start": { "line": 1,  "character": 11 },
+      "end": { "line": 1,  "character": 15 },
+    }),
+  );
+  assert_eq!(json!(&test.steps), json!(null));
+  let res = client.write_request_with_res_as::<TestRunResponseParams>(
+    "deno/testRun",
+    json!({
+      "id": 1,
+      "kind": "run",
+    }),
+  );
+  assert_eq!(res.enqueued.len(), 1);
+  assert_eq!(
+    res.enqueued[0].text_document.uri.as_str(),
+    file.uri().as_str()
+  );
+  assert_eq!(res.enqueued[0].ids.len(), 1);
+  let id = res.enqueued[0].ids[0].clone();
+  let notification =
+    client.read_notification_with_method::<Value>("deno/testRunProgress");
+  assert_eq!(
+    notification,
+    Some(json!({
+      "id": 1,
+      "message": {
+        "type": "started",
+        "test": {
+          "textDocument": { "uri": file.uri() },
+          "id": id,
+        },
+      },
+    })),
+  );
+  let notification =
+    client.read_notification_with_method::<Value>("deno/testRunProgress");
+  let notification = notification.unwrap();
+  let obj = notification.as_object().unwrap();
+  assert_eq!(obj.get("id"), Some(&json!(1)));
+  let message = obj.get("message").unwrap().as_object().unwrap();
+  match message.get("type").and_then(|v| v.as_str()) {
+    Some("failed") => {
+      assert_eq!(
+        message.get("test"),
+        Some(&json!({
+          "textDocument": { "uri": file.uri() },
+          "id": id,
+        })),
+      );
+      assert!(message.contains_key("duration"));
+      assert_eq!(
+        *message.get("messages").unwrap(),
+        json!([
+          {
+            "message": {
+              "kind": "plaintext",
+              "value": format!("Error: Some message.\n        throw new Error(\"Some message.\");\n\u{1b}[0m\u{1b}[31m              ^\u{1b}[0m\n    at \u{1b}[0m\u{1b}[36m{}\u{1b}[0m:\u{1b}[0m\u{1b}[33m3\u{1b}[0m:\u{1b}[0m\u{1b}[33m15\u{1b}[0m", file.url()),
+            },
+            "location": {
+              "uri": file.uri(),
+              "range": {
+                "start": { "line": 2, "character": 14 },
+                "end": { "line": 2, "character": 14 },
+              },
+            },
+          },
+        ]),
+      );
+      let notification =
+        client.read_notification_with_method::<Value>("deno/testRunProgress");
+      assert_eq!(
+        notification,
+        Some(json!({
+          "id": 1,
+          "message": {
+            "type": "end",
+          },
+        })),
+      );
+    }
+    // sometimes on windows, the messages come out of order, but it actually is
+    // working, so if we do get the end before the passed, we will simply let
+    // the test pass
+    Some("end") => (),
+    _ => panic!("unexpected message {}", json!(notification)),
+  }
+  client.shutdown();
+}
+
 #[test]
 #[timeout(300_000)]
 fn lsp_closed_file_find_references() {


Reviews:
user: login: nayeemrmn
body: Looks good, let me know if you need help adapting the test.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T22:23:21Z
user: login: stefnotch
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T11:19:19Z
user: login: stefnotch
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T11:31:47Z
user: login: stefnotch
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T11:32:30Z
user: login: stefnotch
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T11:53:15Z
user: login: stefnotch
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T12:14:48Z
user: login: stefnotch
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T16:36:10Z
user: login: stefnotch
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T16:53:30Z
user: login: stefnotch
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T17:25:51Z
user: login: nayeemrmn
body: LGTM
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-18T16:28:29Z

Review comments:
diff_hunk: @@ -399,6 +399,33 @@ impl TestFailure {
     }
   }
 
+  pub fn error_location(&self) -> Option<TestLocation> {
+    match self {
+      // TODO: What happens if not enough call stack frames were recorded?
+      // TODO: Could anything else trigger the ext:cli/40_test.js file? Maybe we should also test for the method name
+      TestFailure::JsError(js_error) => js_error
+        .frames
+        .iter()
+        .position(|v| v.file_name.as_deref() == Some("ext:cli/40_test.js"))
path: cli/tools/test/mod.rs
user: login: nayeemrmn
body: Maybe we should weaken this to `v.file_name.as_deref().is_some_and(|s| s.starts_with("ext:"))`, so it's less brittle wrt internal changes
created_at: 2025-05-08T22:23:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -102,24 +103,34 @@ fn as_queue_and_filters(
   (queue, filters)
 }
 
-fn as_test_messages<S: AsRef<str>>(
-  message: S,
-  is_markdown: bool,
-) -> Vec<lsp_custom::TestMessage> {
+fn failure_to_test_message(
+  test_uri: lsp::Uri,
+  failure: &TestFailure,
+) -> lsp_custom::TestMessage {
   let message = lsp::MarkupContent {
-    kind: if is_markdown {
-      lsp::MarkupKind::Markdown
-    } else {
-      lsp::MarkupKind::PlainText
-    },
-    value: message.as_ref().to_string(),
+    kind: lsp::MarkupKind::PlainText,
+    value: failure
+      .format(&TestFailureFormatOptions::default())
+      .to_string(),
   };
-  vec![lsp_custom::TestMessage {
+  let location = failure.error_location().map(|v| {
+    let pos = lsp::Position {
+      line: v.line_number,
+      // TODO: The column number might be wrong if Unicode is involved
+      character: v.column_number,
path: cli/lsp/testing/execution.rs
user: login: stefnotch
body: Okay, I tested this. Surprisingly enough, the column number seems to be correct.

I ran a test, and then clicked on the indicator. It placed my cursor at the correct column, and visually points there.

![image](https://github.com/user-attachments/assets/f26b4b0b-a091-4d61-b3b7-e5069aff6fb0)

created_at: 2025-05-09T11:19:18Z
author_association: CONTRIBUTOR
diff_hunk: @@ -399,6 +399,33 @@ impl TestFailure {
     }
   }
 
+  pub fn error_location(&self) -> Option<TestLocation> {
+    match self {
+      // TODO: What happens if not enough call stack frames were recorded?
+      // TODO: Could anything else trigger the ext:cli/40_test.js file? Maybe we should also test for the method name
+      TestFailure::JsError(js_error) => js_error
+        .frames
+        .iter()
+        .position(|v| v.file_name.as_deref() == Some("ext:cli/40_test.js"))
path: cli/tools/test/mod.rs
user: login: stefnotch
body: Sounds good, am doing the change
created_at: 2025-05-09T11:31:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -399,6 +399,33 @@ impl TestFailure {
     }
   }
 
+  pub fn error_location(&self) -> Option<TestLocation> {
+    match self {
+      // TODO: What happens if not enough call stack frames were recorded?
+      // TODO: Could anything else trigger the ext:cli/40_test.js file? Maybe we should also test for the method name
path: cli/tools/test/mod.rs
user: login: stefnotch
body: Yes, it's quite easy to for something else to trigger a `ext:` file. Like an illegal call to `Deno.test` from within a test.

![image](https://github.com/user-attachments/assets/19660e6d-0ca7-402f-a31e-cee14ff6f618)

```
TypeError: The test name can't be empty
  Deno.test("",() => {
       ^
    at testInner (ext:cli/40_test.js:224:13)
    at Object.test (ext:cli/40_test.js:332:10)
    at addTest (file:///C:/Users/Stefnotch/Desktop/dt/main_test.ts:8:8)
```

This particular case happens to be handled correctly, since it fails to find a user function above the `ext` line. So it defaults back to "report it wherever the test is"
created_at: 2025-05-09T11:32:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -399,6 +399,33 @@ impl TestFailure {
     }
   }
 
+  pub fn error_location(&self) -> Option<TestLocation> {
+    match self {
+      // TODO: What happens if not enough call stack frames were recorded?
+      // TODO: Could anything else trigger the ext:cli/40_test.js file? Maybe we should also test for the method name
path: cli/tools/test/mod.rs
user: login: stefnotch
body: I did find a case where only checking for `ext` does fail catastrophically

![image](https://github.com/user-attachments/assets/e00d5984-22f9-4d8c-a8b4-addb454f7c5f)

So we do need a more specific test.

Code is
```
import { A } from "./foo.ts";

// hmm

Deno.test("database operations", (t) => {
  Deno.inspect(new A());
});
```
and that, in another ifle
```
export class A {
 [Symbol.for("Deno.customInspect")]() {
    throw new Error("over here!");
 }
}
```

created_at: 2025-05-09T11:53:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -399,6 +399,33 @@ impl TestFailure {
     }
   }
 
+  pub fn error_location(&self) -> Option<TestLocation> {
+    match self {
+      // TODO: What happens if not enough call stack frames were recorded?
+      // TODO: Could anything else trigger the ext:cli/40_test.js file? Maybe we should also test for the method name
path: cli/tools/test/mod.rs
user: login: stefnotch
body: Finding the first entry in the stack that points at the current test file could be done. To do so, we would repeatedly call `uri_parse_unencoded(v.file_name)` until we find a URI that matches.

However, this wouldn't necessarily point at the correct line in the stack. Instead it'd point at whatever happens to come first in the current file.
![image](https://github.com/user-attachments/assets/ce262194-f00f-4888-a653-324473a91777)

It would need to be combined with a line and column number check. 
But we'd have no way of telling whether we're looking at the correct line, since the stack trace can be cut off.
created_at: 2025-05-09T12:14:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -399,6 +399,33 @@ impl TestFailure {
     }
   }
 
+  pub fn error_location(&self) -> Option<TestLocation> {
+    match self {
+      // TODO: What happens if not enough call stack frames were recorded?
path: cli/tools/test/mod.rs
user: login: stefnotch
body: Turns out that this is a very real issue. If we have about 10 extra call stack frames above the test, then we no longer have enough info to figure out where in the test the error happened.

e.g
```ts
Deno.test("database operations", (t) => {
  throwDeep(10)
});

function throwDeep (d = 0) {
  if(d <= 0) {
  throw new Error("bar");
  } else {
    throwDeep(d - 1)
  }
}
```
created_at: 2025-05-09T16:36:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -399,6 +399,33 @@ impl TestFailure {
     }
   }
 
+  pub fn error_location(&self) -> Option<TestLocation> {
+    match self {
+      // TODO: What happens if not enough call stack frames were recorded?
path: cli/tools/test/mod.rs
user: login: stefnotch
body: This makes implementing a correct "line in test file where error was thrown" significantly harder. 

I initially thought that I could skip all Deno internal frames from the bottom. Logically, the first thing after that must be a user frame.

Except that our stack trace might be cut off. In which case it sometimes doesn't have any Deno internal frames.

Or worse yet, the stack might randomly cut off after a completely unrelated Deno internal frame. 
e.g. `Deno.inspect` is an internal function. And we can throw errors from within a `[Symbol.for("Deno.customInspect")]`
created_at: 2025-05-09T16:53:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -399,6 +399,33 @@ impl TestFailure {
     }
   }
 
+  pub fn error_location(&self) -> Option<TestLocation> {
+    match self {
+      // TODO: What happens if not enough call stack frames were recorded?
+      // TODO: Could anything else trigger the ext:cli/40_test.js file? Maybe we should also test for the method name
+      TestFailure::JsError(js_error) => js_error
+        .frames
+        .iter()
+        .position(|v| v.file_name.as_deref() == Some("ext:cli/40_test.js"))
path: cli/tools/test/mod.rs
user: login: stefnotch
body: I did the necessary testing and it turns out that we should be checking for `ext:cli/40_test.js`. Turns out that there are a lot of edge cases with just `ext:`


created_at: 2025-05-09T17:25:51Z
author_association: CONTRIBUTOR

PR comments:
user: login: stefnotch
created_at: 2025-05-09T18:26:07Z
body: @nayeemrmn I think I'm ready for writing tests for this feature.

I'm assuming that the `lsp_testing_api` test is the one that I care about. How would you suggest that I modify it?

- Do I add a failing test step?
- Or do I add a second `Deno.test` to the contents? If I did that, would I get any ordering guarantees or would I be causing a flaky and racy test?
- Or do I do `temp_dir.write("./test.ts", new contents go here);` and do the whole dance?


author_association: CONTRIBUTOR
user: login: nayeemrmn
created_at: 2025-05-13T00:57:42Z
body: I went ahead and added a test. Let me know if you have questions with the remaining TODOs. It's probably okay to bail and not provide a location where it's hindered by some edge case.
author_association: CONTRIBUTOR
user: login: stefnotch
created_at: 2025-05-15T08:32:59Z
body: I looked at the remaining TODO with fresh eyes. I found a case that it doesn't handle properly. I tweaked it to now also handle that case correctly.


The failure case was this
```ts
import { someFn } from "./bar.ts"

Deno.test(someFn); // The first stack frame above the Deno.test function points at `bar.ts`
```
author_association: CONTRIBUTOR
user: login: stefnotch
created_at: 2025-05-17T14:16:41Z
body: @nayeemrmn The tests apparently fail on Windows because the drive letter in the JS stack trace https://github.com/denoland/deno_core/blob/810fea51a40fbc4c7145cc8b5f18090abc9d9250/core/error.rs#L1825-L1829 is uppercased. Meanwhile the drive letter returned by `file.uri().as_str()` is lowercased since https://github.com/denoland/deno/pull/28737

How do I go about fixing this? Do I ignore that difference? (e.g. by only checking whether the error message starts with the right string)
author_association: CONTRIBUTOR
user: login: nayeemrmn
created_at: 2025-05-17T19:05:28Z
body: > Meanwhile the drive letter returned by `file.uri().as_str()` is lowercased

I guess change it to `file.url()`
author_association: CONTRIBUTOR
user: login: stefnotch
created_at: 2025-05-17T19:34:54Z
body: @nayeemrmn Thank you, I changed it to `.url`. It works on Windows now :)

The CI failure seems to be something unrelated. Seems to be GitHub deciding to rate limit GitHub.

```
 Diff < left / right > :
<error: Import 'https://gist.githubusercontent.com/ry/f12b2aa3409e6b52645bc346a9e22929/raw/79318f239f51d764384a8bded8d7c6a833610dde/print_hello.ts' failed: 429 Too Many Requests
<    at file:///Users/runner/work/deno/deno/tests/specs/run/_017_import_redirect/017_import_redirect.ts:2:28
>Hello
```
author_association: CONTRIBUTOR
user: login: nayeemrmn
created_at: 2025-05-17T19:58:33Z
body: Yeah we're having some flakiness in CI, I'll keep an eye on it and rerun if needed
author_association: CONTRIBUTOR