Repository: yshavit/mdq
Pull request #282 (2025-03-30 02:36:55)

test stderr in integ tests
Rather than having bad queries always just `eprintln` to stderr and then return false, have the `lib.rs` runners return a Result, where the error case is that invalid query's error. Then, use that to test for the error string.

coderabbitai[bot]: <!-- This is an auto-generated comment: summarize by coderabbit.ai -->
<!-- walkthrough_start -->

<details>
<summary>📝 Walkthrough</summary>

## Walkthrough
The pull request introduces enhancements focused on explicit error reporting in both testing and core logic. In the build configuration, new fields have been added to the test case structures to capture expected error outputs, and these are integrated into the test generation logic. The core library code now utilizes a new `Error` enum for handling different error scenarios, with function signatures updated to return `Result` types instead of plain booleans. This change streamlines error propagation by encapsulating Markdown and query parsing errors, and it removes legacy error printing methods. In the integration tests and markdown case files, the test harness is updated to compare the captured error outputs against expected values, ensuring that the test cases accurately reflect syntax and argument parsing errors.

## Possibly related PRs
- **yshavit/mdq#252**: Introduces an `expect_error` field in the `Case` struct in a similar context, aligning with the structural enhancements for error reporting.

</details>

<!-- walkthrough_end -->
<!-- internal state start -->


<!-- DwQgtGAEAqAWCWBnSTIEMB26CuAXA9mAOYCmGJATmriQCaQDG+Ats2bgFyQAOFk+AIwBWJBrngA3EsgEBPRvlqU0AgfFwA6NPEgQAfACgjoCEYDEZyAAUASpADK2VmgqyjcEj2wAbb5AokAI7Y0riQ4rjedJAARDSIYQlKFHzwWGk0ROGhiDGQAO5oyAwB1NFykNiIlJCyiLBoEuroGPSo8EoY4gBm8OXyuLCeGE4CNQBMABzjGjBD1nYMmJBj6AwM0tX01OHz3fi++PlpWQCqNgAyXADasLi43IgcAPTPROqw2AIaTMzPdQ0mrhnsxaIFntwfN5nlNxgBdAAUdweT1e70GXx+LH+9Ua6hBYIhUJh0wAlBp3PMlIgSvBuOJ8Fh8N1dp5bPw8N40tJ0IwGhhSCgsINPPzaFyBfwWQI0PRgpQ+shjoM0qyhZkqAzhTlZgBJDAJEiyqVqhjYFLsdDcXj4NAMWAFIYBIUSNBcuUhCiKyCIeDMbjeeQBRA+MKq5aUCj4PhsRCINCCsYnHherrRAg+3DJPhVZMAAxIvAy3gweYANGqbdx8FtIMxFPBekstT7sERSAlkIMdiLIHmud8KIg8/5sBhyEOfbB8D56AFcOasGg+zZpKG83rhQhkOR8pmKNgxOaSBXe5Ho4wip58jPvPRENxRI34EtfPI0rQX2U1ee+EVEPgDDwGU9DKg6vZpK67qQPKrizHAqD2pggraMwXb4JA3R2vAXK4N+8TiJK+x8GeKQXgkqZZKQE4gSsAxDNULpuh0MGeoqp5OiQFR+jaTSSr2UYCFUuDkHGLT0Aw3guOo8jMpAv6QGKEpZKqEFpkQmrwIy2SdhSBj6MY4BQGQ9ByWgeCENRyg0BJLBsF0XC8PwwiiOIUgyPITDJCoaiaNouhgIYJhQAhyCoMs5kEMQZDWdEvz2Zw/hoHuIbOK4dEKN5qjqFoOgGYZpgGEJOG0BoQ4cAYMRVQYFiQAAgrq0U0TZrZpbJLJIQK0iUp4nUduqUa0IenjLru/D0lpGBuphfR3n2M4PHgAD6kYjhmvZ5tAoQAKIAB6PmII4UYeuAVm63hHMmxFqg+T5NtQk0miQ+2udECkLZCmhzOFtCfi2qABN0URiNEqnzHmpC4EtSzVMOdYkIMigmhtW0JPYB0AGI4SQI48VECUPYyFb5JxfbPQdUO/rjO5HEK6jAVyABe0S5pKeYEStL1iBoH3LZGGhjvkVDcEt0ZLUoWGhgipLlvJBrmsm3ZhmEEvmd4uDoS08n+rg8gUcmjaQBgGFQSEKDILxHR0LMGPmiKFD1gEHGeHmADCV5HbgB5iOblTcLQtEZmkknYEoapjQWXOU2RFAjr0JB3puap5kL6gkEtHPdBgGf4CObCI6ZLIbe71Se97p2QBd7wML7spKPQGZKZ4va82EcmRxTK0xxukC6iyHeuV3Uax77JA67IFbNKn8Ra2P9J617JwANz8PbxzVJPYTEcw1BdvM714J9Pp+jhLiBuEGHTnuA9iKLh94NTimYOKdAVqzWR5h+gEcBwX8MCAccLzMFDPAMAEpPD6wFIgeC24+TIU8GQfkGw97N1CJeRix0jzOgqOdS6RELzky5EBMIClyauUJgaPS5hLB1XVtZSamtexKEki4ChyA5Lk2jC1C8kIBBELlhERURgoBYwTtsX6dAuB5lbkPLgAB5CajJgD2EXgKPQuMsCYLCJtHaUcNF9mKonIceZ9KQFEXNOukiyZRyHtGLgKjKL6K0X2EuOMhQGOwCVMqw5KrVX0oVRAFAGDPAHN4iqVUYg1RoY1KympoipR3ulOSfVuoGA8HArqYUuiDWGryCO21u5yycBfJ+rQoiQE/N0bolBLQKV1o+ZAAQpItQzLBeQ3AXC+klM/SAABZFwABrWgRwsAdKHCcGBLsDzZzSEtNgjtZBxzHGISaAByRpCNFzhFkI+X29ZKl9HoN0KMzBeQLgDC7BEAh8AHArA4k4MsSnLjzKuEM6tgBXJud4O5qiiCkgrAU4e6izqHGOJKZoGZ5xbN/Mgd+4cSDX0BdGEcZAnCTPVJQKafgq4vmRuDaZSyMArO0qgbA/tA4YShRQJcK41zvOubcyASKKDqKFIaY0ZkVhfKNFgU2JBZhGGZaU8UBtkCIMwBsWgXASZkF5P0igQyRk8E6cmd66xzSIE3kbWmVKDRazzMyn+8rFX5AwFYTpJAESRkeWkdlhcUwZEVvvGOJSEjPxcPQBSPTdVXTdNUWY9gT5SQoIGYmQwaXVGBgQPgYyulZDVWaIcWrdXIGWAamOP8ACKnpZDmqHG4sCaomkkFdF0VilAF6UXEvJF1Sg8I4SyXak0RYujJgjC62M8YUKtH8Js6lvrvD+sgD1PsLaoaRsHrG9OVNMLLJbA0GQJBZUBHrFIWgZ0woa1nUSrUzFda+2NnuDIJANK0QyBhUiw9hXKTxVMschLiUYHRfgKQVBfAKGyQcTCF0j2ptoEIYS6YMJVgTN+GFftCgUFoJq9AoKrpAOjJ4QSwka1XoCNWCghEiB6XSXXem2k5IbQACJIADGgRZKB/T43YBQzCF4NrMpRSMU5ltqS8m3tQFqkCsjoeDDRls10jT2lQ/RnZJ45ZIKdb1YNMknods2AmaQVCon1ToZpRkjCqSiGDWwp6+0uHRB4V8fh7B6apKgNtZj6AJFStHV8IppzGPuLzIE4JoTjGmN6Qjac954BECmguZ0ZKA42SkbwzCWBplLVmfM6MsgESSXgFwAAZK7LkFYmBpi6E8SAyWKKkl0HoSAnzGX3IFDarALmgkhPgIOOGgAkwjswICLo4ZnZ1i64BLXIUtpfgBlxkNBsspfy4VyArzQwfIZd8hwvz/lMpjqy1UVW3O1e8SYqAXmC7H389QY8fsQtWLzFnVrwAADqFYMZ6C60l3LvX+tZY1vY35FZIZ30ShjArgUuUHH0ctmrdWRyNaO5FscZ2LtXcSz19LH7BuPZm5RF7CM3tcA+6N8b9KvkAoW791z/21u+Mif4sARgCKIGeMeogGdQhhIJyphqTVYr3icIk9qGSOwjpSVkr2ihcmjQRTNMR1jO4zvWuDVxZcTphpfA6cKmYHo124723g0hTP8X3lHN6LrW7orzPaUQAy87eaRqgfZz4gOXnfXzvc+cfN9gJVL4TKazlkvKZl+tGApNrAXNNEM6xNhy4XDBt1rQPUckWhXHpweA5QZE3wbXcxPDK6aDOLJ4DK74GrnRmNKrJTxWfmA7k6AKBECcOwVNPbpke7V71OyPSF0rCXVgMgSxHg+FomBMGd7SzwwLjrglPfbdNxQQoZwPbnqiDwI9HpCkd5TVIAlCsPrulN/HvueDfAo+h9bu4mGng5K++QTB+jzqr2dsUzvq8JosI4WPOi/8lAtRZOTnrhgBuB9IxcJ4YLFKhQhzDsuX4MZJAAjIueYO0b3PwA+cPdAIgbQA0MIM8DXT1LXe+TQEdF9ZQd9N3KMPwIGWmcKNTc3aoMZfCeYPPMfXaCfFsHFGuY5FgItOlMIF/AZE4CscVIlT3FhL0fdQjcGZgw3XvIwahVTGgdTPVUXTwZhHTR/PTDDbhGNYzXFUzcQczMxWacReuKRMhW+X8FLdZPCcQBXL2JxL2E6FxD2ZzUncndSKnBIfHDbI3DQw7Y7aZBEPLBOboT7IrErabMrIgH5SiCrfcMwt2CwpbKwinWwzQDzCJYRQqKw0EaGK8MnCdW+O4ZgbwDQAgDI8JaqWqenWJWiBJFwVnTnDnfkfqDIHJDYPJfnBSJSZMNgTqJAU5a6XsAidBTwKSMYKIegGIMcYIfAFpBMDQPIdJaMPzNIaabfNo2BDo3fJ+RdZdMeDA9datW1GgWUM6SrGRVaAXOaevMYWVSxUqb6HcfneOOaVjHkZcOtbQXon0WQLoNAXaWPeGOMc/D8L8LDXYHseYNII+RXE3N0bea2BwA6Z8V8UNFARIAwnkJWNUAY7AIY+JX5OsFDN1TDAoD4XkKIe4GoOvPzWALkIgO4YfasX0FsXghBF1TvIUT6HXbQ8dQ8Q/OOKSHjMeOA2FIlCo1+TMHCPwaoBAjCLCQdcTL45sJ1P41BBIUeJAkpa/PwIaZuC9eYRAJ4vCV438HDeYU3e6GQ9gmo3sLg2TDhF1dDLhdfXYVAeYy/CoQhF8dQc+O4nCT3ALPbakx454rUl1ZvGcNMAIUqIQunNTXTCQipbTVhGQjhfTTDQzBQvhJQ1tFQxAYRSAORVAyAV0L0FQcpb/ULeaVA37a4Jk0YpElEhuBMGIOEX7BI2gJI2GZ4VIqGdIzI7I7wQHQs6AgAXkgFWVWXW0gAADl+dszgI+ERobMpFdiUhizSz+iMBBjhiiBqzaycgCQGzpAmyE5B5WysiWAOzacicSd1zEjd8Uidzb4SDT1uBYBoF2zcjIl8iYkYo4kmc2oTQyi0kyDeSudqiaSr0Gj8FUgMAoIWJyDaAC9yAi8S8Eon92i0Fzys8fRLyfjrzhY7yA0IT7o3wKxZiZSmDkjK4VAdy+jjYFALolxuhRCeA8gekYgHw7RPAxhiIa9KKyw8gW5MzDjG86wGx44G4MJg5vBQ4RodjUCh4451DfjEhsLvRlx1TvS3iZUtxE8owmh64+xEsloXAiA4ZhltUmDGRt4SlECKZogOMvp0kk8tIqguzPopLBceLliyNJVsTBhZ5V9uNJ5WhvipSwhyKwLTJMzUBCiWpO9GJzz0UZ8FN+oXS/BewLpJSQCboNSXi3iekTTuh6Idhlwgry10pCgxUkCdTPA9S/KNMJMJVm4yCZMeCWQFILTMNkx7T9oiEnSF4Hpq8jZdtnRPTFLNTY8xUiV/TRDoghoq1exY1kxPSIKoKRpi9S9stlN8jQyZDwypCoyGFZCDNTIEyTNkyhFTEhyMI3RRCwzVTJDIyxD2EGrYz5CvBEya5lDvR8gak6xZRPAwZUBKlugKRYj/FDADBgo5Z7VIpLJXzaJ4p2AuAqAUpmcSiMovJlBso/I8pApgajIR9mB1BotoMloAgmgEU6AlpMSwh8oQaAAWEgAAZnGAAEY6ABBKbaAAA2SmgAdloHGFZpIAAFYAAGVmvm5mhgAATjQHpoEAFsprQG6AEEmD5toDFr5t5oCiBpBvilxo6EQAJpLT6DevrJMnVqAA= -->

<!-- internal state end -->
<!-- finishing_touch_checkbox_start -->

<details>
<summary>✨ Finishing Touches</summary>

- [ ] <!-- {"checkboxId": "7962f53c-55bc-4827-bfbf-6a18da830691"} --> 📝 Generate Docstrings

</details>

<!-- finishing_touch_checkbox_end -->
<!-- tips_start -->

---



<details>
<summary>🪧 Tips</summary>

### Chat

There are 3 ways to chat with [CodeRabbit](https://coderabbit.ai?utm_source=oss&utm_medium=github&utm_campaign=yshavit/mdq&utm_content=282):

- Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  - `I pushed a fix in commit <commit_id>, please review it.`
  - `Generate unit testing code for this file.`
  - `Open a follow-up GitHub issue for this discussion.`
- Files and specific lines of code (under the "Files changed" tab): Tag `@coderabbitai` in a new review comment at the desired location with your query. Examples:
  - `@coderabbitai generate unit testing code for this file.`
  -	`@coderabbitai modularize this function.`
- PR comments: Tag `@coderabbitai` in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  - `@coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.`
  - `@coderabbitai read src/utils.ts and generate unit testing code.`
  - `@coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.`
  - `@coderabbitai help me debug CodeRabbit configuration file.`

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

### CodeRabbit Commands (Invoked using PR comments)

- `@coderabbitai pause` to pause the reviews on a PR.
- `@coderabbitai resume` to resume the paused reviews.
- `@coderabbitai review` to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
- `@coderabbitai full review` to do a full review from scratch and review all the files again.
- `@coderabbitai summary` to regenerate the summary of the PR.
- `@coderabbitai generate docstrings` to [generate docstrings](https://docs.coderabbit.ai/finishing-touches/docstrings) for this PR.
- `@coderabbitai resolve` resolve all the CodeRabbit review comments.
- `@coderabbitai plan` to trigger planning for file edits and PR creation.
- `@coderabbitai configuration` to show the current CodeRabbit configuration for the repository.
- `@coderabbitai help` to get help.

### Other keywords and placeholders

- Add `@coderabbitai ignore` anywhere in the PR description to prevent this PR from being reviewed.
- Add `@coderabbitai summary` to generate the high-level summary at a specific location in the PR description.
- Add `@coderabbitai` anywhere in the PR title to generate the title automatically.

### CodeRabbit Configuration File (`.coderabbit.yaml`)

- You can programmatically configure CodeRabbit by adding a `.coderabbit.yaml` file to the root of your repository.
- Please see the [configuration documentation](https://docs.coderabbit.ai/guides/configure-coderabbit) for more information.
- If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: `# yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json`

### Documentation and Community

- Visit our [Documentation](https://docs.coderabbit.ai) for detailed information on how to use CodeRabbit.
- Join our [Discord Community](http://discord.gg/coderabbit) to get help, request features, and share feedback.
- Follow us on [X/Twitter](https://twitter.com/coderabbitai) for updates and announcements.

</details>

<!-- tips_end -->

Patch:
diff --git a/build.rs b/build.rs
index 2220acc..4f42297 100644
--- a/build.rs
+++ b/build.rs
@@ -133,6 +133,7 @@ struct TestExpect {
     output_json: Option<bool>,
     expect_success: Option<bool>,
     ignore: Option<String>,
+    output_err: Option<String>,
 }
 
 #[derive(Deserialize, Copy, Clone)]
@@ -148,6 +149,7 @@ impl TestSpecFile {
                 case_name,
                 cli_args: test_expect.cli_args,
                 expect_output: test_expect.output,
+                expect_error: test_expect.output_err.unwrap_or_default(),
                 output_json: test_expect.output_json.unwrap_or(false),
                 expect_success: test_expect.expect_success.unwrap_or(true),
                 ignored: test_expect.ignore.is_some(),
@@ -163,6 +165,7 @@ struct Case {
     ignored: bool,
     cli_args: Vec<String>,
     expect_output: String,
+    expect_error: String,
     output_json: bool,
     expect_success: bool,
 }
@@ -195,7 +198,7 @@ impl Case {
                 out.writeln(&format!("cli_args: {:?},", &self.cli_args));
                 out.writeln(&format!("expect_output_json: {},", self.output_json));
                 if self.expect_output.is_empty() {
-                    out.write("expect_output: \"\",");
+                    out.writeln("expect_output: \"\",");
                 } else {
                     out.write("expect_output: indoc::indoc! {r#\"");
                     out.with_indent(|out| {
@@ -210,6 +213,22 @@ impl Case {
                         }
                     });
                 }
+                if self.expect_error.is_empty() {
+                    out.writeln("expect_error: \"\",");
+                } else {
+                    out.write("expect_error: indoc::indoc! {r#\"");
+                    out.with_indent(|out| {
+                        let mut iter = self.expect_error.split('\n').peekable();
+                        while let Some(line) = iter.next() {
+                            out.write(line);
+                            if iter.peek().is_some() {
+                                out.nl();
+                            } else {
+                                out.write("\"#},");
+                            }
+                        }
+                    });
+                }
                 out.write("expect_success: ")
                     .write(&self.expect_success.to_string())
                     .writeln(",");
diff --git a/src/lib.rs b/src/lib.rs
index d7fe790..98bfe9a 100644
--- a/src/lib.rs
+++ b/src/lib.rs
@@ -2,13 +2,15 @@ use crate::fmt_md::MdOptions;
 use crate::fmt_md_inlines::MdInlinesWriterOptions;
 use crate::output::{OutputOpts, Stream};
 use crate::select::{ParseError, SelectorAdapter};
-use crate::tree::{MdDoc, ReadOptions};
+use crate::tree::{InvalidMd, MdDoc, ReadOptions};
 use crate::tree_ref::MdElemRef;
 use crate::tree_ref_serde::SerdeDoc;
 use cli::{Cli, OutputFormat};
 use output::Output;
 use pest::error::ErrorVariant;
 use pest::Span;
+use std::borrow::Cow;
+use std::fmt::{Display, Formatter};
 use std::io;
 use std::io::{stdin, Read, Write};
 
@@ -33,12 +35,46 @@ mod utils_for_test;
 mod vec_utils;
 mod words_buffer;
 
-pub fn run_in_memory(cli: &Cli, contents: &str) -> (bool, String) {
+#[derive(Debug)]
+pub enum Error {
+    QueryParse { query_string: String, error: ParseError },
+    MarkdownParse(InvalidMd),
+}
+
+impl Display for Error {
+    fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+        match self {
+            Error::QueryParse { query_string, error } => {
+                let pest_err = match error {
+                    ParseError::Pest(err) => Cow::Borrowed(err),
+                    ParseError::Other(span, message) => {
+                        let full_span = Span::new(query_string, span.start, span.end);
+                        Cow::Owned(query::Error::new_from_span(
+                            ErrorVariant::CustomError {
+                                message: message.to_string(),
+                            },
+                            full_span.unwrap(),
+                        ))
+                    }
+                };
+
+                writeln!(f, "Syntax error in select specifier:")?;
+                writeln!(f, "{pest_err}")
+            }
+            Error::MarkdownParse(err) => {
+                writeln!(f, "Markdown parse error:")?;
+                writeln!(f, "{err}")
+            }
+        }
+    }
+}
+
+pub fn run_in_memory(cli: &Cli, contents: &str) -> Result<(bool, String), Error> {
     let mut out = Vec::with_capacity(256); // just a guess
 
-    let result = run(&cli, contents.to_string(), || &mut out);
-    let out_str = String::from_utf8(out).map_err(|_| "UTF-8 decode error").unwrap();
-    (result, out_str)
+    let result = run(&cli, contents.to_string(), || &mut out)?;
+    let out_str = String::from_utf8(out).unwrap_or_else(|err| String::from_utf8_lossy(err.as_bytes()).into_owned());
+    Ok((result, out_str))
 }
 
 pub fn run_stdio(cli: &Cli) -> bool {
@@ -47,10 +83,13 @@ pub fn run_stdio(cli: &Cli) -> bool {
     }
     let mut contents = String::new();
     stdin().read_to_string(&mut contents).expect("invalid input (not utf8)");
-    run(&cli, contents, || io::stdout().lock())
+    run(&cli, contents, || io::stdout().lock()).unwrap_or_else(|err| {
+        eprintln!("{err}");
+        false
+    })
 }
 
-fn run<W, F>(cli: &Cli, contents: String, get_out: F) -> bool
+fn run<W, F>(cli: &Cli, contents: String, get_out: F) -> Result<bool, Error>
 where
     F: FnOnce() -> W,
     W: Write,
@@ -63,17 +102,18 @@ where
     let MdDoc { roots, ctx } = match MdDoc::read(ast, &read_options) {
         Ok(mdqs) => mdqs,
         Err(err) => {
-            eprintln!("error: {}", err);
-            return false;
+            return Err(Error::MarkdownParse(err));
         }
     };
 
     let selectors_str = cli.selector_string();
     let selectors = match SelectorAdapter::parse(&selectors_str) {
         Ok(selectors) => selectors,
-        Err(err) => {
-            print_select_parse_error(err, &selectors_str);
-            return false;
+        Err(error) => {
+            return Err(Error::QueryParse {
+                query_string: selectors_str.into_owned(),
+                error,
+            });
         }
     };
 
@@ -117,17 +157,5 @@ where
         }
     }
 
-    found_any
-}
-
-fn print_select_parse_error(err: ParseError, query_string: &str) {
-    eprintln!("Syntax error in select specifier:");
-    let pest_err = match err {
-        ParseError::Pest(err) => err,
-        ParseError::Other(span, message) => {
-            let full_span = Span::new(query_string, span.start, span.end);
-            query::Error::new_from_span(ErrorVariant::CustomError { message }, full_span.unwrap())
-        }
-    };
-    eprintln!("{}", pest_err);
+    Ok(found_any)
 }


Test patch:
diff --git a/tests/integ_test.rs b/tests/integ_test.rs
index 055902d..263b731 100644
--- a/tests/integ_test.rs
+++ b/tests/integ_test.rs
@@ -4,6 +4,7 @@ use clap::Parser;
 struct Case<const N: usize> {
     cli_args: [&'static str; N],
     expect_output: &'static str,
+    expect_error: &'static str,
     expect_output_json: bool,
     md: &'static str,
     expect_success: bool,
@@ -11,9 +12,7 @@ struct Case<const N: usize> {
 
 impl<const N: usize> Case<N> {
     fn check(&self) {
-        let all_cli_args = ["cmd"].iter().chain(&self.cli_args);
-        let cli = mdq::cli::Cli::try_parse_from(all_cli_args).unwrap();
-        let (actual_success, actual_out) = mdq::run_in_memory(&cli, self.md);
+        let (actual_success, actual_out, actual_err) = self.run();
         let (actual_out, expect_out) = if self.expect_output_json {
             let actual_obj = serde_json::from_str::<serde_json::Value>(&actual_out).unwrap();
             let expect_obj = serde_json::from_str::<serde_json::Value>(self.expect_output).unwrap();
@@ -25,8 +24,18 @@ impl<const N: usize> Case<N> {
             (actual_out, self.expect_output.to_string())
         };
         assert_eq!(actual_out, expect_out);
+        assert_eq!(actual_err, self.expect_error);
         assert_eq!(actual_success, self.expect_success);
     }
+
+    fn run(&self) -> (bool, String, String) {
+        let all_cli_args = ["cmd"].iter().chain(&self.cli_args);
+        let cli = mdq::cli::Cli::try_parse_from(all_cli_args).unwrap();
+        match mdq::run_in_memory(&cli, self.md) {
+            Ok((found, stdout)) => (found, stdout, String::new()),
+            Err(err) => (false, String::new(), err.to_string()),
+        }
+    }
 }
 
 include!(concat!(env!("OUT_DIR"), "/tests/integ_test_cases.rs"));
diff --git a/tests/md_cases/select_html.toml b/tests/md_cases/select_html.toml
index 660b782..d570e65 100644
--- a/tests/md_cases/select_html.toml
+++ b/tests/md_cases/select_html.toml
@@ -52,7 +52,14 @@ output = '''
 [expect."unquoted tag"]
 cli_args = ['</> <span>']
 expect_success = false # unquoted string must start with a letter, not a '<'
-output = '''
+output = ''
+output_err = '''Syntax error in select specifier:
+ --> 1:5
+  |
+1 | </> <span>
+  |     ^---
+  |
+  = expected end of input or string
 '''
 
 
diff --git a/tests/md_cases/select_paragraphs.toml b/tests/md_cases/select_paragraphs.toml
index 642c497..ff2847a 100644
--- a/tests/md_cases/select_paragraphs.toml
+++ b/tests/md_cases/select_paragraphs.toml
@@ -56,14 +56,28 @@ This paragraph has _inline_ **formatting**.
 [expect."no colon after p"]
 cli_args = ["P *"]
 expect_success = false
-output = '''
+output = ''
+output_err = '''Syntax error in select specifier:
+ --> 1:1
+  |
+1 | P *
+  | ^---
+  |
+  = expected valid query
 '''
 
 
 [expect."space before colon"]
 cli_args = ["P : *"]
 expect_success = false
-output = '''
+output = ''
+output_err = '''Syntax error in select specifier:
+ --> 1:1
+  |
+1 | P : *
+  | ^---
+  |
+  = expected valid query
 '''