Repository: sourcefrog/cargo-mutants (rust)

Issue:
Option to disable running doctests
Is there an option somewhere to disable running doctests? I am attempting to run this on a fairly large project with over 50 doctests, but it causes each mutant to take a long time to run. That, combined with a lot of mutations (over 300) makes for a long run-time.

For comparison, running normally takes ~15 seconds for each mutation, which ends up being about an hour and a half. However, if I modify my library's `Cargo.toml` to disable doctests (setting `doctests = false`), the time for each mutant reduces to about 1-1.5 seconds each. That ends up being only approximately 7.5 minutes, which is much more desirable, especially if I want to run this in CI.

Is there a way a flag can be added to disable doctests? Or perhaps one already exists? In general, my doctests tend to test things already covered in my unit tests, and serve more as examples in the documentation, so I don't think much is lost by excluding them. 

Pull request:
Pass extra arguments to "cargo test"
Allows for example skipping doctests or limiting the number of threads.

Closes #14, #15 

cc @Anders429 @xd009642

Patch:
diff --git a/Cargo.lock b/Cargo.lock
index 7a83de6..1ca6ba0 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -94,7 +94,7 @@ dependencies = [
 
 [[package]]
 name = "cargo-mutants"
-version = "0.2.0"
+version = "0.2.1-pre"
 dependencies = [
  "anyhow",
  "argh",
diff --git a/Cargo.toml b/Cargo.toml
index 7dbf302..921267e 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "cargo-mutants"
-version = "0.2.0"
+version = "0.2.1-pre"
 edition = "2018"
 authors = ["Martin Pool"]
 license = "MIT"
diff --git a/NEWS.md b/NEWS.md
index 5518922..90e995d 100644
--- a/NEWS.md
+++ b/NEWS.md
@@ -1,19 +1,23 @@
 # cargo-mutants changelog
 
+## Unreleased
+
+- Arguments to `cargo test` can be passed on the command line after `--`.
+  <https://github.com/sourcefrog/cargo-mutants/issues/15>
+
 ## 0.2.0
 
 Released 2022-02-06
 
 - A new `--timeout SECS` option to limit the runtime of any `cargo test`
   invocation, so that mutations that cause tests to hang don't cause
-  `cargo mutants` to hang. 
+  `cargo mutants` to hang.
 
   A default timeout is set based on the time to run tests in an unmutated tree.
-  There is no timeout by default on the unmutated tree. 
+  There is no timeout by default on the unmutated tree.
 
-  On Unix, the `cargo` subprocesses run in a new process group. As a
-  consequence ctrl-c is explicitly caught and propagated to the child
-  processes.
+  On Unix, the `cargo` subprocesses run in a new process group. As a consequence
+  ctrl-c is explicitly caught and propagated to the child processes.
 
 - Show a progress bar while looking for mutation opportunities, and show the
   total number found.
diff --git a/README.md b/README.md
index 077b453..04f5216 100644
--- a/README.md
+++ b/README.md
@@ -104,8 +104,8 @@ flags the function for cargo-mutants.
 - **3**: Some tests timed out: possibly the mutatations caused an infinite loop,
   or the timeout is too low.
 
-- **4**: The tests are already failing or hanging before any mutations are applied,
-  so no mutations were tested.
+- **4**: The tests are already failing or hanging before any mutations are
+  applied, so no mutations were tested.
 
 ### `mutants.out`
 
@@ -119,6 +119,22 @@ A `mutants.out` directory is created in the source directory. It contains:
 
 - An `outcomes.json` file describing the results of all tests.
 
+### Passing arguments to `cargo test`
+
+Command-line options following a `--` delimiter are passed through to
+`cargo test`, which can be used for example to exclude doctests (which tend to
+be slow to build and run):
+
+```sh
+cargo mutants -- --all-targets
+```
+
+You can use a second double-dash to pass options through to the test targets:
+
+```sh
+cargo mutants -- -- --test-threads 1 --nocapture
+```
+
 ### Hangs and timeouts
 
 Some mutations to the tree can cause the test suite to hang. For example, in
@@ -138,8 +154,7 @@ maximum of 5 seconds, or 3x the time to run tests with no mutations.
 You can also set an explicit timout with the `--timeout` option. In this case
 the timeout is also applied to tests run with no mutation.
 
-The timeout does not apply to `cargo check` or `cargo build`, only `cargo
-test`.
+The timeout does not apply to `cargo check` or `cargo build`, only `cargo test`.
 
 When a test times out, you can mark it with `#[mutants::skip]` so that future
 `cargo mutants` runs go faster.
@@ -154,15 +169,14 @@ When a test times out, you can mark it with `#[mutants::skip]` so that future
 
 ### Performance
 
-Anything you can do to make the `cargo build` and `cargo test` suite faster
-will have a multiplicative effect on `cargo mutants` run time, and of course
-will also make normal development more pleasant. There's lots of good advice
-on the web.
+Anything you can do to make the `cargo build` and `cargo test` suite faster will
+have a multiplicative effect on `cargo mutants` run time, and of course will
+also make normal development more pleasant. There's lots of good advice on the
+web.
 
-In particular, on Linux, using the
-[Mold linker](https://github.com/rui314/mold) can improve build times
-significantly: because cargo-mutants does many incremental builds, link time
-is important.
+In particular, on Linux, using the [Mold linker](https://github.com/rui314/mold)
+can improve build times significantly: because cargo-mutants does many
+incremental builds, link time is important.
 
 ### Hard-to-test cases
 
diff --git a/src/lab.rs b/src/lab.rs
index be1d8c9..0832938 100644
--- a/src/lab.rs
+++ b/src/lab.rs
@@ -122,8 +122,8 @@ pub fn test_unmutated_then_all_mutants(
             console,
         )?);
 
-        // Rewrite outcomes.json every time, so we can watch it and so it's not lost if the
-        // program stops or is interrupted.
+        // Rewrite outcomes.json every time, so we can watch it and so it's not
+        // lost if the program stops or is interrupted.
         serde_json::to_writer_pretty(
             BufWriter::new(File::create(output_dir.path().join("outcomes.json"))?),
             &lab_outcome,
@@ -164,16 +164,29 @@ fn run_cargo_phases(
     for &phase in phases {
         let phase_start = Instant::now();
         activity.set_phase(phase.name());
-        let cargo_args: &[&str] = match phase {
-            Phase::Check => &["check", "--tests"],
-            Phase::Build => &["build", "--tests"],
-            Phase::Test => &["test"],
+        let cargo_args = match phase {
+            Phase::Check => vec!["check", "--tests"],
+            Phase::Build => vec!["build", "--tests"],
+            Phase::Test => std::iter::once("test")
+                .chain(
+                    options
+                        .additional_cargo_test_args
+                        .iter()
+                        .map(String::as_str),
+                )
+                .collect(),
         };
         let timeout = match phase {
             Phase::Test => options.test_timeout(),
             _ => Duration::MAX,
         };
-        let cargo_result = run_cargo(cargo_args, build_dir, &mut activity, &mut log_file, timeout)?;
+        let cargo_result = run_cargo(
+            &cargo_args,
+            build_dir,
+            &mut activity,
+            &mut log_file,
+            timeout,
+        )?;
         outcome.add_phase_result(phase, phase_start.elapsed(), cargo_result);
         if (phase == Phase::Check && options.check_only) || !cargo_result.success() {
             break;
diff --git a/src/main.rs b/src/main.rs
index b705443..9756b80 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -34,6 +34,8 @@ pub use crate::run::CargoResult;
 use crate::source::SourceTree;
 
 /// Find inadequately-tested code that can be removed without any tests failing.
+///
+/// See <https://github.com/sourcefrog/cargo-mutants> for more information.
 #[derive(FromArgs, PartialEq, Debug)]
 struct Args {
     /// show cargo output for all invocations (very verbose).
@@ -79,6 +81,12 @@ struct Args {
     /// print mutations that failed to check or build.
     #[argh(switch, short = 'V')]
     unviable: bool,
+
+    // The following option captures all the remaining non-option args, to
+    // send to cargo.
+    /// pass remaining arguments to cargo test after all options and after `--`.
+    #[argh(positional)]
+    cargo_test_args: Vec<String>,
 }
 
 fn main() -> Result<()> {
@@ -109,6 +117,8 @@ fn main() -> Result<()> {
         }
     } else {
         let lab_outcome = lab::test_unmutated_then_all_mutants(&source_tree, &options, &console)?;
+        // TODO: Perhaps print a text summary of how many were tested and whether they were all
+        // caught?
         exit(lab_outcome.exit_code());
     }
     Ok(())
diff --git a/src/mutate.rs b/src/mutate.rs
index 82d641a..3dd5cab 100644
--- a/src/mutate.rs
+++ b/src/mutate.rs
@@ -303,6 +303,7 @@ fn factorial(n: u32) -> u32 {
 
 #[test]
 fn test_factorial() {
+    println!("factorial({}) = {}", 6, factorial(6)); // This line is here so we can see it in --nocapture
     assert_eq!(factorial(6), 720);
 }
 "#
@@ -325,6 +326,7 @@ Default::default() /* ~ changed by cargo-mutants ~ */
 
 #[test]
 fn test_factorial() {
+    println!("factorial({}) = {}", 6, factorial(6)); // This line is here so we can see it in --nocapture
     assert_eq!(factorial(6), 720);
 }
 "#
diff --git a/src/options.rs b/src/options.rs
index ce52b7c..effef11 100644
--- a/src/options.rs
+++ b/src/options.rs
@@ -26,6 +26,9 @@ pub struct Options {
     ///
     /// (Mostly for development so that we don't always exercise the first few mutants.)
     pub shuffle: bool,
+
+    /// Additional arguments to `cargo test`.
+    pub additional_cargo_test_args: Vec<String>,
 }
 
 impl Options {
@@ -58,6 +61,7 @@ impl From<&Args> for Options {
                 .timeout
                 .map(Duration::from_secs_f64)
                 .unwrap_or(Duration::MAX),
+            additional_cargo_test_args: args.cargo_test_args.clone(),
         }
     }
 }


Test patch:
diff --git a/testdata/tree/already_failing_doctests/.gitignore b/testdata/tree/already_failing_doctests/.gitignore
new file mode 100644
index 0000000..2c96eb1
--- /dev/null
+++ b/testdata/tree/already_failing_doctests/.gitignore
@@ -0,0 +1,2 @@
+target/
+Cargo.lock
diff --git a/testdata/tree/already_failing_doctests/Cargo.toml b/testdata/tree/already_failing_doctests/Cargo.toml
new file mode 100644
index 0000000..9176482
--- /dev/null
+++ b/testdata/tree/already_failing_doctests/Cargo.toml
@@ -0,0 +1,12 @@
+[package]
+name = "mutants-testdata-already-failing-doctests"
+edition = "2018"
+version = "0.0.0"
+authors = ["Martin Pool"]
+publish = false
+
+[lib]
+doctest = true # They exist, but we don't want cargo mutants to run them.
+
+[workspace]
+# Don't include this in the overall cargo-mutants workspace, because it fails to build.
diff --git a/testdata/tree/already_failing_doctests/src/lib.rs b/testdata/tree/already_failing_doctests/src/lib.rs
new file mode 100644
index 0000000..795f327
--- /dev/null
+++ b/testdata/tree/already_failing_doctests/src/lib.rs
@@ -0,0 +1,18 @@
+/// Here is a function. It has a doctest, but the doctest does not even build.
+///
+/// cargo-mutants tests use this to check that doctests can be skipped.
+///
+/// ```
+/// # use mutants_testdata_already_failing_doctests::takes_one_arg;
+/// takes_one_arg(123,123,123);
+/// ```
+pub fn takes_one_arg(a: usize) -> usize {
+    a + 1
+}
+
+mod test {
+    #[test]
+    fn takes_one_arg() {
+        assert_eq!(super::takes_one_arg(1), 2);
+    }
+}
diff --git a/testdata/tree/factorial/src/bin/main.rs b/testdata/tree/factorial/src/bin/main.rs
index fcb54f5..e1cc26e 100644
--- a/testdata/tree/factorial/src/bin/main.rs
+++ b/testdata/tree/factorial/src/bin/main.rs
@@ -14,5 +14,6 @@ fn factorial(n: u32) -> u32 {
 
 #[test]
 fn test_factorial() {
+    println!("factorial({}) = {}", 6, factorial(6)); // This line is here so we can see it in --nocapture
     assert_eq!(factorial(6), 720);
 }
diff --git a/tests/cli.rs b/tests/cli.rs
index bc2652d..2f90c9f 100644
--- a/tests/cli.rs
+++ b/tests/cli.rs
@@ -316,6 +316,23 @@ r"src/bin/main\.rs:7: replace factorial -> u32 with Default::default\(\) \.\.\.
         ).unwrap());
 }
 
+#[test]
+fn factorial_mutants_with_all_logs_and_nocapture() {
+    let tmp_src_dir = copy_of_testdata("factorial");
+    run_assert_cmd()
+        .arg("mutants")
+        .args(["--all-logs", "-v", "-V"])
+        .arg("-d")
+        .arg(&tmp_src_dir.path())
+        .args(["--", "--", "--nocapture"])
+        .assert()
+        .code(2)
+        .stderr("")
+        .stdout(contains("factorial(6) = 720")) // println from the test
+        .stdout(contains("factorial(6) = 0")) // The mutated result
+        ;
+}
+
 #[test]
 fn check_succeeds_in_tree_that_builds_but_fails_tests() {
     // --check doesn't actually run the tests so won't discover that they fail.
@@ -378,6 +395,37 @@ fn already_failing_tests_are_detected_before_running_mutants() {
         .stdout(predicate::str::contains("test result: FAILED. 0 passed; 1 failed;").normalize());
 }
 
+#[test]
+fn already_failing_doctests_are_detected() {
+    let tmp_src_dir = copy_of_testdata("already_failing_doctests");
+    run_assert_cmd()
+        .arg("mutants")
+        .current_dir(&tmp_src_dir.path())
+        .env_remove("RUST_BACKTRACE")
+        .assert()
+        .code(4) // CLEAN_TESTS_FAILED
+        .stdout(contains("lib.rs - takes_one_arg (line 5) ... FAILED"))
+        .stdout(contains(
+            "this function takes 1 argument but 3 arguments were supplied",
+        ))
+        .stdout(predicate::str::contains(
+            "cargo test failed in an unmutated tree, so no mutants were tested",
+        ));
+}
+
+#[test]
+fn already_failing_doctests_can_be_skipped_with_cargo_arg() {
+    let tmp_src_dir = copy_of_testdata("already_failing_doctests");
+    run_assert_cmd()
+        .arg("mutants")
+        .args(["--", "--all-targets"])
+        .current_dir(&tmp_src_dir.path())
+        .env_remove("RUST_BACKTRACE")
+        .assert()
+        .code(0)
+        .stdout(contains("found 1 mutation to test"));
+}
+
 #[test]
 fn source_tree_build_fails() {
     let tmp_src_dir = copy_of_testdata("build_fails");
diff --git a/tests/snapshots/cli__list_mutants_with_diffs_in_factorial.snap b/tests/snapshots/cli__list_mutants_with_diffs_in_factorial.snap
index 64b0095..56f20b8 100644
--- a/tests/snapshots/cli__list_mutants_with_diffs_in_factorial.snap
+++ b/tests/snapshots/cli__list_mutants_with_diffs_in_factorial.snap
@@ -1,5 +1,6 @@
 ---
 source: tests/cli.rs
+assertion_line: 41
 expression: "String::from_utf8_lossy(&output.stdout)"
 
 ---
@@ -24,7 +25,7 @@ src/bin/main.rs:1: replace main with ()
 src/bin/main.rs:7: replace factorial -> u32 with Default::default()
 --- src/bin/main.rs
 +++ replace factorial with Default::default()
-@@ -1,18 +1,14 @@
+@@ -1,19 +1,15 @@
  fn main() {
      for i in 1..=6 {
          println!("{}! = {}", i, factorial(i));
@@ -42,6 +43,7 @@ src/bin/main.rs:7: replace factorial -> u32 with Default::default()
  
  #[test]
  fn test_factorial() {
+     println!("factorial({}) = {}", 6, factorial(6)); // This line is here so we can see it in --nocapture
      assert_eq!(factorial(6), 720);
  }