Repository: taiki-e/cargo-hack (language: rust)

Issue:
Option to clean between each run
I'm attempting to use `cargo hack test --feature-powerset` on a project, but given the sheer number of combinations this results in exceeding the available disk space on Github Actions with all the resulting test binaries.

If it were possible to run `cargo clean -p <project>` between each test run, that would delete the test binaries but leave the precompiled dependencies behind. It's probably not wanted as the default since it would slow down rerunning `cargo hack --feature-powerset` locally when you are able to cache all the results, but being an option would be very useful for CI.


Discussion / hints:
(Though TBH I think given the 512 feature combinations in this project I won't be able to use `test --feature-powerset` in CI anyway because of the time it takes, but maybe this would be useful for other projects that are closer to being able to run the full set except for disk space issues).
It may not always be enough in a project where features fine control over dependencies' features, but I think this option is useful to some extent for this use case.

Also, I think this might help as a workaround for https://github.com/rust-lang/rust-clippy/issues/4612. (Probably makes sense to run `cargo clean -p` "before each run" rather than "between each run".)

<br>

> (Though TBH I think given the 512 feature combinations in this project I won't be able to use `test --feature-powerset` in CI anyway because of the time it takes, but maybe this would be useful for other projects that are closer to being able to run the full set except for disk space issues).

AFAIK, if features are additive, then the test only needs to run with `--all-features` once.

Also, If check [if the test compile under all `--feature` combinations](https://github.com/rust-lang/futures-rs/pull/2101), it doesn't need to actually run the test and  I think `check --all-targets` or `test --no-run` is fine. (This probably needs to be separated from checking of the library/binary itself, as dev-dependencies may enable features.)

If features are non-additive, ideally it is preferable to run the test for each feature, but `test --feature-powerset` takes too long, so there is probably no way yet to do this easily. (However, I don't think such features are good for libraries in many cases, as other dependencies may enable it.)
Making `--skip` works for `--all-features` is probably one of the reasonable solutions for this. If one feature is non-additive and other features are additive (e.g., [using different implementations between std and no-std](https://github.com/serde-rs/serde/issues/918)), I think this is enough.


> This probably needs to be separated from checking of the library/binary itself, as dev-dependencies may enable features.

Ah, that's a good point. I had reduced my usage to running `check --all-targets` for the feature-sets since that _only_ takes 18 minutes, and only running the tests once with `--all-features`, but didn't think about having to still have a separate `check --all-targets --no-dev-deps`.

Fix patch:
diff --git a/src/cli.rs b/src/cli.rs
index 6e766566..c5a9642f 100644
--- a/src/cli.rs
+++ b/src/cli.rs
@@ -54,6 +54,10 @@ const HELP: &[(&str, &str, &str, &[&str])] = &[
         "Skip passing --features flag to `cargo` if that feature does not exist in the package",
         &[],
     ),
+    ("", "--clean-per-run", "Remove artifacts for that package before running the command", &[
+        "If used this flag with --workspace, --each-feature, or --feature-powerset, artifacts will be removed before each run.",
+        "Note that dependencies artifacts will be preserved.",
+    ]),
     ("-v", "--verbose", "Use verbose output", &["This flag will be propagated to cargo."]),
     ("", "--color <WHEN>", "Coloring: auto, always, never", &[
         "This flag will be propagated to cargo.",
@@ -107,14 +111,13 @@ impl fmt::Display for Help {
         writeln!(
             f,
             "\
-{0} {1}\n{2}\n\n{3}
+{0} {1}\n\n{2}
 USAGE:
     cargo hack [OPTIONS] [SUBCOMMAND]\n
 Use -h for short descriptions and --help for more details.\n
 OPTIONS:",
             env!("CARGO_PKG_NAME"),
             env!("CARGO_PKG_VERSION"),
-            env!("CARGO_PKG_AUTHORS"),
             env!("CARGO_PKG_DESCRIPTION")
         )?;
 
@@ -145,8 +148,7 @@ Some common cargo commands are (see all commands with --list):
         build       Compile the current package
         check       Analyze the current package and report errors, but don't build object files
         run         Run a binary or example of the local package
-        test        Run the tests
-"
+        test        Run the tests"
         )
     }
 }
@@ -184,6 +186,8 @@ pub(crate) struct Args {
     pub(crate) optional_deps: bool,
     /// --skip-no-default-features
     pub(crate) skip_no_default_features: bool,
+    /// --clean-per-run
+    pub(crate) clean_per_run: bool,
 
     // flags that will be propagated to cargo
     /// --features <FEATURES>...
@@ -264,6 +268,7 @@ pub(crate) fn args(coloring: &mut Option<Coloring>) -> Result<Option<Args>> {
     let mut ignore_non_exist_features = false;
     let mut optional_deps = false;
     let mut skip_no_default_features = false;
+    let mut clean_per_run = false;
 
     let res = (|| -> Result<()> {
         while let Some(arg) = args.next() {
@@ -284,31 +289,31 @@ pub(crate) fn args(coloring: &mut Option<Coloring>) -> Result<Option<Args>> {
                 continue;
             }
 
-            macro_rules! parse_arg1 {
-                ($ident:ident, $propagate:expr, $pat:expr, $help:expr) => {
+            macro_rules! parse_opt {
+                ($opt:ident, $propagate:expr, $pat:expr, $help:expr) => {
                     if arg == $pat {
-                        if $ident.is_some() {
+                        if $opt.is_some() {
                             return Err(multi_arg($help, subcommand.as_ref()));
                         }
                         let next =
                             args.next().ok_or_else(|| req_arg($help, subcommand.as_ref()))?;
                         if $propagate {
-                            $ident = Some(next.clone());
+                            $opt = Some(next.clone());
                             leading.push(arg);
                             leading.push(next);
                         } else {
-                            $ident = Some(next);
+                            $opt = Some(next);
                         }
                         continue;
                     } else if arg.starts_with(concat!($pat, "=")) {
-                        if $ident.is_some() {
+                        if $opt.is_some() {
                             return Err(multi_arg($help, subcommand.as_ref()));
                         }
                         let next = arg
                             .splitn(2, '=')
                             .nth(1)
                             .ok_or_else(|| req_arg($help, subcommand.as_ref()))?;
-                        $ident = Some(next.to_string());
+                        $opt = Some(next.to_string());
                         if $propagate {
                             leading.push(arg);
                         }
@@ -317,18 +322,18 @@ pub(crate) fn args(coloring: &mut Option<Coloring>) -> Result<Option<Args>> {
                 };
             }
 
-            macro_rules! parse_arg2 {
-                ($ident:ident, $allow_split:expr, $pat:expr, $help:expr) => {
+            macro_rules! parse_multi_opt {
+                ($v:ident, $allow_split:expr, $pat:expr, $help:expr) => {
                     if arg == $pat {
                         let arg = args.next().ok_or_else(|| req_arg($help, subcommand.as_ref()))?;
                         if $allow_split {
                             if arg.contains(',') {
-                                $ident.extend(arg.split(',').map(ToString::to_string));
+                                $v.extend(arg.split(',').map(ToString::to_string));
                             } else {
-                                $ident.extend(arg.split(' ').map(ToString::to_string));
+                                $v.extend(arg.split(' ').map(ToString::to_string));
                             }
                         } else {
-                            $ident.push(arg);
+                            $v.push(arg);
                         }
                         continue;
                     } else if arg.starts_with(concat!($pat, "=")) {
@@ -343,26 +348,34 @@ pub(crate) fn args(coloring: &mut Option<Coloring>) -> Result<Option<Args>> {
                                 arg = &arg[1..arg.len() - 1];
                             }
                             if arg.contains(',') {
-                                $ident.extend(arg.split(',').map(ToString::to_string));
+                                $v.extend(arg.split(',').map(ToString::to_string));
                             } else {
-                                $ident.extend(arg.split(' ').map(ToString::to_string));
+                                $v.extend(arg.split(' ').map(ToString::to_string));
                             }
                         } else {
-                            $ident.push(arg.to_string());
+                            $v.push(arg.to_string());
                         }
                         continue;
                     }
                 };
             }
 
-            parse_arg1!(manifest_path, false, "--manifest-path", "--manifest-path <PATH>");
-            parse_arg1!(color, true, "--color", "--color <WHEN>");
+            macro_rules! parse_flag {
+                ($flag:ident) => {
+                    if mem::replace(&mut $flag, true) {
+                        return Err(multi_arg(&arg, subcommand.as_ref()));
+                    }
+                };
+            }
+
+            parse_opt!(manifest_path, false, "--manifest-path", "--manifest-path <PATH>");
+            parse_opt!(color, true, "--color", "--color <WHEN>");
 
-            parse_arg2!(package, false, "--package", "--package <SPEC>");
-            parse_arg2!(package, false, "-p", "--package <SPEC>");
-            parse_arg2!(exclude, false, "--exclude", "--exclude <SPEC>");
-            parse_arg2!(features, true, "--features", "--features <FEATURES>");
-            parse_arg2!(skip, true, "--skip", "--skip <FEATURES>");
+            parse_multi_opt!(package, false, "--package", "--package <SPEC>");
+            parse_multi_opt!(package, false, "-p", "--package <SPEC>");
+            parse_multi_opt!(exclude, false, "--exclude", "--exclude <SPEC>");
+            parse_multi_opt!(features, true, "--features", "--features <FEATURES>");
+            parse_multi_opt!(skip, true, "--skip", "--skip <FEATURES>");
 
             match arg.as_str() {
                 "--workspace" | "--all" => {
@@ -370,41 +383,14 @@ pub(crate) fn args(coloring: &mut Option<Coloring>) -> Result<Option<Args>> {
                         return Err(multi_arg(&arg, subcommand.as_ref()));
                     }
                 }
-                "--no-dev-deps" => {
-                    if mem::replace(&mut no_dev_deps, true) {
-                        return Err(multi_arg(&arg, subcommand.as_ref()));
-                    }
-                }
-                "--remove-dev-deps" => {
-                    if mem::replace(&mut remove_dev_deps, true) {
-                        return Err(multi_arg(&arg, subcommand.as_ref()));
-                    }
-                }
-                "--each-feature" => {
-                    if mem::replace(&mut each_feature, true) {
-                        return Err(multi_arg(&arg, subcommand.as_ref()));
-                    }
-                }
-                "--feature-powerset" => {
-                    if mem::replace(&mut feature_powerset, true) {
-                        return Err(multi_arg(&arg, subcommand.as_ref()));
-                    }
-                }
-                "--ignore-private" => {
-                    if mem::replace(&mut ignore_private, true) {
-                        return Err(multi_arg(&arg, subcommand.as_ref()));
-                    }
-                }
-                "--optional-deps" => {
-                    if mem::replace(&mut optional_deps, true) {
-                        return Err(multi_arg(&arg, subcommand.as_ref()));
-                    }
-                }
-                "--skip-no-default-features" => {
-                    if mem::replace(&mut skip_no_default_features, true) {
-                        return Err(multi_arg(&arg, subcommand.as_ref()));
-                    }
-                }
+                "--no-dev-deps" => parse_flag!(no_dev_deps),
+                "--remove-dev-deps" => parse_flag!(remove_dev_deps),
+                "--each-feature" => parse_flag!(each_feature),
+                "--feature-powerset" => parse_flag!(feature_powerset),
+                "--ignore-private" => parse_flag!(ignore_private),
+                "--optional-deps" => parse_flag!(optional_deps),
+                "--skip-no-default-features" => parse_flag!(skip_no_default_features),
+                "--clean-per-run" => parse_flag!(clean_per_run),
                 "--ignore-unknown-features" => {
                     if ignore_unknown_features || ignore_non_exist_features {
                         return Err(multi_arg(&arg, subcommand.as_ref()));
@@ -426,6 +412,7 @@ pub(crate) fn args(coloring: &mut Option<Coloring>) -> Result<Option<Args>> {
 
     let color = color.map(|c| c.parse()).transpose()?;
     *coloring = color;
+    let verbose = leading.iter().any(|a| a == "--verbose" || a == "-v" || a == "-vv");
 
     res?;
 
@@ -489,7 +476,7 @@ pub(crate) fn args(coloring: &mut Option<Coloring>) -> Result<Option<Args>> {
 
     if subcommand.is_none() {
         if leading.iter().any(|a| a == "--list") {
-            let mut line = ProcessBuilder::new(crate::cargo_binary());
+            let mut line = ProcessBuilder::new(crate::cargo_binary(), verbose);
             line.arg("--list");
             line.exec()?;
             return Ok(None);
@@ -508,7 +495,6 @@ For more information try --help
         }
     }
 
-    let verbose = leading.iter().any(|a| a == "--verbose" || a == "-v" || a == "-vv");
     if ignore_non_exist_features {
         warn!(
             color,
@@ -542,6 +528,7 @@ For more information try --help
         ignore_unknown_features: ignore_unknown_features || ignore_non_exist_features,
         optional_deps,
         skip_no_default_features,
+        clean_per_run,
 
         features,
         color,
@@ -586,3 +573,24 @@ For more information try --help
         }
     )
 }
+
+#[cfg(test)]
+mod tests {
+    use super::Help;
+
+    #[test]
+    fn long_help() {
+        assert_eq!(
+            Help { long: true, term_size: 200 }.to_string(),
+            include_str!("../tests/fixtures/long-help.txt")
+        );
+    }
+
+    #[test]
+    fn short_help() {
+        assert_eq!(
+            Help { long: false, term_size: 200 }.to_string(),
+            include_str!("../tests/fixtures/short-help.txt")
+        );
+    }
+}
diff --git a/src/package.rs b/src/package.rs
index 563de9d2..d335dbdf 100644
--- a/src/package.rs
+++ b/src/package.rs
@@ -1,4 +1,4 @@
-use std::{fmt::Write, ops::Deref};
+use std::{ffi::OsStr, fmt::Write, ops::Deref};
 
 use crate::{metadata, Args, Info, Manifest, ProcessBuilder, Result};
 
@@ -189,6 +189,10 @@ fn exec_cargo(
 ) -> Result<()> {
     info.count += 1;
 
+    if args.clean_per_run {
+        cargo_clean(line.get_program(), args, package)?;
+    }
+
     // running `<command>` on <package> (<count>/<total>)
     let mut msg = String::new();
     write!(msg, "running {} on {}", line, &package.name).unwrap();
@@ -200,6 +204,20 @@ fn exec_cargo(
     line.exec()
 }
 
+fn cargo_clean(cargo: &OsStr, args: &Args, package: &Package<'_>) -> Result<()> {
+    let mut line = ProcessBuilder::new(cargo, args.verbose);
+    line.arg("clean");
+    line.arg("--package");
+    line.arg(&package.name);
+
+    if args.verbose {
+        // running `cargo clean --package <package>`
+        info!(args.color, "running {}", line);
+    }
+
+    line.exec()
+}
+
 fn powerset<T: Clone>(iter: impl IntoIterator<Item = T>) -> Vec<Vec<T>> {
     iter.into_iter().fold(vec![vec![]], |mut acc, elem| {
         let ext = acc.clone().into_iter().map(|mut curr| {
diff --git a/src/process.rs b/src/process.rs
index 8849b9f3..70311215 100644
--- a/src/process.rs
+++ b/src/process.rs
@@ -33,21 +33,21 @@ pub(crate) struct ProcessBuilder {
 
 impl ProcessBuilder {
     /// Creates a new `ProcessBuilder`.
-    pub(crate) fn new(program: OsString) -> Self {
+    pub(crate) fn new(program: impl Into<Rc<OsStr>>, verbose: bool) -> Self {
         Self {
-            program: Rc::from(program),
+            program: program.into(),
             leading_args: Rc::from(&[][..]),
             trailing_args: Rc::from(&[][..]),
             args: Vec::new(),
             features: String::new(),
-            verbose: false,
+            verbose,
         }
     }
 
     /// Creates a new `ProcessBuilder` from `Args`.
-    pub(crate) fn from_args(program: OsString, args: &Args) -> Self {
+    pub(crate) fn from_args(program: impl Into<Rc<OsStr>>, args: &Args) -> Self {
         Self {
-            program: Rc::from(program),
+            program: program.into(),
             leading_args: args.leading_args.clone(),
             trailing_args: args.trailing_args.clone(),
             args: Vec::new(),
@@ -107,6 +107,11 @@ impl ProcessBuilder {
     //     self
     // }
 
+    /// Gets the executable name.
+    pub(crate) fn get_program(&self) -> &OsStr {
+        &self.program
+    }
+
     /// Runs the process, waiting for completion, and mapping non-success exit codes to an error.
     pub(crate) fn exec(&self) -> Result<()> {
         let mut command = self.build_command();


Test patch:
diff --git a/tests/fixtures/long-help.txt b/tests/fixtures/long-help.txt
new file mode 100644
index 00000000..69089167
--- /dev/null
+++ b/tests/fixtures/long-help.txt
@@ -0,0 +1,93 @@
+cargo-hack 0.3.10
+
+A tool to work around some limitations on cargo.
+
+USAGE:
+    cargo hack [OPTIONS] [SUBCOMMAND]
+
+Use -h for short descriptions and --help for more details.
+
+OPTIONS:
+    -p, --package <SPEC>...
+            Package(s) to check.
+
+        --all
+            Alias for --workspace.
+
+        --workspace
+            Perform command for all packages in the workspace.
+
+        --exclude <SPEC>...
+            Exclude packages from the check.
+
+        --manifest-path <PATH>
+            Path to Cargo.toml.
+
+        --features <FEATURES>
+            Space-separated list of features to activate.
+
+        --each-feature
+            Perform for each feature which includes --no-default-features flag and default features of the package.
+
+        --feature-powerset
+            Perform for the feature powerset which includes --no-default-features flag and default features of the package.
+
+        --optional-deps
+            Use optional dependencies as features.
+
+            This flag can only be used with either --each-feature flag or --feature-powerset flag.
+
+        --skip <FEATURES>
+            Space-separated list of features to skip.
+
+            To skip run of default feature, using value `--skip default`.
+
+            This flag can only be used with either --each-feature flag or --feature-powerset flag.
+
+        --skip-no-default-features
+            Skip run of just --no-default-features flag.
+
+            This flag can only be used with either --each-feature flag or --feature-powerset flag.
+
+        --no-dev-deps
+            Perform without dev-dependencies.
+
+            This flag removes dev-dependencies from real `Cargo.toml` while cargo-hack is running and restores it when finished.
+
+        --remove-dev-deps
+            Equivalent to --no-dev-deps flag except for does not restore the original `Cargo.toml` after performed.
+
+        --ignore-private
+            Skip to perform on `publish = false` packages.
+
+        --ignore-unknown-features
+            Skip passing --features flag to `cargo` if that feature does not exist in the package.
+
+        --clean-per-run
+            Remove artifacts for that package before running the command.
+
+            If used this flag with --workspace, --each-feature, or --feature-powerset, artifacts will be removed before each run.
+
+            Note that dependencies artifacts will be preserved.
+
+    -v, --verbose
+            Use verbose output.
+
+            This flag will be propagated to cargo.
+
+        --color <WHEN>
+            Coloring: auto, always, never.
+
+            This flag will be propagated to cargo.
+
+    -h, --help
+            Prints help information.
+
+    -V, --version
+            Prints version information.
+
+Some common cargo commands are (see all commands with --list):
+        build       Compile the current package
+        check       Analyze the current package and report errors, but don't build object files
+        run         Run a binary or example of the local package
+        test        Run the tests
diff --git a/tests/fixtures/short-help.txt b/tests/fixtures/short-help.txt
new file mode 100644
index 00000000..e58783d2
--- /dev/null
+++ b/tests/fixtures/short-help.txt
@@ -0,0 +1,36 @@
+cargo-hack 0.3.10
+
+A tool to work around some limitations on cargo.
+
+USAGE:
+    cargo hack [OPTIONS] [SUBCOMMAND]
+
+Use -h for short descriptions and --help for more details.
+
+OPTIONS:
+    -p, --package <SPEC>...       Package(s) to check
+        --all                     Alias for --workspace
+        --workspace               Perform command for all packages in the workspace
+        --exclude <SPEC>...       Exclude packages from the check
+        --manifest-path <PATH>    Path to Cargo.toml
+        --features <FEATURES>     Space-separated list of features to activate
+        --each-feature            Perform for each feature which includes --no-default-features flag and default features of the package
+        --feature-powerset        Perform for the feature powerset which includes --no-default-features flag and default features of the package
+        --optional-deps           Use optional dependencies as features
+        --skip <FEATURES>         Space-separated list of features to skip
+        --skip-no-default-features Skip run of just --no-default-features flag
+        --no-dev-deps             Perform without dev-dependencies
+        --remove-dev-deps         Equivalent to --no-dev-deps flag except for does not restore the original `Cargo.toml` after performed
+        --ignore-private          Skip to perform on `publish = false` packages
+        --ignore-unknown-features Skip passing --features flag to `cargo` if that feature does not exist in the package
+        --clean-per-run           Remove artifacts for that package before running the command
+    -v, --verbose                 Use verbose output
+        --color <WHEN>            Coloring: auto, always, never
+    -h, --help                    Prints help information
+    -V, --version                 Prints version information
+
+Some common cargo commands are (see all commands with --list):
+        build       Compile the current package
+        check       Analyze the current package and report errors, but don't build object files
+        run         Run a binary or example of the local package
+        test        Run the tests