Repository: sharkdp/hyperfine (language: rust)

Issue:
`--output` overwrites previous commands and runs
Currently I'm working on a set of benchmarks where I want to save the output of (at least) one run from each. `--output` should arguably be allowed N times, where N is the number of commands being benchmarked. 

However, I would also argue that keeping each runs output is extremely useful, specifically for debugging/figuring out why statistical outliers exist.

As an arbitrary example:
```
$ hyperfine --output ./example.log "ls" "pwd"
```
will only show output from a single `pwd`. Passing multiple outputs does not work, as hyperfine only expects one.

As a proposal for how this could change there are two things that would be nice to see:
1. Allow multiple outputs, the same way `--prepare` is allowed per command
   ```
   $ hyperfine --output ./ls.log "ls" --output ./pwd.log "pwd"
   ```
2. Allow some parameter, like N, for the run number
   ```
   $ hyperfine --output "./ls-{N}.log" "ls"
   ```

For short running commands like `ls` this is a problem as it may run somewhere in the neighborhood of 5k times, so it may be a problem to create a new logfile for each run. Longer running commands obviously do not have this problem.


Discussion / hints:
Thank you for your feedback!

The `--output` option was mainly introduced to control where output is being redirected (terminal, /dev/null, a pipe, a file). We didn't really think about your use case, to be honest (FYI @tavianator).

Note that it's rather easy to get the desired functionality if you simply redirect the output yourself:
```bash
hyperfine "ls > ls.log" "pwd > pwd.log"
```

As for the per-iteration output, you could use `>>` to get the output of each and every benchmark run. You could use `--setup` to make sure that the file is empty before starting the benchmark. And you could use `--prepare` to print a separator like `---` before each new output:
```bash
hyperfine \
  --setup "rm -f ls.log" \
  --prepare "echo --- >> ls.log" \
  "ls >> ls.log"
```

If you want a new logfile for every run, I'm sure there would be workarounds as well, but I guess they won't be as concise as the commands suggested above.

As for your feature requests:



> 1. Allow multiple outputs, the same way `--prepare` is allowed per command

I can see how there *might* be cases where one would like to use `--output=pipe` for the first command and `--output=null` for the second, but I guess that's rather the exception. Given that this can be worked around using normal shell redirection, I think I would vote for allowing just one `--output` command.



> 2\. Allow some parameter, like N, for the run number

This *might* be helpful in general, so we could think about setting e.g. `{iteration}` or `{#}` by default (similar to how other variables can be set via `--parameter-scan`/`--parameter-list`. This way, you could do something like
```bash
hyperfine "ls > ls-{#}.log"
```
Another option would be to inject an environment variable, similar to how we already set `HYPERFINE_RANDOMIZED_ENVIRONMENT_OFFSET`. This could be used in such a way:
```bash
hyperfine "ls > ls-${HYPERFINE_ITERATION}.log"
```
> Note that it's rather easy to get the desired functionality if you simply redirect the output yourself:

Indeed this is actually what I have been doing so far, the only issue lies in that multiple iterations are either overwritten or get appended, which can be pretty annoying to parse out with very long outputs even if you mark them.

An injected `HYPERFINE_ITERATION` environment variable would be, in my opinion, the best solution.

I am fine with the idea that `--output` is only allowed once, but I would argue that presenting it as a way to write output to a file becomes rather misleading as it does not do what I expect wrt multiple commands. It's surprising to me that if I redirect the output to a file for later viewing I only see the output of the very last command. I'd expect the output to be appended rather than overwritten. Not to say that it's useless though, it does help in debugging a failed run as hyperfine bails. Perhaps the documentation needs to be clarified in that instance?

I appreciate the consideration, and thank you for this awesome tool
> An injected `HYPERFINE_ITERATION` environment variable would be, in my opinion, the best solution.

Ok. I believe that should be easy to implement.



> I am fine with the idea that `--output` is only allowed once, but I would argue that presenting it as a way to write output to a file becomes rather misleading as it does not do what I expect wrt multiple commands. It's surprising to me that if I redirect the output to a file for later viewing I only see the output of the very last command.

This is of course a very valid point. Sorry that I discarded that point above.

> I'd expect the output to be appended rather than overwritten.

Hm. That's also not the best user experience, right? I think we should go with your "multiple `--output` options allowed" idea after all :+1: 

Another option which might be easier to implement but with an arguably worse UX would be to show a (nice) error message in case `--output=path/to/file` is used with multiple benchmarks, suggesting to use shell redirection instead.
I think we should keep a couple things in mind:

- Overwriting a file is likely to be a more consistent filesystem workload than appending to a file.  I wouldn't want to append by default.
- If we allow `--output` multiple times, what about `--output=null find --output=./file fd`?  That's not a fair benchmark.

Are there compelling use cases other than debugging?  Maybe a `--save-all-outputs` option would be nice that would generate something like

```
+-- command_01_find/
|   +-- 0001.stdout
|   +-- 0001.stderr
|   +-- 0002.stdout
|   +-- ...
+-- command_02_fd/
|   +-- 0001.stdout
|   +-- ...
```
> * Overwriting a file is likely to be a more consistent filesystem workload than appending to a file.  I wouldn't want to append by default.

Is the overwriting Vs. appending such a large difference that it matters over a more reasonable/expected UX? How is overwriting output every time useful? From my perspective the only reason it has any use at all at the moment is that a failed test run will kill hyperfine and leave the failed runs log behind.

Logging the last command of a series of runs as it currently does is practically useless for anything other than a run failure. If that's the only usecase of writing output to file it should be renamed to `--record-errors` and spit out a file per nonzero exit code (assuming ignore errors was on)  Certainly redirecting/piping output is useful, but overwriting file contents for the sake of consistency?

If possible, using a not-insignificant write buffer and flushing between runs would be better for "consistency" than anything if that were a big deal to someone.

> * If we allow `--output` multiple times, what about `--output=null find --output=./file fd`?  That's not a fair benchmark.

Personally I'm apathetic to such an argument. I think that's user error. It might be detectable/something you can warn against but similar results can already be achieved by `"find >> /dev/null" "fd >> ./file"` That's not a fair benchmark either, but that's the test that's being run. Edit: Actually that'd be a good argument *for* handling it with hyperfine, so you can warn about this kind of thing.

> Are there compelling use cases other than debugging?

My use case is being able to try and examine statistical anomalies in the debug output. Being able to go back and see that  a particular run for one reason or another bailed out of a hotpath, or retried is useful to me even beyond "debugging" what went wrong. Knowing why a set of tests took significantly longer might not even be "debugging" but normal operation of said program.

Another potential usecase is determining if a particular run is successful even if the exit code is 0. I know of a few obtuse applications that can fail but not report that as an exit code. Right now I don't know of a way that can be determined with hyperfine other than writing output.
> Is the overwriting Vs. appending such a large difference that it matters over a more reasonable/expected UX?

Maybe not.  I just did some local benchmarking and `>>` was typically 2-3% slower than `>`, but the variance didn't seem any different.  I was mainly worried about the variance.

> How is overwriting output every time useful?

For me, the purpose of `--output=./file` is “I want to know how fast my program is when outputting to a file,” not “I want to look at the output.”

> >  That's not a fair benchmark
>
> Personally I'm apathetic to such an argument

Me too, actually I've kind of changed my mind on this one because
```console
$ hyperfine --output=null fd --output=./file fd
```
*is* an interesting benchmark.

> > Are there compelling use cases other than debugging?
>
> My use case is being able to try and examine statistical anomalies in the debug output.

This makes a lot of sense to me actually.  I usually use `hyperfine` for short-running programs that generate lots of (deterministic) output, and I want to remove the I/O overhead as much as possible.  But if you're doing things where I/O isn't a bottleneck, preserving logs seems very helpful.

---

Anyway, here's what I think now.  Supporting `--output` multiple times scares me because the command line becomes order sensitive.  What should the behaviour of these commands be?

```console
$ hyperfine --output=./file find fd  # Both to ./file right now
$ hyperfine find --output=./file fd  # Does --output apply to fd or both?
$ hyperfine find fd --output=./file  # Does --output apply to anything?
```

I agree we should clarify in the documentation that `--output=./file` is designed for timing stdout-is-a-file execution paths, not examining the output.

For examining the output, I think we should recommend using shell redirections, and provide `$HYPERFINE_ITERATION` to help with that.

I also think something like `--save-all-outputs` is a good idea.  It seems to apply to your non-debugging use cases as well.  We should probably add to it some metadata about each run, so you can find which ones were outliers etc.
>  Supporting --output multiple times scares me because the command line becomes order sensitive. What should the behaviour of these commands be?

I was simply going to assume the same semantics as `--prepare` to be honest. You have to have a single prepare or N `--prepare`s 

Fix patch:
diff --git a/doc/hyperfine.1 b/doc/hyperfine.1
index 73f8e20f4..00fa3521b 100644
--- a/doc/hyperfine.1
+++ b/doc/hyperfine.1
@@ -290,6 +290,11 @@ Don't redirect the output at all (same as \&'\-\-show\-output').
 .IP "<FILE>"
 Write the output to the given file.
 .RE
+.IP
+This option can be specified once for all commands or multiple times,
+once for each command. Note: If you want to log the output of each and
+every iteration, you can use a shell redirection and the $HYPERFINE_ITERATION
+environment variable: 'my-command > output-${HYPERFINE_ITERATION}.log'
 .HP
 \fB\-\-input\fR \fIWHERE\fP
 .IP
diff --git a/src/benchmark/executor.rs b/src/benchmark/executor.rs
index bf30be1f3..42af62231 100644
--- a/src/benchmark/executor.rs
+++ b/src/benchmark/executor.rs
@@ -39,6 +39,7 @@ pub trait Executor {
         command: &Command<'_>,
         iteration: BenchmarkIteration,
         command_failure_action: Option<CmdFailureAction>,
+        output_policy: &CommandOutputPolicy,
     ) -> Result<(TimingResult, ExitStatus)>;
 
     /// Perform a calibration of this executor. For example,
@@ -116,13 +117,14 @@ impl Executor for RawExecutor<'_> {
         command: &Command<'_>,
         iteration: BenchmarkIteration,
         command_failure_action: Option<CmdFailureAction>,
+        output_policy: &CommandOutputPolicy,
     ) -> Result<(TimingResult, ExitStatus)> {
         let result = run_command_and_measure_common(
             command.get_command()?,
             iteration,
             command_failure_action.unwrap_or(self.options.command_failure_action),
             &self.options.command_input_policy,
-            &self.options.command_output_policy,
+            output_policy,
             &command.get_command_line(),
         )?;
 
@@ -167,6 +169,7 @@ impl Executor for ShellExecutor<'_> {
         command: &Command<'_>,
         iteration: BenchmarkIteration,
         command_failure_action: Option<CmdFailureAction>,
+        output_policy: &CommandOutputPolicy,
     ) -> Result<(TimingResult, ExitStatus)> {
         let on_windows_cmd = cfg!(windows) && *self.shell == Shell::Default("cmd.exe");
         let mut command_builder = self.shell.command();
@@ -185,7 +188,7 @@ impl Executor for ShellExecutor<'_> {
             iteration,
             command_failure_action.unwrap_or(self.options.command_failure_action),
             &self.options.command_input_policy,
-            &self.options.command_output_policy,
+            output_policy,
             &command.get_command_line(),
         )?;
 
@@ -229,6 +232,7 @@ impl Executor for ShellExecutor<'_> {
                 &Command::new(None, ""),
                 BenchmarkIteration::NonBenchmarkRun,
                 None,
+                &CommandOutputPolicy::Null,
             );
 
             match res {
@@ -300,6 +304,7 @@ impl Executor for MockExecutor {
         command: &Command<'_>,
         _iteration: BenchmarkIteration,
         _command_failure_action: Option<CmdFailureAction>,
+        _output_policy: &CommandOutputPolicy,
     ) -> Result<(TimingResult, ExitStatus)> {
         #[cfg(unix)]
         let status = {
diff --git a/src/benchmark/mod.rs b/src/benchmark/mod.rs
index 1d41c64d7..2cca85736 100644
--- a/src/benchmark/mod.rs
+++ b/src/benchmark/mod.rs
@@ -8,7 +8,9 @@ use std::cmp;
 
 use crate::benchmark::executor::BenchmarkIteration;
 use crate::command::Command;
-use crate::options::{CmdFailureAction, ExecutorKind, Options, OutputStyleOption};
+use crate::options::{
+    CmdFailureAction, CommandOutputPolicy, ExecutorKind, Options, OutputStyleOption,
+};
 use crate::outlier_detection::{modified_zscores, OUTLIER_THRESHOLD};
 use crate::output::format::{format_duration, format_duration_unit};
 use crate::output::progress_bar::get_progress_bar;
@@ -56,12 +58,14 @@ impl<'a> Benchmark<'a> {
         &self,
         command: &Command<'_>,
         error_output: &'static str,
+        output_policy: &CommandOutputPolicy,
     ) -> Result<TimingResult> {
         self.executor
             .run_command_and_measure(
                 command,
                 executor::BenchmarkIteration::NonBenchmarkRun,
                 Some(CmdFailureAction::RaiseError),
+                output_policy,
             )
             .map(|r| r.0)
             .map_err(|_| anyhow!(error_output))
@@ -71,6 +75,7 @@ impl<'a> Benchmark<'a> {
     fn run_setup_command(
         &self,
         parameters: impl IntoIterator<Item = ParameterNameAndValue<'a>>,
+        output_policy: &CommandOutputPolicy,
     ) -> Result<TimingResult> {
         let command = self
             .options
@@ -82,7 +87,7 @@ impl<'a> Benchmark<'a> {
                             Append ' || true' to the command if you are sure that this can be ignored.";
 
         Ok(command
-            .map(|cmd| self.run_intermediate_command(&cmd, error_output))
+            .map(|cmd| self.run_intermediate_command(&cmd, error_output, output_policy))
             .transpose()?
             .unwrap_or_default())
     }
@@ -91,6 +96,7 @@ impl<'a> Benchmark<'a> {
     fn run_cleanup_command(
         &self,
         parameters: impl IntoIterator<Item = ParameterNameAndValue<'a>>,
+        output_policy: &CommandOutputPolicy,
     ) -> Result<TimingResult> {
         let command = self
             .options
@@ -102,25 +108,33 @@ impl<'a> Benchmark<'a> {
                             Append ' || true' to the command if you are sure that this can be ignored.";
 
         Ok(command
-            .map(|cmd| self.run_intermediate_command(&cmd, error_output))
+            .map(|cmd| self.run_intermediate_command(&cmd, error_output, output_policy))
             .transpose()?
             .unwrap_or_default())
     }
 
     /// Run the command specified by `--prepare`.
-    fn run_preparation_command(&self, command: &Command<'_>) -> Result<TimingResult> {
+    fn run_preparation_command(
+        &self,
+        command: &Command<'_>,
+        output_policy: &CommandOutputPolicy,
+    ) -> Result<TimingResult> {
         let error_output = "The preparation command terminated with a non-zero exit code. \
                             Append ' || true' to the command if you are sure that this can be ignored.";
 
-        self.run_intermediate_command(command, error_output)
+        self.run_intermediate_command(command, error_output, output_policy)
     }
 
     /// Run the command specified by `--conclude`.
-    fn run_conclusion_command(&self, command: &Command<'_>) -> Result<TimingResult> {
+    fn run_conclusion_command(
+        &self,
+        command: &Command<'_>,
+        output_policy: &CommandOutputPolicy,
+    ) -> Result<TimingResult> {
         let error_output = "The conclusion command terminated with a non-zero exit code. \
                             Append ' || true' to the command if you are sure that this can be ignored.";
 
-        self.run_intermediate_command(command, error_output)
+        self.run_intermediate_command(command, error_output, output_policy)
     }
 
     /// Run the benchmark for a single command
@@ -140,6 +154,8 @@ impl<'a> Benchmark<'a> {
         let mut exit_codes: Vec<Option<i32>> = vec![];
         let mut all_succeeded = true;
 
+        let output_policy = &self.options.command_output_policies[self.number];
+
         let preparation_command = self.options.preparation_command.as_ref().map(|values| {
             let preparation_command = if values.len() == 1 {
                 &values[0]
@@ -152,10 +168,11 @@ impl<'a> Benchmark<'a> {
                 self.command.get_parameters().iter().cloned(),
             )
         });
+
         let run_preparation_command = || {
             preparation_command
                 .as_ref()
-                .map(|cmd| self.run_preparation_command(cmd))
+                .map(|cmd| self.run_preparation_command(cmd, output_policy))
                 .transpose()
         };
 
@@ -174,11 +191,11 @@ impl<'a> Benchmark<'a> {
         let run_conclusion_command = || {
             conclusion_command
                 .as_ref()
-                .map(|cmd| self.run_conclusion_command(cmd))
+                .map(|cmd| self.run_conclusion_command(cmd, output_policy))
                 .transpose()
         };
 
-        self.run_setup_command(self.command.get_parameters().iter().cloned())?;
+        self.run_setup_command(self.command.get_parameters().iter().cloned(), output_policy)?;
 
         // Warmup phase
         if self.options.warmup_count > 0 {
@@ -198,6 +215,7 @@ impl<'a> Benchmark<'a> {
                     self.command,
                     BenchmarkIteration::Warmup(i),
                     None,
+                    output_policy,
                 )?;
                 let _ = run_conclusion_command()?;
                 if let Some(bar) = progress_bar.as_ref() {
@@ -229,6 +247,7 @@ impl<'a> Benchmark<'a> {
             self.command,
             BenchmarkIteration::Benchmark(0),
             None,
+            output_policy,
         )?;
         let success = status.success();
 
@@ -289,6 +308,7 @@ impl<'a> Benchmark<'a> {
                 self.command,
                 BenchmarkIteration::Benchmark(i + 1),
                 None,
+                output_policy,
             )?;
             let success = status.success();
 
@@ -418,7 +438,7 @@ impl<'a> Benchmark<'a> {
             println!(" ");
         }
 
-        self.run_cleanup_command(self.command.get_parameters().iter().cloned())?;
+        self.run_cleanup_command(self.command.get_parameters().iter().cloned(), output_policy)?;
 
         Ok(BenchmarkResult {
             command: self.command.get_name(),
diff --git a/src/cli.rs b/src/cli.rs
index 6684ebc86..5dbc56799 100644
--- a/src/cli.rs
+++ b/src/cli.rs
@@ -327,7 +327,7 @@ fn build_command() -> Command {
             Arg::new("output")
                 .long("output")
                 .conflicts_with("show-output")
-                .action(ArgAction::Set)
+                .action(ArgAction::Append)
                 .value_name("WHERE")
                 .help(
                     "Control where the output of the benchmark is redirected. Note \
@@ -343,7 +343,11 @@ fn build_command() -> Command {
                      \n  \
                        inherit:  Don't redirect the output at all (same as '--show-output').\n\
                      \n  \
-                       <FILE>:   Write the output to the given file.",
+                       <FILE>:   Write the output to the given file.\n\n\
+                    This option can be specified once for all commands or multiple times, once for \
+                    each command. Note: If you want to log the output of each and every iteration, \
+                    you can use a shell redirection and the '$HYPERFINE_ITERATION' environment variable:\n    \
+                    hyperfine 'my-command > output-${HYPERFINE_ITERATION}.log'\n\n",
                 ),
         )
         .arg(
diff --git a/src/command.rs b/src/command.rs
index 50fa906aa..c0ae4975b 100644
--- a/src/command.rs
+++ b/src/command.rs
@@ -251,8 +251,8 @@ impl<'a> Commands<'a> {
         self.0.iter()
     }
 
-    pub fn num_commands(&self) -> usize {
-        self.0.len()
+    pub fn num_commands(&self, has_reference_command: bool) -> usize {
+        self.0.len() + if has_reference_command { 1 } else { 0 }
     }
 
     /// Finds all the strings that appear multiple times in the input iterator, returning them in
diff --git a/src/main.rs b/src/main.rs
index a75d4bd07..79acc0bbc 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -32,7 +32,7 @@ fn run() -> Result<()> {
     colored::control::set_virtual_terminal(true).unwrap();
 
     let cli_arguments = get_cli_arguments(env::args_os());
-    let options = Options::from_cli_arguments(&cli_arguments)?;
+    let mut options = Options::from_cli_arguments(&cli_arguments)?;
     let commands = Commands::from_cli_arguments(&cli_arguments)?;
     let export_manager = ExportManager::from_cli_arguments(&cli_arguments, options.time_unit)?;
 
diff --git a/src/options.rs b/src/options.rs
index 72b1e1b0a..c9b0e44eb 100644
--- a/src/options.rs
+++ b/src/options.rs
@@ -234,8 +234,8 @@ pub struct Options {
     /// Where input to the benchmarked command comes from
     pub command_input_policy: CommandInputPolicy,
 
-    /// What to do with the output of the benchmarked command
-    pub command_output_policy: CommandOutputPolicy,
+    /// What to do with the output of the benchmarked commands
+    pub command_output_policies: Vec<CommandOutputPolicy>,
 
     /// Which time unit to use when displaying results
     pub time_unit: Option<Unit>,
@@ -257,7 +257,7 @@ impl Default for Options {
             sort_order_speed_comparison: SortOrder::MeanTime,
             sort_order_exports: SortOrder::Command,
             executor_kind: ExecutorKind::default(),
-            command_output_policy: CommandOutputPolicy::Null,
+            command_output_policies: vec![CommandOutputPolicy::Null],
             time_unit: None,
             command_input_policy: CommandInputPolicy::Null,
         }
@@ -320,23 +320,28 @@ impl Options {
 
         options.cleanup_command = matches.get_one::<String>("cleanup").map(String::from);
 
-        options.command_output_policy = if matches.get_flag("show-output") {
-            CommandOutputPolicy::Inherit
-        } else if let Some(output) = matches.get_one::<String>("output").map(|s| s.as_str()) {
-            match output {
-                "null" => CommandOutputPolicy::Null,
-                "pipe" => CommandOutputPolicy::Pipe,
-                "inherit" => CommandOutputPolicy::Inherit,
-                arg => {
-                    let path = PathBuf::from(arg);
-                    if path.components().count() <= 1 {
-                        return Err(OptionsError::UnknownOutputPolicy(arg.to_string()));
+        options.command_output_policies = if matches.get_flag("show-output") {
+            vec![CommandOutputPolicy::Inherit]
+        } else if let Some(output_values) = matches.get_many::<String>("output") {
+            let mut policies = vec![];
+            for value in output_values {
+                let policy = match value.as_str() {
+                    "null" => CommandOutputPolicy::Null,
+                    "pipe" => CommandOutputPolicy::Pipe,
+                    "inherit" => CommandOutputPolicy::Inherit,
+                    arg => {
+                        let path = PathBuf::from(arg);
+                        if path.components().count() <= 1 {
+                            return Err(OptionsError::UnknownOutputPolicy(arg.to_string()));
+                        }
+                        CommandOutputPolicy::File(path)
                     }
-                    CommandOutputPolicy::File(path)
-                }
+                };
+                policies.push(policy);
             }
+            policies
         } else {
-            CommandOutputPolicy::Null
+            vec![CommandOutputPolicy::Null]
         };
 
         options.output_style = match matches.get_one::<String>("style").map(|s| s.as_str()) {
@@ -346,7 +351,10 @@ impl Options {
             Some("color") => OutputStyleOption::Color,
             Some("none") => OutputStyleOption::Disabled,
             _ => {
-                if options.command_output_policy == CommandOutputPolicy::Inherit
+                if options
+                    .command_output_policies
+                    .iter()
+                    .any(|policy| *policy == CommandOutputPolicy::Inherit)
                     || !io::stdout().is_terminal()
                 {
                     OutputStyleOption::Basic
@@ -436,26 +444,34 @@ impl Options {
         Ok(options)
     }
 
-    pub fn validate_against_command_list(&self, commands: &Commands) -> Result<()> {
-        let num_commands = commands.num_commands()
-            + if self.reference_command.is_some() {
-                1
-            } else {
-                0
-            };
+    pub fn validate_against_command_list(&mut self, commands: &Commands) -> Result<()> {
+        let has_reference_command = self.reference_command.is_some();
+        let num_commands = commands.num_commands(has_reference_command);
+
         if let Some(preparation_command) = &self.preparation_command {
             ensure!(
                 preparation_command.len() <= 1 || num_commands == preparation_command.len(),
-                "The '--prepare' option has to be provided just once or N times, where N is the \
-             number of benchmark commands including a potential reference."
+                "The '--prepare' option has to be provided just once or N times, where N={num_commands} is the \
+                 number of benchmark commands (including a potential reference)."
             );
         }
 
         if let Some(conclusion_command) = &self.conclusion_command {
             ensure!(
                 conclusion_command.len() <= 1 || num_commands == conclusion_command.len(),
-                "The '--conclude' option has to be provided just once or N times, where N is the \
-             number of benchmark commands including a potential reference."
+                "The '--conclude' option has to be provided just once or N times, where N={num_commands} is the \
+                 number of benchmark commands (including a potential reference)."
+            );
+        }
+
+        if self.command_output_policies.len() == 1 {
+            self.command_output_policies =
+                vec![self.command_output_policies[0].clone(); num_commands];
+        } else {
+            ensure!(
+                self.command_output_policies.len() == num_commands,
+                "The '--output' option has to be provided just once or N times, where N={num_commands} is the \
+                 number of benchmark commands (including a potential reference)."
             );
         }
 


Test patch: