Repository: sharkdp/hyperfine (language: rust)

Issue:
Bug ~ quoting? failure on windows-platform
In example, for `bat`:

``` shell
C:\>bat.exe --paging=always --pager="cmd.exe /c more.com" README.md
       │ File: README.md
   1   │ uutils coreutils
   2   │ ================
   3   │
   4   │ [![Discord](https://img.shields.io/badge/discord-join-7289DA.svg?logo=discord&longCache=true&style=flat)](https://discord.
       │ gg/wQVJbvJ)
   5   │ [![License](http://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/uutils/coreutils/blob/master/LICENSE)
   6   │ [![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2Fuutils%2Fcoreutils.svg?type=shield)](https://app.fos
       │ sa.io/projects/git%2Bgithub.com%2Fuutils%2Fcoreutils?ref=badge_shield)
   7   │ [![LOC](https://tokei.rs/b1/github/uutils/coreutils?category=code)](https://github.com/Aaronepower/tokei)
   8   │ [![dependency status](https://deps.rs/repo/github/uutils/coreutils/status.svg)](https://deps.rs/repo/github/uutils/coreuti
       │ ls)
   9   │
  10   │ [![Build Status](https://api.travis-ci.org/uutils/coreutils.svg?branch=master)](https://travis-ci.org/uutils/coreutils)
  11   │ [![Build Status (Windows)](https://ci.appveyor.com/api/projects/status/787ltcxgy86r20le?svg=true)](https://ci.appveyor.com
       │ /project/Arcterus/coreutils)
  12   │ [![Build Status (FreeBSD)](https://api.cirrus-ci.com/github/uutils/coreutils.svg)](https://cirrus-ci.com/github/uutils/cor
       │ eutils/master)

```

When testing with `hyperfine`:

``` shell
C:\>hyperfine "bat.exe --paging=always --pager=\"cmd.exe /c more.com\" README.md"
Benchmark #1: bat.exe --paging=always --pager="cmd.exe /c more.com" README.md                                                     0
Error: Command terminated with non-zero exit code. Use the '-i'/'--ignore-failure' option if you want to ignore this. Alternatively, use the '--show-output' option to debug what went wrong.
```

but, using a command-string without internal quoting (by passing arguments to `bat` via the environment):

``` shell
env BAT_PAGER="cmd.exe /c more.com" hyperfine "bat.exe --paging=always README.md"
Benchmark #1: bat.exe --paging=always README.md                                                                                   0
  Time (mean ± σ):     199.9 ms ±  51.9 ms    [User: 10.3 ms, System: 6.8 ms]                                                      0
  Range (min … max):   179.8 ms … 347.5 ms    10 runs

  Warning: The first benchmarking run for this command was significantly slower than the rest (347.5 ms). This could be caused by (filesystem) caches that were not filled until after the first run. You should consider using the '--warmup' option to fill those caches before the actual benchmark. Alternatively, use the '--prepare' option to clear the caches before each timing run.
```

The text of the target command shown by `hyperfine` looks correct, but the target is not being executed as displayed.

This is likely a shell quoting issue. 


Discussion / hints:
Thank you for reporting this.

So what are the actual quoting/escaping rules for `cmd.exe`? I seem to remember that they are highly non-intuitive (for a Unix user). Can
```bash
"bat.exe --paging=always --pager=\"cmd.exe /c more.com\" README.md"
```
be passed to other programs without any problem?
You're correct; parsing command lines on Windows is complicated and quirky.

And the rules changed in 2008.

David Deley has a good explanation of the various rules and changes (with examples) ["How Command Line Parameters Are Parsed"](http://daviddeley.com/autohotkey/parameters/parameters.htm).

If you're handing the command line off to `cmd`, you should be able to just mechanically escape every character with `^` (eg, `cmd.exe /d/c ^e^c^h^o ^"^t^h^i^s^ ^>^ ^w^o^r^k^s^"`).

If you're handing off directly to another windows process, I think it's ok to assume the 2008+ parsing behavior, *except* that `rust` is currently using the older, pre-2008, parsing behavior (which is *extra* "quirky"). But it does look like that is [likely to change](https://github.com/rust-lang/rust/issues/44650). I'm sure there must be mechanical translation recipe that would work for the direct process approach, but I haven't been able to create or find it.

[`Command` is struggling with this as well](https://github.com/rust-lang/rust/issues/29494#issuecomment-439726937).

That's why I fell back to `cmd.exe /d/c ...` for `env` in [coreutils](https://github.com/uutils/coreutils).

- ref: [How Command Line Parameters Are Parsed](http://daviddeley.com/autohotkey/parameters/parameters.htm) @@ <http://archive.is/9jM4q>
- ref: Discussing [env::args on Windows (GH:rustlang/rust#44650)](https://github.com/rust-lang/rust/issues/44650)
- ref: Discussion about [Powershell command line escaping](https://github.com/PowerShell/PowerShell/issues/1995#issuecomment-562334606)
- ref: [MSDN ~ Parsing C command line args](https://docs.microsoft.com/en-us/cpp/cpp/main-function-command-line-args?view=vs-2019#parsing-c-command-line-arguments)
- ref: [Python's work-around](https://docs.python.org/3/library/subprocess.html#converting-argument-sequence)
- ref: [`Command` does not escape arguments as expected on windows](https://github.com/rust-lang/rust/issues/29494)
Thank you for the detailed explanation and for all the references, especially "How Command Line Parameters Are Parsed". I didn't know it was *that* complicated :frowning: 


I'm inclined to accept the `cmd.exe /d/c ^…^…` workaround for `hyperfine`, if you think that this would solve most issues.

Alternatively, would it help to use PowerShell instead of `cmd.exe`? This could be easily tested via `hyperfine`s `--shell` option (if PowerShell also uses the syntax `/c <cmd>`).
> Alternatively, would it help to use PowerShell instead of `cmd.exe`? This could be easily tested via `hyperfine`s `--shell` option (if PowerShell also uses the syntax `/c <cmd>`).

I don't think PowerShell is a solution at this point. The initial design was not well thought out and it's also [struggling with command line escaping](https://github.com/PowerShell/PowerShell/issues/1995#issuecomment-562334606) and still evolving.

`cmd.exe` is, in comparison, a known, relatively stable, problem space.


It looks like the resolution of https://github.com/rust-lang/rust/issues/29494 (a new `raw_arg` method on Windows) could help us resolve this issue. It's currently only available on nightly though.
> It looks like the resolution of [rust-lang/rust#29494](https://github.com/rust-lang/rust/issues/29494) (a new `raw_arg` method on Windows) could help us resolve this issue. It's currently only available on nightly though.

👍🏻
I've been keeping an eye on that issue as well.
Any update on this?
Looks like this might be stabilized in 1.61, so it will take a few more months (at least).

In the meantime, you might be able to use the new `-N`/`--shell=none` option of hyperfine to disable the shell completely.
FYI @rivy 

`raw_arg` has been stabilized for a while now: https://doc.rust-lang.org/std/os/windows/process/trait.CommandExt.html#tymethod.raw_arg 

Someone with access to a Windows machine might be able to resolve this, ideally backed up by some new (regression) tests.
Hello, 
I can take a shot a this. I've being meaning to get more into OSS and rust, and I run mostly on windows, specially at work.

Pretty cool tool, btw. It's my go-to for performance tests 🥇 

Fix patch:
diff --git a/src/benchmark/executor.rs b/src/benchmark/executor.rs
index 31db47ba6..d0e748e6c 100644
--- a/src/benchmark/executor.rs
+++ b/src/benchmark/executor.rs
@@ -1,3 +1,5 @@
+#[cfg(windows)]
+use std::os::windows::process::CommandExt;
 use std::process::ExitStatus;
 
 use crate::command::Command;
@@ -133,16 +135,17 @@ impl<'a> Executor for ShellExecutor<'a> {
         command: &Command<'_>,
         command_failure_action: Option<CmdFailureAction>,
     ) -> Result<(TimingResult, ExitStatus)> {
+        let on_windows_cmd = cfg!(windows) && *self.shell == Shell::Default("cmd.exe");
         let mut command_builder = self.shell.command();
-        command_builder
-            .arg(
-                if cfg!(windows) && *self.shell == Shell::Default("cmd.exe") {
-                    "/C"
-                } else {
-                    "-c"
-                },
-            )
-            .arg(command.get_command_line());
+        command_builder.arg(if on_windows_cmd { "/C" } else { "-c" });
+
+        // Windows needs special treatment for its behavior on parsing cmd arguments
+        if on_windows_cmd {
+            #[cfg(windows)]
+            command_builder.raw_arg(command.get_command_line());
+        } else {
+            command_builder.arg(command.get_command_line());
+        }
 
         let mut result = run_command_and_measure_common(
             command_builder,


Test patch:
diff --git a/tests/integration_tests.rs b/tests/integration_tests.rs
index 5cbdfa0e9..5a6a67e97 100644
--- a/tests/integration_tests.rs
+++ b/tests/integration_tests.rs
@@ -423,3 +423,21 @@ fn speed_comparison_sort_order() {
             "2.00 ±  0.00  sleep 2\n        1.00          sleep 1",
         ));
 }
+
+#[cfg(windows)]
+#[test]
+fn windows_quote_args() {
+    hyperfine()
+        .arg("more \"example_input_file.txt\"")
+        .assert()
+        .success();
+}
+
+#[cfg(windows)]
+#[test]
+fn windows_quote_before_quote_args() {
+    hyperfine()
+        .arg("dir \"..\\src\\\" \"..\\tests\\\"")
+        .assert()
+        .success();
+}