Change:
For a command-line of the form , where does indeed exist, produces different results than and in a C++ program. Specifically, is returning troublesome results. It looks like the CRT and CommandLineToArgvW . Invocation in Command Prompt: Expected arguments: [] C++ main() arguments: [] Result of GetCommandLineW: Result of CommandLineToArgvW: [, ] Result of : [, ] Obviously, this interpretation makes no sense. does not seem to be affected. A Rust program which demonstrates the mismatch: A C++ program which demonstrates the mismatch:
This probably won't surprise you, but stdlib uses CommandLineToArgvW:
That's fine. It wouldn't be too terrible to write a test case to match with the newer CRT, as long as the libs team was fine with it. This sounds like it's going to end with "you broke backwards compat."
Fundamentally this is a big decision of which standard to follow for argument parsing. I'm not particularly concerned with which standard we follow as long as it is Microsoft's standard and we follow it accurately. Decide between: Maintain status quo of behavior. Change behavior to match what newer CRT versions do.
I personally feel that we should follow the newer standard since that's what all Windows apps are using these days. I'm concerned about the possibility of this being a breaking change but this seems to only affect some rare edge cases in quote handling.
I'm legitimately not sure what "all Windows apps are using these days." Let's try Now let's try PowerShell: . From the way variables behave, it looks like putting two quotes in a row creates an escaped quote, but they don't get properly re-escaped when you pass them to a command line app: Microsoft is making it really difficult to figure out what actually counts as "correct" command line parsing, when there doesn't even seem to be a "right" way that actually works with their stuff. Let's see what WSL does: Bash still acts like you expect it, and apparently the WSL-to-Windows bridge generates backslash escapes when it tries to convert WSL's arguments list into a Windows GetCommandLineW string. Nuts that Bash on Ubuntu on Windows does a better job than PowerShell does (at least, it generates command strings that are capable of being unambiguously parsed). My next thought was to create a file name with quotes in it, and see what kind of parameters Explorer generates for me, but apparently Windows doesn't actually allow quotes in file names. And, finally, let's see what happens if one Rust program calls another one: Apparently, everybody seems to prefer backslashes when generating CLIs for Windows, except for PowerShell which passes quotes unescaped, and which just passes the raw CLI unchanged. So the current approach already seems to have interoperability as good as we would need it to be, but changing it to mimic the current CRT version probably won't break anything, either.
I think it's safe to say that "all Windows apps" are using Microsoft's C runtime for parsing commandline arguments. If "all" means "the majority of software built since 2008". In powershell you can use backticks to escape quotes: .targetdebug "a " c" Or you can surround the argument with single quotes: .targetdebug 'a "b" c' Or you can pass the entire line verbatim, without interpretation by the shell: .targetdebug --% "a "b" c" Or, if you're feeling particularly adventurous: .targetdebug you can write a multi-line "argument" '@ I'm not sure how you reached your last conclusion. It seems odd to use Rust's argument parsing to prove how most Windows programs handle arguments, no?
No I can't. Here's what happens when I try: PS C:Usersmichatestexe.targetdebug "a " c" env::args() = ["C:Usersmichatestexetargetdebug", "a b c"] GetCommandLineW = "C:Usersmichatestexetargetdebug" "a "b" c" It's exactly the same as when I did it with two quotes in a row. It creates a string object with quotes in it (like we want), but then it turns around and passes it verbatim without re-escaping it. PS C:Usersmichatestexe.targetdebug 'a "b" c' env::args() = ["C:Usersmichatestexetargetdebug", "a b c"] GetCommandLineW = "C:Usersmichatestexetargetdebug" "a "b" c" It's not PowerShell's syntax that I'm questioning here. I can easily create a string object with quotes in it. What I want to know is what syntax PowerShell expects us to have. What syntax do they use when re-serializing the command line? Apparently, they don't. Rust and Cargo communicate through command-line parameters, so we probably want to make sure that and can accurately round-trip their data.
The behaviour I was hoping for, by the way, is something like one of these: It's the same behaviour that you were getting when you demoed the python interpreter's argument parsing: it's a single argument with spaces and quotes within it. It's "how do I escape a quote?"
Just to be absolute clear about how Windows works, it might help to completely forget how *nix works. The Windows kernel doesn't know at all about arguments. As far as it's concerned arguments are just a single string that it can pass on to a new process. No more, no less. It's not an array of strings. It's just one string. This string doesn't have to contain the or anything else. The kernel does not care. This string is what gets. It's up to individual applications to send the kernel a string when creating a process. It's up to each individual application to parse it however it likes (or not at all). So what happens is relatively simple: shell sets the string to the being invoked and appends the arguments as the user input them (subject to how the shell itself interprets strings). invoked application parses the string according to its own rules (although in reality it usually defers to the C runtime's interpretation). Step 1 in the above could be replaced by an application. In which case it's entirely up to the application what it sets to. There are no rules. Of course you usually want to make an arguments string that will be correctly interpreted by other applications. So for compatibility reasons double quotes should be escaped if they are actually wanted in the parsed arguments. Or not if they're unwanted. They all work the same, except for Rust which is special. In the following table I've trimmed the enclosing for onwards, just to make it clearer but you can imagine they're still there.
In short the powershell user is constructing the string more or less directly, the particulars of string types aside. But I would add that the intricacies of powershell should have nothing to do with Rust std's handling of command line arguments. So long as the std behaves like other Windows applications it doesn't matter. If powershell makes a nicer way to construct a string then great! Rust will benefit too, assuming it acts the same as other Windows applications. So to sum up, as you say, Rust has two jobs: Parse the incoming command line string (the were linked above) Construct an outgoing command line string (for ) Of course it's easy for Rust to be consistent with itself no matter what rules it uses. It just has to test that construction and parsing match up as expected. However, it's obviously beneficial to be consistent with other applications on the platform. All that said, there's nothing massively wrong with how Rust currently parses arguments. It's just that 12 years ago Microsoft tweaked their CRT's argument parsing rules slightly. Rust uses the old rules so, in some situations, Rust applications may exhibit surprising behaviours to users more accustomed to modern applications.
Yeah, I agree with everything you said there. I'm fine with changing to the new CRT behaviour, as long as we don't break anything. And it looks like we won't.
PowerShell is not a good example, because the rules are still screwed up. . It's very uncomfortable to call apps from PowerShell. I sometimes ended up with generating a bat file and then running that from PowerShell.
I'd highly recommend installing if you can. The current stable version is 7.0.1 which has slightly better command line handling, and the upcoming 7.1 release should improve it further. Otherwise, using (or manually escaping double-quotes using backticks) to set the command string directly produces the most consistent results.

Discussion:
#![feature(const_ip)] #![feature(const_ipv4)] #![feature(const_ipv6)] <ins> #![feature(const_option)] </ins> #![feature(const_raw_ptr_deref)] #![feature(const_socketaddr)] #![feature(container_error_extra)]

<del> #![allow(dead_code)] // runtime init functions not used during testing </del> <ins> //! The Windows command line is just a string //! <https://docs.microsoft.com/en-us/archive/blogs/larryosterman/the-windows-command-line-is-just-a-string> //! //! This module implements the parsing necessary to turn that string into a list of arguments. </ins> #[cfg(test)] mod tests; use crate::ffi::OsString; use crate::fmt; <ins> use crate::marker::PhantomData; use crate::num::NonZeroU16; </ins> use crate::os::windows::prelude::*; use crate::path::PathBuf; <del> use crate::slice; </del> <ins> use crate::ptr::NonNull; </ins> use crate::sys::c; use crate::sys::windows::os::current_exe; use crate::vec;

use core::iter; pub fn args() -> Args { <ins> // SAFETY: `GetCommandLineW` returns a pointer to a null terminated UTF-16 // string so it's safe for `WStrUnits` to use. </ins> unsafe { let lp_cmd_line = c::GetCommandLineW(); <del> let parsed_args_list = parse_lp_cmd_line(lp_cmd_line as *const u16, || { </del> <ins> let parsed_args_list = parse_lp_cmd_line(WStrUnits::new(lp_cmd_line), || { </ins> current_exe().map(PathBuf::into_os_string).unwrap_or_else(|_| OsString::new()) });

/// Implements the Windows command-line argument parsing algorithm. /// /// Microsoft's documentation for the Windows CLI argument format can be found at <del> /// <https://docs.microsoft.com/en-us/previous-versions//17w5ykft(v=vs.85)>. </del> <ins> /// <https://docs.microsoft.com/en-us/cpp/cpp/main-function-command-line-args?view=msvc-160#parsing-c-command-line-arguments> </ins> /// <del> /// Windows includes a function to do this in shell32.dll, /// but linking with that DLL causes the process to be registered as a GUI application. </del> <ins> /// A more in-depth explanation is here: /// <https://daviddeley.com/autohotkey/parameters/parameters.htm#WIN> /// /// Windows includes a function to do command line parsing in shell32.dll. /// However, this is not used for two reasons: /// /// 1. Linking with that DLL causes the process to be registered as a GUI application. </ins> /// GUI applications add a bunch of overhead, even if no windows are drawn. See /// <https://randomascii.wordpress.com/2018/12/03/a-not-called-function-can-cause-a-5x-slowdown/>. /// <del> /// This function was tested for equivalence to the shell32.dll implementation in /// Windows 10 Pro v1803, using an exhaustive test suite available at /// <https://gist.github.com/notriddle/dde431930c392e428055b2dc22e638f5> or /// <https://paste.gg/p/anonymous/47d6ed5f5bd549168b1c69c799825223>. unsafe fn parse_lp_cmd_line<F: Fn() -> OsString>( lp_cmd_line: *const u16, </del> <ins> /// 2. It does not follow the modern C/C++ argv rules outlined in the first two links above. /// /// This function was tested for equivalence to the C/C++ parsing rules using an /// extensive test suite available at /// <https://github.com/ChrisDenton/winarg/tree/std>. fn parse_lp_cmd_line<'a, F: Fn() -> OsString>( lp_cmd_line: Option<WStrUnits<'a>>, </ins> exe_name: F, ) -> Vec<OsString> { <del> const BACKSLASH: u16 = '' as u16; const QUOTE: u16 = '"' as u16; const TAB: u16 = 't' as u16; const SPACE: u16 = ' ' as u16; </del> <ins> const BACKSLASH: NonZeroU16 = NonZeroU16::new(b'' as u16).unwrap(); const QUOTE: NonZeroU16 = NonZeroU16::new(b'"' as u16).unwrap(); const TAB: NonZeroU16 = NonZeroU16::new(b't' as u16).unwrap(); const SPACE: NonZeroU16 = NonZeroU16::new(b' ' as u16).unwrap(); </ins> let mut ret_val = Vec::new(); <del> if lp_cmd_line.is_null() || *lp_cmd_line == 0 { </del> <ins> // If the cmd line pointer is null or it points to an empty string then // return the name of the executable as argv[0]. if lp_cmd_line.as_ref().and_then(|cmd| cmd.peek()).is_none() { </ins> ret_val.push(exe_name()); return ret_val; } <del> let mut cmd_line = { let mut end = 0; while *lp_cmd_line.offset(end) != 0 { end += 1; } slice::from_raw_parts(lp_cmd_line, end as usize) }; </del> <ins> let mut code_units = lp_cmd_line.unwrap(); </ins> // The executable name at the beginning is special. <del> cmd_line = match cmd_line[0] { // The executable name ends at the next quote mark, // no matter what. QUOTE => { let args = { let mut cut = cmd_line[1..].splitn(2, |&c| c == QUOTE); if let Some(exe) = cut.next() { ret_val.push(OsString::from_wide(exe)); } cut.next() }; if let Some(args) = args { args } else { return ret_val; } } // Implement quirk: when they say whitespace here, // they include the entire ASCII control plane: // "However, if lpCmdLine starts with any amount of whitespace, CommandLineToArgvW // will consider the first argument to be an empty string. Excess whitespace at the // end of lpCmdLine is ignored." 0..=SPACE => { ret_val.push(OsString::new()); &cmd_line[1..] } // The executable name ends at the next whitespace, // no matter what. _ => { let args = { let mut cut = cmd_line.splitn(2, |&c| c > 0 && c <= SPACE); if let Some(exe) = cut.next() { ret_val.push(OsString::from_wide(exe)); } cut.next() }; if let Some(args) = args { args } else { return ret_val; } </del> <ins> let mut in_quotes = false; let mut cur = Vec::new(); for w in &mut code_units { match w { // A quote mark always toggles `in_quotes` no matter what because // there are no escape characters when parsing the executable name. QUOTE => in_quotes = !in_quotes, // If not `in_quotes` then whitespace ends argv[0]. SPACE | TAB if !in_quotes => break, // In all other cases the code unit is taken literally. _ => cur.push(w.get()), </ins> } <del> }; </del> <ins> } // Skip whitespace. code_units.advance_while(|w| w == SPACE || w == TAB); ret_val.push(OsString::from_wide(&cur)); // Parse the arguments according to these rules: // * All code units are taken literally except space, tab, quote and backslash. // * When not `in_quotes`, space and tab separate arguments. Consecutive spaces and tabs are // treated as a single separator. // * A space or tab `in_quotes` is taken literally. // * A quote toggles `in_quotes` mode unless it's escaped. An escaped quote is taken literally. // * A quote can be escaped if preceded by an odd number of backslashes. // * If any number of backslashes is immediately followed by a quote then the number of // backslashes is halved (rounding down). // * Backslashes not followed by a quote are all taken literally. // * If `in_quotes` then a quote can also be escaped using another quote // (i.e. two consecutive quotes become one literal quote). </ins> let mut cur = Vec::new(); let mut in_quotes = false; <del> let mut was_in_quotes = false; let mut backslash_count: usize = 0; for &c in cmd_line { match c { // backslash BACKSLASH => { backslash_count += 1; was_in_quotes = false; </del> <ins> while let Some(w) = code_units.next() { match w { // If not `in_quotes`, a space or tab ends the argument. SPACE | TAB if !in_quotes => { ret_val.push(OsString::from_wide(&cur[..])); cur.truncate(0); // Skip whitespace. code_units.advance_while(|w| w == SPACE || w == TAB); </ins> } <del> QUOTE if backslash_count % 2 == 0 => { cur.extend(iter::repeat(b'' as u16).take(backslash_count / 2)); backslash_count = 0; if was_in_quotes { cur.push('"' as u16); was_in_quotes = false; </del> <ins> // Backslashes can escape quotes or backslashes but only if consecutive backslashes are followed by a quote. BACKSLASH => { let backslash_count = code_units.advance_while(|w| w == BACKSLASH) + 1; if code_units.peek() == Some(QUOTE) { cur.extend(iter::repeat(BACKSLASH.get()).take(backslash_count / 2)); // The quote is escaped if there are an odd number of backslashes. if backslash_count % 2 == 1 { code_units.next(); cur.push(QUOTE.get()); } </ins> } else { <del> was_in_quotes = in_quotes; in_quotes = !in_quotes; </del> <ins> // If there is no quote on the end then there is no escaping. cur.extend(iter::repeat(BACKSLASH.get()).take(backslash_count)); </ins> } } <del> QUOTE if backslash_count % 2 != 0 => { cur.extend(iter::repeat(b'' as u16).take(backslash_count / 2)); backslash_count = 0; was_in_quotes = false; cur.push(b'"' as u16); } SPACE | TAB if !in_quotes => { cur.extend(iter::repeat(b'' as u16).take(backslash_count)); if !cur.is_empty() || was_in_quotes { ret_val.push(OsString::from_wide(&cur[..])); cur.truncate(0); </del> <ins> // If `in_quotes` and not backslash escaped (see above) then a quote either // unsets `in_quote` or is escaped by another quote. QUOTE if in_quotes => match code_units.peek() { // Two consecutive quotes when `in_quotes` produces one literal quote. Some(QUOTE) => { cur.push(QUOTE.get()); code_units.next(); </ins> } <del> backslash_count = 0; was_in_quotes = false; } _ => { cur.extend(iter::repeat(b'' as u16).take(backslash_count)); backslash_count = 0; was_in_quotes = false; cur.push(c); } </del> <ins> // Otherwise set `in_quotes`. Some(_) => in_quotes = false, // The end of the command line. // Push `cur` even if empty, which we do by breaking while `in_quotes` is still set. None => break, }, // If not `in_quotes` and not BACKSLASH escaped (see above) then a quote sets `in_quote`. QUOTE => in_quotes = true, // Everything else is always taken literally. _ => cur.push(w.get()), </ins> } } <del> cur.extend(iter::repeat(b'' as u16).take(backslash_count)); // include empty quoted strings at the end of the arguments list if !cur.is_empty() || was_in_quotes || in_quotes { </del> <ins> // Push the final argument, if any. if !cur.is_empty() || in_quotes { </ins> ret_val.push(OsString::from_wide(&cur[..])); } ret_val

self.parsed_args_list.len() } } <ins> /// A safe iterator over a LPWSTR /// (aka a pointer to a series of UTF-16 code units terminated by a NULL). struct WStrUnits<'a> { // The pointer must never be null... lpwstr: NonNull<u16>, // ...and the memory it points to must be valid for this lifetime. lifetime: PhantomData<&'a [u16]>, } impl WStrUnits<'_> { /// Create the iterator. Returns `None` if `lpwstr` is null. /// /// SAFETY: `lpwstr` must point to a null-terminated wide string that lives /// at least as long as the lifetime of this struct. unsafe fn new(lpwstr: *const u16) -> Option<Self> { Some(Self { lpwstr: NonNull::new(lpwstr as _)?, lifetime: PhantomData }) } fn peek(&self) -> Option<NonZeroU16> { // SAFETY: It's always safe to read the current item because we don't // ever move out of the array's bounds. unsafe { NonZeroU16::new(*self.lpwstr.as_ptr()) } } /// Advance the iterator while `predicate` returns true. /// Returns the number of items it advanced by. fn advance_while<P: FnMut(NonZeroU16) -> bool>(&mut self, mut predicate: P) -> usize { let mut counter = 0; while let Some(w) = self.peek() { if !predicate(w) { break; } counter += 1; self.next(); } counter } } impl Iterator for WStrUnits<'_> { // This can never return zero as that marks the end of the string. type Item = NonZeroU16; fn next(&mut self) -> Option<NonZeroU16> { // SAFETY: If NULL is reached we immediately return. // Therefore it's safe to advance the pointer after that. unsafe { let next = self.peek()?; self.lpwstr = NonNull::new_unchecked(self.lpwstr.as_ptr().add(1)); Some(next) } } } </ins>

let mut wide: Vec<u16> = OsString::from(string).encode_wide().collect(); wide.push(0); let parsed = <del> unsafe { parse_lp_cmd_line(wide.as_ptr() as *const u16, || OsString::from("TEST.EXE")) }; </del> <ins> unsafe { parse_lp_cmd_line(WStrUnits::new(wide.as_ptr()), || OsString::from("TEST.EXE")) }; </ins> let expected: Vec<OsString> = parts.iter().map(|k| OsString::from(k)).collect(); <del> assert_eq!(parsed.as_slice(), expected.as_slice()); </del> <ins> assert_eq!(parsed.as_slice(), expected.as_slice(), "{:?}", string); </ins> } #[test]

#[test] fn official_examples() { chk(r#"EXE "abc" d e"#, &["EXE", "abc", "d", "e"]); <del> chk(r#"EXE ab d"e f"g h"#, &["EXE", r#"ab"#, "de fg", "h"]); </del> <ins> chk(r#"EXE ab d"e f"g h"#, &["EXE", r"ab", "de fg", "h"]); </ins> chk(r#"EXE a"b c d"#, &["EXE", r#"a"b"#, "c", "d"]); <del> chk(r#"EXE a"b c" d e"#, &["EXE", r#"ab c"#, "d", "e"]); </del> <ins> chk(r#"EXE a"b c" d e"#, &["EXE", r"ab c", "d", "e"]); </ins> } #[test] fn whitespace_behavior() { <del> chk(r#" test"#, &["", "test"]); chk(r#"  test"#, &["", "test"]); chk(r#" test test2"#, &["", "test", "test2"]); chk(r#" test  test2"#, &["", "test", "test2"]); chk(r#"test test2 "#, &["test", "test2"]); chk(r#"test  test2 "#, &["test", "test2"]); chk(r#"test "#, &["test"]); </del> <ins> chk(" test", &["", "test"]); chk("  test", &["", "test"]); chk(" test test2", &["", "test", "test2"]); chk(" test  test2", &["", "test", "test2"]); chk("test test2 ", &["test", "test2"]); chk("test  test2 ", &["test", "test2"]); chk("test ", &["test"]); </ins> } #[test] fn genius_quotes() { chk(r#"EXE "" """#, &["EXE", "", ""]); <del> chk(r#"EXE "" """"#, &["EXE", "", """]); </del> <ins> chk(r#"EXE "" """"#, &["EXE", "", r#"""#]); </ins> chk( r#"EXE "this is """all""" in the same argument""#, <del> &["EXE", "this is "all" in the same argument"], </del> <ins> &["EXE", r#"this is "all" in the same argument"#], </ins> ); <del> chk(r#"EXE "a"""#, &["EXE", "a""]); chk(r#"EXE "a"" a"#, &["EXE", "a"", "a"]); </del> <ins> chk(r#"EXE "a"""#, &["EXE", r#"a""#]); chk(r#"EXE "a"" a"#, &["EXE", r#"a" a"#]); </ins> // quotes cannot be escaped in command names chk(r#""EXE" check"#, &["EXE", "check"]); chk(r#""EXE check""#, &["EXE check"]); <del> chk(r#""EXE """for""" check"#, &["EXE ", r#"for""#, "check"]); chk(r#""EXE "for" check"#, &[r#"EXE "#, r#"for""#, "check"]); </del> <ins> chk(r#""EXE """for""" check"#, &["EXE for check"]); chk(r#""EXE "for" check"#, &[r"EXE for check"]); chk(r#""EXE " for " check"#, &[r"EXE ", "for", r#"""#, "check"]); chk(r#"E"X"E test"#, &["EXE", "test"]); chk(r#"EX""E test"#, &["EXE", "test"]); } // from https://daviddeley.com/autohotkey/parameters/parameters.htm#WINCRULESEX #[test] fn post_2008() { chk("EXE CallMeIshmael", &["EXE", "CallMeIshmael"]); chk(r#"EXE "Call Me Ishmael""#, &["EXE", "Call Me Ishmael"]); chk(r#"EXE Cal"l Me I"shmael"#, &["EXE", "Call Me Ishmael"]); chk(r#"EXE CallMe"Ishmael"#, &["EXE", r#"CallMe"Ishmael"#]); chk(r#"EXE "CallMe"Ishmael""#, &["EXE", r#"CallMe"Ishmael"#]); chk(r#"EXE "Call Me Ishmael""#, &["EXE", r"Call Me Ishmael"]); chk(r#"EXE "CallMe"Ishmael""#, &["EXE", r#"CallMe"Ishmael"#]); chk(r#"EXE ab"#, &["EXE", r"ab"]); chk(r#"EXE "ab""#, &["EXE", r"ab"]); chk(r#"EXE ""Call Me Ishmael"""#, &["EXE", r#""Call Me Ishmael""#]); chk(r#"EXE "C:TEST A""#, &["EXE", r"C:TEST A"]); chk(r#"EXE ""C:TEST A"""#, &["EXE", r#""C:TEST A""#]); chk(r#"EXE "a b c"  d  e"#, &["EXE", "a b c", "d", "e"]); chk(r#"EXE "ab"c"  ""  d"#, &["EXE", r#"ab"c"#, r"", "d"]); chk(r#"EXE ab d"e f"g h"#, &["EXE", r"ab", "de fg", "h"]); chk(r#"EXE a"b c d"#, &["EXE", r#"a"b"#, "c", "d"]); chk(r#"EXE a"b c" d e"#, &["EXE", r"ab c", "d", "e"]); // Double Double Quotes chk(r#"EXE "a b c"""#, &["EXE", r#"a b c""#]); chk(r#"EXE """CallMeIshmael"""  b  c"#, &["EXE", r#""CallMeIshmael""#, "b", "c"]); chk(r#"EXE """Call Me Ishmael""""#, &["EXE", r#""Call Me Ishmael""#]); chk(r#"EXE """"Call Me Ishmael"" b c"#, &["EXE", r#""Call"#, "Me", "Ishmael", "b", "c"]); </ins> }

pub fn RemoveDirectoryW(lpPathName: LPCWSTR) -> BOOL; pub fn SetFileAttributesW(lpFileName: LPCWSTR, dwFileAttributes: DWORD) -> BOOL; pub fn SetLastError(dwErrCode: DWORD); <del> pub fn GetCommandLineW() -> *mut LPCWSTR; </del> <ins> pub fn GetCommandLineW() -> LPWSTR; </ins> pub fn GetTempPathW(nBufferLength: DWORD, lpBuffer: LPCWSTR) -> DWORD; pub fn GetCurrentProcess() -> HANDLE; pub fn GetCurrentThread() -> HANDLE;