Repository: xonsh/xonsh
Pull request: tutorial for subproc strings
Hopefully this closes #621 and #1432

Problem statement:
quote removal in subprocess mode does not behave as expected
Please have a look at the following two examples::

```
    xonsh$ /bin/echo 'foo'
    foo

    xonsh$ /bin/echo 'foo'bar
    'foo'bar
```

In the first case quotes are stripped from the second argument; in the
second, quotes are kept.

This is not consistent with what other shells do, and makes it quite
inconvenient to pass option arguments containing spaces.  For example,
the following would include the quotes in the commit message::

```
    svn commit --message='A short description here.'
```

Hints from issue discussion:
Hi @riccardomurri, you are correct that we are not consistent with how other shells treat string.  Xonsh makes a willful departure from what other shells do, in part because what other shells do is not self-consistent. Rather, xonsh's literal string syntax sticks as close as possible to Python string behaviour, which we believe to be cleaner and more consistent.  

The only departure from Python strings in subproc mode is that literal strings are globbed and have their environment variables expanded, as one would expect from other shells.

The examples that you give of echo and svn are not entirely consistent themselves.  It is generally the responsibility of each program to parse its own arguments.  So if svn is including quotes this is svn fault.  As an demonstration, consider the following Bash script which simply prints its arguments:

**temp.sh**

``` bash
#!/bin/bash
echo $*
```

Now let's run this with multiple arguments.

``` console
scopatz@localhost ~ $ chmod 755 temp.sh 

# see bash keeps the quotes
scopatz@localhost ~ $ ./temp.sh --message='hello mom'
--message='hello mom'  

# bash keeps the quotes even when running though bash directly
scopatz@localhost ~ $ bash temp.sh --message='hello mom'
--message='hello mom'

# same thing with double quotes
scopatz@localhost ~ $ bash temp.sh --message="hello mom"
--message="hello mom"

# Let's add a 'foo'bar argument, and see that the quotes remain.
scopatz@localhost ~ $ bash temp.sh 'foo'bar --message="hello mom"
'foo'bar --message="hello mom"
```

So bash is doing something strange here with echo and a literal string + characters that I don't think is worth replicating.

As per getting the SVN example to work in a natural way, recall that arguments are whitespace separated.  Thus, I would leave off the `=` and you have two distinct tokens:

``` console
$ svn commit --message 'A short description here.'
```

Alternatively, if you truly must have the equals sign and want xonsh to group the internal whitespace, I believe that you can put the whole token in quotes.

``` console
$ svn commit '--message=A short description here.'
```

This is probably less satisfying, but works because your original command is actually equivalent to:

``` console
$ svn commit "--message='A short description here.'"
```

In general for xonsh, if a subprocess argument does not start _and_ end with quote characters or other known syntax like `$()`, `$[]`, or `@()` then that argument is interpreted as being wrapped in Python-like quotes.

I hope this helps clear things up.

Hello @scopatz, thank for looking into this!

Actually, I cannot replicate the "temp.sh" example you showed. (Did
you run that from windows?) My bash (and zsh too) removes the quotes,
both single and double, wherever they are in the argument being
processed, which is consistent with what the [bash man page
states](https://www.gnu.org/software/bash/manual/bashref.html#Quote-Removal)::

```
rmurri@xenia:~$ cat /tmp/test.sh
#! /bin/sh

echo $*

rmurri@xenia:~$ /tmp/test.sh foo'bar'
foobar

rmurri@xenia:~$ /tmp/test.sh --message='foo'
--message=foo

rmurri@xenia:~$ /tmp/test.sh --message="foo"
--message=foo
```

I understand Xonsh does not need to retain compatibility; actually, I
would go as far as to argue that, since Xonsh does not do
word-splitting after variable substitution, it does not need quote
removal _at all_::

```
    xonsh$ def cnt(args, stdin=None):
               print(len(args))

    xonsh$ aliases['cnt'] = cnt

    # three separate arguments
    xonsh$ cnt a b c
    3

    # when coming from a var, this is one argument instead
    xonsh$ $var='a b c'
    xonsh$ cnt $var
    1
```

Instead sh/bash needs quote removal to control whether variables are
expanded into one or several arguments:

```
    bash$ cnt () { echo $#; }

    # three args
    bash$ cnt a b c
    3

    # still three args because of word splitting
    bash$ var='a b c'
    bash$ cnt $var
    3
    bash$ cnt "$var"
    1
```

Still, if you keep quote removal, I think it should behave like
sh/bash does, for the principle of least surprise. (And because
rewriting simple cases of quote removal using Xonsh' `@(...)` is too
verbose.)

I stand corrected: of course we need quote removal in order to have
arguments with embedded spaces.

So, to sum up, there are two styles of quote removal:
- What Xonsh currently implements (Windows-style?): remove quotes only
  if an argument starts and ends with them.
- UNIX shell style: remove quotes anywhere in an argument as the last
  step of command-line processing.

Being used to UNIX shells, I would like to see the latter style
implemented, at least as an option.

Hi @riccardomurri, I think I got the bash behaviour wrong myself. temp.sh does act as you describe if I am in a Bash interactive instance.

On the subject of what to do about it, I would really call the first one "Python-style."  My personal opinion is that the way the sh-lang spec is written is rather arcane and not up to modern notions of how strings should work. (Though I am not saying that what we have now is perfect.)

Now, I may be wrong about this, but I believe that adding in optional Unix-sh-style strings would touch the parser. I am a little wary of having option-dependent syntax. This means that xonsh code could not be universally parsed.

On the flipside, adding in the behvaiour for strings you are suggesting would have overlapping syntax  some Python literals, namely `b"x"`, `r"x"`, `u"x"`, and in v3.6 now `f"x"`.  Whatever would be done would have to avoid these specific cases.

Now, that said, I am open to suggestions - though for something like this I would also like for it to be vetted by @adqm, @gforsyth, and the mailing list.  A suggestion backed by a PR would be even more compelling. Here are a couple of thoughts that I had that might be more or less compelling:
1. Use double quotes.  I don't believe that Python uses double quote so `foo""bar""` and `foo''bar''` may be fair game.
2. Match a backtick and a single quote, like in LaTeX.  
3. Make `=""` and `=''` an operator in subproc mode that removes the quote characters and keeps the equals sign.  This doesn't cover the `foo'bar'` case, and therefore not as hard to understand or implement.
4. Come up with some other crazy syntax that is better than the above.

In any event, I guess I'd like folks to weigh in with their thoughts.

Hey @scopatz, 

I agree that Python doesn't use double-up quotes -- are you envisioning something along the lines of

```
> echo foo""bar""
foo"bar"
```

?

That doesn't seem too terribly hard, but I'm also at a loss for an actual use-case.  @riccardomurri -- did you have a specific idea in mind that led you to this example?  If it's the `svn` case, then my inclination is to go with number 3 above and create the subproc operators.  

Hi @gforsyth, sorry to be clear, I am envisioning double quotes as the following:

``` console
$ echo foo""bar""
foobar
```

Which would mirror Bash's quoting behaviour.  However, I don't think that this makes anyone happy because it isn't as concise as the sh-lang syntax for the people that want it and it makes something possible that should not be possible for the people who don't want it (and are generally anti-arcana).

Also, I am most pro option 3 as well.

Commit message: Thanks to @melund, tailing

Patch to review:
diff --git a/docs/guides.rst b/docs/guides.rst
index c402ba016b..0f42e1f32a 100644
--- a/docs/guides.rst
+++ b/docs/guides.rst
@@ -13,6 +13,7 @@ Guides
     tutorial_events
     tutorial_completers
     tutorial_history_backend
+    tutorial_subproc_strings
     tutorial_ptk
     bash_to_xsh
     python_virtual_environments
\ No newline at end of file
diff --git a/docs/tutorial_subproc_strings.rst b/docs/tutorial_subproc_strings.rst
new file mode 100644
index 0000000000..ce4f3f1426
--- /dev/null
+++ b/docs/tutorial_subproc_strings.rst
@@ -0,0 +1,131 @@
+.. _tutorial_subproc_strings:
+
+************************************
+Tutorial: Subprocess Strings
+************************************
+Strings in xonsh follow two simple rules:
+
+1. Strings in xonsh are always parsed in the same way, and
+2. Python always wins!
+
+Together these rules mean that **even strings in subprocess mode are treated
+like Python strings!** This will (help) preserve your sanity.
+
+No Escape
+=========
+Xonsh strings are exactly like Python strings everywhere. Xonsh uses
+exactly the same escape characters that Python dones; no more and no less.
+This is different from other shells, which have a different set of escape
+sequences than Python has.  Notably, many sh-langs allow you to escape
+spaces with ``\ `` (backslash-space).
+
+**bash**
+
+.. code-block:: bash
+
+    $ echo A\ Single\ Argument
+    A Single Argument
+
+In the above example, since the spaces are escaped, the ``echo`` command
+only receves a single argument. Xonsh does not allow this. If you were
+to try this in xonsh, you'd see:
+
+**xonsh**
+
+.. code-block:: xonsh
+
+    $ echo Actually\ Three\ Arguments
+    Actually\ Three\ Arguments
+
+In this example, echo recives three arguments:: ``"Actually\\"``, ``"Three\\"``,
+and ``"Arguments"``. Instead, xonsh requires you to use quotes in order to
+pass in a single argument:
+
+**xonsh** or **bash**
+
+.. code-block:: xonsh
+
+    $ echo "A Single Argument"
+    A Single Argument
+
+Using quotes is arguably what should have been done in sh-lang in the
+first place.
+
+.. note::
+
+    When in doubt in subprocess mode, use quotes!
+
+
+Justification
+=============
+The reasons for not having additional escape sequences, as in sh-langs, are:
+
+1. Escape charaters can get overwhemlingly ugly, fast.
+2. We have escape characters, they are called quotes :)
+3. We have literal input in subprocess mode via macros.
+
+On this last point, if you don't already know about
+`Subprocess Macros <tutorial_macros.html#subprocess-macros>`_,
+these allow all input following an ``!`` to be treated as a single argument.
+For example,
+
+**xonsh**
+
+.. code-block:: xonsh
+
+    $ echo! A  Single     Argument
+    A  Single     Argument
+
+Subprocess macros are the ultimate escape mechanism.
+
+The Quotes Stay
+===============
+In sh-langs, internal quote characters are removed. For instance:
+
+.. code-block:: bash
+
+    $ echo foo"bar"baz
+    foobarbaz
+
+    $ echo --key="value"
+    --key=value
+
+Xonsh considers this behaviour insane. Instead, xonsh treats these
+arguments as if they were surrounded in another, outer level of
+quotation (``'foo"bar"baz'``). Xonsh will keep the quotation marks
+when leading and trailing quotes are not matched.
+
+**xonsh**
+
+.. code-block:: xonsh
+
+    $ echo foo"bar"baz
+    foo"bar"baz
+
+    $ echo --key="value"
+    --key="value"
+
+You can think of these being equivalent to,
+
+
+**xonsh**
+
+.. code-block:: xonsh
+
+    $ echo 'foo"bar"baz'
+    foo"bar"baz
+
+    $ echo '--key="value"'
+    --key="value"
+
+This is yet-another major point of departure for xonsh from traditional
+shells. However, the xonsh subprocess string handling is
+consistent and predictable.
+
+Further Reading
+===============
+For deeper details on the great string debate, please feel free to read
+and comment at:
+
+* `To Quote or Not Quote <https://github.com/xonsh/xonsh/issues/1432>`_
+* `Quote removal in subprocess mode does not behave as expected <https://github.com/xonsh/xonsh/issues/621>`_


Code review comments:
diff_hunk: @@ -0,0 +1,131 @@
+.. _tutorial_subproc_strings:
+
+************************************
+Tutorial: Subprocess Strings
+************************************
+Strings in xonsh follow two simple rules:
+
+1. Strings in xonsh are always parsed in the same way, and
+2. Python always wins!
+
+Together these rules mean that **even strings in subprocess mode are treated
+like Python strings!** This will (help) preserve your sanity.
+
+No Escape
+=========
+Xonsh strings are exactly like Python strings everywhere. Xonsh uses
+exactly the same escape characters that Python dones; no more and no less.
path: docs/tutorial_subproc_strings.rst
text: @user1:
*dones -> does

diff_hunk: @@ -0,0 +1,131 @@
+.. _tutorial_subproc_strings:
+
+************************************
+Tutorial: Subprocess Strings
+************************************
+Strings in xonsh follow two simple rules:
+
+1. Strings in xonsh are always parsed in the same way, and
+2. Python always wins!
+
+Together these rules mean that **even strings in subprocess mode are treated
+like Python strings!** This will (help) preserve your sanity.
+
+No Escape
+=========
+Xonsh strings are exactly like Python strings everywhere. Xonsh uses
+exactly the same escape characters that Python dones; no more and no less.
+This is different from other shells, which have a different set of escape
+sequences than Python has.  Notably, many sh-langs allow you to escape
+spaces with ``\ `` (backslash-space).
+
+**bash**
+
+.. code-block:: bash
+
+    $ echo A\ Single\ Argument
+    A Single Argument
+
+In the above example, since the spaces are escaped, the ``echo`` command
+only receves a single argument. Xonsh does not allow this. If you were
+to try this in xonsh, you'd see:
+
+**xonsh**
+
+.. code-block:: xonsh
+
+    $ echo Actually\ Three\ Arguments
+    Actually\ Three\ Arguments
+
+In this example, echo recives three arguments:: ``"Actually\\"``, ``"Three\\"``,
+and ``"Arguments"``. Instead, xonsh requires you to use quotes in order to
+pass in a single argument:
+
+**xonsh** or **bash**
+
+.. code-block:: xonsh
+
+    $ echo "A Single Argument"
+    A Single Argument
+
+Using quotes is arguably what should have been done in sh-lang in the
+first place.
+
+.. note::
+
+    When in doubt in subprocess mode, use quotes!
+
+
+Justification
+=============
+The reasons for not having additional escape sequences, as in sh-langs, are:
+
+1. Escape charaters can get overwhemlingly ugly, fast.
+2. We have escape characters, they are called quotes :)
+3. We have literal input in subprocess mode via macros.
+
+On this last point, if you don't already know about
+`Subprocess Macros <tutorial_macros.html#subprocess-macros>`_,
+these allow all input following an ``!`` to be treated as a single argument.
+For example,
+
+**xonsh**
+
+.. code-block:: xonsh
+
+    $ echo! A  Single     Argument
+    A  Single     Argument
+
+Subprocess macros are the ultimate escape mechanism.
+
+The Quotes Stay
+===============
+In sh-langs, internal quote characters are removed. For instance:
+
+.. code-block:: bash
+
+    $ echo foo"bar"baz
+    foobarbaz
+
+    $ echo --key="value"
+    --key=value
+
+Xonsh considers this behaviour insane. Instead, xonsh treats these
+arguments as if they were surrounded in another, outer level of
+quotation (``'foo"bar"baz'``). Xonsh will keep the quotation marks
+when leading and trailing quotes are not matched.
+
+**xonsh**
+
+.. code-block:: xonsh
+
+    $ echo foo"bar"baz
+    foo"bar"baz
+
+    $ echo --key="value"
+    --key="value"
+
+You can think of these being equivalent to,
+
+
+**xonsh**
+
+.. code-block:: xonsh
+
+    $ echo 'foo"bar"baz'
+    foo"bar"baz
+
+    $ echo '--key="value"'
+    --key="value"
+
+This is yet-another major point of departure for xonsh from traditional
path: docs/tutorial_subproc_strings.rst
text: @user1:
No hyphen