Repository: pallets/click

Issue:
Shell completion does't parse escaped slashes or spaces correctly
When a value with an escape string, for instance `my\ file`, is passed to a `click.ParamType`, the `incomplete` value which is passed to `click.ParamType.shell_complete` will only be `"my\ "` instead of `my file`.

### Expected Behavior

```python
import click

@click.command()
@click.argument("name", type=click.Choice(["my file", "my folder", "my\\ file", "other"]))
def cli(name):
    click.echo(name)

if __name__ == '__main__':
    cli()
```

Completing `my\ f` should be interpreted as the Python string `"my f"` and match the first two values.

```
$ example my\ f<TAB>
my\ file
my\ folder
```

### Actual Behavior

Completing `my\ f` is interpreted as the string `"my\\ f"` and matches the third value.

```shell
$ example my\ f<TAB>
my\\\ file
```

### Environment

* Python version: 3.8.6
* Click version: 8.0.0.dev0


Discussion:
`"\ "` isn't a valid escape sequence in a Python string, so Python will escape the escape character. If you try it in the interpreter, you get.

```pycon
>>> "\ "
'\\ '
```

Finally, the shell you're using escapes spaces on its own, so you get the third escape character.
Maybe you misunderstood, my issues is not with the triple backslash. The issue is that `incomplete = "my\"` while it should be `incomplete  = "my f"` because the actual argument to be completed is "my f".
I've updated the issue with a simpler example and direct explanations for expected and actual behavior.

The issue seems to be with how `split_arg_string` handles slashes and spaces. It's trying to emulate how `sys.argv` would be populated to apply it to the `CWORDS` env var, but doesn't get it right in all cases.

Adding some debugging code to `ZshComplete.get_completion_args`, when `my\ f` is completed:

```
os.environ['COMP_WORDS']='example my\\ f'
cwords=['example', 'my\\', 'f']
cword=1
args=[]
incomplete='my\\'
```

The shell is passing `my\ a`, which is the Python string `"my\\ f"`. This is getting parsed into two values `"my\\"` and `"f"`. Instead, it should get parsed into one value `"my\\ f"`.

You can find some other weird behaviors by tab completing earlier, or with a single opening quote, or within quotes. The splitter should be able to handle these.

That's about the extent of the time I can devote to this for now though. Your best bet for getting this fixed before 8.0 is released is to make a PR.
Ok, thanks. Yes, that example was a bit of a mess.

From a quick look, `split_arg_string` attempts to do something similar to the quite complex `shlex.split`. Any reason why you are not using the latter? Bad performance?
Because `shlex.split` expects a complete command line string, but completion by definition needs to work with an incomplete string.

```python
shlex.split("example my\\")
```
```pytb
ValueError: No escaped character
```
```python
shlex.split("example 'my ")
```
```pytb
ValueError: No closing quotation
```
You might be able to get around that by manually iterating over a `shlex.shlex` object and doing some cleanup when it raises an error.
I was about to propose exactly that! Any remainder after a `ValueError` can be taken as an incomplete argument.
Those two examples should be the only cases where `shlex.split` gets into a failed state, so I think handling those cases with `shlex.shelx` should be enough.
Hopefully. Also, unpaired quotes should ideally be preserved during the completion process: An input of `my\ f` should be completed to `my\ file` while an input of `'my f` should be completed to `'my file'`.

I'll need to think a bit about this and see if can submit a PR that fixes more than it breaks. But not tonight, it's already late here.

Resolving patch:
diff --git a/src/click/parser.py b/src/click/parser.py
--- a/src/click/parser.py
+++ b/src/click/parser.py
@@ -21,7 +21,6 @@
 # maintained by the Python Software Foundation.
 # Copyright 2001-2006 Gregory P. Ward
 # Copyright 2002-2006 Python Software Foundation
-import re
 from collections import deque
 
 from .exceptions import BadArgumentUsage
@@ -103,22 +102,37 @@ def normalize_opt(opt, ctx):
 
 
 def split_arg_string(string):
-    """Given an argument string this attempts to split it into small parts."""
-    rv = []
-    for match in re.finditer(
-        r"('([^'\\]*(?:\\.[^'\\]*)*)'|\"([^\"\\]*(?:\\.[^\"\\]*)*)\"|\S+)\s*",
-        string,
-        re.S,
-    ):
-        arg = match.group().strip()
-        if arg[:1] == arg[-1:] and arg[:1] in "\"'":
-            arg = arg[1:-1].encode("ascii", "backslashreplace").decode("unicode-escape")
-        try:
-            arg = type(string)(arg)
-        except UnicodeError:
-            pass
-        rv.append(arg)
-    return rv
+    """Split an argument string as with :func:`shlex.split`, but don't
+    fail if the string is incomplete. Ignores a missing closing quote or
+    incomplete escape sequence and uses the partial token as-is.
+
+    .. code-block:: python
+
+        split_arg_string("example 'my file")
+        ["example", "my file"]
+
+        split_arg_string("example my\\")
+        ["example", "my"]
+
+    :param string: String to split.
+    """
+    import shlex
+
+    lex = shlex.shlex(string, posix=True)
+    lex.whitespace_split = True
+    lex.commenters = ""
+    out = []
+
+    try:
+        for token in lex:
+            out.append(token)
+    except ValueError:
+        # Raised when end-of-string is reached in an invalid state. Use
+        # the partial token as-is. The quote or escape character is in
+        # lex.state, not lex.token.
+        out.append(lex.token)
+
+    return out
 
 
 class Option:


Test patch:
diff --git a/tests/test_parser.py b/tests/test_parser.py
new file mode 100644
--- /dev/null
+++ b/tests/test_parser.py
@@ -0,0 +1,17 @@
+import pytest
+
+from click.parser import split_arg_string
+
+
+@pytest.mark.parametrize(
+    ("value", "expect"),
+    [
+        ("cli a b c", ["cli", "a", "b", "c"]),
+        ("cli 'my file", ["cli", "my file"]),
+        ("cli 'my file'", ["cli", "my file"]),
+        ("cli my\\", ["cli", "my"]),
+        ("cli my\\ file", ["cli", "my file"]),
+    ],
+)
+def test_split_arg_string(value, expect):
+    assert split_arg_string(value) == expect