Repository: pallets/click (python)

Issue:
Shell completion does't parse escaped slashes or spaces correctly
When a value with an escape string, for instance `my\ file`, is passed to a `click.ParamType`, the `incomplete` value which is passed to `click.ParamType.shell_complete` will only be `"my\ "` instead of `my file`.

### Expected Behavior

```python
import click

@click.command()
@click.argument("name", type=click.Choice(["my file", "my folder", "my\\ file", "other"]))
def cli(name):
    click.echo(name)

if __name__ == '__main__':
    cli()
```

Completing `my\ f` should be interpreted as the Python string `"my f"` and match the first two values.

```
$ example my\ f<TAB>
my\ file
my\ folder
```

### Actual Behavior

Completing `my\ f` is interpreted as the string `"my\\ f"` and matches the third value.

```shell
$ example my\ f<TAB>
my\\\ file
```

### Environment

* Python version: 3.8.6
* Click version: 8.0.0.dev0

Pull request description:
Shell completion escape handling
This PR fixes #1708 by using Python's `shlex` module to parse and split argument strings for shell completion. Incomplete arguments where splitting with `shlex.split` fails with a `ValueError` are handled by inspecting the state of the parser after the failure. There are only two such failures: 

1. A missing closing quote for an incomplete argument. This is handled by appending the missing quote. Therefore, shell completion suggestions for an argument `'path to` will be generated with `incomplete = "path to"`.
2. An incomplete escape sequence. This is handled by removing the incomplete sequence. Therefore, shell completion suggestions for an argument `'path\` will be generated with `incomplete = "path"`. This mirrors the behaviour of zsh and possibly other shells.

<!--
Ensure each step in CONTRIBUTING.rst is complete by adding an "x" to each box below.
If only docs were changed, these aren't relevant and can be removed.
-->

Checklist:

- [x] Add tests that demonstrate the correct behavior of the change. Tests should fail without the change.
- [x] Add or update relevant docs, in the docs folder and in code.
- [x] Add an entry in `CHANGES.rst` summarizing the change and linking to the issue.
- [x] Add `.. versionchanged::` entries in any relevant code docs.
- [x] Run `pre-commit` hooks and fix any issues.
- [x] Run `pytest` and `tox`, no tests failed.


Patch:
diff --git a/CHANGES.rst b/CHANGES.rst
index cb6cd13..6faaa89 100644
--- a/CHANGES.rst
+++ b/CHANGES.rst
@@ -107,6 +107,8 @@ Unreleased
         The shell now comes first, such as ``{shell}_source`` rather
         than ``source_{shell}``, and is always required.
 
+-   Completion correctly parses command line strings with incomplete
+    quoting or escape sequences. :issue:`1708`
 -   Extra context settings (``obj=...``, etc.) are passed on to the
     completion system. :issue:`942`
 -   Include ``--help`` option in completion. :pr:`1504`
diff --git a/src/click/parser.py b/src/click/parser.py
index 92c9354..d730e01 100644
--- a/src/click/parser.py
+++ b/src/click/parser.py
@@ -21,7 +21,6 @@ Copyright 2002-2006 Python Software Foundation. All rights reserved.
 # maintained by the Python Software Foundation.
 # Copyright 2001-2006 Gregory P. Ward
 # Copyright 2002-2006 Python Software Foundation
-import re
 from collections import deque
 
 from .exceptions import BadArgumentUsage
@@ -103,22 +102,37 @@ def normalize_opt(opt, ctx):
 
 
 def split_arg_string(string):
-    """Given an argument string this attempts to split it into small parts."""
-    rv = []
-    for match in re.finditer(
-        r"('([^'\\]*(?:\\.[^'\\]*)*)'|\"([^\"\\]*(?:\\.[^\"\\]*)*)\"|\S+)\s*",
-        string,
-        re.S,
-    ):
-        arg = match.group().strip()
-        if arg[:1] == arg[-1:] and arg[:1] in "\"'":
-            arg = arg[1:-1].encode("ascii", "backslashreplace").decode("unicode-escape")
-        try:
-            arg = type(string)(arg)
-        except UnicodeError:
-            pass
-        rv.append(arg)
-    return rv
+    """Split an argument string as with :func:`shlex.split`, but don't
+    fail if the string is incomplete. Ignores a missing closing quote or
+    incomplete escape sequence and uses the partial token as-is.
+
+    .. code-block:: python
+
+        split_arg_string("example 'my file")
+        ["example", "my file"]
+
+        split_arg_string("example my\\")
+        ["example", "my"]
+
+    :param string: String to split.
+    """
+    import shlex
+
+    lex = shlex.shlex(string, posix=True)
+    lex.whitespace_split = True
+    lex.commenters = ""
+    out = []
+
+    try:
+        for token in lex:
+            out.append(token)
+    except ValueError:
+        # Raised when end-of-string is reached in an invalid state. Use
+        # the partial token as-is. The quote or escape character is in
+        # lex.state, not lex.token.
+        out.append(lex.token)
+
+    return out
 
 
 class Option:


Test patch:
diff --git a/tests/test_parser.py b/tests/test_parser.py
new file mode 100644
index 0000000..964f9c8
--- /dev/null
+++ b/tests/test_parser.py
@@ -0,0 +1,17 @@
+import pytest
+
+from click.parser import split_arg_string
+
+
+@pytest.mark.parametrize(
+    ("value", "expect"),
+    [
+        ("cli a b c", ["cli", "a", "b", "c"]),
+        ("cli 'my file", ["cli", "my file"]),
+        ("cli 'my file'", ["cli", "my file"]),
+        ("cli my\\", ["cli", "my"]),
+        ("cli my\\ file", ["cli", "my file"]),
+    ],
+)
+def test_split_arg_string(value, expect):
+    assert split_arg_string(value) == expect