Repository: pallets/click

Issue:
Expand globs in arguments on Windows?
I'm using a variadic `Path` argument:

```python
@click.command()
@click.argument(
    "paths", nargs=-1, type=click.Path(exists=True, dir_okay=False)
)
def whatever(paths)
   ...
```

When I run `myprogram.py *.txt`, then on unix of course it works, because the shell expands on the glob. But on Windows, glob expansion is the responsibility of the program being invoked, and click doesn't appear to do it – even in a case like this where it has all the information needed to tell that glob expansion is appropriate. And click kind of has to do it, because click wants to validate the paths, and this has to happen *after* glob expansion – if I pass `*.txt`, it says `Error: Invalid value for "paths": Path "*.txt" does not exist.`, and my code doesn't even get invoked, so I can't even fix up the globs myself, except by disabling all of click's validation and then doing it all myself.


Discussion:
* Should it be Click's (built-in) job to make up for cmd not having globs?
* How can we safely tell that a name is a glob?  There are other patterns besides `*`.
* What if the name comes from Bash and `*.txt` was actually the name of the file after expansion?
* Do we also want to expand `~`?
* If we add glob expansion to parameters, do we also need to do it for env vars on Unix too? For example, if a parameter has an env var `NAME="*.txt"`, no expansion is done, Click currently expects you to handle that.

My first reaction is that this shouldn't be done by Click, although it should be possible to subclass `Argument.process_value()` or `Path` to work with globs.
> What if the name comes from Bash and *.txt was actually the name of the file after expansion?

IMHO such behavior would have to be completely disabled on non-Windows systems.

Maybe the least strange/error-prone solution would be to have a `windows_expand_globs` argument that can be set to True if you want glob expansion on Windows.
> Should it be Click's (built-in) job to make up for cmd not having globs?

The Windows convention that it is every program's job to make up for cmd not having globs, yes. "Annoying thing that every cli app has to do" seems like it falls within click's scope.

> How can we safely tell that a name is a glob? There are other patterns besides *.

I believe in traditional Windows globbing the metacharacters are `*`, `?` only. The python `glob` module also does `[]`. You probably can't exactly match what, like, MSVC would do, but that's ok, it's Windows, it's not your fault and no one expects everything to work consistently.

> What if the name comes from Bash and *.txt was actually the name of the file after expansion?

As @thiefmaster notes, you'd only enable enable this on Windows. But what if someone is using mingw bash on Windows? Well, then you might double-expand, just like every other traditional Windows cli program would in the same situation.

> Do we also want to expand `~`?

That would be an interesting option for `File` and `Path` arguments yeah, though it's more of a Unix thing so the history is a bit different. I can see a good argument for doing `expanduser` by default on `File` and `Path` arguments, with a switch available to turn it off.

> If we add glob expansion to parameters, do we also need to do it for env vars on Unix too? For example, if a parameter has an env var NAME="*.txt", no expansion is done, Click currently expects you to handle that.

I'm not really familiar with how click uses envvars. As a Unix user I would expect `NAME=~/foo.txt` to work, and be surprised if `NAME=*.txt` worked. I don't know what people would expect for envvar handling on Windows.
> "Annoying thing that every cli app has to do" seems like it falls within click's scope.

Agree.

> I believe in traditional Windows globbing the metacharacters are *, ? only.

It would be less efficient, but the easiest way would be to always pass values through expanduser and glob first.

Is there a way to signal that something glob-like *isn't* a glob, like single quotes on Unix?

One last thing I thought of. What happens when using `invoke` (for testing or for dispatching other commands) now? On Windows, `invoke(args=["subcommand", "*.txt"])` would expand, but on Unix it wouldn't, which I can see causing inconsistent behavior.
> Is there a way to signal that something glob-like isn't a glob, like single quotes on Unix?

I'm not sure. In Windows, quote handling is also done in process (basically you just get a raw command line string and are free to do whatever with it), so in principle you could use quotes to disable globbing for specific arguments. But in Python, the quote handling happens early in interpreter startup, so you can't see them in `sys.argv`. I guess if you're really masochistic you could call [`GetCommandLine`](https://msdn.microsoft.com/en-us/library/windows/desktop/ms683156(v=vs.85).aspx) and try to reparse it, but i can't imagine this working well given all the ways you python can be invoked (e.g. `python -m mycli`), plus it'd be a ton of fiddly work for unclear benefit.

Python's glob module lets you use `[]` for quoting. (E.g., the pattern `[*].txt` matches the literal filename `*.txt`.) Not very intuitive, but it is possible, and I guess Windows users are used to weird edge cases when trying to use the cli to refer to files with `*` in the name?

> One last thing I thought of. What happens when using invoke (for testing or for dispatching other commands) now? On Windows, `invoke(args=["subcommand", "*.txt"])` would expand, but on Unix it wouldn't, which I can see causing inconsistent behavior.

I don't know enough about click and `invoke` to say anything useful here. (I've spent about an hour with click so far :-).) Tough I guess you could make the argument that the behavior *is* inconsistent on the two platforms, intentionally so, and your tests should reflect that?
Hello, I came here from the Black project, because running `black *.py` doesn't work on Windows Command Prompt or PowerShell and I've traced the problem to how Click handles wildcards. Click not handling this results in wildcards simply not working on Windows for **every** program that uses Click, which seems like a large hole.

> How can we safely tell that a name is a glob? There are other patterns besides *.

I think leaning on Python's `glob` module is suitable enough for handling * and ? (njsmith is correct that [] isn't used on Windows, but we'd get it for free with `glob`.) And anyways, let's not let perfect be the enemy of good; even just getting some features working would be good. I just want `black *.py` to work on Windows.

> What if the name comes from Bash and *.txt was actually the name of the file after expansion?

To reiterate njsmith, this would have to be a Windows-only feature. The * and ? characters are illegal to have in files on Windows anyway (I assume this is an NTFS thing.)

> Do we also want to expand `~`?

Nah. This isn't a Windows command line idiom. It'd be easier to leave it unimplemented, given that `glob` also doesn't implement it.

Anyway, my comments mostly reiterate what njsmith said. I'd just like to shed more light on this issue because "wildcards don't work on Windows" seems like a pretty big deal for every program that uses Click.
> What happens when using `invoke` (for testing or for dispatching other commands) now? On Windows, `invoke(args=["subcommand", "*.txt"])` would expand, but on Unix it wouldn't, which I can see causing inconsistent behavior.

Thinking about this again, processing should only happen in `BaseCommand.main`, so `Context.invoke` isn't affected. `invoke` is already expected to be passed Python values, not unparsed command line values.

So the solution for this is to add a call to `glob` in `BaseCommand.main` only on Windows.
I'll like to pick this up
Note that `*.txt` will only expand if there actually are any `.txt` files, otherwise Unix shells pass on the literal `*.txt` and you get the same error as in the op. Not going to do anything about this, since it's standard shell behavior, but it can be unexpected so I wanted to mention it. `glob.glob` returns an empty list if nothing matches, so I'll replace that with the original pattern in that case to match this behavior.

I think we should also call `os.path.expanduser` and `os.path.expandvars` on each argument. If we're expanding globs for better consistency between Windows and Unix, it makes sense to do the other expansions that Unix shells do and Python provides. I'll also enable `recursive=True` for `glob` so that `**` is expanded.

Resolving patch:
diff --git a/src/click/core.py b/src/click/core.py
--- a/src/click/core.py
+++ b/src/click/core.py
@@ -30,6 +30,7 @@
 from .types import convert_type
 from .types import IntRange
 from .utils import _detect_program_name
+from .utils import _expand_args
 from .utils import echo
 from .utils import make_default_short_help
 from .utils import make_str
@@ -903,9 +904,6 @@ def main(
         This method is also available by directly calling the instance of
         a :class:`Command`.
 
-        .. versionadded:: 3.0
-           Added the `standalone_mode` flag to control the standalone mode.
-
         :param args: the arguments that should be used for parsing.  If not
                      provided, ``sys.argv[1:]`` is used.
         :param prog_name: the program name that should be used.  By default
@@ -926,6 +924,13 @@ def main(
                                 of :meth:`invoke`.
         :param extra: extra keyword arguments are forwarded to the context
                       constructor.  See :class:`Context` for more information.
+
+        .. versionchanged:: 8.0
+            When taking arguments from ``sys.argv`` on Windows, glob
+            patterns, user dir, and env vars are expanded.
+
+        .. versionchanged:: 3.0
+           Added the ``standalone_mode`` parameter.
         """
         # Verify that the environment is configured correctly, or reject
         # further execution to avoid a broken script.
@@ -933,6 +938,9 @@ def main(
 
         if args is None:
             args = sys.argv[1:]
+
+            if os.name == "nt":
+                args = _expand_args(args)
         else:
             args = list(args)
 
diff --git a/src/click/utils.py b/src/click/utils.py
--- a/src/click/utils.py
+++ b/src/click/utils.py
@@ -1,5 +1,6 @@
 import os
 import sys
+import typing as t
 
 from ._compat import _default_text_stderr
 from ._compat import _default_text_stdout
@@ -482,3 +483,48 @@ def _detect_program_name(path=None, _main=sys.modules["__main__"]):
         py_module = f"{py_module}.{name}"
 
     return f"python -m {py_module.lstrip('.')}"
+
+
+def _expand_args(
+    args: t.Iterable[str],
+    *,
+    user: bool = True,
+    env: bool = True,
+    glob_recursive: bool = True,
+) -> t.List[str]:
+    """Simulate Unix shell expansion with Python functions.
+
+    See :func:`glob.glob`, :func:`os.path.expanduser`, and
+    :func:`os.path.expandvars`.
+
+    This intended for use on Windows, where the shell does not do any
+    expansion. It may not exactly match what a Unix shell would do.
+
+    :param args: List of command line arguments to expand.
+    :param user: Expand user home directory.
+    :param env: Expand environment variables.
+    :param glob_recursive: ``**`` matches directories recursively.
+
+    .. versionadded:: 8.0
+
+    :meta private:
+    """
+    from glob import glob
+
+    out = []
+
+    for arg in args:
+        if user:
+            arg = os.path.expanduser(arg)
+
+        if env:
+            arg = os.path.expandvars(arg)
+
+        matches = glob(arg, recursive=glob_recursive)
+
+        if not matches:
+            out.append(arg)
+        else:
+            out.extend(matches)
+
+    return out


Test patch:
diff --git a/tests/test_utils.py b/tests/test_utils.py
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -429,3 +429,13 @@ def __init__(self, package_name):
 )
 def test_detect_program_name(path, main, expected):
     assert click.utils._detect_program_name(path, _main=main) == expected
+
+
+def test_expand_args(monkeypatch):
+    user = os.path.expanduser("~")
+    assert user in click.utils._expand_args(["~"])
+    monkeypatch.setenv("CLICK_TEST", "hello")
+    assert "hello" in click.utils._expand_args(["$CLICK_TEST"])
+    assert "setup.cfg" in click.utils._expand_args(["*.cfg"])
+    assert os.path.join("docs", "conf.py") in click.utils._expand_args(["**/conf.py"])
+    assert "*.not-found" in click.utils._expand_args(["*.not-found"])