Repository: pypa/setuptools

Issue:
glob patterns for data_files
Would be great to be able to use glob patterns for `data_files`, so that one can write something like this:

```
[options.data_files]
share/myapp/doc = doc/*.md
```

This would be consistent with the way `package_data` works.


Discussion:
I have no great objections to this, though do we need some indication that it's to be interpreted as a glob pattern and not a literal string? Even if we limit it just to supporting `*`, asterisk is a valid character in a file or folder.
`*` in file names sounds like a weird edge case to me. It is going to cause trouble for users in many places. I don't think it is worth supporting, but then I haven't checked if it is taken into account in other parts of setuptools.
@agateau I'm not really willing to break backwards compatibility by changing the way these strings are interpreted from being paths to being glob patterns. An asterisk is a valid character for a file name, it's definitely something we want to support, regardless of whether it will break in other places as well.

We just need a way to differentiate between "interpret this as a path" and "interpret this as a glob", hopefully it's not terribly difficult.
@pganssle `*` is already interpreted as a wildcard in a number of places, just see the number of occurrences in the [documentation](https://setuptools.readthedocs.io/en/latest/setuptools.html) all over the place.
It's really **not** interpreting it as a wildcard in a single place that breaks consistency with all the rest of the API.
@hydrargyrum Like I said above, I'm in favor of adding a way to do this, but it would be a backwards-incompatible change *and* would be removing the ability to literally specify files with asterisks in their names if we just start interpreting all file paths as glob literals.

As much as I am in favor of making common patterns *easier* in `setup.cfg`, it's still pretty easy to use `glob.glob` in `setup.py`, whereas if we make this backwards-incompatible change, it will be impossible to get the exact current behavior (likely in either `setup.py` or `setup.cfg`).

If we want to make actual progress on this issue, I think we need to come up with possible backwards-compatible interfaces that would allow this to move forward. I think they will broadly fall into four categories:

1. Special syntax for the specifier string that would not be a valid path on any supported OS, but would allow us to indicate that something is a glob pattern, e.g. `::glob::*.md` (not sure if that is an invalid path on all supported OSes, just guessing)
2. Some sort of wrapper type, like `GlobPattern('*.md')`
3. A boolean option toggle defaulting to `False` that specifies that `data_files` should be interpreted as glob patterns.
4. A new option that specifies glob patterns, like `data_files_pattern`, which is *always* interpreted as a glob pattern.

Number 1 is probably the best option, but it's not terribly discoverable, and right now at least *only* `data_files` would requires this special syntax, which is not great. Number 2 would be great if this were for `setup.py` only, but I don't think there's any good way to specify a custom wrapper type in `setup.cfg`.

Numbers 3 and 4 are ugly mainly because you now have two different options to specify the same thing, and it may not be obvious to people what they do.

If anyone has better options or wants to argue for one of these, please do let me know, because none of the options seem terribly appealing to me.
Number 1 looks like the best option to me as well, because it would work for setup.py and setup.cfg, but this syntax should be usable for all keys were a glob pattern can be used.

While I respect your desire to not break compatibility, I have yet to find a package shipping files with asterisks in their names. I may be wrong on this, but I would be surprised if simply changing the code to accept asterisks would wreak so much havoc. Quoting the Zen of Python:

> practicality beats purity.
The only invalid path on Unix is a null byte (and "/" cannot exist in filenames but validly denotes a directory separator).

It's theoretically possible but highly unlikely people wanted a literal asterisk in their file names, but by the same token that ":" is illegal on Windows, and in fact also illegal even on Unix if you happen to be using fat32 or NTFS *without* the POSIX namespace (the restriction is part of the filesystem spec)...

... the asterisk is *also* illegal in filenames on Windows (and at the filesystem layer on fat32 and NTFS).

So, such code depending on literal asterisks would unexpectedly fail on Windows, and also on that one person's USB drive used for copying around a personal python environment from Unix system to Unix system.

One could argue backwards compatibility with something inherently broken is not such an important priority, and even if someone did need that behavior, the asterisk would simply expand to multiple files including the one with the literal asterisk. So the package would collect too many files, but at least the desired file would in fact be there.

If we want to make it super unlikely for anyone to ever stumble across this by using some prefix marker like option 1, I don't think `::glob::` provides additional utility over `glob:` which would have symmetry with existing directives like `file:`, `attr:`, `find:`, and `find_namespace:`.
I discovered this issue while trying to migrate some other project off of pbr and onto the builtin setup.cfg support (and wondering why the obvious thing did not work). Unfortunately, the project uses pbr's support for
```
[files]
data_files =
    share/foo = <glob>
```

so it cannot be mapped to `[options.data_files]` no matter how much I'd like... I'm not sure if I should propose a change that involves "oh, also reinstate setup.py as the logic driver in order to glob for files with glob.glob()".
Recently discovered the same thing and found this thread.

I have begrudgingly reinstituted a setup.py just to do some glob() calls, but I feel a new directive like `glob:` could be in-line with mannerisms of the setup.cfg syntax.

Resolving patch:
diff --git a/setuptools/config.py b/setuptools/config.py
--- a/setuptools/config.py
+++ b/setuptools/config.py
@@ -9,6 +9,7 @@
 from collections import defaultdict
 from functools import partial
 from functools import wraps
+from glob import iglob
 import contextlib
 
 from distutils.errors import DistutilsOptionError, DistutilsFileError
@@ -255,6 +256,34 @@ def _parse_list(cls, value, separator=','):
 
         return [chunk.strip() for chunk in value if chunk.strip()]
 
+    @classmethod
+    def _parse_list_glob(cls, value, separator=','):
+        """Equivalent to _parse_list() but expands any glob patterns using glob().
+
+        However, unlike with glob() calls, the results remain relative paths.
+
+        :param value:
+        :param separator: List items separator character.
+        :rtype: list
+        """
+        glob_characters = ('*', '?', '[', ']', '{', '}')
+        values = cls._parse_list(value, separator=separator)
+        expanded_values = []
+        for value in values:
+
+            # Has globby characters?
+            if any(char in value for char in glob_characters):
+                # then expand the glob pattern while keeping paths *relative*:
+                expanded_values.extend(sorted(
+                    os.path.relpath(path, os.getcwd())
+                    for path in iglob(os.path.abspath(value))))
+
+            else:
+                # take the value as-is:
+                expanded_values.append(value)
+
+        return expanded_values
+
     @classmethod
     def _parse_dict(cls, value):
         """Represents value as a dict.
@@ -711,5 +740,5 @@ def parse_section_data_files(self, section_options):
 
         :param dict section_options:
         """
-        parsed = self._parse_section_to_dict(section_options, self._parse_list)
+        parsed = self._parse_section_to_dict(section_options, self._parse_list_glob)
         self['data_files'] = [(k, v) for k, v in parsed.items()]


Test patch:
diff --git a/setuptools/tests/test_config.py b/setuptools/tests/test_config.py
--- a/setuptools/tests/test_config.py
+++ b/setuptools/tests/test_config.py
@@ -893,6 +893,41 @@ def test_data_files(self, tmpdir):
             ]
             assert sorted(dist.data_files) == sorted(expected)
 
+    def test_data_files_globby(self, tmpdir):
+        fake_env(
+            tmpdir,
+            '[options.data_files]\n'
+            'cfg =\n'
+            '      a/b.conf\n'
+            '      c/d.conf\n'
+            'data = *.dat\n'
+            'icons = \n'
+            '      *.ico\n'
+            'audio = \n'
+            '      *.wav\n'
+            '      sounds.db\n'
+        )
+
+        # Create dummy files for glob()'s sake:
+        tmpdir.join('a.dat').write('')
+        tmpdir.join('b.dat').write('')
+        tmpdir.join('c.dat').write('')
+        tmpdir.join('a.ico').write('')
+        tmpdir.join('b.ico').write('')
+        tmpdir.join('c.ico').write('')
+        tmpdir.join('beep.wav').write('')
+        tmpdir.join('boop.wav').write('')
+        tmpdir.join('sounds.db').write('')
+
+        with get_dist(tmpdir) as dist:
+            expected = [
+                ('cfg', ['a/b.conf', 'c/d.conf']),
+                ('data', ['a.dat', 'b.dat', 'c.dat']),
+                ('icons', ['a.ico', 'b.ico', 'c.ico']),
+                ('audio', ['beep.wav', 'boop.wav', 'sounds.db']),
+            ]
+            assert sorted(dist.data_files) == sorted(expected)
+
     def test_python_requires_simple(self, tmpdir):
         fake_env(
             tmpdir,