Repository: pypa/setuptools

Issue:
package_data not doing recursive glob calls
Even though glob supports it, glob support for recursive ** searching is not enabled when specifying package_data. This is not clear in the documentation.

https://github.com/pypa/setuptools/blob/a94ccbf404a79d56f9b171024dee361de9a948da/setuptools/command/build_py.py#L106

This means that only single `*` searches in a specific directory are supported.  By changing the `glob` function to support recursive, this would allow us to specify `**` patterns to a directory instead of having to specify every sub-directory in the structure.

Reference:
https://github.com/python/cpython/blob/master/Lib/glob.py#L18


Discussion:
Yes, I can confirm that without calling `glob(pattern, recursive=True)`, the `**` pattern isn't expanded.

What this issue needs is more background. What is a use-case where having recursive globbing support would be helpful?

More importantly, this routine is for `find_data_files` in a package. Since a Python package is already defined as a directory containing files, it may not even be viable for a package to contain directories of data. Even if you're able to cause subdirectories and their contents to be added as package data, I'm not sure that would be a supported scenario. Before adding facilities to allow a packager to more easily add this data to a package, we should be sure that's something that the packaging systems endorses/supports.
Thanks for getting back. I have a nested directory of configuration files (.yml) and a nested directory of validation files (.json). Unfortunately, I have to go through each sub directory in order to include them in the package. This works and is working for my current use case, but every time I add addition sub-directories, I have to remember to add another path.
```
[options.package_data]
module = config/A/1/*.yml, config/B/1/*.yml, validation/A/*.json, validation/B/*.json
```
I would prefer to just have to do this (in fact I originally tried this, but didn't work which is why I reported the issue).
```
[options.package_data]
module = config/**/*.yml, validation/**/*.json
```
I also tried `*.yml, *.json` that the docs alluded to working, but it does not (same issue non recursive so only works on root directory files).
```
[options.package_data]
# If any package contains *.txt or *.rst files, include them:
* = *.txt, *.rst
hello = *.msg
```

Agreed, it would be massively helpful if I could use the `**` glob in setup.py's `package_data`. I deploy Django apps which include many templates in potentially endless subdirectories. There is no glob pattern except `**/*.html` to describe their paths.
I just found this issue working with Django templates as well.  Recursive glob support would be tremendously helpful when packaging Django apps that include templates
I got the same problem with stub files for typing. These data files are *.pyi and may be spread at multiple folder levels.

I have two workarounds to this difficulty.

One is with a custom implementation:

```python
import os
from typing import List

def find_stubs(top: str) -> List[str]:
    stubs = []
    for root, _, files in os.walk(top):
        dir = root.split(os.path.sep, 1)[-1] # rid of 'top'
        for f in files:
            if f.endswith('.pyi'):
                if os.path.sep in root:
                    f = os.path.join(dir, f)
                stubs.append(f)
    return stubs

setup(
    …
    package_data={'foo': find_stubs('foo')},
    …
)
```

Another one is more generic, with a manifest template:

```python
setup(
    …
    include_package_data=True,
    …
)
```
MANIFEST.in
```
recursive-include foo *.pyi
```

Working with Django, I prefer the second way because I also need to include many other data files:
```
recursive-include foo/locale *.po *.mo
recursive-include foo/static *
recursive-include foo/templates *.html *.txt
graft docs
prune docs/_build
```

I would like to add the issue that for my cython code setuptools does not copy the data files to `site-packages`. 

Sources list my `pxd` files but they are not transferred to the install  directory. I have both tried using `MANIFEST.in` as well as `package_data = {"": "**pxd **pyx".split()}`. A workaround form was to `use data_files` which is not preferred ...
Bump.

At a glance, the situation looks worse than described when there are multiple packages in a single source tree.

Unfortunately I can't quite get resolution on what's happening when I run `bdist_wheel`. Is there an environment variable or something that I can set to see how the `package_data` is being assembled?
It sounds as if there's a reasonable need for this issue, and now that I see that using `include_package_data` would effectively include the same contents as a recursive glob, I'm a lot less concerned about creating a potential new issue. I'd entertain a PR that addresses this issue.

> Is there an environment variable or something that I can set to see how the `package_data` is being assembled?

I'm not aware of one. I guess you'll have to go to the source.

As a workaround while this is implemented, I've done the following, it might be useful to someone else (note the example uses `src` layout but is easy to tweak):
```python
import os
from pathlib import Path
from typing import Dict, List

def _expand_recursive_globs(package_data: Dict[str, List[str]]) -> Dict[str, List[str]]:
    root = (Path(__file__).parent / "src").resolve()
    for module, patterns in package_data.items():
        new_patterns = []
        m_root = None
        for idx, p in enumerate(patterns):
            if "**" in p:
                m_root = m_root or root / module.replace(".", os.sep)
                for f in m_root.glob("**"):  # all subdirectories
                    if f.name == "__pycache__":
                        continue
                    subdir_pattern = p.replace("**", str(f.relative_to(m_root)))
                    new_patterns.append(subdir_pattern)
            else:
                new_patterns.append(p)
        package_data[module] = new_patterns
    return package_data

setup(
    name="my-package",
    packages=find_packages("src"),
    package_dir={"": "src"},
    package_data=_expand_recursive_globs({"my_module": ["**/*.csv"]})
)
```
and if you have this project structure:
```
setup.py
src/
    my_module/
        __init__.py
        a/
            file1.csv
        b/
            file2.csv
```
then the package data will expand to:
```python
package_data={"my_module": ["./a/*.csv", "./b/*.csv"]})
```
         
Bump.
I came here googling the same issue. Are there any news on implementing the recursive search?
I need this in a setup.cfg/pyproject.toml flow, so I can't work around by writing my own function in setup.py :-(
Hi @mat-rs thanks for the ping. If you would like to give it a try, we would love receiving a PR!

Regarding the `setup.py`/`setup.cfg`, I believe that in the end both configurations are merged into one. So you can keep most of your config in `setup.cfg` and just have the `package_data` in `setup.py`.

Alternatively, you can also consider `include_package_data=True`.
Actually I didn't want to add ``setup.py`` to my projects and I haven't yet found out what exactly ``include_package_data`` does, but as per the discussion above, I assumed that implementing the recursive search for ``package_data`` was agreed upon to make sense.

Would it be as simple as replacing:

```
globs_expanded = map(glob, patterns)
```

with

```
from functools import partial
...
globs_expanded = map(partial(glob, recursive=True), patterns)
```

Or would that break anything of the current expected behavior?
Thank you very much @mat-rs for having a look on that!

> Would it be as simple as replacing

It seems like a good shot, by looking at https://github.com/pypa/setuptools/blob/0ec53b228800300992ba1c53c2f089a435d4970c/setuptools/command/build_py.py#L101-L104

> Or would that break anything of the current expected behavior?

You can start by running the test suite (e.g. via `tox -- -p no:cov` to filter out the noise) and see what happens. It would be also nice if you add some tests to capture the new behaviour/feature.

Resolving patch:
diff --git a/setuptools/command/build_py.py b/setuptools/command/build_py.py
--- a/setuptools/command/build_py.py
+++ b/setuptools/command/build_py.py
@@ -1,3 +1,4 @@
+from functools import partial
 from glob import glob
 from distutils.util import convert_path
 import distutils.command.build_py as orig
@@ -98,7 +99,7 @@ def find_data_files(self, package, src_dir):
             package,
             src_dir,
         )
-        globs_expanded = map(glob, patterns)
+        globs_expanded = map(partial(glob, recursive=True), patterns)
         # flatten the expanded globs into an iterable of matches
         globs_matches = itertools.chain.from_iterable(globs_expanded)
         glob_files = filter(os.path.isfile, globs_matches)


Test patch:
diff --git a/setuptools/tests/test_build_py.py b/setuptools/tests/test_build_py.py
--- a/setuptools/tests/test_build_py.py
+++ b/setuptools/tests/test_build_py.py
@@ -25,6 +25,29 @@ def test_directories_in_package_data_glob(tmpdir_cwd):
     dist.run_commands()
 
 
+def test_recursive_in_package_data_glob(tmpdir_cwd):
+    """
+    Files matching recursive globs (**) in package_data should
+    be included in the package data.
+
+    #1806
+    """
+    dist = Distribution(dict(
+        script_name='setup.py',
+        script_args=['build_py'],
+        packages=[''],
+        package_data={'': ['path/**/data']},
+    ))
+    os.makedirs('path/subpath/subsubpath')
+    open('path/subpath/subsubpath/data', 'w').close()
+
+    dist.parse_command_line()
+    dist.run_commands()
+
+    assert stat.S_ISREG(os.stat('build/lib/path/subpath/subsubpath/data').st_mode), \
+        "File is not included"
+
+
 def test_read_only(tmpdir_cwd):
     """
     Ensure read-only flag is not preserved in copy