Repository: pypa/setuptools

Issue:
Ignoring package_data when include_package_data is True
Hi, firstly, thanks for the work you maintainers do on this library :)

I could be incorrect in saying this, but it's come to my understanding that setting `include_package_data=True` in setup includes data files specified by MANIFEST.in and *only* in MANIFEST.in. At the very least, this behaviour is not clearly explained in the docs, and at the worst it is unexpected and implicit behaviour. 

Here is a sentence from the docs which seems to be trying to say the above, although elsewhere in the docs `include_package_data` is not explained in this way:

> If using the setuptools-specific `include_package_data` argument, files specified by `package_data` will *not* be automatically added to the manifest unless they are listed in the MANIFEST.in file.

Doesn't this make `package_data` redundant when `include_package_data` is True? Does it really make sense to have a param called `include_package_data` which implicitly ignores `package_data`?

In summary, this is the behaviour I seem to have come across when using these options in various combinations:
 1. Including files in `package_data` with `include_package_data=False`:
    - Files specified by `package_data` are included as well as everything in MANIFEST.in.
 2. Including files in `package_data` with `include_package_data=True`:
    - Only the files specified by MANIFEST.in are included, files in `package_data` not specified by MANIFEST.in are excluded.
 3. Omitting `package_data` with `include_package_data=True`:
    - Same result as point 2.

I'm sorry if I've got this wrong, any correspondence is much appreciated :)


Discussion:
I tend to agree that the intuitive behavior would either be:

1. `include_package_data` + `package_data` does the union of everything found between the two of them
2. an error is raised as these are incompatible arguments

My preference would be 1, but that is a backwards-incompatible change that could probably cause all kinds of issues with existing packages, and a shocking number of projects don't test their packages *as installed*. I think we can detect if both are specified and switch it over to a warning for a few releases, then an error.

Also, I find `include_package_data` to be a **very** misleading name. I suspect we'd be better off deprecating this keyword entirely. It could be replaced with a `find_package_data` function similar to `find_packages`. Alternatively (and I like this less), we could do something like add an `include_manifest_data` or `include_manifest_specified_data`, which would have the "union of specified package data and manifest data" behavior.
As a newcomer to the Python packaging world I experienced that confusion first hand
This is completely confusing and frustrating. Hope to see this resolved very soon
I suspect this issue is related to [this hacky behavior](https://github.com/pypa/setuptools/blob/dfd155218751787fecd2bb3a11fef6e510ed8ec3/setuptools/command/sdist.py#L119-L120).
Hi @Tobotimus. I just would like to clarify, that `package_data` is not exactly ignored.

According to [some experiments](https://github.com/abravalheri/experiment-setuptools-package-data), this is the expected behaviour:

- In wheels:

  - ``exclude_package_data`` will **ALWAYS** prevent "data files" from being included in the distribution;
  - after considering that, "data files" will be included in the distribution if:

    - ``package_data`` lists them **OR**
    - ``include_package_data=True`` **AND** ``MANIFEST.in`` includes them.

- In sdists, "data files" will be included in the distribution if:

  - ``MANIFEST.in`` includes them **OR**
  - ``include_package_data=False`` (or not present)
    **AND** ``package_data`` lists them
    **AND** ``exclude_package_data`` does not list them


Please notice this considers the extreme case, when the data files are placed inside directories that are not valid Python packages (e.g. missing ``__init__.py`` files or whose names are not valid python identifiers) [*1].

Also have in mind that "data files" outside the package directory are no longer recommended [*2].

Finally, I might have done something wrong in the experiments, so any review/correction is appreciated.

[*1]: https://setuptools.pypa.io/en/latest/userguide/datafiles.html
[*2]: https://setuptools.pypa.io/en/latest/userguide/datafiles.html#non-package-data-files

Resolving patch:
diff --git a/setuptools/command/build_py.py b/setuptools/command/build_py.py
--- a/setuptools/command/build_py.py
+++ b/setuptools/command/build_py.py
@@ -67,6 +67,16 @@ def _get_data_files(self):
         self.analyze_manifest()
         return list(map(self._get_pkg_data_files, self.packages or ()))
 
+    def get_data_files_without_manifest(self):
+        """
+        Generate list of ``(package,src_dir,build_dir,filenames)`` tuples,
+        but without triggering any attempt to analyze or build the manifest.
+        """
+        # Prevent eventual errors from unset `manifest_files`
+        # (that would otherwise be set by `analyze_manifest`)
+        self.__dict__.setdefault('manifest_files', {})
+        return list(map(self._get_pkg_data_files, self.packages or ()))
+
     def _get_pkg_data_files(self, package):
         # Locate package source directory
         src_dir = self.get_package_dir(package)
diff --git a/setuptools/command/egg_info.py b/setuptools/command/egg_info.py
--- a/setuptools/command/egg_info.py
+++ b/setuptools/command/egg_info.py
@@ -608,6 +608,18 @@ def prune_file_list(self):
         self.filelist.exclude_pattern(r'(^|' + sep + r')(RCS|CVS|\.svn)' + sep,
                                       is_regex=1)
 
+    def _safe_data_files(self, build_py):
+        """
+        The parent class implementation of this method
+        (``sdist``) will try to include data files, which
+        might cause recursion problems when
+        ``include_package_data=True``.
+
+        Therefore, avoid triggering any attempt of
+        analyzing/building the manifest again.
+        """
+        return build_py.get_data_files_without_manifest()
+
 
 def write_file(filename, contents):
     """Create a file with the specified name and write 'contents' (a
diff --git a/setuptools/command/sdist.py b/setuptools/command/sdist.py
--- a/setuptools/command/sdist.py
+++ b/setuptools/command/sdist.py
@@ -114,12 +114,15 @@ def _add_defaults_python(self):
 
     def _safe_data_files(self, build_py):
         """
-        Extracting data_files from build_py is known to cause
-        infinite recursion errors when `include_package_data`
-        is enabled, so suppress it in that case.
+        Since the ``sdist`` class is also used to compute the MANIFEST
+        (via :obj:`setuptools.command.egg_info.manifest_maker`),
+        there might be recursion problems when trying to obtain the list of
+        data_files and ``include_package_data=True`` (which in turn depends on
+        the files included in the MANIFEST).
+
+        To avoid that, ``manifest_maker`` should be able to overwrite this
+        method and avoid recursive attempts to build/analyze the MANIFEST.
         """
-        if self.distribution.include_package_data:
-            return ()
         return build_py.data_files
 
     def _add_data_files(self, data_files):


Test patch:
diff --git a/setuptools/tests/test_sdist.py b/setuptools/tests/test_sdist.py
--- a/setuptools/tests/test_sdist.py
+++ b/setuptools/tests/test_sdist.py
@@ -126,6 +126,28 @@ def test_package_data_in_sdist(self):
         assert os.path.join('sdist_test', 'c.rst') not in manifest
         assert os.path.join('d', 'e.dat') in manifest
 
+    def test_package_data_and_include_package_data_in_sdist(self):
+        """
+        Ensure package_data and include_package_data work
+        together.
+        """
+        setup_attrs = {**SETUP_ATTRS, 'include_package_data': True}
+        assert setup_attrs['package_data']
+
+        dist = Distribution(setup_attrs)
+        dist.script_name = 'setup.py'
+        cmd = sdist(dist)
+        cmd.ensure_finalized()
+
+        with quiet():
+            cmd.run()
+
+        manifest = cmd.filelist.files
+        assert os.path.join('sdist_test', 'a.txt') in manifest
+        assert os.path.join('sdist_test', 'b.txt') in manifest
+        assert os.path.join('sdist_test', 'c.rst') not in manifest
+        assert os.path.join('d', 'e.dat') in manifest
+
     def test_setup_py_exists(self):
         dist = Distribution(SETUP_ATTRS)
         dist.script_name = 'foo.py'