Repository: pypa/setuptools

Issue:
Accidentally passing package_data a str value triggers error on Windows but not Linux
As part of https://github.com/greenelab/manubot/pull/51, I specified the `package_data` argument of `setuptools.setup()` in our [`setup.py`](https://github.com/greenelab/manubot/blob/bfac0e10b452a2556487de250e1acd47fe94d1da/setup.py#L73-L76):

```py
    # Specify additional patterns to match files
    package_data={
        'manubot': 'cite/*.lua',
    },
```

The Linux Travis CI build passes, but the Windows [AppVeyer builds](https://ci.appveyor.com/project/greenelab/manubot/build/1.0.99) error when attempting `pip install .`. Here is the error:

<details>

```
%CMD_IN_ENV% pip install .
Processing c:\projects\manubot
    Complete output from command python setup.py egg_info:
    running egg_info
    creating pip-egg-info\manubot.egg-info
    writing pip-egg-info\manubot.egg-info\PKG-INFO
    writing dependency_links to pip-egg-info\manubot.egg-info\dependency_links.txt
    writing entry points to pip-egg-info\manubot.egg-info\entry_points.txt
    writing requirements to pip-egg-info\manubot.egg-info\requires.txt
    writing top-level names to pip-egg-info\manubot.egg-info\top_level.txt
    writing manifest file 'pip-egg-info\manubot.egg-info\SOURCES.txt'
    c:\python36\lib\distutils\dist.py:261: UserWarning: Unknown distribution option: 'long_description_content_type'
      warnings.warn(msg)
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\appveyor\AppData\Local\Temp\1\pip-req-build-akczu3gv\setup.py", line 75, in <module>
        'manubot': 'cite/*.lua',
      File "c:\python36\lib\site-packages\setuptools\__init__.py", line 129, in setup
        return distutils.core.setup(**attrs)
      File "c:\python36\lib\distutils\core.py", line 148, in setup
        dist.run_commands()
      File "c:\python36\lib\distutils\dist.py", line 955, in run_commands
        self.run_command(cmd)
      File "c:\python36\lib\distutils\dist.py", line 974, in run_command
        cmd_obj.run()
      File "c:\python36\lib\site-packages\setuptools\command\egg_info.py", line 278, in run
        self.find_sources()
      File "c:\python36\lib\site-packages\setuptools\command\egg_info.py", line 293, in find_sources
        mm.run()
      File "c:\python36\lib\site-packages\setuptools\command\egg_info.py", line 524, in run
        self.add_defaults()
      File "c:\python36\lib\site-packages\setuptools\command\egg_info.py", line 560, in add_defaults
        sdist.add_defaults(self)
      File "c:\python36\lib\site-packages\setuptools\command\py36compat.py", line 34, in add_defaults
        self._add_defaults_python()
      File "c:\python36\lib\site-packages\setuptools\command\sdist.py", line 134, in _add_defaults_python
        for _, src_dir, _, filenames in build_py.data_files:
      File "c:\python36\lib\site-packages\setuptools\command\build_py.py", line 66, in __getattr__
        self.data_files = self._get_data_files()
      File "c:\python36\lib\site-packages\setuptools\command\build_py.py", line 83, in _get_data_files
        return list(map(self._get_pkg_data_files, self.packages or ()))
      File "c:\python36\lib\site-packages\setuptools\command\build_py.py", line 95, in _get_pkg_data_files
        for file in self.find_data_files(package, src_dir)
      File "c:\python36\lib\site-packages\setuptools\command\build_py.py", line 114, in find_data_files
        return self.exclude_data_files(package, src_dir, files)
      File "c:\python36\lib\site-packages\setuptools\command\build_py.py", line 198, in exclude_data_files
        files = list(files)
      File "c:\python36\lib\site-packages\setuptools\command\build_py.py", line 234, in <genexpr>
        for pattern in raw_patterns
      File "c:\python36\lib\distutils\util.py", line 125, in convert_path
        raise ValueError("path '%s' cannot be absolute" % pathname)
    ValueError: path '/' cannot be absolute
```

</details>

In summary, [this line in `distutils.util.convert_path`](https://github.com/python/cpython/blob/02c4d4b6dcd772ec3a7fdca517118f3fa53b0b88/Lib/distutils/util.py#L124-L125) is throwing the following error:

```
ValueError: path '/' cannot be absolute
```

So at some point, it seems that the package_data relative path is converted to absolute triggering this error. Any advice on how to resolve this issue would be greatly appreciated! This is occurring with pip-18.0 in Python 3.6 & 3.7.


Discussion:
As @agitter discovered, the issue was with the forward-slash path separator. In other words, `pip install` does not throw an error if we switch to:

```
    # Specify additional patterns to match files
    package_data={
        'manubot': 'cite\\*.lua',
    },
```

In the case, of a cross-platform `setup.py`, presumably we could use `os.path.join('cite', '*.lua')` or pathlib. However, after making this adjustment, the specified file did not end up getting included in the package distribution. We gave up and switched to using `MANIFEST.in`.

So I guess this issue could be updated to be "package_data does not accept forward-slash separated paths on Windows". I'm not sure whether this is intentional or a bug.
> I'm not sure whether this is intentional or a bug.

Indeed, I would be happy to know how to resolve/work around this issue.
This is very puzzling, but it does not sound like deliberate behavior. Possibly an over-zealous regex somewhere?
I recently met with this problem as well where a module wasn't installing, and i had to download the module to manually install it. 

As a stopgap measure to change the absolute paths being defined, what i did was to alter the _manifest_normalize function in egg_info.py to null replace all paths to produce the relative paths instead, and assign that to self.filelist.files. On the other hand this might not be a good thing if self.filelist.files is not be altered. 

It appears that, at least in my case, while <module>.egg-info/SOURCES.txt is written, unfortunately setuptools relies on the in memory set of files to carry on with the setup instead of rereading the <module>.egg-info/SOURCES.txt.


@dhimmel according to the docs [here](https://docs.python.org/2/distutils/setupscript.html#installing-package-data) the values in the `package_data` dict should be arrays. Try

```
package_data={
    'manubot': ['cite/*.lua'],
}
```

@aschmied looks like passing a list as a value rather than a raw string fixed the issue on Windows: see https://github.com/manubot/manubot/pull/111. 

`package_data` ends up being passed to the `_get_platform_patterns` as the `spec` argument:

https://github.com/pypa/setuptools/blob/a94ccbf404a79d56f9b171024dee361de9a948da/setuptools/command/build_py.py#L220-L235

Therefore, what I think was happening is that each character of `cite/*.lua` was being iterated through and treated as it's own glob. It seems that the Windows setup choked when in encountered the glob `/` and threw `ValueError: path '/' cannot be absolute`.

But now I'm confused as to why Linux works. In fact, it seems that Linux may include the package data file regardless of whether `package_data` or `include_package_data` is specified in `setup.py`. So there is something weird potentially happening on Linux.

For `setuptools`, I think the best course of action would be to add a warning if the values for `package_data` or `exclude_package_data` are strings rather than a list. I'm happy to make a PR for this. Would appreciate any guidance on how / where this package usually put these sanity checks.

Looks like the validation of the `package_data` and `exclude_package_data` fields occurs here:

https://github.com/pypa/setuptools/blob/3adda966e2e76ad59cf79fd8bdeab77fb11d308f/setuptools/dist.py#L307-L322

Resolving patch:
diff --git a/setuptools/dist.py b/setuptools/dist.py
--- a/setuptools/dist.py
+++ b/setuptools/dist.py
@@ -213,8 +213,12 @@ def check_importable(dist, attr, value):
 
 
 def assert_string_list(dist, attr, value):
-    """Verify that value is a string list or None"""
+    """Verify that value is a string list"""
     try:
+        # verify that value is a list or tuple to exclude unordered
+        # or single-use iterables
+        assert isinstance(value, (list, tuple))
+        # verify that elements of value are strings
         assert ''.join(value) != value
     except (TypeError, ValueError, AttributeError, AssertionError):
         raise DistutilsSetupError(
@@ -307,20 +311,17 @@ def check_test_suite(dist, attr, value):
 
 def check_package_data(dist, attr, value):
     """Verify that value is a dictionary of package names to glob lists"""
-    if isinstance(value, dict):
-        for k, v in value.items():
-            if not isinstance(k, str):
-                break
-            try:
-                iter(v)
-            except TypeError:
-                break
-        else:
-            return
-    raise DistutilsSetupError(
-        attr + " must be a dictionary mapping package names to lists of "
-        "wildcard patterns"
-    )
+    if not isinstance(value, dict):
+        raise DistutilsSetupError(
+            "{!r} must be a dictionary mapping package names to lists of "
+            "string wildcard patterns".format(attr))
+    for k, v in value.items():
+        if not isinstance(k, six.string_types):
+            raise DistutilsSetupError(
+                "keys of {!r} dict must be strings (got {!r})"
+                .format(attr, k)
+            )
+        assert_string_list(dist, 'values of {!r} dict'.format(attr), v)
 
 
 def check_packages(dist, attr, value):


Test patch:
diff --git a/setuptools/tests/test_dist.py b/setuptools/tests/test_dist.py
--- a/setuptools/tests/test_dist.py
+++ b/setuptools/tests/test_dist.py
@@ -3,7 +3,13 @@
 from __future__ import unicode_literals
 
 import io
-from setuptools.dist import DistDeprecationWarning, _get_unpatched
+import re
+from distutils.errors import DistutilsSetupError
+from setuptools.dist import (
+    _get_unpatched,
+    check_package_data,
+    DistDeprecationWarning,
+)
 from setuptools import Distribution
 from setuptools.extern.six.moves.urllib.request import pathname2url
 from setuptools.extern.six.moves.urllib_parse import urljoin
@@ -263,3 +269,48 @@ def test_maintainer_author(name, attrs, tmpdir):
         else:
             line = '%s: %s' % (fkey, val)
             assert line in pkg_lines_set
+
+
+CHECK_PACKAGE_DATA_TESTS = (
+    # Valid.
+    ({
+        '': ['*.txt', '*.rst'],
+        'hello': ['*.msg'],
+    }, None),
+    # Not a dictionary.
+    ((
+        ('', ['*.txt', '*.rst']),
+        ('hello', ['*.msg']),
+    ), (
+        "'package_data' must be a dictionary mapping package"
+        " names to lists of string wildcard patterns"
+    )),
+    # Invalid key type.
+    ({
+        400: ['*.txt', '*.rst'],
+    }, (
+        "keys of 'package_data' dict must be strings (got 400)"
+    )),
+    # Invalid value type.
+    ({
+        'hello': str('*.msg'),
+    }, (
+        "\"values of 'package_data' dict\" must be a list of strings (got '*.msg')"
+    )),
+    # Invalid value type (generators are single use)
+    ({
+        'hello': (x for x in "generator"),
+    }, (
+        "\"values of 'package_data' dict\" must be a list of strings "
+        "(got <generator object"
+    )),
+)
+
+
+@pytest.mark.parametrize('package_data, expected_message', CHECK_PACKAGE_DATA_TESTS)
+def test_check_package_data(package_data, expected_message):
+    if expected_message is None:
+        assert check_package_data(None, 'package_data', package_data) is None
+    else:
+        with pytest.raises(DistutilsSetupError, match=re.escape(expected_message)):
+            check_package_data(None, str('package_data'), package_data)