Repository: celery/kombu

Issue:
Gathering data_files in setup.py is broken (affects current wheel on PyPI)
The `setup.py` file in this package uses the `data_files` argument to install "data files" on setup. Looking at the code that's responsible for gathering those files in
https://github.com/celery/kombu/blob/v4.2.0/setup.py#L71-L80
one can assume that the intention was to add every file which name is not ending with `.py` under the `kombu` directory. But the code there actually generates a list of _every_ file in the given directory when it encounters at least one file that's not ending with `.py`. It can get tricky if the environment is not clean - for example when there are some `.pyc` files.

Now, when creating packages from the source files we have a discrepancy when it comes to wheel format and tar.gz one (the actual cause for it is not yet clear to me). 

I have tried to install both wheel and source packages on two separate virtualenvs from the following "unclean" state of this repo:
```sh
$ git status --short
$ git clean -nx
Would remove kombu/utils/functional.pyc
``` 

I have done it like that:
```sh
$ python setup.py sdist
$ python setup.py bdist_wheel
$ /tmp/kombutests-wheel/bin/pip install dist/kombu-4.2.0-py2.py3-none-any.whl
$ /tmp/kombutests-targz/bin/pip install dist/kombu-4.2.0.tar.gz
$ tree /tmp/kombutests-wheel/kombu/
/tmp/kombutests-wheel/kombu/
└── utils
    ├── __init__.py
    ├── __init__.pyc
    ├── amq_manager.py
    ├── amq_manager.pyc
    ...
    ├── uuid.py
    └── uuid.pyc
$ tree /tmp/kombutests-targz/kombu
/tmp/kombutests-targz/kombu [error opening dir]

0 directories, 0 files
```
where `/tmp/kombutests-targz` and `/tmp/kombutests-wheel` are the virtualenvs I've mentioned.
This shows that there's an additional `kombu/utils` directory "installed" in the virtualenv's base directory only because we had one additional `.pyc` file in the repo (`kombu/utils/functional.pyc`).

Now, on top of that, it seems that the wheel that's published on PyPi is broken because (I suspect) it was created from a non-clean repo clone.

When downloading the wheel from https://files.pythonhosted.org/packages/2f/0d/416d396ee75b7e47e620005ef81a5b8a36c53c85bb148bd82459e8e61b6b/kombu-4.2.0-py2.py3-none-any.whl (found on https://pypi.org/project/kombu/#files) and unpacking it with `wheel unpack` we can find `kombu-4.2.0/kombu-4.2.0.data/data/kombu` directory. This directory gets copied into the virtualenv base directory. The contents of that directory look similar to what actually gets installed inside `lib/python2.7/site-packages/kombu`, but there are some substantial differences. 
For example there's still an old `async` directory there (it was renamed in https://github.com/celery/kombu/commit/75695205f6e7af8e7e9178e010debc3871b19106) and `asynchronous` dir has only `.pyc` files inside `__pycache__` directories. 
I've pasted the contents of that directory in http://dpaste.com/1C2C7D5

This has bitten us quite hard since we run our processes from the virtualenv base directory so we've used that incomplete version of kombu from that data directory and encountered some "unexplainable" `ImportError`s. 

As I've mentioned I'm not sure what's the original reason for such difference when installing from wheel and from .tar.gz package. Although I'm pretty sure the `data_files` argument from setup is not used correctly here. 

Our current workaround for this issue is to put
```
--no-binary kombu
```
on top of our `requirements.txt` file.


Discussion:
how about removing data_files argument? I dont see django and other libs dont use this
Yep, I'm up for that solution too. Will try to open a PR.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -31,6 +31,7 @@ def add_default(m):
 def add_doc(m):
     return (('doc', m.groups()[0]),)
 
+
 pats = {re_meta: add_default, re_doc: add_doc}
 here = os.path.abspath(os.path.dirname(__file__))
 meta_fh = open(os.path.join(here, 'kombu/__init__.py'))
@@ -47,12 +48,6 @@ def add_doc(m):
     meta_fh.close()
 # --
 
-data_files = []
-root_dir = os.path.dirname(__file__)
-if root_dir != '':
-    os.chdir(root_dir)
-src_dir = 'kombu'
-
 
 def fullsplit(path, result=None):
     if result is None:
@@ -68,17 +63,6 @@ def fullsplit(path, result=None):
 for scheme in list(INSTALL_SCHEMES.values()):
     scheme['data'] = scheme['purelib']
 
-for dirpath, dirnames, filenames in os.walk(src_dir):
-    # Ignore dirnames that start with '.'
-    for i, dirname in enumerate(dirnames):
-        if dirname.startswith('.'):
-            del dirnames[i]
-    for filename in filenames:
-        if not filename.endswith('.py'):
-            data_files.append(
-                [dirpath, [os.path.join(dirpath, f) for f in filenames]],
-            )
-
 if os.path.exists('README.rst'):
     long_description = codecs.open('README.rst', 'r', 'utf-8').read()
 else:
@@ -117,6 +101,7 @@ def run_tests(self):
         import pytest
         sys.exit(pytest.main(self.pytest_args))
 
+
 setup(
     name='kombu',
     packages=setuptools.find_packages(exclude=['t', 't.*']),
@@ -128,7 +113,6 @@ def run_tests(self):
     author_email=meta['contact'],
     url=meta['homepage'],
     platforms=['any'],
-    data_files=data_files,
     zip_safe=False,
     license='BSD',
     cmdclass={'test': pytest},


Test patch: