Repository: ansible/ansible

Issue:
ansible install via setup.py fails with 'can't copy 'lib/ansible/galaxy/data/default'' depending on setuptools version
<!---
Please do not report issues/requests related to Ansible modules here !!

Report them to the appropriate modules-core or modules-extras project:
  - https://github.com/ansible/ansible-modules-core/issues
  - https://github.com/ansible/ansible-modules-extras/issues

Also verify first that your issue/request is not already reported in GitHub
-->
##### ISSUE TYPE

<!--- Pick one below and delete the rest: -->
- Bug Report
##### COMPONENT NAME

<!--- Name of the plugin/task/feature -->

installing via "python setup.py install"
##### ANSIBLE VERSION

<!--- Paste verbatim output from “ansible --version” between quotes below -->

```
git HEAD
```
##### CONFIGURATION

<!---
Mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->
##### OS / ENVIRONMENT

<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say “N/A” for anything that is not platform-specific.
-->

ubuntu 14.04
##### SUMMARY

<!--- Explain the problem briefly -->

installing via "python setup.py install" fails with 'can't copy 'lib/ansible/galaxy/data/default': doesn't exist or not a regular file'

unable to install
##### STEPS TO REPRODUCE

<!---
For bugs, show exactly how to reproduce the problem.
For new features, show how the feature would be used.
-->

just attempt to install

<!--- Paste example playbooks or commands between quotes below -->

```

```

<!--- You can also paste gist.github.com links for larger files -->
##### EXPECTED RESULTS

<!--- What did you expect to happen when running the steps above? -->

install going all the way to the end
##### ACTUAL RESULTS

<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

install crash with message in subject

<!--- Paste verbatim command output between quotes below -->

```

```



Discussion:
this is the broken commit
https://github.com/ansible/ansible/pull/18157

Could not reproduce on F23, but did see the error on u1404 ...

```
copying lib/ansible/modules/extras/windows/win_scheduled_task.ps1 -> build/lib.linux-x86_64-2.7/ansible/modules/extras/windows
copying lib/ansible/modules/extras/windows/win_iis_webbinding.ps1 -> build/lib.linux-x86_64-2.7/ansible/modules/extras/windows
copying lib/ansible/modules/extras/windows/win_unzip.ps1 -> build/lib.linux-x86_64-2.7/ansible/modules/extras/windows
copying lib/ansible/modules/extras/windows/win_package.ps1 -> build/lib.linux-x86_64-2.7/ansible/modules/extras/windows
copying lib/ansible/modules/extras/windows/win_say.ps1 -> build/lib.linux-x86_64-2.7/ansible/modules/extras/windows
copying lib/ansible/modules/extras/windows/win_nssm.ps1 -> build/lib.linux-x86_64-2.7/ansible/modules/extras/windows
copying lib/ansible/modules/extras/windows/win_regmerge.ps1 -> build/lib.linux-x86_64-2.7/ansible/modules/extras/windows
copying lib/ansible/modules/extras/windows/win_regedit.ps1 -> build/lib.linux-x86_64-2.7/ansible/modules/extras/windows
copying lib/ansible/modules/extras/windows/win_chocolatey.ps1 -> build/lib.linux-x86_64-2.7/ansible/modules/extras/windows
copying lib/ansible/modules/extras/windows/win_file_version.ps1 -> build/lib.linux-x86_64-2.7/ansible/modules/extras/windows
copying lib/ansible/modules/extras/windows/win_timezone.ps1 -> build/lib.linux-x86_64-2.7/ansible/modules/extras/windows
creating build/lib.linux-x86_64-2.7/ansible/galaxy/data
error: can't copy 'lib/ansible/galaxy/data/container_enabled': doesn't exist or not a regular file
```

The error went away once I updated setuptools.

@sxpert what version of setuptools is on your machine? Are you using the default 2.2?

needs_info

this is what's installed on the box, on what is basically a fresh install through pxe/netboot of ubuntu 14.04: 

$ aptitude show python-setuptools
Package: python-setuptools  
State: installed
Automatically installed: no
Version: 3.3-1ubuntu2
Priority: optional
Section: python
Maintainer: Ubuntu Developers ubuntu-devel-discuss@lists.ubuntu.com
Architecture: all
Uncompressed Size: 830 k
Depends: python:any (>= 2.7), python:any (< 2.8), python-pkg-resources (= 3.3-1ubuntu2)
Conflicts: python-distribute (< 0.7)
Replaces: python-distribute (< 0.6.6)
Provides: python-distribute, python2.7-setuptools
Description: Python Distutils Enhancements
 Extensions to the python-distutils for large or complex distributions.
Homepage: https://pypi.python.org/pypi/setuptools

It seems that this is at least somewhat related to the version of setuptools installed:
- I can reproduce this issue on the devel branch or d60bc492 (#18157) on CentOS 7 with the yum version of setuptools-0.9.8-4.el7 (but an upgraded pip). Reverting to 679da002 installs ansible fine.
- However using newer version of setuptools installed via get-pip.py (apip-8.1.2 setuptools-28.7.0 wheel-0.29.0) also works, even with the devel branch or d60bc492.

Here is a minimal Dockerfile I am using to test with the yum setuptools-0.9.8-4.el7 - replace `devel` with commit hash to test d60bc492 or 679da002:

```
FROM centos:centos7

RUN yum -y update && yum -y install epel-release && yum clean all
RUN yum -y install gcc make python-devel python-pip openssl-devel openssh-clients git && yum clean all

RUN pip install --upgrade pip && pip install git+git://github.com/ansible/ansible.git@VERSION

RUN mkdir -p /opt/ansible
WORKDIR /opt/ansible

# Show version.
RUN ansible --version
```

Here is a Dockerfile using get-pip.py that installs devel cleanly:

```
FROM centos:centos7

RUN yum -y update && yum -y install epel-release && yum clean all
RUN yum -y install gcc make python-devel openssl-devel openssh-clients git && yum clean all
RUN curl -O https://bootstrap.pypa.io/get-pip.py
RUN python get-pip.py

RUN pip install git+git://github.com/ansible/ansible.git@devel

RUN mkdir -p /opt/ansible
WORKDIR /opt/ansible

# Show version.
RUN ansible --version
```

@chouseknecht @jctanner This is causing failures across the board when building Ansible debs. 

Can we make this work with the version of `python-setuptools` provided by the distro repos?

What is the solution. I am also facing this problem. Steps done
sudo apt-get install python-pip git python-dev
   git clone https://github.com/ansible/ansible.git
   git -C ansible submodule update --init --recursive
   sudo pip install ./ansible
It fails with error: can't copy 'lib/ansible/galaxy/data/default': doesn't exist or not a regular file
Anyone?

Several things to look into here:
-  if someone does "python setup.py sdist" does a tarball get created?
  Does it contain all the files that are in that source tree?
  *if not, then try looking for problems in MANIFEST.in. (I see an
  include-recursive for the data directory already so I don't think this is
  the problem)
- if you add  `__init__.py` files to the data directories (all of
  them), does the problem go away?  I thought that setuptools required those
  to recognize package data dirs.  Perhaps the requirement has gone away in
  more recent versions of setuptools leading to the different behavior
  depending on the setuptools version.

@abadger I've tried your suggestion of creating `__init__.py` in all subdirectories of `lib/ansible/galaxy`, but the problem persists.

`'galaxy/data/*'` in this line seems like it may be at least part of the problem: https://github.com/ansible/ansible/blob/devel/setup.py#L29 

Yeah.. maybe older versions of setuptools can't handle directories in that
data structure.  Does making it more explicit about what files in those
directories to include work?  Or specifying the directory name without the
wildcard to match all files in the directory?

Yeah, that seems to be the case:

https://github.com/pypa/setuptools/issues/261

Fixed in setuptools 21.2.1

@abadger Thanks for the link. That was it. It works when changing that to `'galaxy/data/*.*'`. The following files don't have extensions and wont be matched:

```
lib/ansible/galaxy/data/default/tests/inventory
lib/ansible/galaxy/data/container_enabled/tests/inventory
```

I'm not familiar with what's going on here, so I'm not sure what the implication(s) of this approach would be. Suggestions for getting this to work with older versions of setuptools?


Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -26,7 +26,7 @@
       package_dir={ '': 'lib' },
       packages=find_packages('lib'),
       package_data={
-         '': ['module_utils/*.ps1', 'modules/core/windows/*.ps1', 'modules/extras/windows/*.ps1', 'galaxy/data/*'],
+         '': ['module_utils/*.ps1', 'modules/core/windows/*.ps1', 'modules/extras/windows/*.ps1', 'galaxy/data/*/*.*', 'galaxy/data/*/*/*.*', 'galaxy/data/*/tests/inventory'],
       },
       classifiers=[
           'Development Status :: 5 - Production/Stable',


Test patch: