Repository: ansible/ansible

Issue:
setup.py sdist converts symlinks to files breaking webdoc generation and other features
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report

##### COMPONENT NAME
<!--- Name of the module/plugin/task/feature -->
webdocs

##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
```
ansible 2.4.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/fred/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/fred/external/ansible/.tox/py27/lib/python2.7/site-packages/ansible
  executable location = /home/fred/external/ansible/.tox/py27/bin/ansible
  python version = 2.7.13 (default, May 10 2017, 20:04:28) [GCC 6.3.1 20161221 (Red Hat 6.3.1-1)]
```

##### CONFIGURATION
<!---
Mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->

##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say “N/A” for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.
-->
Fedora 25

##### SUMMARY
<!--- Explain the problem briefly -->

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->
python setup.py sdist

# and then from the extracted tarball:

make webdocs

<!--- Paste example playbooks or commands between quotes below -->
```yaml

```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
doc built

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->
```
$ make webdocs
(cd docs/docsite/; CPUS=4 make docs)
make[1]: Entering directory '/home/fred/external/ansible/ansible-2.4.0/docs/docsite'
rm -rf htmlout
rm -rf _build
rm -f .buildinfo
rm -f objects.inv
rm -rf *.doctrees
Cleaning up minified css files
find . -type f -name "*.min.css" -delete
Cleaning up byte compiled python stuff
find . -regex ".*\.py[co]$" -delete
Cleaning up editor backup files
find . -type f \( -name "*~" -or -name "#*" \) -delete
find . -type f \( -name "*.swp" \) -delete
Cleaning up generated rst
rm rst/list_of_*.rst
rm: cannot remove 'rst/list_of_*.rst': No such file or directory
Makefile:28: recipe for target 'clean' failed
make[1]: [clean] Error 1 (ignored)
rm rst/*_by_category.rst
rm rst/*_module.rst
rm rst/playbooks_directives.rst
rm: cannot remove 'rst/playbooks_directives.rst': No such file or directory
Makefile:28: recipe for target 'clean' failed
make[1]: [clean] Error 1 (ignored)
rm rst/playbooks_keywords.rst
../bin/testing_formatter.sh
PYTHONPATH=../../lib ../bin/dump_keywords.py --template-dir=../templates --output-dir=rst/ -d ./keyword_desc.yml
PYTHONPATH=../../lib ../bin/plugin_formatter.py -t rst --template-dir=../templates --module-dir=../../lib/ansible/modules -o rst/
rendering: ipadm_ifprop
rendering: ecs_taskdefinition
rendering: selinux
rendering: cl_interface_policy
rendering: gce_lb
rendering: os_keystone_domain_facts
rendering: os_recordset
rendering: junos_rpc
rendering: pn_ospf
rendering: gcpubsub_facts
rendering: zabbix_maintenance
rendering: svc
rendering: nxos_gir_profile_management
rendering: kernel_blacklist
rendering: cs_snapshot_policy
rendering: nxos_command
rendering: os_user_facts
rendering: bigip_ucs
rendering: net_l3_interface
rendering: logentries
rendering: bigip_pool_member
rendering: avi_api_session
rendering: avi_networkprofile
rendering: os_floating_ip
rendering: avi_backup
rendering: win_feature
rendering: ce_evpn_global
rendering: win_service
rendering: ec2_elb
rendering: win_tempfile
rendering: ovirt_templates_facts
rendering: iam_policy
rendering: win_reboot
rendering: jenkins_plugin
rendering: avi_useraccountprofile
rendering: supervisorctl
rendering: rax_cbs_attachments
rendering: mysql_user
rendering: ec2_group
rendering: junos_l3_interface
rendering: consul_session
rendering: parted
rendering: ec2_win_password
rendering: ec2_vpc_vgw
rendering: win_domain_controller
rendering: vmware_maintenancemode
rendering: jenkins_job
rendering: ec2_lc_find
rendering: ce_netstream_global
rendering: sf_snapshot_schedule_manager
rendering: ec2_vpc_peer
rendering: ec2_vpc_subnet_facts
rendering: gunicorn
rendering: ovirt_storage_domains_facts
rendering: airbrake_deployment
rendering: digital_ocean_block_storage
rendering: netscaler_cs_action
rendering: ovirt_vmpools_facts
rendering: avi_gslbapplicationpersistenceprofile
rendering: yum_repository
rendering: bigmon_policy
rendering: vyos_interface
rendering: aos_template
rendering: ec2_ami_find
rendering: ec2_facts
*** ERROR: DEPRECATED MODULE MISSING 'deprecated' DOCUMENTATION: ../../lib/ansible/modules/cloud/amazon/_ec2_facts.py, ec2_facts ***

Makefile:53: recipe for target 'modules' failed
make[1]: *** [modules] Error 1
make[1]: Leaving directory '/home/fred/external/ansible/ansible-2.4.0/docs/docsite'
Makefile:311: recipe for target 'webdocs' failed
make: *** [webdocs] Error 2
```



Discussion:
removed "from sdist tarball" as that does not matter for this error as it is present in the `/devel` branch no matter how you consume it.
it matters as the sdist process is creating a regular file and btw making
the doc build process fail as it checks that it's a symlink.

On Thu, Jul 20, 2017 at 2:54 PM, Brian Coca <notifications@github.com>
wrote:

> removed "from sdist tarball" as that does not matter for this error as it
> is present in the /devel branch no matter how you consume it.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/ansible/ansible/issues/27105#issuecomment-316693807>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ADBdPprkSOxlvc-HZi-SbKXCXDLPYxPGks5sP04JgaJpZM4OeDC_>
> .
>

that is diff issue then, sdist should preserve symlinks, Ansible treats them in special ways so having it as a file is a problem.
Could be but that's not the case. That's why I opened it this way as it'll
fail at release time and that's not a good thing ;-)

On Thu, Jul 20, 2017 at 4:39 PM, Brian Coca <notifications@github.com>
wrote:

> that is diff issue then, sdist should preserve symlinks
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/ansible/ansible/issues/27105#issuecomment-316724065>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ADBdPuxz9sRVdjwHJBStXVn7wwtUWq6qks5sP2atgaJpZM4OeDC_>
> .
>

That's how distutils works and always has.  I believe i added some code to replace the files with symlinks but I believe that code activates at install time, rather than build time.  We could look at moving that code to run at a different/additional times but it seems strange to me that it would only be breaking now....
it is rare that people do docs from sdist instead of git checkout, also it is rare that we rename modules
Oh, and this isn't the bin scripts; it's the modules that are renamed...  we don't have any code that changes those.
Found an upstream (setuptools) bug report: https://github.com/pypa/setuptools/issues/415#issuecomment-316788191  I've left a comment and told the maintainer about it so they can think about it.  We'll still need a workaround for the foreseeable future, though.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -1,17 +1,132 @@
+
+import json
 import os
 import os.path
 import sys
+from collections import defaultdict
+from distutils.command.build_scripts import build_scripts as BuildScripts
+from distutils.command.sdist import sdist as SDist
 
-sys.path.insert(0, os.path.abspath('lib'))
-from ansible.release import __version__, __author__
 try:
     from setuptools import setup, find_packages
+    from setuptools.command.build_py import build_py as BuildPy
+    from setuptools.command.install_lib import install_lib as InstallLib
+    from setuptools.command.install_scripts import install_scripts as InstallScripts
 except ImportError:
     print("Ansible now needs setuptools in order to build. Install it using"
           " your package manager (usually python-setuptools) or via pip (pip"
           " install setuptools).")
     sys.exit(1)
 
+sys.path.insert(0, os.path.abspath('lib'))
+from ansible.release import __version__, __author__
+
+
+SYMLINK_CACHE = 'SYMLINK_CACHE.json'
+
+
+def _find_symlinks(topdir, extension=''):
+    """Find symlinks that should be maintained
+
+    Maintained symlinks exist in the bin dir or are modules which have
+    aliases.  Our heuristic is that they are a link in a certain path which
+    point to a file in the same directory.
+    """
+    symlinks = defaultdict(list)
+    for base_path, dirs, files in os.walk(topdir):
+        for filename in files:
+            filepath = os.path.join(base_path, filename)
+            if os.path.islink(filepath) and filename.endswith(extension):
+                target = os.readlink(filepath)
+                if os.path.dirname(target) == '':
+                    link = filepath[len(topdir):]
+                    if link.startswith('/'):
+                        link = link[1:]
+                    symlinks[os.path.basename(target)].append(link)
+    return symlinks
+
+
+def _cache_symlinks(symlink_data):
+    with open(SYMLINK_CACHE, 'w') as f:
+        f.write(json.dumps(symlink_data))
+
+
+def _maintain_symlinks(symlink_type, base_path):
+    """Switch a real file into a symlink"""
+    try:
+        # Try the cache first because going from git checkout to sdist is the
+        # only time we know that we're going to cache correctly
+        with open(SYMLINK_CACHE, 'r') as f:
+            symlink_data = json.loads(f.read())
+    except IOError as e:
+        if e.errno == 2:
+            # SYMLINKS_CACHE doesn't exist.  Fallback to trying to create the
+            # cache now.  Will work if we're running directly from a git
+            # checkout or from an sdist created earlier.
+            symlink_data = {'script': _find_symlinks('bin'),
+                            'library': _find_symlinks('lib', '.py'),
+                            }
+
+            # Sanity check that something we know should be a symlink was
+            # found.  We'll take that to mean that the current directory
+            # structure properly reflects symlinks in the git repo
+            if 'ansible-playbook' in symlink_data['script']['ansible']:
+                _cache_symlinks(symlink_data)
+            else:
+                raise
+        else:
+            raise
+    symlinks = symlink_data[symlink_type]
+
+    for source in symlinks:
+        for dest in symlinks[source]:
+            dest_path = os.path.join(base_path, dest)
+            if not os.path.islink(dest_path):
+                try:
+                    os.unlink(dest_path)
+                except OSError as e:
+                    if e.errno == 2:
+                        # File does not exist which is all we wanted
+                        pass
+                os.symlink(source, dest_path)
+
+
+class BuildPyCommand(BuildPy):
+    def run(self):
+        BuildPy.run(self)
+        _maintain_symlinks('library', self.build_lib)
+
+
+class BuildScriptsCommand(BuildScripts):
+    def run(self):
+        BuildScripts.run(self)
+        _maintain_symlinks('script', self.build_dir)
+
+
+class InstallLibCommand(InstallLib):
+    def run(self):
+        InstallLib.run(self)
+        _maintain_symlinks('library', self.install_dir)
+
+
+class InstallScriptsCommand(InstallScripts):
+    def run(self):
+        InstallScripts.run(self)
+        _maintain_symlinks('script', self.install_dir)
+
+
+class SDistCommand(SDist):
+    def run(self):
+        # have to generate the cache of symlinks for release as sdist is the
+        # only command that has access to symlinks from the git repo
+        symlinks = {'script': _find_symlinks('bin'),
+                    'library': _find_symlinks('lib', '.py'),
+                    }
+        _cache_symlinks(symlinks)
+
+        SDist.run(self)
+
+
 with open('requirements.txt') as requirements_file:
     install_requirements = requirements_file.read().splitlines()
     if not install_requirements:
@@ -33,26 +148,16 @@
     install_requirements.append(crypto_backend)
 
 
-SYMLINKS = {'ansible': frozenset(('ansible-console',
-                                  'ansible-doc',
-                                  'ansible-galaxy',
-                                  'ansible-playbook',
-                                  'ansible-pull',
-                                  'ansible-vault'))}
-
-for source in SYMLINKS:
-    for dest in SYMLINKS[source]:
-        dest_path = os.path.join('bin', dest)
-        if not os.path.islink(dest_path):
-            try:
-                os.unlink(dest_path)
-            except OSError as e:
-                if e.errno == 2:
-                    # File does not exist which is all we wanted
-                    pass
-            os.symlink(source, dest_path)
-
 setup(
+    # Use the distutils SDist so that symlinks are not expanded
+    # Use a custom Build for the same reason
+    cmdclass={
+        'build_py': BuildPyCommand,
+        'build_scripts': BuildScriptsCommand,
+        'install_lib': InstallLibCommand,
+        'install_scripts': InstallScriptsCommand,
+        'sdist': SDistCommand,
+    },
     name='ansible',
     version=__version__,
     description='Radically simple IT automation',


Test patch: