Repository: conda/conda-build

Issue:
patching issue with test_api_skeleton.py
Hi, 

I am having issued with tests/test_api_skeleton.py.

On the first failure:

```bash
============================= test session starts ==============================
platform linux -- Python 3.5.2, pytest-2.9.2, py-1.4.31, pluggy-0.3.1
rootdir: <...>/conda-build, inifile: setup.cfg
plugins: capturelog-0.7, cov-2.3.1
collected 11 items

tests/test_api_skeleton.py F

 generated xml file: <...>/conda-build/junit.xml 
=================================== FAILURES ===================================
__________________________ test_repo[-pypi-pip-8.1.2] __________________________
Traceback (most recent call last):
  File "<...>/conda-build/tests/test_api_skeleton.py", line 21, in test_repo
    api.skeletonize(package, repo, version=version, output_dir=testing_workdir, config=test_config)
  File "<...>/conda-build/conda_build/api.py", line 193, in skeletonize
    recursive=recursive, config=config, **kwargs)
  File "<...>/conda-build/conda_build/skeletons/pypi.py", line 406, in skeletonize
    noprompt, packages, config=config, setup_options=setup_options)
  File "<...>/conda-build/conda_build/skeletons/pypi.py", line 664, in get_package_metadata
    config=config)
  File "<...>/conda-build/conda_build/skeletons/pypi.py", line 924, in get_pkginfo
    run_setuppy(src_dir, tempdir, python_version, config=config, setup_options=setup_options)
  File "<...>/conda-build/conda_build/skeletons/pypi.py", line 983, in run_setuppy
    apply_patch(join(stdlib_dir, 'distutils'), patch, config=config)
  File "<...>/conda-build/conda_build/source.py", line 483, in apply_patch
    check_call_env([patch] + patch_args, cwd=src_dir)
  File "<...>/conda-build/conda_build/utils.py", line 552, in check_call_env
    return _func_defaulting_env_to_os_environ(subprocess.check_call, *popenargs, **kwargs)
  File "<...>/conda-build/conda_build/utils.py", line 548, in _func_defaulting_env_to_os_environ
    return func(_args, **kwargs)
  File "<...>/lib/python3.5/subprocess.py", line 581, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/patch', '-p0', '-i', '/tmp/tmp3rh8k2j4conda_skeleton_pip-8.1.2.tar.gz/pypi-distutils.patch']' returned non-zero exit status 1
----------------------------- Captured stdout call -----------------------------
Using url https://pypi.python.org/packages/e7/a8/7556133689add8d1a54c0b14aeff0acb03c64707ce100ecd53934da1aa13/pip-8.1.2.tar.gz (1.1 MB) for pip.
Downloading pip
Unpacking pip...
done
working in /tmp/tmp3rh8k2j4conda_skeleton_pip-8.1.2.tar.gz
updating index in: /tmp/pytest-of-<...>/pytest-7/test_repo__pypi_pip_8_1_2_0/linux-64
updating index in: /tmp/pytest-of-<...>/pytest-7/test_repo__pypi_pip_8_1_2_0/noarch

The following NEW packages will be INSTALLED:

    openssl:    1.0.2j-0      (soft-link)
    pip:        8.1.2-py35_0  (soft-link)
    python:     3.5.2-0       (soft-link)
    pyyaml:     3.12-py35_0   (soft-link)
    readline:   6.2-2         (soft-link)
    setuptools: 27.2.0-py35_0 (soft-link)
    sqlite:     3.13.0-0      (soft-link)
    tk:         8.5.18-0      (soft-link)
    wheel:      0.29.0-py35_0 (soft-link)
    xz:         5.2.2-0       (soft-link)
    yaml:       0.1.6-0       (soft-link)
    zlib:       1.2.8-3       (soft-link)

Applying patch: '/tmp/tmp3rh8k2j4conda_skeleton_pip-8.1.2.tar.gz/pypi-distutils.patch' in /tmp/pytest-of-<...>/pytest-7/test_repo__pypi_pip_8_1_2_0/skeleton_1478167437529/_b_env_placehold_<...>/lib/python3.5/distutils
File core.py is not a regular file -- refusing to patch
1 out of 1 hunk ignored -- saving rejects to file core.py.rej
```

patching fails because core.py is a symlink ...

A temporary fix would be adding --follow-symlinks in the patch command, but this is ugly
**source.py l.480 in apply_patch**
```python
patch_args = ['-p%d' % patch_strip_level, '-i', path, '--follow-symlinks']
``` 


Discussion:
I'm OK with the --follow-symlinks, but it does impose a future constraint on any other patch implementation we might want to switch to.  There was some effort in the past to use a pure-python implementation, but that didn't support fuzzing, so we didn't end up using it.

@mingwandroid do you have any opinion here?

I don't like the idea of following symlinks since it involves changing files where it is not supposed to (i.e. here in tmp). 

A much more elegant solution would be replacing the symlink with the file it points to, but that would require to a-priori know which files are involved in the patching process. 

Maybe use lsdiff on the patch, but that would require an additional dependency (patchutils) ?

Yes, the pure python implementation would solve this.

@neok-m4700 I'm confused as to how this error arises.  I run these tests very often, but have never seen this.  What settings do you have for conda?  How is it that these files are being symlinked?  I'm asking because this is actually a huge problem elsewhere, because symlinking binaries that have RPATH breaks them.

Hi wanted to go further on #1492, when this issue arose.

Using a fresh install of Anaconda2-4.2.0-Linux-x86_64.sh, on Ubuntu 16.10.

The issue is arising because of symlinks in **test_repo__pypi_pip_8_1_2_0**/skeleton_1478167437529/_b_env_placehold_<...>/lib/python3.5/distutils

Here is a reduced script to reproduce the problem

``` bash
#!/bin/bash
{
  exec &> >(tee test.log)

  mkdir -p orig new
  init=false
  for d in orig new; do
    if [ ! -d $d/conda_build_test_recipe ]; then
      init=true; cd $d
      git clone https://github.com/conda/conda_build_test_recipe.git
      cd $OLDPWD
    fi
    if [ ! -d $d/conda-build ]; then
      init=true; cd $d
      git clone https://github.com/conda/conda-build.git
      cd $OLDPWD
    fi
  done

  if $init; then
    conda update -n root conda-build
    conda install -y -n root --no-pin pytest-cov mock
    pip install pytest-capturelog pytest-xdist
  fi

  do_test() {
    (
      cd $1/conda-build
      py.test tests
      # py.test -n $(nproc) tests
    )
  }

  do_test orig

  exit
}
```


Resolving patch:
diff --git a/conda_build/conda_interface.py b/conda_build/conda_interface.py
--- a/conda_build/conda_interface.py
+++ b/conda_build/conda_interface.py
@@ -55,6 +55,9 @@
     NoPackagesFoundError = conda.exceptions.NoPackagesFoundError
     CondaValueError = conda.exceptions.CondaValueError
 
+    # disallow softlinks.  This avoids a lot of dumb issues, at the potential cost of disk space.
+    conda.base.context.context.allow_softlinks = False
+
 else:
     from conda.config import get_default_urls, non_x86_linux_machines, load_condarc  # NOQA
     from conda.cli.common import get_prefix  # NOQA
@@ -74,6 +77,8 @@
     get_rc_urls = cc.get_rc_urls
     get_local_urls = cc.get_local_urls
 
+    cc.allow_softlinks = False
+
     class PaddingError(Exception):
         pass
 


Test patch: