Repository: conda/conda-build

Issue:
file chmods fail with conda-build 2.0.2
Given the forest of "placehold" strings, I'm having a hard time figuring out what's going on here.  But it looks like a file protection issue of some sort.  I checked; all my recipes have the same protection for bld.bat.  Could this be due to multiple unpacks of the "goodstuff" packages, due to multiple scans of the setup.py file?  This is on a buildbot, with a clean OS X Vagrant image, with conda-build 2.0.2.  Do I have to roll back to conda-build 1.x again?

```
BUILD START: goodstuff-1.0a5.post218+g344ae1df4486f1448d088fabb88cfa9a785e38ce.development-0

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    pandoc-1.15.2              |                0        25.0 MB
    pyparsing-2.1.4            |           py27_0          70 KB
    six-1.10.0                 |           py27_0          16 KB
    packaging-16.7             |           py27_0          41 KB  https://goodstuff.parc.xerox.com/conda
    ------------------------------------------------------------
                                           Total:        25.1 MB

The following NEW packages will be INSTALLED:

    openssl:    1.0.2j-0                                       
    packaging:  16.7-py27_0   https://goodstuff.parc.xerox.com/conda
    pandoc:     1.15.2-0                                       
    pip:        8.1.2-py27_0                                   
    pyparsing:  2.1.4-py27_0                                   
    python:     2.7.12-1                                       
    readline:   6.2-2                                          
    setuptools: 27.2.0-py27_0                                  
    six:        1.10.0-py27_0                                  
    sqlite:     3.13.0-0                                       
    tk:         8.5.18-0                                       
    wheel:      0.29.0-py27_0                                  
    zlib:       1.2.8-3                                        

Package: goodstuff-1.0a5.post218+g344ae1df4486f1448d088fabb88cfa9a785e38ce.development-0
source tree in: /var/folders/8r/w07v646575j16n5bjcfrk44h0000gn/T/miniconda2/conda-bld/goodstuff_1475512243326/work
compiling .pyc files...
number of files: 299
Fixing permissions
Traceback (most recent call last):
  File "/var/folders/8r/w07v646575j16n5bjcfrk44h0000gn/T/miniconda2/bin/conda-build", line 6, in <module>
    sys.exit(conda_build.cli.main_build.main())
  File "/var/folders/8r/w07v646575j16n5bjcfrk44h0000gn/T/miniconda2/lib/python2.7/site-packages/conda_build/cli/main_build.py", line 246, in main
    execute(sys.argv[1:])
  File "/var/folders/8r/w07v646575j16n5bjcfrk44h0000gn/T/miniconda2/lib/python2.7/site-packages/conda_build/cli/main_build.py", line 238, in execute
    already_built=None, config=config)
  File "/var/folders/8r/w07v646575j16n5bjcfrk44h0000gn/T/miniconda2/lib/python2.7/site-packages/conda_build/api.py", line 83, in build
    need_source_download=need_source_download, config=config)
  File "/var/folders/8r/w07v646575j16n5bjcfrk44h0000gn/T/miniconda2/lib/python2.7/site-packages/conda_build/build.py", line 1004, in build_tree
    config=recipe_config)
  File "/var/folders/8r/w07v646575j16n5bjcfrk44h0000gn/T/miniconda2/lib/python2.7/site-packages/conda_build/build.py", line 716, in build
    prefix=config.build_prefix)
  File "/var/folders/8r/w07v646575j16n5bjcfrk44h0000gn/T/miniconda2/lib/python2.7/site-packages/conda_build/build.py", line 336, in create_info_files
    copy_recipe(m, config)
  File "/var/folders/8r/w07v646575j16n5bjcfrk44h0000gn/T/miniconda2/lib/python2.7/site-packages/conda_build/build.py", line 190, in copy_recipe
    copy_into(src_path, dst_path, timeout=config.timeout)
  File "/var/folders/8r/w07v646575j16n5bjcfrk44h0000gn/T/miniconda2/lib/python2.7/site-packages/conda_build/utils.py", line 107, in copy_into
    shutil.copy2(src, dst_fn)
  File "/var/folders/8r/w07v646575j16n5bjcfrk44h0000gn/T/miniconda2/lib/python2.7/shutil.py", line 131, in copy2
    copystat(src, dst)
  File "/var/folders/8r/w07v646575j16n5bjcfrk44h0000gn/T/miniconda2/lib/python2.7/shutil.py", line 103, in copystat
    os.chflags(dst, st.st_flags)
OSError: [Errno 1] Operation not permitted: '/var/folders/8r/w07v646575j16n5bjcfrk44h0000gn/T/miniconda2/conda-bld/goodstuff_1475512243326/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/info/recipe/bld.bat'
```



Discussion:
Completely mystified here.  Same script works fine on my Ubuntu buildbot.

We've seen some other mystifying things to do with shared folders passed into virtual machines.  There's some logic in there that tries to cope with how different OS's handle symlinks and such, and those heuristics are based on how Python reports itself - which may not match the underlying filesystem when shared folders and VMs are involved.

The other issue so far is https://github.com/conda/conda-build/issues/1302

I'm also a little wary of using the "shutil" module at all.  Neither the design nor the implementation of "shutil" fill me with admiration.

This is an OS X Yosemite Vagrant image running under Parallels on an OS X Yosemite host.  Run from a terminal on a MB Pro it runs just fine.

If you have better ideas, I'm wide open to them.  shutil is rife with strangeness and issues, but it works right now for the vast majority of the time.

By the way, this OS X buildbot has been working fine for months until I shifted to conda-build 2.0.2.  Did you change the copying action somehow?

What version did you update from?

1.21.14.

Nothing really stands out to me, just addition of file locking for better support of parallel builds: https://gist.github.com/msarahan/73e21e87acd9822fc5e81a42e541b3c9

Better ideas:  unipath's Path.copy() would work here.  It doesn't try to copy flags.

I'm hesitant to take on a new dependency just for this, and unipath's Path.copy() ends up using shutil anyway: https://github.com/mikeorr/Unipath/blob/master/unipath/path.py#L267-L279 .  Is it worth trying to copy flags first, but falling back to just `copy`, and then `copyfile` so that your use case might work?

Unipath uses shutil, but not the overly ambitious part of shutil that attempts to copy flags.  It copies permissions and mtime, but not flags.

I think it would be fine, instead, to break down your use of shutil in a similar fashion; basically, just repeat the code from https://github.com/mikeorr/Unipath/blob/master/unipath/path.py#L267-L287 in your code to make your own copy.  The key is either to not use os.chflags, or catch the error if you do use it.

I've reverted my OS X buildbot to 1.21.14 till this gets fixed.  Seems to work OK.


Resolving patch:
diff --git a/conda_build/utils.py b/conda_build/utils.py
--- a/conda_build/utils.py
+++ b/conda_build/utils.py
@@ -104,7 +104,16 @@ def copy_into(src, dst, timeout=90, symlinks=False):
                 os.makedirs(os.path.dirname(dst_fn))
             if lock:
                 lock.acquire(timeout=timeout)
-            shutil.copy2(src, dst_fn)
+            # with each of these, we are copying less metadata.  This seems to be necessary
+            #   to cope with some shared filesystems with some virtual machine setups.
+            #  See https://github.com/conda/conda-build/issues/1426
+            try:
+                shutil.copy2(src, dst_fn)
+            except OSError:
+                try:
+                    shutil.copy(src, dst_fn)
+                except OSError:
+                    shutil.copyfile(src, dst_fn)
         except shutil.Error:
             log.debug("skipping %s - already exists in %s", os.path.basename(src), dst)
         finally:


Test patch: