Repository: conda/conda

Issue:
os.rename throws exception when renaming directories on lower overlayfs layers
OS: CentOS Linux release 7.2.1511 (Core)
docker version: 1.12.6
docker storage driver: overlayfs

Docker Image has existing conda environment baked into layer

```
    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/opt/continuum/anaconda/lib/python3.6/site-packages/conda/exceptions.py", line 640, in conda_exception_handler
        return_value = func(*args, **kwargs)
      File "/opt/continuum/anaconda/lib/python3.6/site-packages/conda/cli/main.py", line 140, in _main
        exit_code = args.func(args, p)
      File "/opt/continuum/anaconda/lib/python3.6/site-packages/conda/cli/main_install.py", line 80, in execute
        install(args, parser, 'install')
      File "/opt/continuum/anaconda/lib/python3.6/site-packages/conda/cli/install.py", line 326, in install
        execute_actions(actions, index, verbose=not context.quiet)
      File "/opt/continuum/anaconda/lib/python3.6/site-packages/conda/plan.py", line 830, in execute_actions
        execute_instructions(plan, index, verbose)
      File "/opt/continuum/anaconda/lib/python3.6/site-packages/conda/instructions.py", line 247, in execute_instructions
        cmd(state, arg)
      File "/opt/continuum/anaconda/lib/python3.6/site-packages/conda/instructions.py", line 108, in UNLINKLINKTRANSACTION_CMD
        txn.execute()
      File "/opt/continuum/anaconda/lib/python3.6/site-packages/conda/core/link.py", line 297, in execute
        rollback_excs,
    conda.CondaMultiError: [Errno 22] Invalid argument: '/opt/continuum/anaconda/envs/anaconda44_py35/lib/python3.5/version.txt' -> '/opt/continuum/anaconda/envs/anaconda44_py35/lib/python3.5/version.txt.c~'
```

cc @msarahan @kalefranz @mmarchetti @dsludwig 


Discussion:
Can you give me the full stack trace?  That just looks like part of it...

> `During handling of the above exception, another exception occurred:`
```
$ /opt/continuum/anaconda/envs/anaconda44_py35/bin/conda install --yes --prefix /opt/continuum/anaconda/envs/anaconda44_py35 --channel anaconda-enterprise anaconda-enterprise jupyter_kernel_gateway anaconda-enterprise-web-publisher --debug`




    Traceback (most recent call last):
      File "/opt/continuum/anaconda/lib/python3.6/site-packages/conda/core/link.py", line 326, in _execute_actions
        action.execute()
      File "/opt/continuum/anaconda/lib/python3.6/site-packages/conda/core/path_actions.py", line 620, in execute
        backoff_rename(self.target_full_path, self.holding_full_path, force=True)
      File "/opt/continuum/anaconda/lib/python3.6/site-packages/conda/gateways/disk/update.py", line 53, in backoff_rename
        exp_backoff_fn(rename, source_path, destination_path, force)
      File "/opt/continuum/anaconda/lib/python3.6/site-packages/conda/gateways/disk/__init__.py", line 21, in exp_backoff_fn
        return fn(*args, **kwargs)
      File "/opt/continuum/anaconda/lib/python3.6/site-packages/conda/gateways/disk/update.py", line 47, in rename
        os_rename(source_path, destination_path)
    OSError: [Errno 22] Invalid argument: '/opt/continuum/anaconda/envs/anaconda44_py35/bin/anaconda-project' -> '/opt/continuum/anaconda/envs/anaconda44_py35/bin/anaconda-project.c~'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/opt/continuum/anaconda/lib/python3.6/site-packages/conda/core/link.py", line 281, in execute
        pkg_data, actions)
      File "/opt/continuum/anaconda/lib/python3.6/site-packages/conda/core/link.py", line 344, in _execute_actions
        reverse_excs,
    conda.CondaMultiError: [Errno 22] Invalid argument: '/opt/continuum/anaconda/envs/anaconda44_py35/bin/anaconda-project' -> '/opt/continuum/anaconda/envs/anaconda44_py35/bin/anaconda-project.c~'


    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/opt/continuum/anaconda/lib/python3.6/site-packages/conda/exceptions.py", line 640, in conda_exception_handler
        return_value = func(*args, **kwargs)
      File "/opt/continuum/anaconda/lib/python3.6/site-packages/conda/cli/main.py", line 140, in _main
        exit_code = args.func(args, p)
      File "/opt/continuum/anaconda/lib/python3.6/site-packages/conda/cli/main_install.py", line 80, in execute
        install(args, parser, 'install')
      File "/opt/continuum/anaconda/lib/python3.6/site-packages/conda/cli/install.py", line 326, in install
        execute_actions(actions, index, verbose=not context.quiet)
      File "/opt/continuum/anaconda/lib/python3.6/site-packages/conda/plan.py", line 830, in execute_actions
        execute_instructions(plan, index, verbose)
      File "/opt/continuum/anaconda/lib/python3.6/site-packages/conda/instructions.py", line 247, in execute_instructions
        cmd(state, arg)
      File "/opt/continuum/anaconda/lib/python3.6/site-packages/conda/instructions.py", line 108, in UNLINKLINKTRANSACTION_CMD
        txn.execute()
      File "/opt/continuum/anaconda/lib/python3.6/site-packages/conda/core/link.py", line 297, in execute
        rollback_excs,
    conda.CondaMultiError: [Errno 22] Invalid argument: '/opt/continuum/anaconda/envs/anaconda44_py35/bin/anaconda-project' -> '/opt/continuum/anaconda/envs/anaconda44_py35/bin/anaconda-project.c~'
```
Related reference: https://github.com/moby/moby/issues/25409
errno 22 is `EINVAL`
    #define EINVAL          22      /* Invalid argument */
NPM discussion: https://github.com/npm/npm/issues/9863

Might no just be errno 22.  moby/moby#25409 has

    OSError: [Errno 18] Invalid cross-device link: '/foo' -> '/foo2'

which is `EXDEV`.
> rename(2): OverlayFS does not fully support the rename(2) system call. Your application needs to detect its failure and fall back to a “copy and unlink” strategy.

https://docs.docker.com/engine/userguide/storagedriver/overlayfs-driver/#limitations-on-overlayfs-compatibility

Resolving patch:
diff --git a/conda/gateways/disk/update.py b/conda/gateways/disk/update.py
--- a/conda/gateways/disk/update.py
+++ b/conda/gateways/disk/update.py
@@ -1,6 +1,7 @@
 # -*- coding: utf-8 -*-
 from __future__ import absolute_import, division, print_function, unicode_literals
 
+from errno import EINVAL, EXDEV
 from logging import getLogger
 from os import rename as os_rename, utime
 from os.path import dirname, isdir, lexists
@@ -8,6 +9,7 @@
 
 from . import exp_backoff_fn
 from .delete import rm_rf
+from .link import islink
 from ...common.path import expand
 
 log = getLogger(__name__)
@@ -39,12 +41,30 @@ def update_file_in_place_as_binary(file_full_path, callback):
             fh.close()
 
 
+def _copy_then_remove(source_path, destination_path):
+    from .create import copy, create_hard_link_or_copy
+    if islink(source_path):
+        copy(source_path, destination_path)
+    else:
+        create_hard_link_or_copy(source_path, destination_path)
+    rm_rf(source_path)
+
+
 def rename(source_path, destination_path, force=False):
     if lexists(destination_path) and force:
         rm_rf(destination_path)
     if lexists(source_path):
         log.trace("renaming %s => %s", source_path, destination_path)
-        os_rename(source_path, destination_path)
+        try:
+            os_rename(source_path, destination_path)
+        except EnvironmentError as e:
+            if e.errno in (EINVAL, EXDEV):
+                # see https://github.com/conda/conda/issues/6711
+                log.trace("Could not rename do to errno [%s]. Falling back to copy/remove.",
+                          e.errno)
+                _copy_then_remove(source_path, destination_path)
+            else:
+                raise
     else:
         log.trace("cannot rename; source path does not exist '%s'", source_path)
 


Test patch: