Repository: conda/conda

Issue:
conda 4.3.6 broken - Uncaught backoff with errno EEXIST 17
Trying to update all broke with a *file already exists* error when trying to install the packages:
```
C:\dev\code\sandbox> conda update --all
Fetching package metadata .................
Solving package specifications: .

Package plan for installation in environment C:\bin\Anaconda:

The following packages will be UPDATED:
<SNIP>
ipywidgets-5.2 100% |###############################| Time: 0:00:00   8.57 MB/s
pymc3-3.0-py35 100% |###############################| Time: 0:00:01 152.30 kB/s
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno EEXIST 17
ERROR conda.core.link:_execute_actions(318): An error occurred while uninstalling package 'conda-forge::jupyterlab-0.12.
1-py35_0'.
FileExistsError(17, 'Cannot create a file when that file already exists')
Attempting to roll back.

WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno EEXIST 17
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno EEXIST 17
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno EEXIST 17
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno EEXIST 17
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno EEXIST 17
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno EEXIST 17
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno EEXIST 17
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno EEXIST 17
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno EEXIST 17
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno EEXIST 17
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno EEXIST 17
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno EEXIST 17
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno EEXIST 17
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno EEXIST 17
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno EEXIST 17


FileExistsError(17, 'Cannot create a file when that file already exists')
FileExistsError(17, 'Cannot create a file when that file already exists')
FileExistsError(17, 'Cannot create a file when that file already exists')
FileExistsError(17, 'Cannot create a file when that file already exists')
FileExistsError(17, 'Cannot create a file when that file already exists')
FileExistsError(17, 'Cannot create a file when that file already exists')
FileExistsError(17, 'Cannot create a file when that file already exists')
FileExistsError(17, 'Cannot create a file when that file already exists')
FileExistsError(17, 'Cannot create a file when that file already exists')
FileExistsError(17, 'Cannot create a file when that file already exists')
FileExistsError(17, 'Cannot create a file when that file already exists')
FileExistsError(17, 'Cannot create a file when that file already exists')
FileExistsError(17, 'Cannot create a file when that file already exists')
FileExistsError(17, 'Cannot create a file when that file already exists')
FileExistsError(17, 'Cannot create a file when that file already exists')
FileExistsError(17, 'Cannot create a file when that file already exists')
```
Trying again results in the same error (with a different package this time) leaving me unable to update my environment
```
C:\dev\code\sandbox> conda update --all
Fetching package metadata .................
Solving package specifications: .

Package plan for installation in environment C:\bin\Anaconda:

The following packages will be UPDATED:
<SNIP>
Proceed ([y]/n)? y

icu-58.1-vc14_ 100% |###############################| Time: 0:00:15 782.64 kB/s
libxml2-2.9.3- 100% |###############################| Time: 0:00:04 754.62 kB/s
qt-4.8.7-vc14_ 100% |###############################| Time: 0:00:34   1.51 MB/s
pyqt-4.11.4-py 100% |###############################| Time: 0:00:00   9.97 MB/s
matplotlib-2.0 100% |###############################| Time: 0:00:06   1.06 MB/s
anaconda-navig 100% |###############################| Time: 0:00:02   1.51 MB/s
qtconsole-4.2. 100% |###############################| Time: 0:00:00  14.07 MB/s
WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno EEXIST 17
ERROR conda.core.link:_execute_actions(318): An error occurred while uninstalling package 'conda-forge::flask-cors-3.0.2-py35_0'.
FileExistsError(17, 'Cannot create a file when that file already exists')
Attempting to roll back.

WARNING conda.gateways.disk:exp_backoff_fn(47): Uncaught backoff with errno EEXIST 17


FileExistsError(17, 'Cannot create a file when that file already exists')
FileExistsError(17, 'Cannot create a file when that file already exists')
```


Discussion:
Using conda 4.3.6 on py35 / win64.

Downgrading to conda 4.2.15 allowed me to complete the update.
I'd love to know where this error came from and fix it, but I need more information.  Can you still reproduce it?  If so, can you give me the output with the addition of `-v -v -v`?
@kalefranz - I reverted to the previous revision, reinstalled 4.3.6 and re-ran conda update with `-v -v -v` which appeared to fail with the same error.

Log available at - https://gist.github.com/dhirschfeld/2fa4c615e09c33d03f86f9d95deefcbf
NB: 4.3.7 also has the same problem for me

Resolving patch:
diff --git a/conda/core/path_actions.py b/conda/core/path_actions.py
--- a/conda/core/path_actions.py
+++ b/conda/core/path_actions.py
@@ -601,12 +601,12 @@ def __init__(self, transaction_context, linked_package_data, target_prefix, targ
     def execute(self):
         if self.link_type != LinkType.directory:
             log.trace("renaming %s => %s", self.target_short_path, self.holding_short_path)
-            backoff_rename(self.target_full_path, self.holding_full_path)
+            backoff_rename(self.target_full_path, self.holding_full_path, force=True)
 
     def reverse(self):
         if self.link_type != LinkType.directory and lexists(self.holding_full_path):
             log.trace("reversing rename %s => %s", self.holding_short_path, self.target_short_path)
-            backoff_rename(self.holding_full_path, self.target_full_path)
+            backoff_rename(self.holding_full_path, self.target_full_path, force=True)
 
     def cleanup(self):
         if self.link_type == LinkType.directory:
@@ -718,7 +718,7 @@ def execute(self):
                 # the source and destination are the same file, so we're done
                 return
             else:
-                backoff_rename(self.target_full_path, self.hold_path)
+                backoff_rename(self.target_full_path, self.hold_path, force=True)
 
         if self.url.startswith('file:/'):
             source_path = url_to_path(self.url)
@@ -760,8 +760,7 @@ def execute(self):
     def reverse(self):
         if lexists(self.hold_path):
             log.trace("moving %s => %s", self.hold_path, self.target_full_path)
-            rm_rf(self.target_full_path)
-            backoff_rename(self.hold_path, self.target_full_path)
+            backoff_rename(self.hold_path, self.target_full_path, force=True)
 
     def cleanup(self):
         rm_rf(self.hold_path)
diff --git a/conda/gateways/disk/update.py b/conda/gateways/disk/update.py
--- a/conda/gateways/disk/update.py
+++ b/conda/gateways/disk/update.py
@@ -2,19 +2,17 @@
 from __future__ import absolute_import, division, print_function, unicode_literals
 
 from logging import getLogger
-from os import rename, utime
+from os import rename as os_rename, utime
 from os.path import lexists
 import re
 
 from conda._vendor.auxlib.path import expand
+from conda.gateways.disk.delete import rm_rf
 
 from . import exp_backoff_fn
 
 log = getLogger(__name__)
 
-# in the rest of conda's code, os.rename is preferably imported from here
-rename = rename
-
 SHEBANG_REGEX = re.compile(br'^(#!((?:\\ |[^ \n\r])+)(.*))')
 
 
@@ -42,13 +40,18 @@ def update_file_in_place_as_binary(file_full_path, callback):
             fh.close()
 
 
-def backoff_rename(source_path, destination_path):
+def rename(source_path, destination_path, force=False):
+    if lexists(destination_path) and force:
+        rm_rf(destination_path)
     if lexists(source_path):
         log.trace("renaming %s => %s", source_path, destination_path)
-        exp_backoff_fn(rename, source_path, destination_path)
+        os_rename(source_path, destination_path)
     else:
         log.trace("cannot rename; source path does not exist '%s'", source_path)
-    return
+
+
+def backoff_rename(source_path, destination_path, force=False):
+    exp_backoff_fn(rename, source_path, destination_path, force)
 
 
 def touch(path):


Test patch:
diff --git a/conda/gateways/disk/test.py b/conda/gateways/disk/test.py
--- a/conda/gateways/disk/test.py
+++ b/conda/gateways/disk/test.py
@@ -53,15 +53,20 @@ def hardlink_supported(source_file, dest_dir):
     # file system configuration, a symbolic link may be created
     # instead. If a symbolic link is created instead of a hard link,
     # return False.
-    log.trace("checking hard link capability for %s => %s", source_file, dest_dir)
     test_file = join(dest_dir, '.tmp.' + basename(source_file))
     assert isfile(source_file), source_file
     assert isdir(dest_dir), dest_dir
     assert not lexists(test_file), test_file
     try:
         create_link(source_file, test_file, LinkType.hardlink, force=True)
-        return not islink(test_file)
+        is_supported = not islink(test_file)
+        if is_supported:
+            log.trace("hard link supported for %s => %s", source_file, dest_dir)
+        else:
+            log.trace("hard link IS NOT supported for %s => %s", source_file, dest_dir)
+        return is_supported
     except (IOError, OSError):
+        log.trace("hard link IS NOT supported for %s => %s", source_file, dest_dir)
         return False
     finally:
         rm_rf(test_file)