Repository: conda/conda

Issue:
Verify hangs when a package is corrupted
**I'm submitting a...**
  - [x] bug report
  - [ ] feature request


## Current Behavior

When installing a corrupted package, the verify step of conda hangs for a long time without telling anything.

When running the same with `-v -v -v` one can see errors of type: 

```
TRACE conda.core.path_actions:verify(269): retrying lexists(C:\Miniconda3\pkgs\boost-1.61.0-vc14h6b12553_2\Library\bin\boost_atomic-vc140-mt-1_61.dll) in 0.141842 sec
TRACE conda.core.path_actions:verify(269): retrying lexists(C:\Miniconda3\pkgs\boost-1.61.0-vc14h6b12553_2\Library\bin\boost_atomic-vc140-mt-1_61.dll) in 0.272421 sec
TRACE conda.core.path_actions:verify(269): retrying lexists(C:\Miniconda3\pkgs\boost-1.61.0-vc14h6b12553_2\Library\bin\boost_atomic-vc140-mt-1_61.dll) in 0.404178 sec
TRACE conda.core.path_actions:verify(269): retrying lexists(C:\Miniconda3\pkgs\boost-1.61.0-vc14h6b12553_2\Library\bin\boost_atomic-vc140-mt-1_61.dll) in 0.882638 sec
TRACE conda.core.path_actions:verify(269): retrying lexists(C:\Miniconda3\pkgs\boost-1.61.0-vc14h6b12553_2\Library\bin\boost_atomic-vc140-mt-1_61.dll) in 1.67327 sec
TRACE conda.core.path_actions:verify(269): retrying lexists(C:\Miniconda3\pkgs\boost-1.61.0-vc14h6b12553_2\Library\bin\boost_atomic-vc140-mt-1_61.dll) in 3.27765 sec
DEBUG conda.core.link:_verify_individual_level(337): Verification error in action LinkPathAction(target_prefix='C:\\Miniconda3', target_short_path='Library/bin/boost_atomic-vc140-mt-1_61.dll', source_short_path='Library/bin/boost_atomic-vc140-mt-1_61.dll', source_path_data=PathDataV1(_path='Library/bin/boost_atomic-vc140-mt-1_61.dll', path_type='hardlink', sha256='8b4a03d09a79377afd2a761175341ec9fdb6a13d867b44a2bca243a84541a391', size_in_bytes=10240), source_prefix='C:\\Miniconda3\\pkgs\\boost-1.61.0-vc14h6b12553_2', _execute_successful=False, link_type=<LinkType.hardlink: 1>, prefix_path_data=None)
conda.exceptions.CondaVerificationError: The package for boost located at C:\Miniconda3\pkgs\boost-1.61.0-vc14h6b12553_2
appears to be corrupted. The path 'Library/bin/boost_atomic-vc140-mt-1_61.dll'
specified in the package manifest cannot be found.


TRACE conda.core.path_actions:verify(269): retrying lexists(C:\Miniconda3\pkgs\boost-1.61.0-vc14h6b12553_2\Library\bin\boost_atomic-vc140-mt-gd-1_61.dll) in 0.156715 sec
TRACE conda.core.path_actions:verify(269): retrying lexists(C:\Miniconda3\pkgs\boost-1.61.0-vc14h6b12553_2\Library\bin\boost_atomic-vc140-mt-gd-1_61.dll) in 0.271798 sec
TRACE conda.core.path_actions:verify(269): retrying lexists(C:\Miniconda3\pkgs\boost-1.61.0-vc14h6b12553_2\Library\bin\boost_atomic-vc140-mt-gd-1_61.dll) in 0.462018 sec
TRACE conda.core.path_actions:verify(269): retrying lexists(C:\Miniconda3\pkgs\boost-1.61.0-vc14h6b12553_2\Library\bin\boost_atomic-vc140-mt-gd-1_61.dll) in 0.885679 sec
TRACE conda.core.path_actions:verify(269): retrying lexists(C:\Miniconda3\pkgs\boost-1.61.0-vc14h6b12553_2\Library\bin\boost_atomic-vc140-mt-gd-1_61.dll) in 1.67949 sec
TRACE conda.core.path_actions:verify(269): retrying lexists(C:\Miniconda3\pkgs\boost-1.61.0-vc14h6b12553_2\Library\bin\boost_atomic-vc140-mt-gd-1_61.dll) in 3.26532 sec
DEBUG conda.core.link:_verify_individual_level(337): Verification error in action LinkPathAction(target_prefix='C:\\Miniconda3', target_short_path='Library/bin/boost_atomic-vc140-mt-gd-1_61.dll', source_short_path='Library/bin/boost_atomic-vc140-mt-gd-1_61.dll', source_path_data=PathDataV1(_path='Library/bin/boost_atomic-vc140-mt-gd-1_61.dll', path_type='hardlink', sha256='df13333e7f1d40aa29fc0193a62d58d730691b09a0d291c981514206db9d25cd', size_in_bytes=16896), source_prefix='C:\\Miniconda3\\pkgs\\boost-1.61.0-vc14h6b12553_2', _execute_successful=False, link_type=<LinkType.hardlink: 1>, prefix_path_data=None)
conda.exceptions.CondaVerificationError: The package for boost located at C:\Miniconda3\pkgs\boost-1.61.0-vc14h6b12553_2
appears to be corrupted. The path 'Library/bin/boost_atomic-vc140-mt-gd-1_61.dll'
specified in the package manifest cannot be found.
``` 

So what happens is: conda finds out that the linking doesn't seem to work for a file, it retries this file for 3.2 seconds, fails and then continues with the next file - which also fails. 

Now as you can see, the package is boost and it has 12475 files. This would go on for about 10-12hours, without saying anything to the user.

We could not really reproduce why the package was corrupted in the first place but it seems after we cleaned the cache it worked..

### Steps to Reproduce
```
conda install <a package with missing files>
```


## Expected Behavior

Conda should stop after the first verification error, maybe rollback and tell the user that verification failed.



Discussion:
I just ran into this last night too.  I think we can probably remove all of that backoff code, because I don't think it actually ever helped with its intended purpose.
Same goes for me. I think what hit this off for my machine was when I was running a `conda update --all` and realized a mistake and ctrl-c stopped the thing. The problem persisted with clearing the cache, running `conda clean --packages --tarballs` or even `conda clean --all`.

What helped was going to the specific package in `/anaconda3/pkgs/` and removing it from there, reinstalling it later.

Resolving patch:
diff --git a/conda/core/path_actions.py b/conda/core/path_actions.py
--- a/conda/core/path_actions.py
+++ b/conda/core/path_actions.py
@@ -5,9 +5,7 @@
 from errno import EXDEV
 from logging import getLogger
 from os.path import basename, dirname, getsize, join
-from random import random
 import re
-from time import sleep
 from uuid import uuid4
 
 from .envs_manager import USER_ENVIRONMENTS_TXT_FILE, register_env, unregister_env
@@ -17,7 +15,7 @@
 from .._vendor.auxlib.ish import dals
 from ..base.constants import CONDA_TARBALL_EXTENSION
 from ..base.context import context
-from ..common.compat import iteritems, on_win, range, text_type
+from ..common.compat import iteritems, on_win, text_type
 from ..common.path import (get_bin_directory_short_path, get_leaf_directories,
                            get_python_noarch_target_path, get_python_short_path,
                            parse_entry_point_def,
@@ -255,31 +253,13 @@ def __init__(self, transaction_context, package_info,
 
     def verify(self):
         if self.link_type != LinkType.directory and not lexists(self.source_full_path):  # pragma: no cover  # NOQA
-            # This backoff loop is added because of some weird race condition conda-build
-            # experiences. Would be nice at some point to get to the bottom of why it happens.
-
-            # sum(((2 ** n) + random()) * 0.1 for n in range(2))
-            # with max_retries = 2, max total time ~= 0.4 sec
-            # with max_retries = 3, max total time ~= 0.8 sec
-            # with max_retries = 6, max total time ~= 6.5 sec
-            count = self.transaction_context.get('_verify_backoff_count', 0)
-            max_retries = 6 if count < 4 else 3
-            for n in range(max_retries):
-                sleep_time = ((2 ** n) + random()) * 0.1
-                log.trace("retrying lexists(%s) in %g sec", self.source_full_path, sleep_time)
-                sleep(sleep_time)
-                if lexists(self.source_full_path):
-                    break
-            else:
-                # only run the 6.5 second backoff time once
-                self.transaction_context['_verify_backoff_count'] = count + 1
-                return CondaVerificationError(dals("""
-                The package for %s located at %s
-                appears to be corrupted. The path '%s'
-                specified in the package manifest cannot be found.
-                """ % (self.package_info.index_json_record.name,
-                       self.package_info.extracted_package_dir,
-                       self.source_short_path)))
+            return CondaVerificationError(dals("""
+            The package for %s located at %s
+            appears to be corrupted. The path '%s'
+            specified in the package manifest cannot be found.
+            """ % (self.package_info.index_json_record.name,
+                   self.package_info.extracted_package_dir,
+                   self.source_short_path)))
 
         source_path_data = self.source_path_data
         try:


Test patch: