Repository: conda/conda

Issue:
conda appears to be stuck during update/install: Access Denied Error on Windows
Hi,

At work we have on occasion noticed that some of our build workers were taking unusually long during some `conda update` and `conda install` operations.

After enabling debugging with `-vv` and `-debug`, we identified that Conda was having trouble removing the `pkgs/.trash` folder. Here's the output we were seeing:

```
10:49:00 INFO conda.common.disk:delete_trash(235): Could not delete path in trash dir W:\Miniconda\pkgs\.trash\b095ce24-6c5c-4df1-aeb1-b726106d1f2b
10:49:00 WindowsError(5, 'Access is denied')
10:49:00 DEBUG conda.common.disk:exp_backoff_fn(154): WindowsError(5, 'Access is denied')
10:49:00 DEBUG conda.common.disk:exp_backoff_fn(163): retrying disk.py/59 <lambda>() in 0.147886 sec
10:49:00 DEBUG conda.common.disk:exp_backoff_fn(154): WindowsError(5, 'Access is denied')
10:49:00 DEBUG conda.common.disk:exp_backoff_fn(163): retrying disk.py/59 <lambda>() in 0.252876 sec
10:49:00 DEBUG conda.common.disk:exp_backoff_fn(154): WindowsError(5, 'Access is denied')
10:49:00 DEBUG conda.common.disk:exp_backoff_fn(163): retrying disk.py/59 <lambda>() in 0.43156 sec
10:49:00 DEBUG conda.common.disk:exp_backoff_fn(154): WindowsError(5, 'Access is denied')
10:49:00 DEBUG conda.common.disk:exp_backoff_fn(163): retrying disk.py/59 <lambda>() in 0.834403 sec
10:49:01 DEBUG conda.common.disk:exp_backoff_fn(154): WindowsError(5, 'Access is denied')
10:49:01 DEBUG conda.common.disk:exp_backoff_fn(163): retrying disk.py/59 <lambda>() in 1.69182 sec
10:49:03 DEBUG conda.common.disk:exp_backoff_fn(154): WindowsError(5, 'Access is denied')
10:49:03 DEBUG conda.common.disk:exp_backoff_fn(163): retrying disk.py/59 <lambda>() in 3.20222 sec
10:49:06 DEBUG conda.common.disk:exp_backoff_fn(154): WindowsError(5, 'Access is denied')
10:49:06 INFO conda.common.disk:delete_trash(235): Could not delete path in trash dir W:\Miniconda\pkgs\.trash\b1e43e9c-d723-4323-8ceb-3b834d67e4e4
10:49:06 WindowsError(5, 'Access is denied')
```

This could go on for dozens of minutes. 

Looking at this worker we noticed a few things:
- `conda` was the only Python process running at the time;
- Looking at code we noticed the docs for `exp_backoff_fn`:
  
  ``` python
  """Mostly for retrying file operations that fail on Windows due to virus scanners"""
  ```
  
  Some issues (#2947, #2983) also mention virus scanners, so we disabled the virus scanner we have ("ESET Endpoint Antivirus") but the problem persisted. 

Here's the output of `conda info`:

```
Current conda install:

               platform : win-64
          conda version : 4.2.9+1
       conda is private : False
      conda-env version : 4.2.9+1
    conda-build version : 1.20.0+1
         python version : 2.7.12.final.0
       requests version : 2.11.1
       root environment : W:\Miniconda  (writable)
    default environment : W:\Miniconda
       envs directories : W:\Miniconda\envs
          package cache : W:\Miniconda\pkgs
           channel URLs : https://eden.esss.com.br/conda-channel/esss/win-64/
                          https://eden.esss.com.br/conda-channel/esss/noarch/
                          https://eden.esss.com.br/conda-channel/mirror/win-64/
                          https://eden.esss.com.br/conda-channel/mirror/noarch/
                          https://eden.esss.com.br/conda-channel/mirror-conda-forge/win-64/
                          https://eden.esss.com.br/conda-channel/mirror-conda-forge/noarch/
            config file : W:\Miniconda\.condarc
           offline mode : False

```

Our current workaround was to remove the `pkgs/.trash` folder just before calling `conda update` or `conda install`.

Any ideas on how to debug this perhaps? Any help is greatly appreciated.



Discussion:
First, thanks for the awesome, detailed report.  I wish every issue filed was this descriptive :)

As you can tell from our code, we've been battling permissions issues like this on windows for a while now.

Question 1: What are the details of your file system/mount for your `W:\` drive?  Local? Network? NTFS? Other?

Question 2: 

> Our current workaround was to remove the pkgs/.trash folder just before calling conda update or conda install.

Are you doing this with the same automated script that is calling `conda update` or `conda install`?  What is the actual command you're using?

> First, thanks for the awesome, detailed report. I wish every issue filed was this descriptive :)

Thanks, it's the least I can do. 😁 

> Question 1: What are the details of your file system/mount for your W:\ drive? Local? Network? NTFS? Other?

That is a normal, local partition. In almost all of our slaves that's a separate hard-drive than the one where the system is installed.

A detail that perhaps it wasn't very clear is that we checked the permissions of the `.trash` folder and the files on it and they were fine. We can delete the files normally when logging in with the same user which runs our CI job once we kill the python/conda process.

> Are you doing this with the same automated script that is calling conda update or conda install? 
> What is the actual command you're using?

Yes, we are doing that in two scripts at the moment:
1. A powershell script which is run daily to provision the workers (ensure that conda is installed, update some choco packages, cleanup some workspaces, etc). That script is now running this command right before attempting to execute `conda install`:
   
   ``` ps
   $cir = conda info --root
   $trash_dir = "$($cir)\pkgs\.trash"
   if (Test-Path $trash_dir){
       Remove-Item -Recurse -Force $trash_dir
   }
   conda --debug update --all --yes --quiet 
   ```
2. A "setup worker" BAT file which executed at the start of each job, and calls `conda env update` to create a conda-environment:
   
   ``` bat
   for /F %%i in ('conda info --root') do set CONDA_ROOT=%%i
   set CONDA_TRASH=%CONDA_ROOT%\pkgs\.trash
   if exist %CONDA_TRASH% (
       echo Removing %CONDA_TRASH%
       rd /s /q %CONDA_TRASH%
   )
   @REM update conda and conda-env to the versions specified on this branch
   call conda install -n root --file "%WORKSPACE%\eden\conda_version.txt" || goto ERROR
   ```

The `conda install/update` calls above are the points where we were observing that conda was getting stuck.

We have a separate `W:` drive because we can easily quick format it and run the provisioning script to "wipe" a slave. The `W:` drive contains Miniconda, all packages and environments, as well as the jobs' workspaces.

Helpful comment from @ChrisBarker-NOAA on the conda message board.
https://groups.google.com/a/continuum.io/forum/#!msg/conda/9dPHBsAGqoQ/CrKdGLyfBgAJ

> Windows file locking in a pain in the %#^%
> 
> We have had problems that we have never been able to resolve. On my workstation, I ended up installing "unlocker"
> 
> http://www.iobit.com/en/iobit-unlocker.php
> 
> which is very nice for interactive GUI use -- but not helpful for conda. But I wll say that I end up using it a LOT!
> 
> Anyway, aside from virus scanners, backup software can be a really pain this way:
> 
> it detects a change in some files.
> it locks them while updating the backup
> now you can't delete them until it is done.
> 
> The proper solution here is configure your backup software to not back up stuff it shouldn't, but conda has no control over that.
> 
> However, it looks like at least in this case it's a .trash folder that's causing the problem ,so it seems conda should just give up (quickly)  if it can't delete that -- it's not fatal, is it?
> 
> Also -- maybe there is a way to make a system call to unlock files? woudl conda have the permission to do that?


Resolving patch:
diff --git a/conda/common/disk.py b/conda/common/disk.py
--- a/conda/common/disk.py
+++ b/conda/common/disk.py
@@ -20,6 +20,8 @@
 log = getLogger(__name__)
 
 
+MAX_TRIES = 7
+
 def try_write(dir_path, heavy=False):
     """Test write access to a directory.
 
@@ -50,20 +52,21 @@ def try_write(dir_path, heavy=False):
         return access(dir_path, W_OK)
 
 
-def backoff_unlink(file_or_symlink_path):
+def backoff_unlink(file_or_symlink_path, max_tries=MAX_TRIES):
     def _unlink(path):
         make_writable(path)
         unlink(path)
 
     try:
-        exp_backoff_fn(lambda f: lexists(f) and _unlink(f), file_or_symlink_path)
+        exp_backoff_fn(lambda f: lexists(f) and _unlink(f), file_or_symlink_path,
+                       max_tries=max_tries)
     except (IOError, OSError) as e:
         if e.errno not in (ENOENT,):
             # errno.ENOENT File not found error / No such file or directory
             raise
 
 
-def backoff_rmdir(dirpath):
+def backoff_rmdir(dirpath, max_tries=MAX_TRIES):
     if not isdir(dirpath):
         return
 
@@ -75,13 +78,13 @@ def backoff_rmdir(dirpath):
     def retry(func, path, exc_info):
         if getattr(exc_info[1], 'errno', None) == ENOENT:
             return
-        recursive_make_writable(dirname(path))
+        recursive_make_writable(dirname(path), max_tries=max_tries)
         func(path)
 
     def _rmdir(path):
         try:
             recursive_make_writable(path)
-            exp_backoff_fn(rmtree, path, onerror=retry)
+            exp_backoff_fn(rmtree, path, onerror=retry, max_tries=max_tries)
         except (IOError, OSError) as e:
             if e.errno == ENOENT:
                 log.debug("no such file or directory: %s", path)
@@ -120,7 +123,7 @@ def make_writable(path):
             raise
 
 
-def recursive_make_writable(path):
+def recursive_make_writable(path, max_tries=MAX_TRIES):
     # The need for this function was pointed out at
     #   https://github.com/conda/conda/issues/3266#issuecomment-239241915
     # Especially on windows, file removal will often fail because it is marked read-only
@@ -128,25 +131,25 @@ def recursive_make_writable(path):
         for root, dirs, files in walk(path):
             for path in chain.from_iterable((files, dirs)):
                 try:
-                    exp_backoff_fn(make_writable, join(root, path))
+                    exp_backoff_fn(make_writable, join(root, path), max_tries=max_tries)
                 except (IOError, OSError) as e:
                     if e.errno == ENOENT:
                         log.debug("no such file or directory: %s", path)
                     else:
                         raise
     else:
-        exp_backoff_fn(make_writable, path)
+        exp_backoff_fn(make_writable, path, max_tries=max_tries)
 
 
 def exp_backoff_fn(fn, *args, **kwargs):
     """Mostly for retrying file operations that fail on Windows due to virus scanners"""
+    max_tries = kwargs.pop('max_tries', MAX_TRIES)
     if not on_win:
         return fn(*args, **kwargs)
 
     import random
     # with max_tries = 6, max total time ~= 3.2 sec
     # with max_tries = 7, max total time ~= 6.5 sec
-    max_tries = 7
     for n in range(max_tries):
         try:
             result = fn(*args, **kwargs)
@@ -228,9 +231,9 @@ def delete_trash(prefix=None):
             path = join(trash_dir, p)
             try:
                 if isdir(path):
-                    backoff_rmdir(path)
+                    backoff_rmdir(path, max_tries=1)
                 else:
-                    backoff_unlink(path)
+                    backoff_unlink(path, max_tries=1)
             except (IOError, OSError) as e:
                 log.info("Could not delete path in trash dir %s\n%r", path, e)
         if listdir(trash_dir):


Test patch: