Repository: conda/conda

Issue:
[Regression] Conda create environment fails on lock if root environment is not under user control
This issue is introduced in Conda 4.1.0 (Conda 4.0.8 works fine).

```
$ conda create -n root2 python=2                                                                                                                                                                   [123/1811]
Fetching package metadata .......
Solving package specifications .............

Package plan for installation in environment /home/frol/.conda/envs/root2:

The following NEW packages will be INSTALLED:

    openssl:    1.0.2h-1      (soft-link)
    pip:        8.1.2-py27_0  (soft-link)
    python:     2.7.11-0      (soft-link)
    readline:   6.2-2         (soft-link)
    setuptools: 23.0.0-py27_0 (soft-link)
    sqlite:     3.13.0-0      (soft-link)
    tk:         8.5.18-0      (soft-link)
    wheel:      0.29.0-py27_0 (soft-link)
    zlib:       1.2.8-3       (soft-link)

Proceed ([y]/n)?

Linking packages ...
An unexpected error has occurred, please consider sending the
following traceback to the conda GitHub issue tracker at:

    https://github.com/conda/conda/issues

Include the output of the command 'conda info' in your report.


Traceback (most recent call last):
  File "/usr/local/miniconda/bin/conda", line 6, in <module>
    sys.exit(main())
  File "/usr/local/miniconda/lib/python2.7/site-packages/conda/cli/main.py", line 120, in main
    args_func(args, p)
  File "/usr/local/miniconda/lib/python2.7/site-packages/conda/cli/main.py", line 127, in args_func
    args.func(args, p)
  File "/usr/local/miniconda/lib/python2.7/site-packages/conda/cli/main_create.py", line 57, in execute
    install(args, parser, 'create')
  File "/usr/local/miniconda/lib/python2.7/site-packages/conda/cli/install.py", line 407, in install
    execute_actions(actions, index, verbose=not args.quiet)
  File "/usr/local/miniconda/lib/python2.7/site-packages/conda/plan.py", line 566, in execute_actions
    inst.execute_instructions(plan, index, verbose)
  File "/usr/local/miniconda/lib/python2.7/site-packages/conda/instructions.py", line 137, in execute_instructions
    cmd(state, arg)
  File "/usr/local/miniconda/lib/python2.7/site-packages/conda/instructions.py", line 80, in LINK_CMD
    link(state['prefix'], dist, lt, index=state['index'], shortcuts=shortcuts)
  File "/usr/local/miniconda/lib/python2.7/site-packages/conda/install.py", line 1035, in link
    with Locked(prefix), Locked(pkgs_dir):
  File "/usr/local/miniconda/lib/python2.7/site-packages/conda/lock.py", line 60, in __enter__
    os.makedirs(self.lock_path)
  File "/usr/local/miniconda/lib/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/usr/local/miniconda/pkgs/.conda_lock-949'
```

`/usr/local/miniconda/` is a system-wide installation of miniconda, so obviously, users cannot create lock files there.

P.S. I have a dream that updating conda software won't break things on every release...



Discussion:
It seems that I cannot even do `source activate ...` as a regular user now. It just hangs. Here is what I get when I interrupt it with `^C`:

```
$ source activate root2
^CTraceback (most recent call last):
  File "/usr/local/miniconda/bin/conda", line 6, in <module>
    sys.exit(main())
  File "/usr/local/miniconda/lib/python2.7/site-packages/conda/cli/main.py", line 48, in main
    activate.main()
  File "/usr/local/miniconda/lib/python2.7/site-packages/conda/cli/activate.py", line 121, in main
    path = get_path(shelldict)
  File "/usr/local/miniconda/lib/python2.7/site-packages/conda/cli/activate.py", line 108, in get_path
    return run_in(shelldict["printpath"], shelldict)[0]
  File "/usr/local/miniconda/lib/python2.7/site-packages/conda/utils.py", line 174, in run_in
    stdout, stderr = p.communicate()
  File "/usr/local/miniconda/lib/python2.7/subprocess.py", line 799, in communicate
    return self._communicate(input)
  File "/usr/local/miniconda/lib/python2.7/subprocess.py", line 1409, in _communicate
    stdout, stderr = self._communicate_with_poll(input)
  File "/usr/local/miniconda/lib/python2.7/subprocess.py", line 1463, in _communicate_with_poll
    ready = poller.poll()
KeyboardInterrupt
```

It also seems like there is no way to pin Conda version. Installing any package to the root env tries to update Conda to the latest version:

```
$ conda install python
Using Anaconda Cloud api site https://api.anaconda.org
Fetching package metadata: ....
Solving package specifications: .........

Package plan for installation in environment /usr/local/miniconda:

The following packages will be UPDATED:

    conda:     4.0.8-py27_0 --> 4.1.0-py27_0
    conda-env: 2.4.5-py27_0 --> 2.5.0-py27_0

Proceed ([y]/n)?
```

You can now pin 4.0.9.

```
conda install conda=4.0.9
conda config --set auto_update_conda false
```

`auto_update_conda` setting will be added to 4.1.1 also, which is coming out tonight or tomorrow morning.

Freaking lock on the whole package cache.  We're going to get rid of that soon.  I can't really tell what initially tripped here.  What are the permissions on `/usr/local/miniconda/pkgs/.conda_lock-949`?  Are they yours, or root, or does it even exist?

Ohhh I think I get it, from your issue title.  I actually thought this was already broken anyway.  I plan on having 4.2 out in a couple weeks now, and should be a simple fix at that point.  Ok with staying on 4.0.9 for this use case until then?

@kalefranz It seems that I don't have options here but wait. I use Docker containers and this bug doesn't bother me that much. However, I would say that it is not a minor regression to postpone it to the next release. (From my experience, next release will break things in another way, so people will stuck with 4.0.*)

In the last several months our coverage has gone from ~48% to almost 70%. The code base is still far more fragile and brittle than I'd like it to be, but we're making progress I think.

Reverting #2320 fixed the regression. However, it seems that it just fails silently at locking, but at least it works in non-concurrent scenarios.

cc @alanhdu

@frol: Yeah, that's about right. Before #2320, conda would just swallow all `OsError`s (including `PermissionError`s). 

For now, we could add a `try ... except` around https://github.com/conda/conda/blob/master/conda/lock.py#L60, catch a `PermissionError` (or whatever the equivalent Python 2 error is), and try to do something smart (or just silently fail... depends how important that lock actually is).

I am seeing this issue as well @kalefranz 

After reading this, I thought I would be able to downgrade conda to 4.0.9. However after doing that, I am still unable to activate centrally administered conda environments as a user. Is there a prior version of conda you would reccommend? Or did the 4.1.3 version leave something in my miniconda install that is causing the problem? Do I need to re-install miniconda2 from scratch? Should I just wait for this to be fixed before proceeding with trying to build a central conda install for our users? 

@davidslac There are two options you may try:
1. Downgrade conda-env together with conda:
   
   ``` bash
   $ conda install conda=4.0.9 'conda-env<2.5'
   ```
2. Patch (revert changes) conda lock in Conda 4.1.x:
   
   ``` bash
   $ curl -o "/usr/local/miniconda/lib/python"*"/site-packages/conda/lock.py" \
            "https://raw.githubusercontent.com/conda/conda/9428ad0b76be55e8070e04dd577c96e7dab571e0/conda/lock.py"
   ```

Thanks! I tried both, but it sill did not work. It's probably something 
I'm overlooking on my part - after the first failure I deleted the 
lock.pyc, but still no luck.

I'll just hope for a fix in the future. I suspect though, that central 
administration is not as standard of a use case, so one is more likely 
to run into problems. It may be we should just provide a channel of our 
packages to our users and let them administer their own software stacks, 
give them some environments we know work.

best,

David

On 06/27/16 10:58, Vlad Frolov wrote:

> @davidslac https://github.com/davidslac There are two options you 
> may try:
> 
> 1.
> 
> ```
> Downgrade conda-env:
> 
> $ conda install'conda-env<2.5'
> ```
> 
> 2.
> 
> ```
> Patch (revert changes) conda lock:
> 
> $ curl -o"/usr/local/miniconda/lib/python"*"/site-packages/conda/lock.py"  \
>           "https://raw.githubusercontent.com/conda/conda/9428ad0b76be55e8070e04dd577c96e7dab571e0/conda/lock.py"
> ```


Resolving patch:
diff --git a/conda/egg_info.py b/conda/egg_info.py
--- a/conda/egg_info.py
+++ b/conda/egg_info.py
@@ -15,6 +15,7 @@
 from .misc import rel_path
 
 
+
 def get_site_packages_dir(installed_pkgs):
     for info in itervalues(installed_pkgs):
         if info['name'] == 'python':
diff --git a/conda/exceptions.py b/conda/exceptions.py
--- a/conda/exceptions.py
+++ b/conda/exceptions.py
@@ -8,6 +8,3 @@ class InvalidInstruction(CondaException):
     def __init__(self, instruction, *args, **kwargs):
         msg = "No handler for instruction: %r" % instruction
         super(InvalidInstruction, self).__init__(msg, *args, **kwargs)
-
-class LockError(RuntimeError, CondaException):
-    pass
diff --git a/conda/lock.py b/conda/lock.py
--- a/conda/lock.py
+++ b/conda/lock.py
@@ -17,11 +17,11 @@
 """
 from __future__ import absolute_import, division, print_function
 
-import logging
 import os
-import time
-
-from .exceptions import LockError
+import logging
+from os.path import join
+import glob
+from time import sleep
 
 LOCKFN = '.conda_lock'
 
@@ -33,13 +33,15 @@ class Locked(object):
     """
     Context manager to handle locks.
     """
-    def __init__(self, path, retries=10):
+    def __init__(self, path):
         self.path = path
         self.end = "-" + str(os.getpid())
-        self.lock_path = os.path.join(self.path, LOCKFN + self.end)
-        self.retries = retries
+        self.lock_path = join(self.path, LOCKFN + self.end)
+        self.pattern = join(self.path, LOCKFN + '-*')
+        self.remove = True
 
     def __enter__(self):
+        retries = 10
         # Keep the string "LOCKERROR" in this string so that external
         # programs can look for it.
         lockstr = ("""\
@@ -48,24 +50,33 @@ def __enter__(self):
     If you are sure that conda is not running, remove it and try again.
     You can also use: $ conda clean --lock\n""")
         sleeptime = 1
-
-        for _ in range(self.retries):
-            if os.path.isdir(self.lock_path):
-                stdoutlog.info(lockstr % self.lock_path)
+        files = None
+        while retries:
+            files = glob.glob(self.pattern)
+            if files and not files[0].endswith(self.end):
+                stdoutlog.info(lockstr % str(files))
                 stdoutlog.info("Sleeping for %s seconds\n" % sleeptime)
-
-                time.sleep(sleeptime)
+                sleep(sleeptime)
                 sleeptime *= 2
+                retries -= 1
             else:
-                os.makedirs(self.lock_path)
-                return self
+                break
+        else:
+            stdoutlog.error("Exceeded max retries, giving up")
+            raise RuntimeError(lockstr % str(files))
 
-        stdoutlog.error("Exceeded max retries, giving up")
-        raise LockError(lockstr % self.lock_path)
+        if not files:
+            try:
+                os.makedirs(self.lock_path)
+            except OSError:
+                pass
+        else:  # PID lock already here --- someone else will remove it.
+            self.remove = False
 
     def __exit__(self, exc_type, exc_value, traceback):
-        try:
-            os.rmdir(self.lock_path)
-            os.rmdir(self.path)
-        except OSError:
-            pass
+        if self.remove:
+            for path in self.lock_path, self.path:
+                try:
+                    os.rmdir(path)
+                except OSError:
+                    pass


Test patch:
diff --git a/tests/test_exceptions.py b/tests/test_exceptions.py
--- a/tests/test_exceptions.py
+++ b/tests/test_exceptions.py
@@ -18,7 +18,3 @@ def test_creates_message_with_instruction_name(self):
         e = exceptions.InvalidInstruction(random_instruction)
         expected = "No handler for instruction: %s" % random_instruction
         self.assertEqual(expected, str(e))
-
-def test_lockerror_hierarchy():
-    assert issubclass(exceptions.LockError, exceptions.CondaException)
-    assert issubclass(exceptions.LockError, RuntimeError)
diff --git a/tests/test_lock.py b/tests/test_lock.py
deleted file mode 100644
--- a/tests/test_lock.py
+++ /dev/null
@@ -1,32 +0,0 @@
-import os.path
-import pytest
-
-from conda.lock import Locked, LockError
-
-
-def test_lock_passes(tmpdir):
-    with Locked(tmpdir.strpath) as lock:
-        path = os.path.basename(lock.lock_path)
-        assert tmpdir.join(path).exists() and tmpdir.join(path).isdir()
-
-    # lock should clean up after itself
-    assert not tmpdir.join(path).exists()
-    assert not tmpdir.exists()
-
-def test_lock_locks(tmpdir):
-    with Locked(tmpdir.strpath) as lock1:
-        path = os.path.basename(lock1.lock_path)
-        assert tmpdir.join(path).exists() and tmpdir.join(path).isdir()
-
-        with pytest.raises(LockError) as execinfo:
-            with Locked(tmpdir.strpath, retries=1) as lock2:
-                assert False  # this should never happen
-            assert lock2.lock_path == lock1.lock_path
-        assert "LOCKERROR" in str(execinfo)
-        assert "conda is already doing something" in str(execinfo)
-
-        assert tmpdir.join(path).exists() and tmpdir.join(path).isdir()
-
-    # lock should clean up after itself
-    assert not tmpdir.join(path).exists()
-    assert not tmpdir.exists()