Repository: conda/conda

Issue:
Conda 4.2.x attempts to create lock files in readonly file-based channels and does not respect CONDA_ENVS_PATH
I am getting warnings and eventually an error after upgrading from 4.1.11 to 4.2.7 relating to lock files and packages attempting to write to shared read-only directories in our continuous build.

I modified `/shared/path/to/miniconda/lib/python2.7/site-packages/conda/lock.py` to find out where conda 4.2.x was putting lock files and found the following:

`WARNING conda.lock:touch(55): Failed to create lock, do not run conda in parallel processes [errno 13] /shared/path/to/vendor/conda/linux-64.pid24915.conda_lock`

`WARNING conda.lock:touch(55): Failed to create lock, do not run conda in parallel processes [errno 13] /shared/path/to/miniconda/pkgs/python-dateutil-2.4.2-py27_0.tar.bz2.pid24915.conda_lock`

`/shared/path/to` is read-only because it is used by multiple build-agents concurrently.

Additional I encountered the following error:

`CondaRuntimeError: Runtime error: RuntimeError: Runtime error: Could not open u'/shared/path/to/miniconda/pkgs/python-dateutil-2.4.2-py27_0.tar.bz2.part' for writing ([Errno 13] Permission denied: u'/shared/path/to/miniconda/pkgs/python-dateutil-2.4.2-py27_0.tar.bz2.part').`

In conda <= 4.1.11 we were able to set the `CONDA_ENVS_PATH` to have a shared installation of conda, but with separate package caches/lock files per build agent on our continuous build server, in order to avoid concurrent use of the package cache.

Two questions:
- How do we disable lock files being placed in read-only shared file-based channels.
- Does `CONDA_ENVS_PATH` no longer override the package cache/lock file directory from the root?

Conda 4.2.x attempts to create lock files in readonly file-based channels and does not respect CONDA_ENVS_PATH
I am getting warnings and eventually an error after upgrading from 4.1.11 to 4.2.7 relating to lock files and packages attempting to write to shared read-only directories in our continuous build.

I modified `/shared/path/to/miniconda/lib/python2.7/site-packages/conda/lock.py` to find out where conda 4.2.x was putting lock files and found the following:

`WARNING conda.lock:touch(55): Failed to create lock, do not run conda in parallel processes [errno 13] /shared/path/to/vendor/conda/linux-64.pid24915.conda_lock`

`WARNING conda.lock:touch(55): Failed to create lock, do not run conda in parallel processes [errno 13] /shared/path/to/miniconda/pkgs/python-dateutil-2.4.2-py27_0.tar.bz2.pid24915.conda_lock`

`/shared/path/to` is read-only because it is used by multiple build-agents concurrently.

Additional I encountered the following error:

`CondaRuntimeError: Runtime error: RuntimeError: Runtime error: Could not open u'/shared/path/to/miniconda/pkgs/python-dateutil-2.4.2-py27_0.tar.bz2.part' for writing ([Errno 13] Permission denied: u'/shared/path/to/miniconda/pkgs/python-dateutil-2.4.2-py27_0.tar.bz2.part').`

In conda <= 4.1.11 we were able to set the `CONDA_ENVS_PATH` to have a shared installation of conda, but with separate package caches/lock files per build agent on our continuous build server, in order to avoid concurrent use of the package cache.

Two questions:
- How do we disable lock files being placed in read-only shared file-based channels.
- Does `CONDA_ENVS_PATH` no longer override the package cache/lock file directory from the root?



Discussion:
Re: lock files, ran into the same problem today -- conda tries to create lock files for readonly file-based channels which yields a ton of warnings:

``` sh
$ conda create -p ./foobar python=3.5
WARNING conda.lock:touch(53): Failed to create lock, do not run conda in parallel processes [errno 13]
WARNING conda.lock:touch(53): Failed to create lock, do not run conda in parallel processes [errno 13]
WARNING conda.lock:touch(53): Failed to create lock, do not run conda in parallel processes [errno 13]
```

This is a regression from 4.1.x, please fix.

@HugoTian @kalefranz ping

By the way, why does it try to lock at all when all we're doing is reading from an external channel? I get that package cache lock makes sense since it's inherently a mutable thing, but why this?

Anyone?..

We're bumping into this one as well. Has anyone got any info yet?

This is blocking our upgrade to 4.2 as well.

This issue with creating locks in read-only filesystem channels is a breaking our workflow as well

@kalefranz :panda_face: :panda_face: :panda_face: :panda_face: :panda_face: :panda_face: :panda_face: :panda_face: :panda_face: :panda_face: 

@aldanor , I just check the history, previously, failing create lock will be handled with debug level logging . So there is no warning info. 

I think there may be 2 solutions, first we change it back to debug, and there will be no warning. Second, we give another detailed warning based on exception types.

@gnarlie , @eyew , can you give us more info about the exception or error that you encountered during upgrade or workflow ?

Thanks in advance

@HugoTian why does Conda attempt to put a lockfile in a file-based channel in the first place?  Are you saying all clients of a file-based channel need write access?  How does that work with http channels?

@HugoTian The error message I get is: Runtime error: Could not open u/path_to_package/pkgs/packagename.tar.bz2.part' for writing ([Errno 13] Permission denied: ...)

The  pkgs directory (and miniconda installation itself) is on a shared filesystem and read-only because that conda instance is shared among multiple continuous build agents. In conda 4.1.x each build agent can maintain its own (temporary) package cache and locks by setting `CONDA_ENVS_PATH`.

If we're going about this incorrectly, what is the proper way to share a conda installation among many agents? Having them each install miniconda to a temporary location before running a build is too slow to be workable in our case.

There's two separate issues discussed here:

1) locking `pkgs` directory which generally is a legit thing to do and which may require further discussion
2) locking remote file channels -- which is a very strange thing to do, anyone care to explain?

1) Locking the package directory makes sense.  However, in versions < 4.2.x, one could specify where the package directory was located via CONDA_ENV_PATH.  This no longer seems to work as of 4.2.x
2) Agreed.

I think we may bring back the CONDA_ENV_PATH setting, and I am not quite familiar with the cases when dealing with remote file-based channels. 

When I design the filelock and directory lock, the primary goal is to lock local file and directory to avoid parallel execution. Sorry that I did not consider the remote file channel case.

This has affected me too. I install to a known location using `-p` so even having to set the `CONDA_ENV_PATH` to a writable location is counter-intuitive. However, it's the only workaround I have, and in 4.2 it's gone too.

Hi there,

we seem to run into the same issue. How is the CONDA_ENV_PATH variable supposed to work? I set "env_dirs" in .condarc to two values:

envs_dirs:
- ~/conda-envs
- /location/of/conda/installation/envs

This works so far as expected (so users installing something get this installed into their home directories (~/conda-envs)). But obviously some locking is still done in the installation dir of conda (which is not writeable for the users).

Would setting "CONDA_ENV_PATH" to some directory solve this issue (if it worked for 4.2.0)?

Cheers, ocarino

Re: lock files, ran into the same problem today -- conda tries to create lock files for readonly file-based channels which yields a ton of warnings:

``` sh
$ conda create -p ./foobar python=3.5
WARNING conda.lock:touch(53): Failed to create lock, do not run conda in parallel processes [errno 13]
WARNING conda.lock:touch(53): Failed to create lock, do not run conda in parallel processes [errno 13]
WARNING conda.lock:touch(53): Failed to create lock, do not run conda in parallel processes [errno 13]
```

This is a regression from 4.1.x, please fix.

@HugoTian @kalefranz ping

By the way, why does it try to lock at all when all we're doing is reading from an external channel? I get that package cache lock makes sense since it's inherently a mutable thing, but why this?

Anyone?..

We're bumping into this one as well. Has anyone got any info yet?

This is blocking our upgrade to 4.2 as well.

This issue with creating locks in read-only filesystem channels is a breaking our workflow as well

@kalefranz :panda_face: :panda_face: :panda_face: :panda_face: :panda_face: :panda_face: :panda_face: :panda_face: :panda_face: :panda_face: 

@aldanor , I just check the history, previously, failing create lock will be handled with debug level logging . So there is no warning info. 

I think there may be 2 solutions, first we change it back to debug, and there will be no warning. Second, we give another detailed warning based on exception types.

@gnarlie , @eyew , can you give us more info about the exception or error that you encountered during upgrade or workflow ?

Thanks in advance

@HugoTian why does Conda attempt to put a lockfile in a file-based channel in the first place?  Are you saying all clients of a file-based channel need write access?  How does that work with http channels?

@HugoTian The error message I get is: Runtime error: Could not open u/path_to_package/pkgs/packagename.tar.bz2.part' for writing ([Errno 13] Permission denied: ...)

The  pkgs directory (and miniconda installation itself) is on a shared filesystem and read-only because that conda instance is shared among multiple continuous build agents. In conda 4.1.x each build agent can maintain its own (temporary) package cache and locks by setting `CONDA_ENVS_PATH`.

If we're going about this incorrectly, what is the proper way to share a conda installation among many agents? Having them each install miniconda to a temporary location before running a build is too slow to be workable in our case.

There's two separate issues discussed here:

1) locking `pkgs` directory which generally is a legit thing to do and which may require further discussion
2) locking remote file channels -- which is a very strange thing to do, anyone care to explain?

1) Locking the package directory makes sense.  However, in versions < 4.2.x, one could specify where the package directory was located via CONDA_ENV_PATH.  This no longer seems to work as of 4.2.x
2) Agreed.

I think we may bring back the CONDA_ENV_PATH setting, and I am not quite familiar with the cases when dealing with remote file-based channels. 

When I design the filelock and directory lock, the primary goal is to lock local file and directory to avoid parallel execution. Sorry that I did not consider the remote file channel case.

This has affected me too. I install to a known location using `-p` so even having to set the `CONDA_ENV_PATH` to a writable location is counter-intuitive. However, it's the only workaround I have, and in 4.2 it's gone too.

Hi there,

we seem to run into the same issue. How is the CONDA_ENV_PATH variable supposed to work? I set "env_dirs" in .condarc to two values:

envs_dirs:
- ~/conda-envs
- /location/of/conda/installation/envs

This works so far as expected (so users installing something get this installed into their home directories (~/conda-envs)). But obviously some locking is still done in the installation dir of conda (which is not writeable for the users).

Would setting "CONDA_ENV_PATH" to some directory solve this issue (if it worked for 4.2.0)?

Cheers, ocarino


Resolving patch:
diff --git a/conda/base/context.py b/conda/base/context.py
--- a/conda/base/context.py
+++ b/conda/base/context.py
@@ -104,6 +104,9 @@ class Context(Configuration):
     binstar_upload = PrimitiveParameter(None, aliases=('anaconda_upload',),
                                         parameter_type=(bool, NoneType))
 
+    _envs_dirs = SequenceParameter(string_types, aliases=('envs_dirs', 'envs_path'),
+                                   string_delimiter=os.pathsep)
+
     @property
     def default_python(self):
         ver = sys.version_info
@@ -156,8 +159,6 @@ def root_dir(self):
     def root_writable(self):
         return try_write(self.root_dir)
 
-    _envs_dirs = SequenceParameter(string_types, aliases=('envs_dirs',))
-
     @property
     def envs_dirs(self):
         return tuple(abspath(expanduser(p))
diff --git a/conda/cli/main_config.py b/conda/cli/main_config.py
--- a/conda/cli/main_config.py
+++ b/conda/cli/main_config.py
@@ -289,6 +289,7 @@ def execute_config(args, parser):
                                            'debug',
                                            'default_channels',
                                            'disallow',
+                                           'envs_dirs',
                                            'json',
                                            'offline',
                                            'proxy_servers',


Test patch:
diff --git a/tests/base/test_context.py b/tests/base/test_context.py
--- a/tests/base/test_context.py
+++ b/tests/base/test_context.py
@@ -1,6 +1,8 @@
 # -*- coding: utf-8 -*-
 from __future__ import absolute_import, division, print_function, unicode_literals
 
+import os
+
 import pytest
 from conda._vendor.auxlib.ish import dals
 from conda.base.context import context, reset_context
@@ -8,6 +10,7 @@
 from conda.common.configuration import YamlRawParameter
 from conda.common.yaml import yaml_load
 from conda.models.channel import Channel
+from conda.utils import on_win
 from unittest import TestCase
 
 
@@ -58,3 +61,23 @@ def test_old_channel_alias(self):
             "ftp://new.url:8082/conda-forge/label/dev/linux-64",
             "ftp://new.url:8082/conda-forge/label/dev/noarch",
         ]
+
+    def test_conda_envs_path(self):
+        saved_envs_path = os.environ.get('CONDA_ENVS_PATH')
+        beginning = "C:" + os.sep if on_win else os.sep
+        path1 = beginning + os.sep.join(['my', 'envs', 'dir', '1'])
+        path2 = beginning + os.sep.join(['my', 'envs', 'dir', '2'])
+        try:
+            os.environ['CONDA_ENVS_PATH'] = path1
+            reset_context()
+            assert context.envs_dirs[0] == path1
+
+            os.environ['CONDA_ENVS_PATH'] = os.pathsep.join([path1, path2])
+            reset_context()
+            assert context.envs_dirs[0] == path1
+            assert context.envs_dirs[1] == path2
+        finally:
+            if saved_envs_path:
+                os.environ['CONDA_ENVS_PATH'] = saved_envs_path
+            else:
+                del os.environ['CONDA_ENVS_PATH']