Repository: conda/conda-build

Issue:
--no-locking looks like it doesn't work
conda-build 2.1.0 has a --no-locking flag. This looks extremely useful, especially if users of a centrally administered conda installation are going to use conda-build to create their own packages.

I tried to use it to build a recipe in a directory I don't have write access too, as well as using conda-build from a centrally administered installation I don't have write access to.

However late in the build process, I get the message

`OSError: [Errno 13] Permission denied: '/reg/g/psdm/sw/conda/inst/miniconda2-dev-rhel7/locks'
`
that is, it looks like conda-build tries to make a locks directory in the central installation? Can this be turned off so that users can run conda-build?


Discussion:
I also tried to use the --output-folder option, but then ran into another problem:

```
  File "/reg/g/psdm/sw/conda/inst/miniconda2-dev-rhel7/lib/python2.7/site-packages/conda_build/build.py", line 201, in copy_recipe
    utils.copy_into(src_path, dst_path, timeout=config.timeout, locking=config.locking)
  File "/reg/g/psdm/sw/conda/inst/miniconda2-dev-rhel7/lib/python2.7/site-packages/conda_build/utils.py", line 143, in copy_into
    with try_acquire_locks(locks, timeout):
UnboundLocalError: local variable 'locks' referenced before assignment
(
```
The latter is a bug. The former is a duplicate feature request. See https://github.com/conda/conda-build/issues/1601

Leaving this issue open for the bug.
Also, locking is important. Without it, conda will almost certainly not let you do more than one simultaneous build.  I don't think that going away from locking makes multi user support easier, rather the opposite.  Do you have reason to believe otherwise?
Thanks, I tried to make the installation locks subdir world readable, but didn't have the same luck that as described in that comment. Anyways, nice to hear that it is on the books for 3.0!

best,

David Schneider
SLAC/LCLS
________________________________________
From: Mike Sarahan [notifications@github.com]
Sent: Monday, January 9, 2017 1:57 PM
To: conda/conda-build
Cc: Schneider, David A.; Author
Subject: Re: [conda/conda-build] --no-locking looks like it doesn't work (#1633)

The latter is a bug. The former is a duplicate feature request. See #1601<https://github.com/conda/conda-build/issues/1601>

Leaving this issue open for the bug.

�
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<https://github.com/conda/conda-build/issues/1633#issuecomment-271420499>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AQAThb8IJZmZ3FJJDwcgEf5cwPeRRwZ8ks5rQq1VgaJpZM4LewY7>.


I agree, I'd like to do locking with multi-user support, so I take that back, turning off locking is not a useful feature for the long haul, it would be nice if there was a way for users to run conda-build from my central install today - but this is not critical.

I don't know how conda-build works in detail - I was imaging that with the --outuptdir option, simultaneous conda-build's from different users may read the same files but otherwise would not interfere with one another, so turning off locking seemed safe.

I imagine the issue is when a central administrator updates conda and conda-build - if a user is in the midst of using one of those tools, that is not so good. If one wants to be robust in that situation, then it seems that the central conda install needs to know what the users are doing -- that seems too complicated, I might just send a warning to people that I'm going to update the central tools (unless it is possible to have different conda and conda-build versions in different environments).

best,

David Schneider
SLAC/LCLS
________________________________________
From: Mike Sarahan [notifications@github.com]
Sent: Monday, January 9, 2017 2:08 PM
To: conda/conda-build
Cc: Schneider, David A.; Author
Subject: Re: [conda/conda-build] --no-locking looks like it doesn't work (#1633)

Also, locking is important. Without it, conda will almost certainly not let you do more than one simultaneous build. I don't think that going away from locking makes multi user support easier, rather the opposite. Do you have reason to believe otherwise?

�
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<https://github.com/conda/conda-build/issues/1633#issuecomment-271423209>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AQAThX66z4KQjFNBCbtQxvXMpG_WL1QBks5rQq_QgaJpZM4LewY7>.


Did you see the conda-build 2.0 blog post?  There's a bit at the end about this: https://www.continuum.io/blog/developer-blog/whats-old-and-new-conda-build

Conda build 2 keeps separate build folders for each build.  In that regard, it is already doing what you want.  Unfortunately, locks are necessary because not everything is independent.  Namely, conda's package cache throws a massive wrench in the gears.  It's very hard to get conda to not erase something out from under another build.  This is what the locks are for, primarily.  Also for indexing folders, where multiple builds should never try to index a folder at the same time, or else corruption occurs.  If you don't care about package caching (very fast internet connection?) then you can probably set a per-build package cache location, and you'll be closer to solving this.  Unfortunately, conda-build will still try to write its locks folder, but you can nuke that bit of code in your install if you really wanted to.

``--output-folder`` is a somewhat esoteric option that I put in for https://github.com/conda/conda-concourse-ci - that system depends on specific folders to shuttle build intermediates around.  ``--output-folder`` makes it easier to comply with that in a dynamic way - the build system specifies the output folder to conda-build, and conda-build dumps its output directly there.   No need to figure out the platform subdirectory and all that - that is all kept track of by the CI system.

Resolving patch:
diff --git a/conda_build/utils.py b/conda_build/utils.py
--- a/conda_build/utils.py
+++ b/conda_build/utils.py
@@ -138,8 +138,7 @@ def copy_into(src, dst, timeout=90, symlinks=False, lock=None, locking=True):
 
         if not lock:
             lock = get_lock(src_folder, timeout=timeout)
-        if locking:
-            locks = [lock]
+        locks = [lock] if locking else []
         with try_acquire_locks(locks, timeout):
             # if intermediate folders not not exist create them
             dst_folder = os.path.dirname(dst)


Test patch: