Repository: conda/conda

Issue:
PermissionError in /tmp on `conda create`
`conda create -n myenv python` fails with a PermissionError. I updated conda recently to 4.4.8 (python 3.6.4.final.0).

Conda root is installed on a read-only file system, so I expect it to install the env in my home dir. There is plenty of space left.

```
    Traceback (most recent call last):
      File "/miniconda/lib/python3.6/site-packages/conda/exceptions.py", line 772, in __call__
        return func(*args, **kwargs)
      File "/miniconda/lib/python3.6/site-packages/conda/cli/main.py", line 78, in _main
        exit_code = do_call(args, p)
      File "/miniconda/lib/python3.6/site-packages/conda/cli/conda_argparse.py", line 77, in do_call
        exit_code = getattr(module, func_name)(args, parser)
      File "/miniconda/lib/python3.6/site-packages/conda/cli/main_create.py", line 11, in execute
        install(args, parser, 'create')
      File "/miniconda/lib/python3.6/site-packages/conda/cli/install.py", line 255, in install
        handle_txn(progressive_fetch_extract, unlink_link_transaction, prefix, args, newenv)
      File "/miniconda/lib/python3.6/site-packages/conda/cli/install.py", line 285, in handle_txn
        unlink_link_transaction.execute()
      File "/miniconda/lib/python3.6/site-packages/conda/core/link.py", line 220, in execute
        self.verify()
      File "/miniconda/lib/python3.6/site-packages/conda/common/io.py", line 441, in decorated
        return f(*args, **kwds)
      File "/miniconda/lib/python3.6/site-packages/conda/core/link.py", line 207, in verify
        exceptions = self._verify(self.prefix_setups, self.prefix_action_groups)
      File "/miniconda/lib/python3.6/site-packages/conda/core/link.py", line 460, in _verify
        cls._verify_transaction_level(prefix_setups),
      File "/miniconda/lib/python3.6/site-packages/conda/core/link.py", line 455, in <genexpr>
        exceptions = tuple(exc for exc in concatv(
      File "/miniconda/lib/python3.6/site-packages/conda/core/link.py", line 323, in _verify_individual_level
        error_result = axn.verify()
      File "/miniconda/lib/python3.6/site-packages/conda/core/path_actions.py", line 410, in verify
        self.file_mode)
      File "/miniconda/lib/python3.6/site-packages/conda/core/portability.py", line 57, in update_prefix
        update_file_in_place_as_binary(realpath(path), _update_prefix)
      File "/miniconda/lib/python3.6/site-packages/conda/gateways/disk/update.py", line 33, in update_file_in_place_as_binary
        fh = exp_backoff_fn(open, file_full_path, 'rb+')
      File "/miniconda/lib/python3.6/site-packages/conda/gateways/disk/__init__.py", line 23, in exp_backoff_fn
        return fn(*args, **kwargs)
    PermissionError: [Errno 13] Permission denied: '/tmp/tmpp4kwvm4j/be616029-6651-4227-a448-4e595cd8e473'
```

That file was created during package installation by conda. Oddly it is missing write permissions:

```
$ ls -l /tmp/tmpp4kwvm4j/be616029-6651-4227-a448-4e595cd8e473
-r-xr-xr-x. 1 joostk wheel 932 Nov  8 22:42 /tmp/tmpp4kwvm4j/be616029-6651-4227-a448-4e595cd8e473
```

How can I resolve this, without going through conda's code, making sure it sets the permissions correctly?


Discussion:
I am able to create a new env without installing packages:

```
$ conda create -n myenv

Solving environment: done

## Package Plan ##

  environment location: /home/joostk/.conda/envs/myenv


Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use:
# > source activate myenv
#
# To deactivate an active environment, use:
# > source deactivate
#
```

But installation of packages fails from then.

```
$ source activate myenv
(myenv) $ conda install python

Solving environment: done

## Package Plan ##

  environment location: /home/joostk/.conda/envs/myenv

  added / updated specs:
    - python


The following NEW packages will be INSTALLED:

    ca-certificates: 2017.08.26-h1d4fec5_0
    certifi:         2018.1.18-py36_0
    libedit:         3.1-heed3624_0
    libffi:          3.2.1-hd88cf55_4
    libgcc-ng:       7.2.0-h7cc24e2_2
    libstdcxx-ng:    7.2.0-h7a57d05_2
    ncurses:         6.0-h9df7e31_2
    openssl:         1.0.2n-hb7f436b_0
    pip:             9.0.1-py36h6c6f9ce_4
    python:          3.6.4-hc3d631a_1
    readline:        7.0-ha6073c6_4
    setuptools:      38.4.0-py36_0
    sqlite:          3.22.0-h1bed415_0
    tk:              8.6.7-hc745277_3
    wheel:           0.30.0-py36hfd4bba0_1
    xz:              5.2.3-h55aa19d_2
    zlib:            1.2.11-ha838bed_2

Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: failed

# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<
```
Same here.
Does python's `tempfile.mkdtemp()` function return a non-writable location on your system?
I just tried here and it created a folder with the following permissions:

`drwx------`

with my username and group.
So the answer is no.
I believe I found the problem. Conda is likely creating some files in that temporary folder with the same permissions as the files from anaconda folder, which in my case no-writable. Doing a

`chmod -R 755 /path/to/anaconda/folder`

fixes the issue.

However, that is not a solution as some people will have (to prevent accidental file removal, for example) anaconda folder non-writable. It was possible with old conda versions.
It seems to happen only when conda is ran from a mount (hard-links fail).

I have inspected `strace` of conda just after confirming downloading packages and installation. It writes the file, then chmods it to 555 and tries opening it in 'rb+':

```
open("/tmp/tmpjhgo25ir/929e78c1-07f9-4100-8fa1-7de08e0a1e2d", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 8
fstat(8, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
ioctl(8, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, 0x7ffe450d5e70) = -1 ENOTTY (Inappropriate ioctl for device)
lseek(8, 0, SEEK_CUR)                   = 0
brk(0x7ff1f9d83000)                     = 0x7ff1f9d83000
read(7, "# libffi.la - a libtool library "..., 4194304) = 932
read(7, "", 3145728)                    = 0
read(7, "", 4194304)                    = 0
write(8, "# libffi.la - a libtool library "..., 932) = 932
close(8)                                = 0
close(7)                                = 0
stat("/miniconda/pkgs/libffi-3.2.1-hd88cf55_4/lib/libffi.la", {st_mode=S_IFREG|0555, st_size=932, ...}) = 0
utimensat(AT_FDCWD, "/tmp/tmpjhgo25ir/929e78c1-07f9-4100-8fa1-7de08e0a1e2d", {{1518024073, 468838246}, {1510177337, 0}}, 0) = 0
chmod("/tmp/tmpjhgo25ir/929e78c1-07f9-4100-8fa1-7de08e0a1e2d", 0555) = 0
listxattr("/miniconda/pkgs/libffi-3.2.1-hd88cf55_4/lib/libffi.la", "", 256) = 0
lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=36864, ...}) = 0
lstat("/tmp/tmpjhgo25ir", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
lstat("/tmp/tmpjhgo25ir/929e78c1-07f9-4100-8fa1-7de08e0a1e2d", {st_mode=S_IFREG|0555, st_size=932, ...}) = 0
open("/tmp/tmpjhgo25ir/929e78c1-07f9-4100-8fa1-7de08e0a1e2d", O_RDWR|O_CLOEXEC) = -1 EACCES (Permission denied)
futex(0x7ff1d4000c10, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
write(1, "\10\10failed\n", 9)           = 9
```

Why the chmod 555????
Likely because it mimics the permission result of 

`stat("/miniconda/pkgs/libffi-3.2.1-hd88cf55_4/lib/libffi.la", {st_mode=S_IFREG|0555, st_size=932, ...}) = 0`


humm
There might be a quick fix: remove all files from

`/miniconda/pkgs`

I will try it here.
Ah! conda was copying that file and setting the permissions as such.  `/miniconda/*` is (again) a read-only tree and owned by root. So, `chmod` the files there as 'u+x', which would fix the problem as well..

```
sudo chown -R root:root /miniconda/pkgs  # optional
sudo chmod -R ug+w /miniconda/pkgs
```
Which fixed the problem!
Edit: I was in the middle of writing this up when @hzpc-joostk made his latest comment.  There's probably more detail than necessary here, but I'm going to go ahead and post it anyway, and probably come back again in the future to use for other situations.

---

Let me explain for you guys what's happening in this part of the code--give you some background and context--and then maybe we can figure it out together.

Probably should start by focusing on what's happening in https://github.com/conda/conda/blob/4.4.9/conda/core/path_actions.py#L389-L425

```python
def verify(self):
    validation_error = super(PrefixReplaceLinkAction, self).verify()
    if validation_error:
        return validation_error

    if islink(self.source_full_path):
        log.trace("ignoring prefix update for symlink with source path %s",
                  self.source_full_path)
        # return
        assert False, "I don't think this is the right place to ignore this"

    self.intermediate_path = join(self.transaction_context['temp_dir'], text_type(uuid4()))

    log.trace("copying %s => %s", self.source_full_path, self.intermediate_path)
    create_link(self.source_full_path, self.intermediate_path, LinkType.copy)

    try:
        log.trace("rewriting prefixes in %s", self.target_full_path)
        update_prefix(self.intermediate_path,
                      context.target_prefix_override or self.target_prefix,
                      self.prefix_placeholder,
                      self.file_mode)
    except _PaddingError:
        raise PaddingError(self.target_full_path, self.prefix_placeholder,
                           len(self.prefix_placeholder))

    sha256_in_prefix = compute_sha256sum(self.intermediate_path)

    self.prefix_path_data = PathDataV1.from_objects(
        self.prefix_path_data,
        file_mode=self.file_mode,
        path_type=PathType.hardlink,
        prefix_placeholder=self.prefix_placeholder,
        sha256_in_prefix=sha256_in_prefix,
    )
```

When we get here, conda's solver has already ran to decide what needs to be uninstalled and installed into an environment, and all the packages have been downloaded and extracted into the package cache.  The uninstall-install process is done in what we call an [`UnlinkLinkTransaction`](https://github.com/conda/conda/blob/4.4.9/conda/core/link.py#L142), which has three stages: (1) prepare, (2) verify, (3) execute.  During the prepare stage, for *every single* path we're going to modify, we instantiate some flavor of [`PathAction`](https://github.com/conda/conda/blob/4.4.9/conda/core/path_actions.py#L59), which itself has four required methods: (1) verify, (2) execute, (3) reverse, (4) cleanup.  So in theory, we should have full knowledge of every disk mutation we're going to make after we're done with the prepare stage.

During the verify stage, we do verification of certain things at the [whole transaction level](https://github.com/conda/conda/blob/4.4.9/conda/core/link.py#L400) and at the [per-prefix level](https://github.com/conda/conda/blob/4.4.9/conda/core/link.py#L330) (transactions are designed with the capability to be multi-prefix transactions, for the anticipated future feature addition of [private application environments](https://github.com/conda/conda/issues/3912)), and then we also call the `verify()` method for every `PathAction` we've instantiated during the prepare phase.  More specifics on this last part in a bit, because that's where we're getting tripped up.

After the verify stage is done, we actually *begin* the transaction, where we start to call the `execute()` method for every `PathAction`.  If we get through them all without an unexpected exception happening, the transaction has completed successfully, and we finish by calling the `cleanup()` method for every `PathAction`.  However, if anything unexpected happens during the transaction, we roll back the transaction by calling `reverse()` on every `PathAction` that had been executed so far.

Okay back to details of the `verify()` step for individual `PathAction`s.  By far the most used is `LinkPathAction`, and the [`verify()`](https://github.com/conda/conda/blob/4.4.9/conda/core/path_actions.py#L257-L358) method there.  `LinkPathAction` takes a specific path in the package cache, and correctly links it into the target prefix.  Depending on configuration, the type of the path being linked, etc., we might be hard-linking, soft-linking, copying, just creating a directory, creating a new relative soft link, etc.  That's all taken care of by `LinkPathAction`.  The `verify()` method makes sure the path we're going to be linking *actually exists* in the package cache, and makes sure it hasn't been modified from what's expected by checking the sha256.

There's a variant of `LinkPathAction` that we also use a lot, `PrefixReplaceLinkAction`, which we use when we'd ideally like to just hard-link the path into the prefix, but instead we have to copy it, because we also have to do prefix rewrites within the content of the file.  Here during the `verify()` step, we're copying the content of the path in the package cache to a temporary location on disk, and doing the prefix rewriting on the file in that temporary location.  The `execute()` method then uses the content of that temporary location to link into the prefix, and `cleanup()` removes that temporary location.

Write permissions of this temporary location is where we're having problem here.
I'm also not comfortable with forcing users and admins to manually make what should be a read-only package cache (i.e. `/miniconda/pkgs`) user- and group-writable.  In general, I don't consider that an acceptable solution to this problem.  We can do better on the conda side.  So I'm going to re-open the issue.

I do think it's a pretty simple fix though.  We just need to be more careful with permissions for the file in that temporary location.
@kalefranz many thanks for your extensive description of the problem, explanation of the (un)installing/(un)linking process and verifications involved in them.  
All in all these seems to be complex, but sophisticated and well-designed processes. Good job devs! 

My apologies for closing this issue too early.  
I agree, setting the permissions to group writable is a bit overkill and I haven't tried only setting them user (root) writable. It is fixed, as conda copies the file's stats as well ([create_link](https://github.com/conda/conda/blob/4.4.9/conda/core/path_actions.py#L403) > [copy](https://github.com/conda/conda/blob/4.4.9/conda/gateways/disk/create.py#L309) > [_do_copy](https://github.com/conda/conda/blob/4.4.9/conda/gateways/disk/create.py#L247) > [copystat](https://github.com/conda/conda/blob/4.4.9/conda/gateways/disk/create.py#L264)).

Letting conda set its own permissions in the temporary location seems to be a simple yet effective fix. 👍 

Resolving patch:
diff --git a/conda/core/path_actions.py b/conda/core/path_actions.py
--- a/conda/core/path_actions.py
+++ b/conda/core/path_actions.py
@@ -29,6 +29,7 @@
                                     create_link, create_python_entry_point, extract_tarball,
                                     make_menu, write_as_json_to_file)
 from ..gateways.disk.delete import rm_rf, try_rmdir_all_empty
+from ..gateways.disk.permissions import make_writable
 from ..gateways.disk.read import (compute_md5sum, compute_sha256sum, islink, lexists,
                                   read_index_json)
 from ..gateways.disk.update import backoff_rename, touch
@@ -401,6 +402,7 @@ def verify(self):
 
         log.trace("copying %s => %s", self.source_full_path, self.intermediate_path)
         create_link(self.source_full_path, self.intermediate_path, LinkType.copy)
+        make_writable(self.intermediate_path)
 
         try:
             log.trace("rewriting prefixes in %s", self.target_full_path)


Test patch: