Repository: spack/spack

Issue:
Permission error when activating environment by a different user
We have a Spack installation on our NFSv4 cluster. When a user tries to activate an environment, sometimes it fails with a PermissionError as Spack tries acquire a read-only lock on the .spack-env/transaction_lock file.

### Steps to reproduce the issue
1. Log in to a compute node under a different user than the owner of the .spack-env directory.

2. Try to activate an environment:
```console
$ spack env activate gromacs_patterson
==> Error: [Errno 13] Permission denied: '/opt/spack/var/spack/environments/gromacs_patterson/.spack-env'
```

3. See if the directory is already there
```console
$ ls /opt/spack/var/spack/environments/gromacs_patterson
spack.lock  spack.yaml  spack.yaml~
```

4. Try to activate once more: success!
```console
$ spack env activate gromacs_patterson
```


### Error Message

```console
$ spack -d env activate gromacs_patterson
==> [2020-07-07-16:05:28.170359, 27289] Imported env from built-in commands
==> [2020-07-07-16:05:28.172212, 27289] Imported env from built-in commands
==> [2020-07-07-16:05:28.172850, 27289] Reading config file /opt/spack/etc/spack/defaults/config.yaml
==> [2020-07-07-16:05:28.182003, 27289] Reading config file /opt/spack/etc/spack/config.yaml
==> [2020-07-07-16:05:28.182706, 27289] READ LOCK: /opt/spack/var/spack/environments/gromacs_patterson/.spack-env/transaction_lock[0:0] [Acquiring] 
Traceback (most recent call last):
  File "/opt/spack/bin/spack", line 64, in <module>
    sys.exit(spack.main.main())
  File "/opt/spack/lib/spack/spack/main.py", line 761, in main
    return _invoke_command(command, parser, args, unknown)
  File "/opt/spack/lib/spack/spack/main.py", line 489, in _invoke_command
    return_val = command(parser, args)
  File "/opt/spack/lib/spack/spack/cmd/env.py", line 428, in env
    action(args)
  File "/opt/spack/lib/spack/spack/cmd/env.py", line 112, in env_activate
    active_env = ev.get_env(namedtuple('args', ['env'])(env),
  File "/opt/spack/lib/spack/spack/environment.py", line 319, in get_env
    return read(env)
  File "/opt/spack/lib/spack/spack/environment.py", line 373, in read
    return Environment(root(name))
  File "/opt/spack/lib/spack/spack/environment.py", line 582, in __init__
    with lk.ReadTransaction(self.txlock):
  File "/opt/spack/lib/spack/llnl/util/lock.py", line 576, in __enter__
    if self._enter() and self._acquire_fn:
  File "/opt/spack/lib/spack/llnl/util/lock.py", line 603, in _enter
    return self._lock.acquire_read(self._timeout)
  File "/opt/spack/lib/spack/llnl/util/lock.py", line 304, in acquire_read
    wait_time, nattempts = self._lock(fcntl.LOCK_SH, timeout=timeout)
  File "/opt/spack/lib/spack/spack/util/lock.py", line 31, in _lock
    return super(Lock, self)._lock(op, timeout)
  File "/opt/spack/lib/spack/llnl/util/lock.py", line 154, in _lock
    parent = self._ensure_parent_directory()
  File "/opt/spack/lib/spack/llnl/util/lock.py", line 240, in _ensure_parent_directory
    os.makedirs(parent)
  File "/usr/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/opt/spack/var/spack/environments/gromacs_patterson/.spack-env'
```

### Information on your system

```console
$ spack debug report
* **Spack:** 0.15.0-38-be1009db6
* **Python:** 3.8.3
* **Platform:** linux-archrolling-skylake
```

### Additional information

This only happens on compute nodes where the Spack folder is shared using NFS4. I suspect the attribute caching mechanism of NFS to be the culprit. 

```python
>>> import os
>>> os.makedirs('/opt/spack/var/spack/environments/gromacs_patterson/.spack-env')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/opt/spack/var/spack/environments/gromacs_patterson/.spack-env'
>>> os.path.isdir('/opt/spack/var/spack/environments/gromacs_patterson/.spack-env')
True
>>> os.makedirs('/opt/spack/var/spack/environments/gromacs_patterson/.spack-env')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/opt/spack/var/spack/environments/gromacs_patterson/.spack-env'
```

As demonstrated above, if the user does not have write permission for the root directory of the environment, ```os.makedirs()``` fails with PermissionError before actually checking if the `.spack_env` subdirectory exists. However, if its existence is checked beforehand, `os.makedirs()` raises the FileExistsError, which is already handled in the `_ensure_parent_directory()` method of the `Lock` class in the `llnl.util.lock` module.

Although it goes against the Pythonic wisdom of "Better ask for forgiveness than permission", it might solve the problem to insert a 
```python
os.path.is_dir(parent)
```
check at the beginning of the above mentioned method.

The other, non-Spack solution would be to make the attribute caching of NFS4 more aggressive. However, this negatively affects the performance of the network file system.

Thank you in advance,

Andras

<!-- These boxes can be checked by replacing [ ] with [x] or by clicking them after submitting the issue. -->
- [x] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [x] I have searched the issues of this repo and believe this is not a duplicate
- [x] I have run the failing commands in debug mode and reported the output




Discussion:
Have you solve the issue somehow? With spack 0.15.4 happens still the same to me same:

```console
# source $PATH/spack/share/spack/setup-env.sh 
# spack env activate gromacs2020
==> Error: [Errno 13] Permission denied: '$PATH/spack/var/spack/environments/gromacs2020/.spack-env'
```

The following code solves the problem:
```console
# source $PATH/spack/share/spack/setup-env.sh 
# ls $PATH/spack/var/spack/environments/gromacs2020/.spack-env
# spack env activate gromacs2020
```

> **NOTE:** I will say that accessing the folder in any way before the `source env` is issued solves the problem. 


> **NOTE2:** We have an old installation of spack (git) and the same issue doe not happen. 
At this point, I am forced to run with a patched version of spack to be able to allow the users to export the environment without forcing the users to write this dirty `ls` commands as reported in my previous comment.

I have patched `${MOUNTPATH}/spack/lib/spack/llnl/util/lock.py` as @awacha suggested:

```python
 def _ensure_parent_directory(self):
        parent = os.path.dirname(self.path)
        os.path.isdir(parent) #This line solves the problem.

        # relative paths to lockfiles in the current directory have no parent
        if not parent:
            return '.'

        try:
            os.makedirs(parent)
        except OSError as e:
            # makedirs can fail when diretory already exists.
            if not (e.errno == errno.EEXIST and os.path.isdir(parent) or
                    e.errno == errno.EISDIR):
                raise
        return parent
```

> **NOTE:** ${MOUNTPATH} is an NFS 4.2 mount of a zfs dataset over an omipath network. Degrading the mount to use NFS version to 3 does not solve the issue contrary to what is suggested by @awacha.
I am running into this as well - our development team and CI use shared spack environments on a shared FS. Has this been addressed in a pr yet?
I've also seen this on some of our systems:

```
==> [2022-08-29-11:08:20.864291] Imported env from built-in commands
==> [2022-08-29-11:08:20.870369] Imported env from built-in commands
==> [2022-08-29-11:08:20.873466] Reading config file /projects/spack/etc/spack/defaults/config.yaml
==> [2022-08-29-11:08:20.905505] Reading config file /projects/spack/etc/spack/config.yaml
Traceback (most recent call last):
  File "/projects/spack/bin/spack", line 98, in <module>
    sys.exit(spack.main.main())
  File "/projects/spack/lib/spack/spack/main.py", line 893, in main
    return _main(argv)
  File "/projects/spack/lib/spack/spack/main.py", line 848, in _main
    return finish_parse_and_run(parser, cmd_name, env_format_error)
  File "/projects/spack/lib/spack/spack/main.py", line 876, in finish_parse_and_run
    return _invoke_command(command, parser, args, unknown)
  File "/projects/spack/lib/spack/spack/main.py", line 533, in _invoke_command
    return_val = command(parser, args)
  File "/projects/spack/lib/spack/spack/cmd/env.py", line 711, in env
    action(args)
  File "/projects/spack/lib/spack/spack/cmd/env.py", line 153, in env_activate
    active_env = ev.Environment(env_path)
  File "/projects/spack/lib/spack/spack/environment/environment.py", line 653, in __init__
    with lk.ReadTransaction(self.txlock):
  File "/projects/spack/lib/spack/llnl/util/lock.py", line 716, in __enter__
    if self._enter() and self._acquire_fn:
  File "/projects/spack/lib/spack/llnl/util/lock.py", line 743, in _enter
    return self._lock.acquire_read(self._timeout)
  File "/projects/spack/lib/spack/llnl/util/lock.py", line 449, in acquire_read
    wait_time, nattempts = self._lock(LockType.READ, timeout=timeout)
  File "/projects/spack/lib/spack/spack/util/lock.py", line 35, in _lock
    return super(Lock, self)._lock(op, timeout)
  File "/projects/spack/lib/spack/llnl/util/lock.py", line 313, in _lock
    self._ensure_parent_directory()
  File "/projects/spack/lib/spack/llnl/util/lock.py", line 384, in _ensure_parent_directory
    os.makedirs(parent)
  File "/usr/lib64/python3.6/os.py", line 220, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/projects/spack/var/spack/environments/detribits/.spack-env'
```

The `PermissionError` is a subclass of `OSError`, which `_ensure_parent_directory` is supposed to catch... so why isn't it? ...

Resolving patch:
diff --git a/lib/spack/llnl/util/lock.py b/lib/spack/llnl/util/lock.py
--- a/lib/spack/llnl/util/lock.py
+++ b/lib/spack/llnl/util/lock.py
@@ -386,8 +386,12 @@ def _ensure_parent_directory(self):
         try:
             os.makedirs(parent)
         except OSError as e:
-            # makedirs can fail when diretory already exists.
-            if not (e.errno == errno.EEXIST and os.path.isdir(parent) or e.errno == errno.EISDIR):
+            # os.makedirs can fail in a number of ways when the directory already exists.
+            # With EISDIR, we know it exists, and others like EEXIST, EACCES, and EROFS
+            # are fine if we ensure that the directory exists.
+            # Python 3 allows an exist_ok parameter and ignores any OSError as long as
+            # the directory exists.
+            if not (e.errno == errno.EISDIR or os.path.isdir(parent)):
                 raise
         return parent
 


Test patch: