Repository: spack/spack

Issue:
Issue with `spack clean -a` and `build_stage/.lock` file 
### Steps to reproduce

```console
spack clean -a 
spack install hypre@develop (or any package) 
```

### Error message

Output after `spack clean -a` 
```
==> Removing all temporary build stages
==> Warning: [Errno 20] Not a directory: '/tmp/osborn9/spack-stage/.lock'
==> Warning: [Errno 20] Not a directory: '/tmp/osborn9/spack-stage/.lock'
==> Removing cached downloads
==> Removing install failure marks
==> Removing cached information on repositories
==> Removing python cache files
```

Then when installing a package after executing `spack clean -a`, e.g., `spack -d install hypre@develop`
```
==> [2022-03-23-09:33:12.829744] hypre: Building hypre-2.24.0-fbool6s5axfnsrx63sabg5zfkuvy4dvb [AutotoolsPackage]
==> [2022-03-23-09:33:12.861902] Error: PermissionError: [Errno 13] Permission denied: '/tmp/osborn9/spack-stage/.lock'

/home/osborn9/Code/spack/lib/spack/spack/build_environment.py:1065, in _setup_pkg_and_run:
       1062        tb_string = traceback.format_exc()
       1063
       1064        # build up some context from the offending package so we can
  >>   1065        # show that, too.
       1066        package_context = get_package_context(tb)
       1067
       1068        logfile = None


Traceback (most recent call last):
  File "/home/osborn9/Code/spack/lib/spack/spack/build_environment.py", line 1049, in _setup_pkg_and_run
    return_value = function(pkg, kwargs)
  File "/home/osborn9/Code/spack/lib/spack/spack/installer.py", line 2017, in build_process
    return installer.run()
  File "/home/osborn9/Code/spack/lib/spack/spack/installer.py", line 1861, in run
    with self.pkg.stage:
  File "/home/osborn9/Code/spack/lib/spack/spack/stage.py", line 728, in __enter__
    item.__enter__()
  File "/home/osborn9/Code/spack/lib/spack/spack/stage.py", line 337, in __enter__
    self._lock.acquire_write(timeout=60)
  File "/home/osborn9/Code/spack/lib/spack/llnl/util/lock.py", line 474, in acquire_write
    wait_time, nattempts = self._lock(LockType.WRITE, timeout=timeout)
  File "/home/osborn9/Code/spack/lib/spack/spack/util/lock.py", line 35, in _lock
    return super(Lock, self)._lock(op, timeout)
  File "/home/osborn9/Code/spack/lib/spack/llnl/util/lock.py", line 314, in _lock
    self._file = file_tracker.get_fh(self.path)
  File "/home/osborn9/Code/spack/lib/spack/llnl/util/lock.py", line 124, in get_fh
    fd = os.open(path, os_mode)
PermissionError: [Errno 13] Permission denied: '/tmp/osborn9/spack-stage/.lock'
==> [2022-03-23-09:33:12.862418] Flagging hypre-2.24.0-fbool6s5axfnsrx63sabg5zfkuvy4dvb as failed: PermissionError: [Errno 13] Permission denied: '/tmp/osborn9/spack-stage/.lock'
==> [2022-03-23-09:33:12.878095] ChildError: PermissionError: [Errno 13] Permission denied: '/tmp/osborn9/spack-stage/.lock'
```

This behavior was noticed after https://github.com/spack/spack/pull/27021 (potential issue with addition of `cleanup` in `lock.py` ?) 

### Information on your system

* **Spack:** 0.18.0.dev0 (c300b9204778a4ed21f32d9d77dc11a17c4f46a0)
* **Python:** 3.6.8
* **Platform:** linux-rhel7-sandybridge
* **Concretizer:** clingo

### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have searched the issues of this repo and believe this is not a duplicate
- [X] I have run the failing commands in debug mode and reported the output


Discussion:
Thanks for the report, I was about to submit a similar issue with a warning from `os.open`:
```console
$ spack  clean -a
==> Removing all temporary build stages
==> Warning: open() missing required argument 'flags' (pos 2)
==> Removing cached downloads
==> Removing install failure marks
==> Removing cached information on repositories
==> Removing python cache files
```
Using `git bisect` seems to point to 4aee27816e7101753aeb392e868096236a26d84d but I'm not totally sure of the exact commit, since a couple of them were entirely broken.
Using this diff:
```diff
diff --git a/lib/spack/llnl/util/tty/__init__.py b/lib/spack/llnl/util/tty/__init__.py
index de49f3f77f..5b827a3490 100644
--- a/lib/spack/llnl/util/tty/__init__.py
+++ b/lib/spack/llnl/util/tty/__init__.py
@@ -269,6 +269,7 @@ def warn(message, *args, **kwargs):
 
     kwargs.setdefault('format', '*Y')
     kwargs.setdefault('stream', sys.stderr)
+    raise RuntimeError(message)
     info("Warning: " + six.text_type(message), *args, **kwargs)
 
```
I have the following stacktrace:

<details>

<summary>$ spack -d clean -a</summary>

```
=> [2022-03-23-18:42:26.702531] Imported clean from built-in commands
==> [2022-03-23-18:42:26.703639] Imported clean from built-in commands
==> [2022-03-23-18:42:26.704137] Removing all temporary build stages
==> [2022-03-23-18:42:26.704783] Reading config file /home/culpo/PycharmProjects/spack/etc/spack/defaults/config.yaml
Traceback (most recent call last):
  File "/usr/lib/python3.8/shutil.py", line 712, in rmtree
    fd = os.open(path, os.O_RDONLY)
PermissionError: [Errno 13] Permission denied: '/tmp/culpo/spack-stage/.lock'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/culpo/PycharmProjects/spack/lib/spack/llnl/util/filesystem.py", line 1126, in onerror
    func(path)
TypeError: open() missing required argument 'flags' (pos 2)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/culpo/PycharmProjects/spack/bin/spack", line 98, in <module>
    sys.exit(spack.main.main())
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/main.py", line 916, in main
    return _main(argv)
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/main.py", line 871, in _main
    return finish_parse_and_run(parser, cmd_name, env_format_error)
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/main.py", line 899, in finish_parse_and_run
    return _invoke_command(command, parser, args, unknown)
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/main.py", line 551, in _invoke_command
    return_val = command(parser, args)
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/cmd/clean.py", line 78, in clean
    spack.stage.purge()
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/stage.py", line 826, in purge
    remove_linked_tree(stage_path)
  File "/home/culpo/PycharmProjects/spack/lib/spack/spack/util/path.py", line 117, in path_filter_caller
    return func(*args, **kwargs)
  File "/home/culpo/PycharmProjects/spack/lib/spack/llnl/util/filesystem.py", line 1136, in remove_linked_tree
    shutil.rmtree(path, onerror=onerror)
  File "/usr/lib/python3.8/shutil.py", line 714, in rmtree
    onerror(os.open, path, sys.exc_info())
  File "/home/culpo/PycharmProjects/spack/lib/spack/llnl/util/filesystem.py", line 1128, in onerror
    tty.warn(e)
  File "/home/culpo/PycharmProjects/spack/lib/spack/llnl/util/tty/__init__.py", line 272, in warn
    raise RuntimeError(message)
RuntimeError: open() missing required argument 'flags' (pos 2)
```

</details>
This seems to fix the issue:
```diff
diff --git a/lib/spack/spack/stage.py b/lib/spack/spack/stage.py
index b92c0fbbc8..f289305392 100644
--- a/lib/spack/spack/stage.py
+++ b/lib/spack/spack/stage.py
@@ -823,7 +823,10 @@ def purge():
         for stage_dir in os.listdir(root):
             if stage_dir.startswith(stage_prefix) or stage_dir == '.lock':
                 stage_path = os.path.join(root, stage_dir)
-                remove_linked_tree(stage_path)
+                if os.path.isdir(stage_path):
+                    remove_linked_tree(stage_path)
+                else:
+                    os.remove(stage_path)
 
 
 def get_checksums_for_versions(url_dict, name, **kwargs):
```
So, complete diagnosis of the issues is:
1. During `spack clean -a` we were trying to remove all the stage dirs
2. When doing that we are (erroneously?) passing directories AND files to `remove_linked_tree` and `shutil.rmtree` (which should accept only directories)
3. Before the Windows PR errors in that function were ignored, so we didn't notice
4. After the Windows PR we try to handle the error and part of that is to `os.chmod(path, stat.S_IWUSR)` the path that couldn't be removed
5. When the file lock is set to ` stat.S_IWUSR` that causes issues later when trying to acquire the lock

Resolving patch:
diff --git a/lib/spack/llnl/util/filesystem.py b/lib/spack/llnl/util/filesystem.py
--- a/lib/spack/llnl/util/filesystem.py
+++ b/lib/spack/llnl/util/filesystem.py
@@ -27,7 +27,7 @@
 from spack.util.executable import Executable
 from spack.util.path import path_to_os_path, system_path_filter
 
-is_windows  = _platform == 'win32'
+is_windows = _platform == 'win32'
 
 if not is_windows:
     import grp
@@ -1201,12 +1201,16 @@ def onerror(func, path, exe_info):
             tty.warn(e)
             pass
 
+    kwargs = {'ignore_errors': True}
+    if is_windows:
+        kwargs = {'onerror': onerror}
+
     if os.path.exists(path):
         if os.path.islink(path):
-            shutil.rmtree(os.path.realpath(path), onerror=onerror)
+            shutil.rmtree(os.path.realpath(path), **kwargs)
             os.unlink(path)
         else:
-            shutil.rmtree(path, onerror=onerror)
+            shutil.rmtree(path, **kwargs)
 
 
 @contextmanager
diff --git a/lib/spack/spack/stage.py b/lib/spack/spack/stage.py
--- a/lib/spack/spack/stage.py
+++ b/lib/spack/spack/stage.py
@@ -823,7 +823,10 @@ def purge():
         for stage_dir in os.listdir(root):
             if stage_dir.startswith(stage_prefix) or stage_dir == '.lock':
                 stage_path = os.path.join(root, stage_dir)
-                remove_linked_tree(stage_path)
+                if os.path.isdir(stage_path):
+                    remove_linked_tree(stage_path)
+                else:
+                    os.remove(stage_path)
 
 
 def get_checksums_for_versions(url_dict, name, **kwargs):


Test patch:
diff --git a/lib/spack/spack/test/llnl/util/filesystem.py b/lib/spack/spack/test/llnl/util/filesystem.py
--- a/lib/spack/spack/test/llnl/util/filesystem.py
+++ b/lib/spack/spack/test/llnl/util/filesystem.py
@@ -866,3 +866,21 @@ def test_visit_directory_tree_follow_none(noncyclical_dir_structure):
         j('b'),
     ]
     assert not visitor.symlinked_dirs_after
+
+
+@pytest.mark.regression('29687')
+@pytest.mark.parametrize('initial_mode', [
+    stat.S_IRUSR | stat.S_IXUSR,
+    stat.S_IWGRP
+])
+@pytest.mark.skipif(sys.platform == 'win32', reason='Windows might change permissions')
+def test_remove_linked_tree_doesnt_change_file_permission(tmpdir, initial_mode):
+    # Here we test that a failed call to remove_linked_tree, due to passing a file
+    # as an argument instead of a directory, doesn't leave the file with different
+    # permissions as a side effect of trying to handle the error.
+    file_instead_of_dir = tmpdir.ensure('foo')
+    file_instead_of_dir.chmod(initial_mode)
+    initial_stat = os.stat(str(file_instead_of_dir))
+    fs.remove_linked_tree(str(file_instead_of_dir))
+    final_stat = os.stat(str(file_instead_of_dir))
+    assert final_stat == initial_stat