Repository: spack/spack

Issue:
Index creation not process safe
### Steps to reproduce

```console
$ cd $(mktemp -d)
$ cat spack.yaml
spack:
  specs: [zlib+shared, zlib~shared, zlib+pic, zlib~pic]
  concretizer:
    unify: false
$ git clone --depth=1 https://github.com/spack/spack.git
$ bwrap --dev-bind / / --tmpfs ~  ./spack/bin/spack -e . concretize
==> Bootstrapping clingo from pre-built binaries
==> Fetching https://mirror.spack.io/bootstrap/github-actions/v0.2/build_cache/linux-centos7-x86_64-gcc-10.2.1-clingo-bootstrap-spack-shqedxgvjnhiwdcdrvjhbd73jaevv7wt.spec.json
==> Fetching https://mirror.spack.io/bootstrap/github-actions/v0.2/build_cache/linux-centos7-x86_64/gcc-10.2.1/clingo-bootstrap-spack/linux-centos7-x86_64-gcc-10.2.1-clingo-bootstrap-spack-shqedxgvjnhiwdcdrvjhbd73jaevv7wt.spack
==> Installing "clingo-bootstrap@spack%gcc@10.2.1~docs~ipo+python+static_libstdcpp build_type=Release arch=linux-centos7-x86_64" from a buildcache
==> Bootstrapping patchelf from pre-built binaries
==> Bootstrapping patchelf@0.13.1:0.13.99 %gcc target=x86_64 from sources
==> Installing patchelf-0.13.1-ge7kco6c6zucflebr6x37jcgkzbey6wv
==> No binary for patchelf-0.13.1-ge7kco6c6zucflebr6x37jcgkzbey6wv found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/08/08c0237e89be74d61ddf8f6ff218439cdd62af572d568fb38913b53e222831de.tar.gz
==> No patches needed for patchelf
==> patchelf: Executing phase: 'autoreconf'
==> patchelf: Executing phase: 'configure'
==> patchelf: Executing phase: 'build'
==> patchelf: Executing phase: 'install'
==> patchelf: Successfully installed patchelf-0.13.1-ge7kco6c6zucflebr6x37jcgkzbey6wv
  Fetch: 0.47s.  Build: 5.48s.  Total: 5.96s.
[+] /home/harmen/.spack/bootstrap/store/linux-ubuntu20.04-x86_64/gcc-11.1.0/patchelf-0.13.1-ge7kco6c6zucflebr6x37jcgkzbey6wv
==> Starting concretization pool with 4 processes
==> Error: errors occurred during concretization of the environment:
[Errno 2] No such file or directory: '/home/harmen/.spack/cache/indices/62a4a153a8_c373ec5d29.json'
[Errno 2] No such file or directory: '/home/harmen/.spack/cache/indices/62a4a153a8_c373ec5d29.json'
[Errno 2] No such file or directory: '/home/harmen/.spack/cache/indices/62a4a153a8_c373ec5d29.json'
```

### Error message

```
==> Error: errors occurred during concretization of the environment:
[Errno 2] No such file or directory: '/home/harmen/.spack/cache/indices/62a4a153a8_c373ec5d29.json'
[Errno 2] No such file or directory: '/home/harmen/.spack/cache/indices/62a4a153a8_c373ec5d29.json'
[Errno 2] No such file or directory: '/home/harmen/.spack/cache/indices/62a4a153a8_c373ec5d29.json'
```

### Information on your system

.

### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have searched the issues of this repo and believe this is not a duplicate
- [X] I have run the failing commands in debug mode and reported the output


Discussion:

Resolving patch:
diff --git a/lib/spack/spack/util/file_cache.py b/lib/spack/spack/util/file_cache.py
--- a/lib/spack/spack/util/file_cache.py
+++ b/lib/spack/spack/util/file_cache.py
@@ -3,6 +3,7 @@
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
+import errno
 import os
 import shutil
 
@@ -170,13 +171,17 @@ def mtime(self, key):
             return sinfo.st_mtime
 
     def remove(self, key):
+        file = self.cache_path(key)
         lock = self._get_lock(key)
         try:
             lock.acquire_write()
-            os.unlink(self.cache_path(key))
+            os.unlink(file)
+        except OSError as e:
+            # File not found is OK, so remove is idempotent.
+            if e.errno != errno.ENOENT:
+                raise
         finally:
             lock.release_write()
-            lock.cleanup()
 
 
 class CacheError(SpackError):


Test patch:
diff --git a/lib/spack/spack/test/util/file_cache.py b/lib/spack/spack/test/util/file_cache.py
--- a/lib/spack/spack/test/util/file_cache.py
+++ b/lib/spack/spack/test/util/file_cache.py
@@ -55,9 +55,12 @@ def test_write_and_remove_cache_file(file_cache):
 
     file_cache.remove('test.yaml')
 
-    # After removal both the file and the lock file should not exist
+    # After removal the file should not exist
     assert not os.path.exists(file_cache.cache_path('test.yaml'))
-    assert not os.path.exists(file_cache._lock_path('test.yaml'))
+
+    # Whether the lock file exists is more of an implementation detail, on Linux they
+    # continue to exist, on Windows they don't.
+    # assert os.path.exists(file_cache._lock_path('test.yaml'))
 
 
 @pytest.mark.skipif(sys.platform == 'win32',
@@ -94,3 +97,10 @@ def test_cache_write_readonly_cache_fails(file_cache):
 
     with pytest.raises(CacheError, match='Insufficient permissions to write'):
         file_cache.write_transaction(filename)
+
+
+@pytest.mark.regression('31475')
+def test_delete_is_idempotent(file_cache):
+    """Deleting a non-existent key should be idempotent, to simplify life when
+    running delete with multiple processes"""
+    file_cache.remove('test.yaml')