Repository: spack/spack

Issue:
spack.store.db.read_transaction might take a write lock
### Steps to reproduce

Working on #26264 I had [this stacktrace](https://gitlab.spack.io/spack/spack/-/jobs/1127884) on Gitlab Pipelines. The stacktrace shows multiple processes trying a read operation and ending up trying to acquire a write lock (and timing out for that). While not easily reproducible, I think this is a bug at least in the API, since it is unexpected that a context manager that starts a "read" transaction ever need to acquire a write lock.

### Error message

```console
[PID=232] LockTimeoutError: Timed out waiting for a write lock.
5985TASK STACKTRACE:
5986  File "/builds/spack/spack/lib/spack/spack/environment/environment.py", line 2012, in _concretize_task
5987    value = _concretize_from_constraints(spec_constraints, tests)
5988  File "/builds/spack/spack/lib/spack/spack/environment/environment.py", line 1989, in _concretize_from_constraints
5989    return s.concretized(tests=tests)
5990  File "/builds/spack/spack/lib/spack/spack/spec.py", line 2678, in concretized
5991    clone.concretize(tests=tests)
5992  File "/builds/spack/spack/lib/spack/spack/spec.py", line 2640, in concretize
5993    self._new_concretize(tests)
5994  File "/builds/spack/spack/lib/spack/spack/spec.py", line 2610, in _new_concretize
5995    result = spack.solver.asp.solve([self], tests=tests)
5996  File "/builds/spack/spack/lib/spack/spack/solver/asp.py", line 1691, in solve
5997    return driver.solve(setup, specs, dump, models, timers, stats, tests)
5998  File "/builds/spack/spack/lib/spack/spack/solver/asp.py", line 396, in solve
5999    answers = builder.build_specs(tuples)
6000  File "/builds/spack/spack/lib/spack/spack/solver/asp.py", line 1647, in build_specs
6001    spack.spec.Spec.ensure_no_deprecated(s)
6002  File "/builds/spack/spack/lib/spack/spack/spec.py", line 2586, in ensure_no_deprecated
6003    with spack.store.db.read_transaction():
6004  File "/builds/spack/spack/lib/spack/llnl/util/lock.py", line 685, in __enter__
6005    self._as = self._acquire_fn()
6006  File "/builds/spack/spack/lib/spack/spack/database.py", line 1054, in _read
6007    with lk.WriteTransaction(self.lock):
6008  File "/builds/spack/spack/lib/spack/llnl/util/lock.py", line 684, in __enter__
6009    if self._enter() and self._acquire_fn:
6010  File "/builds/spack/spack/lib/spack/llnl/util/lock.py", line 720, in _enter
6011    return self._lock.acquire_write(self._timeout)
6012  File "/builds/spack/spack/lib/spack/llnl/util/lock.py", line 448, in acquire_write
6013    wait_time, nattempts = self._lock(fcntl.LOCK_EX, timeout=timeout)
6014  File "/builds/spack/spack/lib/spack/spack/util/lock.py", line 32, in _lock
6015    return super(Lock, self)._lock(op, timeout)
6016  File "/builds/spack/spack/lib/spack/llnl/util/lock.py", line 316, in _lock
6017    .format(lock_type[op]))
```


### Information on your system

Not relevant. Happens in #26264 since it adds multiprocessing capabilities to concretization.

### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have searched the issues of this repo and believe this is not a duplicate
- [X] I have run the failing commands in debug mode and reported the output


Discussion:

Resolving patch:
diff --git a/lib/spack/spack/database.py b/lib/spack/spack/database.py
--- a/lib/spack/spack/database.py
+++ b/lib/spack/spack/database.py
@@ -1024,12 +1024,7 @@ def _write(self, type, value, traceback):
             raise
 
     def _read(self):
-        """Re-read Database from the data in the set location.
-
-        This does no locking, with one exception: it will automatically
-        try to regenerate a missing DB if local. This requires taking a
-        write lock.
-        """
+        """Re-read Database from the data in the set location. This does no locking."""
         if os.path.isfile(self._index_path):
             current_verifier = ''
             if _use_uuid:
@@ -1049,12 +1044,6 @@ def _read(self):
                 "No database index file is present, and upstream"
                 " databases cannot generate an index file")
 
-        # The file doesn't exist, try to traverse the directory.
-        # reindex() takes its own write lock, so no lock here.
-        with lk.WriteTransaction(self.lock):
-            self._write(None, None, None)
-        self.reindex(spack.store.layout)
-
     def _add(
             self,
             spec,


Test patch:
diff --git a/lib/spack/spack/test/database.py b/lib/spack/spack/test/database.py
--- a/lib/spack/spack/test/database.py
+++ b/lib/spack/spack/test/database.py
@@ -894,3 +894,18 @@ def _raise(db, spec):
     with pytest.raises(Exception):
         with spack.store.db.prefix_write_lock(s):
             assert False
+
+
+@pytest.mark.regression('26600')
+def test_database_works_with_empty_dir(tmpdir):
+    # Create the lockfile and failures directory otherwise
+    # we'll get a permission error on Database creation
+    db_dir = tmpdir.ensure_dir('.spack-db')
+    db_dir.ensure('lock')
+    db_dir.ensure_dir('failures')
+    tmpdir.chmod(mode=0o555, rec=1)
+    db = spack.database.Database(str(tmpdir))
+    with db.read_transaction():
+        db.query()
+    # Check that reading an empty directory didn't create a new index.json
+    assert not os.path.exists(db._index_path)