Repository: SCons/scons

Issue:
Possible race in CacheDir config
This was filed as a comment to closed PR #3353 by @mtvjr.  Moving to a fresh issue because I believe it to be a different problem, plus that PR has been closed for over four years.

================

Looks like this might not completely make things thread safe. We just ran into an issue where it appears the cache file is attempted to be read between file creation and the `json.dump` call finishing. The `json.load` call failed due to an empty input file.

```
13:57:22  Traceback (most recent call last):
13:57:22    File "/usr/lib/python3.6/site-packages/SCons/CacheDir.py", line 195, in _readconfig
13:57:22      self.config = json.load(config)
13:57:22    File "/usr/lib64/python3.6/json/__init__.py", line 299, in load
13:57:22      parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
13:57:22    File "/usr/lib64/python3.6/json/__init__.py", line 354, in loads
13:57:22      return _default_decoder.decode(s)
13:57:22    File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode
13:57:22      obj, end = self.raw_decode(s, idx=_w(s, 0).end())
13:57:22    File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode
13:57:22      raise JSONDecodeError("Expecting value", s, err.value) from None
13:57:22  json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
13:57:22  
13:57:22  During handling of the above exception, another exception occurred:
13:57:22  
13:57:22  Traceback (most recent call last):
13:57:22    File "/usr/lib/python3.6/site-packages/SCons/Taskmaster.py", line 230, in execute
13:57:22      if not t.retrieve_from_cache():
13:57:22    File "/usr/lib/python3.6/site-packages/SCons/Node/FS.py", line 2991, in retrieve_from_cache
13:57:22      return self.get_build_env().get_CacheDir().retrieve(self)
13:57:22    File "/usr/lib/python3.6/site-packages/SCons/Environment.py", line 1033, in get_CacheDir
13:57:22      cd = SCons.CacheDir.CacheDir(path)
13:57:22    File "/usr/lib/python3.6/site-packages/SCons/CacheDir.py", line 159, in __init__
13:57:22      self._readconfig(path)
13:57:22    File "/usr/lib/python3.6/site-packages/SCons/CacheDir.py", line 198, in _readconfig
13:57:22      raise SCons.Errors.SConsEnvironmentError(msg)
13:57:22  SCons.Errors.SConsEnvironmentError: Failed to read cache configuration for /var/lib/scons-ext/cache
```


Discussion:
Note the MSVC config caching recently has locking added, using the new `FileLock` class. We should be able to do the same for the `CacheDir` config.
Thank you for the quick response and my apologies for not responding myself, I tend not to check GitHub very often. Let me know if you need any help or any additional information about the issue.

In our sconstruct, we've bypassed the issue by calling CacheDir directly as soon as we know what directory our cache will use, but before spawning any jobs.
> Thank you for the quick response and my apologies for not responding myself, I tend not to check GitHub very often. Let me know if you need any help or any additional information about the issue.
> 
> In our sconstruct, we've bypassed the issue by calling CacheDir directly as soon as we know what directory our cache will use, but before spawning any jobs.

Are you spawning off other scons runs? or just regular build tasks?
Yeah, I'd say unless you're planning to have distinct CacheDir setups per environment, getting it set up early is good practice anyway.

Resolving patch:
diff --git a/SCons/CacheDir.py b/SCons/CacheDir.py
--- a/SCons/CacheDir.py
+++ b/SCons/CacheDir.py
@@ -34,7 +34,7 @@
 import SCons.Action
 import SCons.Errors
 import SCons.Warnings
-import SCons
+import SCons.Util
 
 cache_enabled = True
 cache_debug = False
@@ -169,15 +169,16 @@ def _readconfig(self, path):
         """
         config_file = os.path.join(path, 'config')
         try:
+            # still use a try block even with exist_ok, might have other fails
             os.makedirs(path, exist_ok=True)
-        except FileExistsError:
-            pass
         except OSError:
             msg = "Failed to create cache directory " + path
             raise SCons.Errors.SConsEnvironmentError(msg)
 
         try:
-            with open(config_file, 'x') as config:
+            with SCons.Util.FileLock(config_file, timeout=5, writer=True), open(
+                config_file, "x"
+            ) as config:
                 self.config['prefix_len'] = 2
                 try:
                     json.dump(self.config, config)
@@ -186,9 +187,11 @@ def _readconfig(self, path):
                     raise SCons.Errors.SConsEnvironmentError(msg)
         except FileExistsError:
             try:
-                with open(config_file) as config:
+                with SCons.Util.FileLock(config_file, timeout=5, writer=False), open(
+                    config_file
+                ) as config:
                     self.config = json.load(config)
-            except ValueError:
+            except (ValueError, json.decoder.JSONDecodeError):
                 msg = "Failed to read cache configuration for " + path
                 raise SCons.Errors.SConsEnvironmentError(msg)
 
diff --git a/SCons/Tool/MSCommon/common.py b/SCons/Tool/MSCommon/common.py
--- a/SCons/Tool/MSCommon/common.py
+++ b/SCons/Tool/MSCommon/common.py
@@ -173,7 +173,7 @@ def read_script_env_cache() -> dict:
     p = Path(CONFIG_CACHE)
     if not CONFIG_CACHE or not p.is_file():
         return envcache
-    with SCons.Util.FileLock(CONFIG_CACHE, timeout=5, writer=True), p.open('r') as f:
+    with SCons.Util.FileLock(CONFIG_CACHE, timeout=5, writer=False), p.open('r') as f:
         # Convert the list of cache entry dictionaries read from
         # json to the cache dictionary. Reconstruct the cache key
         # tuple from the key list written to json.
diff --git a/SCons/Util/filelock.py b/SCons/Util/filelock.py
--- a/SCons/Util/filelock.py
+++ b/SCons/Util/filelock.py
@@ -42,7 +42,7 @@ class SConsLockFailure(Exception):
 class FileLock:
     """Lock a file using a lockfile.
 
-    Locking for the case where multiple processes try to hit an externally
+    Basic locking for when multiple processes may hit an externally
     shared resource that cannot depend on locking within a single SCons
     process. SCons does not have a lot of those, but caches come to mind.
 
@@ -51,8 +51,9 @@ class FileLock:
     and :meth:`release_lock`.
 
     Lock can be a write lock, which is held until released, or a read
-    lock, which releases immediately upon aquisition - the interesting
-    thing being to not read a file which somebody else may be writing,
+    lock, which releases immediately upon aquisition - we want to not
+    read a file which somebody else may be writing, but not create the
+    writers starvation problem of the classic readers/writers lock.
 
     TODO: Should default timeout be None (non-blocking), or 0 (block forever),
        or some arbitrary number?


Test patch: