Repository: llvm/llvm-project
Pull request #135703: [clang][Dependency Scanning] Adding an API to Diagnose Invalid Negative Stat Cache Entries
Author: login: qiongsiwu
Labels: name: clang
description: Clang issues not falling into any other category
Created: 2025-04-15T00:05:39Z | Merged: 2025-04-18T17:52:40Z

We have had numerous situations where the negatively stat cached paths are invalidated during the build, and such invalidations lead to build errors. 

This PR adds an API to diagnose such cases. `DependencyScanningFilesystemSharedCache::diagnoseNegativeStatCachedPaths` allows users of the cache to ask the cache to examine all negatively stat cached paths, and re-stat the paths using the passed-in file system. If any re-stat succeeds, the API emits diagnostics. 

rdar://149147920


Diff:
diff --git a/clang/include/clang/Tooling/DependencyScanning/DependencyScanningFilesystem.h b/clang/include/clang/Tooling/DependencyScanning/DependencyScanningFilesystem.h
index d12814e7c9253..74b40f7452edb 100644
--- a/clang/include/clang/Tooling/DependencyScanning/DependencyScanningFilesystem.h
+++ b/clang/include/clang/Tooling/DependencyScanning/DependencyScanningFilesystem.h
@@ -220,6 +220,14 @@ class DependencyScanningFilesystemSharedCache {
   CacheShard &getShardForFilename(StringRef Filename) const;
   CacheShard &getShardForUID(llvm::sys::fs::UniqueID UID) const;
 
+  /// Visits all cached entries and re-stat an entry using FS if
+  /// it is negatively stat cached. If re-stat succeeds on a path,
+  /// the path is added to InvalidPaths, indicating that the cache
+  /// may have erroneously negatively cached it. The caller can then
+  /// use InvalidPaths to issue diagnostics.
+  std::vector<StringRef>
+  getInvalidNegativeStatCachedPaths(llvm::vfs::FileSystem &UnderlyingFS) const;
+
 private:
   std::unique_ptr<CacheShard[]> CacheShards;
   unsigned NumShards;
diff --git a/clang/lib/Tooling/DependencyScanning/DependencyScanningFilesystem.cpp b/clang/lib/Tooling/DependencyScanning/DependencyScanningFilesystem.cpp
index 4d738e4bea41a..191b6ef439860 100644
--- a/clang/lib/Tooling/DependencyScanning/DependencyScanningFilesystem.cpp
+++ b/clang/lib/Tooling/DependencyScanning/DependencyScanningFilesystem.cpp
@@ -108,6 +108,33 @@ DependencyScanningFilesystemSharedCache::getShardForUID(
   return CacheShards[Hash % NumShards];
 }
 
+std::vector<StringRef>
+DependencyScanningFilesystemSharedCache::getInvalidNegativeStatCachedPaths(
+    llvm::vfs::FileSystem &UnderlyingFS) const {
+  // Iterate through all shards and look for cached stat errors.
+  std::vector<StringRef> InvalidPaths;
+  for (unsigned i = 0; i < NumShards; i++) {
+    const CacheShard &Shard = CacheShards[i];
+    std::lock_guard<std::mutex> LockGuard(Shard.CacheLock);
+    for (const auto &[Path, CachedPair] : Shard.CacheByFilename) {
+      const CachedFileSystemEntry *Entry = CachedPair.first;
+
+      if (Entry->getError()) {
+        // Only examine cached errors.
+        llvm::ErrorOr<llvm::vfs::Status> Stat = UnderlyingFS.status(Path);
+        if (Stat) {
+          // This is the case where we have cached the non-existence
+          // of the file at Path first, and a a file at the path is created
+          // later. The cache entry is not invalidated (as we have no good
+          // way to do it now), which may lead to missing file build errors.
+          InvalidPaths.push_back(Path);
+        }
+      }
+    }
+  }
+  return InvalidPaths;
+}
+
 const CachedFileSystemEntry *
 DependencyScanningFilesystemSharedCache::CacheShard::findEntryByFilename(
     StringRef Filename) const {
diff --git a/clang/unittests/Tooling/DependencyScanning/DependencyScanningFilesystemTest.cpp b/clang/unittests/Tooling/DependencyScanning/DependencyScanningFilesystemTest.cpp
index 111351fb90cee..5ea8d02353dc3 100644
--- a/clang/unittests/Tooling/DependencyScanning/DependencyScanningFilesystemTest.cpp
+++ b/clang/unittests/Tooling/DependencyScanning/DependencyScanningFilesystemTest.cpp
@@ -178,3 +178,27 @@ TEST(DependencyScanningFilesystem, CacheStatFailures) {
   DepFS.exists("/cache/a.pcm");
   EXPECT_EQ(InstrumentingFS->NumStatusCalls, 5u);
 }
+
+TEST(DependencyScanningFilesystem, DiagnoseStaleStatFailures) {
+  auto InMemoryFS = llvm::makeIntrusiveRefCnt<llvm::vfs::InMemoryFileSystem>();
+  InMemoryFS->setCurrentWorkingDirectory("/");
+
+  DependencyScanningFilesystemSharedCache SharedCache;
+  DependencyScanningWorkerFilesystem DepFS(SharedCache, InMemoryFS);
+
+  bool Path1Exists = DepFS.exists("/path1");
+  EXPECT_EQ(Path1Exists, false);
+
+  // Adding a file that has been stat-ed,
+  InMemoryFS->addFile("/path1", 0, llvm::MemoryBuffer::getMemBuffer(""));
+  Path1Exists = DepFS.exists("/path1");
+  // Due to caching in SharedCache, path1 should not exist in
+  // DepFS's eyes.
+  EXPECT_EQ(Path1Exists, false);
+
+  std::vector<llvm::StringRef> InvalidPaths =
+      SharedCache.getInvalidNegativeStatCachedPaths(*InMemoryFS.get());
+
+  EXPECT_EQ(InvalidPaths.size(), 1u);
+  ASSERT_STREQ("/path1", InvalidPaths[0].str().c_str());
+}


Reviews:
user: login: jansvoboda11
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T15:57:32Z
user: login: jansvoboda11
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T15:59:18Z
user: login: jansvoboda11
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T16:01:17Z
user: login: jansvoboda11
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T16:02:11Z
user: login: qiongsiwu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T17:39:04Z
user: login: qiongsiwu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T17:39:58Z
user: login: qiongsiwu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T17:41:43Z
user: login: jansvoboda11
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T17:49:05Z
user: login: jansvoboda11
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T18:04:51Z
user: login: qiongsiwu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T21:35:45Z
user: login: jansvoboda11
body: On a second thought, what's the current intent for build system integration? I'm curious why we're emitting a diagnostic here rather than just passing the list of paths back to the build system and letting it decide how to report these.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T17:12:04Z
user: login: jansvoboda11
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T22:00:34Z
user: login: qiongsiwu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T23:01:09Z
user: login: qiongsiwu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T23:02:00Z
user: login: qiongsiwu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T23:02:14Z
user: login: jansvoboda11
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T23:06:19Z
user: login: jansvoboda11
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T23:07:19Z
user: login: jansvoboda11
body: LGTM once the extra `std::string` constructor gets removed & the test passes.
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T23:10:08Z
user: login: qiongsiwu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T23:13:20Z

Review comments:
diff_hunk: @@ -108,6 +108,32 @@ DependencyScanningFilesystemSharedCache::getShardForUID(
   return CacheShards[Hash % NumShards];
 }
 
+void DependencyScanningFilesystemSharedCache::diagnoseNegativeStatCachedPaths(
+    llvm::raw_ostream &OS, llvm::vfs::FileSystem &UnderlyingFS) const {
+  // Iterate through all shards and look for cached stat errors.
+  for (unsigned i = 0; i < NumShards; i++) {
+    const CacheShard &Shard = CacheShards[i];
path: clang/lib/Tooling/DependencyScanning/DependencyScanningFilesystem.cpp
user: login: jansvoboda11
body: Why not like this?
```suggestion
  for (const CacheShard &Shard : CacheShards) {
```
created_at: 2025-04-15T15:57:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -108,6 +108,32 @@ DependencyScanningFilesystemSharedCache::getShardForUID(
   return CacheShards[Hash % NumShards];
 }
 
+void DependencyScanningFilesystemSharedCache::diagnoseNegativeStatCachedPaths(
+    llvm::raw_ostream &OS, llvm::vfs::FileSystem &UnderlyingFS) const {
+  // Iterate through all shards and look for cached stat errors.
+  for (unsigned i = 0; i < NumShards; i++) {
+    const CacheShard &Shard = CacheShards[i];
+    for (const auto &Path : Shard.CacheByFilename.keys()) {
+      auto CachedPairIt = Shard.CacheByFilename.find(Path);
path: clang/lib/Tooling/DependencyScanning/DependencyScanningFilesystem.cpp
user: login: jansvoboda11
body: Why the extra lookup?
```suggestion
    for (const auto &[Filename, Entry] : Shard.CacheByFilename) {
      const CachedFileSystemEntry *Entry = Entry.first;
```
created_at: 2025-04-15T15:59:18Z
author_association: CONTRIBUTOR
diff_hunk: @@ -108,6 +108,32 @@ DependencyScanningFilesystemSharedCache::getShardForUID(
   return CacheShards[Hash % NumShards];
 }
 
+void DependencyScanningFilesystemSharedCache::diagnoseNegativeStatCachedPaths(
+    llvm::raw_ostream &OS, llvm::vfs::FileSystem &UnderlyingFS) const {
+  // Iterate through all shards and look for cached stat errors.
+  for (unsigned i = 0; i < NumShards; i++) {
+    const CacheShard &Shard = CacheShards[i];
+    for (const auto &Path : Shard.CacheByFilename.keys()) {
+      auto CachedPairIt = Shard.CacheByFilename.find(Path);
+      auto CachedPair = CachedPairIt->getValue();
+      const CachedFileSystemEntry *Entry = CachedPair.first;
+
+      if (Entry->getError()) {
+        // Only examine cached errors.
+        llvm::ErrorOr<llvm::vfs::Status> Stat = UnderlyingFS.status(Path);
+        if (Stat) {
+          // This is the case where we have negatively cached the non-existence
+          // of the file at Path, but the cache is later invalidated. Report
+          // diagnostics.
path: clang/lib/Tooling/DependencyScanning/DependencyScanningFilesystem.cpp
user: login: jansvoboda11
body: ```suggestion
          // This is the case where we have negatively the non-existence of the
          // path, but it's later created without invalidating the cache. Report
          // diagnostics.
```
created_at: 2025-04-15T16:01:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -108,6 +108,32 @@ DependencyScanningFilesystemSharedCache::getShardForUID(
   return CacheShards[Hash % NumShards];
 }
 
+void DependencyScanningFilesystemSharedCache::diagnoseNegativeStatCachedPaths(
path: clang/lib/Tooling/DependencyScanning/DependencyScanningFilesystem.cpp
user: login: jansvoboda11
body: Shouldn't this be taking locks? If there's a precondition that no other threads use this object, we should at least reflect that in the documentation.
created_at: 2025-04-15T16:02:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -108,6 +108,32 @@ DependencyScanningFilesystemSharedCache::getShardForUID(
   return CacheShards[Hash % NumShards];
 }
 
+void DependencyScanningFilesystemSharedCache::diagnoseNegativeStatCachedPaths(
+    llvm::raw_ostream &OS, llvm::vfs::FileSystem &UnderlyingFS) const {
+  // Iterate through all shards and look for cached stat errors.
+  for (unsigned i = 0; i < NumShards; i++) {
+    const CacheShard &Shard = CacheShards[i];
+    for (const auto &Path : Shard.CacheByFilename.keys()) {
+      auto CachedPairIt = Shard.CacheByFilename.find(Path);
path: clang/lib/Tooling/DependencyScanning/DependencyScanningFilesystem.cpp
user: login: qiongsiwu
body: Thanks! Fixed. 
created_at: 2025-04-15T17:39:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -108,6 +108,32 @@ DependencyScanningFilesystemSharedCache::getShardForUID(
   return CacheShards[Hash % NumShards];
 }
 
+void DependencyScanningFilesystemSharedCache::diagnoseNegativeStatCachedPaths(
path: clang/lib/Tooling/DependencyScanning/DependencyScanningFilesystem.cpp
user: login: qiongsiwu
body: Yes good catch! This should have a lock. 
created_at: 2025-04-15T17:39:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -108,6 +108,32 @@ DependencyScanningFilesystemSharedCache::getShardForUID(
   return CacheShards[Hash % NumShards];
 }
 
+void DependencyScanningFilesystemSharedCache::diagnoseNegativeStatCachedPaths(
+    llvm::raw_ostream &OS, llvm::vfs::FileSystem &UnderlyingFS) const {
+  // Iterate through all shards and look for cached stat errors.
+  for (unsigned i = 0; i < NumShards; i++) {
+    const CacheShard &Shard = CacheShards[i];
path: clang/lib/Tooling/DependencyScanning/DependencyScanningFilesystem.cpp
user: login: qiongsiwu
body: I could not do a range-based for because `CacheShards` is an array and it does not have iterator support. Maybe there are other simple ways to make a range-based for loop work here?
created_at: 2025-04-15T17:41:43Z
author_association: CONTRIBUTOR
diff_hunk: @@ -108,6 +108,32 @@ DependencyScanningFilesystemSharedCache::getShardForUID(
   return CacheShards[Hash % NumShards];
 }
 
+void DependencyScanningFilesystemSharedCache::diagnoseNegativeStatCachedPaths(
+    llvm::raw_ostream &OS, llvm::vfs::FileSystem &UnderlyingFS) const {
+  // Iterate through all shards and look for cached stat errors.
+  for (unsigned i = 0; i < NumShards; i++) {
+    const CacheShard &Shard = CacheShards[i];
path: clang/lib/Tooling/DependencyScanning/DependencyScanningFilesystem.cpp
user: login: jansvoboda11
body: Ah, makes sense. I'm not aware of standard (or LLVM) utilities to iterate over `unique_ptr<T[]>` in range-based for.
created_at: 2025-04-15T17:49:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -108,6 +108,31 @@ DependencyScanningFilesystemSharedCache::getShardForUID(
   return CacheShards[Hash % NumShards];
 }
 
+void DependencyScanningFilesystemSharedCache::diagnoseNegativeStatCachedPaths(
+    llvm::raw_ostream &OS, llvm::vfs::FileSystem &UnderlyingFS) const {
+  // Iterate through all shards and look for cached stat errors.
+  for (unsigned i = 0; i < NumShards; i++) {
+    const CacheShard &Shard = CacheShards[i];
+    std::lock_guard<std::mutex> LockGuard(Shard.CacheLock);
+    for (const auto &[Path, CachedPair] : Shard.CacheByFilename) {
+      const CachedFileSystemEntry *Entry = CachedPair.first;
+
+      if (Entry->getError()) {
+        // Only examine cached errors.
+        llvm::ErrorOr<llvm::vfs::Status> Stat = UnderlyingFS.status(Path);
+        if (Stat) {
+          // This is the case where we have cached the non-existence
+          // of the file at Path, but the file is created but the cache is
+          // not invalidated. Report diagnostics.
+          OS << Path << " did not exist when it was first searched "
+             << "but was created later. This may have led to a build failure "
+                "due to missing files.\n";
path: clang/lib/Tooling/DependencyScanning/DependencyScanningFilesystem.cpp
user: login: jansvoboda11
body: This might get very busy with multiple files. Can we make this shorter?

created_at: 2025-04-15T18:04:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -108,6 +108,31 @@ DependencyScanningFilesystemSharedCache::getShardForUID(
   return CacheShards[Hash % NumShards];
 }
 
+void DependencyScanningFilesystemSharedCache::diagnoseNegativeStatCachedPaths(
+    llvm::raw_ostream &OS, llvm::vfs::FileSystem &UnderlyingFS) const {
+  // Iterate through all shards and look for cached stat errors.
+  for (unsigned i = 0; i < NumShards; i++) {
+    const CacheShard &Shard = CacheShards[i];
+    std::lock_guard<std::mutex> LockGuard(Shard.CacheLock);
+    for (const auto &[Path, CachedPair] : Shard.CacheByFilename) {
+      const CachedFileSystemEntry *Entry = CachedPair.first;
+
+      if (Entry->getError()) {
+        // Only examine cached errors.
+        llvm::ErrorOr<llvm::vfs::Status> Stat = UnderlyingFS.status(Path);
+        if (Stat) {
+          // This is the case where we have cached the non-existence
+          // of the file at Path, but the file is created but the cache is
+          // not invalidated. Report diagnostics.
+          OS << Path << " did not exist when it was first searched "
+             << "but was created later. This may have led to a build failure "
+                "due to missing files.\n";
path: clang/lib/Tooling/DependencyScanning/DependencyScanningFilesystem.cpp
user: login: qiongsiwu
body: Good point. This logic is revised so we output a list of them per API call to reduce duplication. 
created_at: 2025-04-15T21:35:45Z
author_association: CONTRIBUTOR
diff_hunk: @@ -108,6 +108,32 @@ DependencyScanningFilesystemSharedCache::getShardForUID(
   return CacheShards[Hash % NumShards];
 }
 
+void DependencyScanningFilesystemSharedCache::
+    diagnoseInvalidNegativeStatCachedPaths(
+        std::vector<std::string> &InvalidPaths,
path: clang/lib/Tooling/DependencyScanning/DependencyScanningFilesystem.cpp
user: login: jansvoboda11
body: Any reason to take an out-parameter instead of returning a value?
created_at: 2025-04-17T20:36:18Z
author_association: CONTRIBUTOR
diff_hunk: @@ -108,6 +108,32 @@ DependencyScanningFilesystemSharedCache::getShardForUID(
   return CacheShards[Hash % NumShards];
 }
 
+void DependencyScanningFilesystemSharedCache::
+    diagnoseInvalidNegativeStatCachedPaths(
+        std::vector<std::string> &InvalidPaths,
+        llvm::vfs::FileSystem &UnderlyingFS) const {
+  // Iterate through all shards and look for cached stat errors.
+  for (unsigned i = 0; i < NumShards; i++) {
+    const CacheShard &Shard = CacheShards[i];
+    std::lock_guard<std::mutex> LockGuard(Shard.CacheLock);
+    for (const auto &[Path, CachedPair] : Shard.CacheByFilename) {
+      const CachedFileSystemEntry *Entry = CachedPair.first;
+
+      if (Entry->getError()) {
+        // Only examine cached errors.
+        llvm::ErrorOr<llvm::vfs::Status> Stat = UnderlyingFS.status(Path);
+        if (Stat) {
+          // This is the case where we have cached the non-existence
+          // of the file at Path first, and a a file at the path is created
+          // later. The cache entry is not invalidated (as we have no good
+          // way to do it now), which may lead to missing file build errors.
+          InvalidPaths.push_back(std::string(Path));
path: clang/lib/Tooling/DependencyScanning/DependencyScanningFilesystem.cpp
user: login: jansvoboda11
body: Why allocate the string here? I'd imagine we can just add `StringRef` and defer the decision whether to allocate or not to the client (based on whether they need the strings to outlive the service or not).
created_at: 2025-04-17T21:55:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -108,6 +108,32 @@ DependencyScanningFilesystemSharedCache::getShardForUID(
   return CacheShards[Hash % NumShards];
 }
 
+void DependencyScanningFilesystemSharedCache::
+    diagnoseInvalidNegativeStatCachedPaths(
path: clang/lib/Tooling/DependencyScanning/DependencyScanningFilesystem.cpp
user: login: jansvoboda11
body: This function no longer diagnoses. Maybe "collect" fits better?
created_at: 2025-04-17T21:55:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -108,6 +108,32 @@ DependencyScanningFilesystemSharedCache::getShardForUID(
   return CacheShards[Hash % NumShards];
 }
 
+void DependencyScanningFilesystemSharedCache::
+    diagnoseInvalidNegativeStatCachedPaths(
+        std::vector<std::string> &InvalidPaths,
+        llvm::vfs::FileSystem &UnderlyingFS) const {
+  // Iterate through all shards and look for cached stat errors.
+  for (unsigned i = 0; i < NumShards; i++) {
+    const CacheShard &Shard = CacheShards[i];
+    std::lock_guard<std::mutex> LockGuard(Shard.CacheLock);
+    for (const auto &[Path, CachedPair] : Shard.CacheByFilename) {
+      const CachedFileSystemEntry *Entry = CachedPair.first;
+
+      if (Entry->getError()) {
+        // Only examine cached errors.
+        llvm::ErrorOr<llvm::vfs::Status> Stat = UnderlyingFS.status(Path);
+        if (Stat) {
+          // This is the case where we have cached the non-existence
+          // of the file at Path first, and a a file at the path is created
+          // later. The cache entry is not invalidated (as we have no good
+          // way to do it now), which may lead to missing file build errors.
+          InvalidPaths.push_back(std::string(Path));
path: clang/lib/Tooling/DependencyScanning/DependencyScanningFilesystem.cpp
user: login: qiongsiwu
body: Sure! Fixed. I think it might be less burdensome for the client to not worry about how to allocate the string, but I don't have a strong opinion on this. Is it a performance reason where we do not want to allocate the strings here so the client can save the allocation if necessary? 
created_at: 2025-04-17T23:01:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -108,6 +108,32 @@ DependencyScanningFilesystemSharedCache::getShardForUID(
   return CacheShards[Hash % NumShards];
 }
 
+void DependencyScanningFilesystemSharedCache::
+    diagnoseInvalidNegativeStatCachedPaths(
+        std::vector<std::string> &InvalidPaths,
path: clang/lib/Tooling/DependencyScanning/DependencyScanningFilesystem.cpp
user: login: qiongsiwu
body: Nope - fixed. Is it a style preference that we prefer a return value? I use either freely and I don't have a strong preference either way. 
created_at: 2025-04-17T23:02:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -108,6 +108,32 @@ DependencyScanningFilesystemSharedCache::getShardForUID(
   return CacheShards[Hash % NumShards];
 }
 
+void DependencyScanningFilesystemSharedCache::
+    diagnoseInvalidNegativeStatCachedPaths(
path: clang/lib/Tooling/DependencyScanning/DependencyScanningFilesystem.cpp
user: login: qiongsiwu
body: Good point! Fixed. 
created_at: 2025-04-17T23:02:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -108,6 +108,32 @@ DependencyScanningFilesystemSharedCache::getShardForUID(
   return CacheShards[Hash % NumShards];
 }
 
+void DependencyScanningFilesystemSharedCache::
+    diagnoseInvalidNegativeStatCachedPaths(
+        std::vector<std::string> &InvalidPaths,
path: clang/lib/Tooling/DependencyScanning/DependencyScanningFilesystem.cpp
user: login: jansvoboda11
body: Yeah, I think in general we prefer to return by value unless there's a good reason not to.
created_at: 2025-04-17T23:06:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -108,6 +108,32 @@ DependencyScanningFilesystemSharedCache::getShardForUID(
   return CacheShards[Hash % NumShards];
 }
 
+void DependencyScanningFilesystemSharedCache::
+    diagnoseInvalidNegativeStatCachedPaths(
+        std::vector<std::string> &InvalidPaths,
+        llvm::vfs::FileSystem &UnderlyingFS) const {
+  // Iterate through all shards and look for cached stat errors.
+  for (unsigned i = 0; i < NumShards; i++) {
+    const CacheShard &Shard = CacheShards[i];
+    std::lock_guard<std::mutex> LockGuard(Shard.CacheLock);
+    for (const auto &[Path, CachedPair] : Shard.CacheByFilename) {
+      const CachedFileSystemEntry *Entry = CachedPair.first;
+
+      if (Entry->getError()) {
+        // Only examine cached errors.
+        llvm::ErrorOr<llvm::vfs::Status> Stat = UnderlyingFS.status(Path);
+        if (Stat) {
+          // This is the case where we have cached the non-existence
+          // of the file at Path first, and a a file at the path is created
+          // later. The cache entry is not invalidated (as we have no good
+          // way to do it now), which may lead to missing file build errors.
+          InvalidPaths.push_back(std::string(Path));
path: clang/lib/Tooling/DependencyScanning/DependencyScanningFilesystem.cpp
user: login: jansvoboda11
body: Yeah, in general, we prefer reduce the number of allocations where we can for better performance.
created_at: 2025-04-17T23:07:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -108,6 +108,33 @@ DependencyScanningFilesystemSharedCache::getShardForUID(
   return CacheShards[Hash % NumShards];
 }
 
+std::vector<StringRef>
+DependencyScanningFilesystemSharedCache::getInvalidNegativeStatCachedPaths(
+    llvm::vfs::FileSystem &UnderlyingFS) const {
+  // Iterate through all shards and look for cached stat errors.
+  std::vector<StringRef> InvalidPaths;
+  for (unsigned i = 0; i < NumShards; i++) {
+    const CacheShard &Shard = CacheShards[i];
+    std::lock_guard<std::mutex> LockGuard(Shard.CacheLock);
+    for (const auto &[Path, CachedPair] : Shard.CacheByFilename) {
+      const CachedFileSystemEntry *Entry = CachedPair.first;
+
+      if (Entry->getError()) {
+        // Only examine cached errors.
+        llvm::ErrorOr<llvm::vfs::Status> Stat = UnderlyingFS.status(Path);
+        if (Stat) {
+          // This is the case where we have cached the non-existence
+          // of the file at Path first, and a a file at the path is created
+          // later. The cache entry is not invalidated (as we have no good
+          // way to do it now), which may lead to missing file build errors.
+          InvalidPaths.push_back(std::string(Path));
path: clang/lib/Tooling/DependencyScanning/DependencyScanningFilesystem.cpp
user: login: jansvoboda11
body: This will cause the `StringRef` in `InvalidPaths` to point to the temporary `std::string` that gets immediately deallocated.
created_at: 2025-04-17T23:09:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -108,6 +108,33 @@ DependencyScanningFilesystemSharedCache::getShardForUID(
   return CacheShards[Hash % NumShards];
 }
 
+std::vector<StringRef>
+DependencyScanningFilesystemSharedCache::getInvalidNegativeStatCachedPaths(
+    llvm::vfs::FileSystem &UnderlyingFS) const {
+  // Iterate through all shards and look for cached stat errors.
+  std::vector<StringRef> InvalidPaths;
+  for (unsigned i = 0; i < NumShards; i++) {
+    const CacheShard &Shard = CacheShards[i];
+    std::lock_guard<std::mutex> LockGuard(Shard.CacheLock);
+    for (const auto &[Path, CachedPair] : Shard.CacheByFilename) {
+      const CachedFileSystemEntry *Entry = CachedPair.first;
+
+      if (Entry->getError()) {
+        // Only examine cached errors.
+        llvm::ErrorOr<llvm::vfs::Status> Stat = UnderlyingFS.status(Path);
+        if (Stat) {
+          // This is the case where we have cached the non-existence
+          // of the file at Path first, and a a file at the path is created
+          // later. The cache entry is not invalidated (as we have no good
+          // way to do it now), which may lead to missing file build errors.
+          InvalidPaths.push_back(std::string(Path));
path: clang/lib/Tooling/DependencyScanning/DependencyScanningFilesystem.cpp
user: login: qiongsiwu
body: Good catch! Fixed! 
created_at: 2025-04-17T23:13:20Z
author_association: CONTRIBUTOR

PR comments:
user: login: llvmbot
created_at: 2025-04-15T00:06:11Z
body: 
<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-clang

Author: Qiongsi Wu (qiongsiwu)

<details>
<summary>Changes</summary>

We have had numerous situations where the negatively stat cached paths are invalidated during the build, and such invalidations lead to build errors. 

This PR adds an API to diagnose such cases. `DependencyScanningFilesystemSharedCache::diagnoseNegativeStatCachedPaths` allows users of the cache to ask the cache to examine all negatively stat cached paths, and re-stat the paths using the passed-in file system. If any re-stat succeeds, the API emits diagnostics. 

---
Full diff: https://github.com/llvm/llvm-project/pull/135703.diff


3 Files Affected:

- (modified) clang/include/clang/Tooling/DependencyScanning/DependencyScanningFilesystem.h (+8) 
- (modified) clang/lib/Tooling/DependencyScanning/DependencyScanningFilesystem.cpp (+26) 
- (modified) clang/unittests/Tooling/DependencyScanning/DependencyScanningFilesystemTest.cpp (+28) 


``````````diff
diff --git a/clang/include/clang/Tooling/DependencyScanning/DependencyScanningFilesystem.h b/clang/include/clang/Tooling/DependencyScanning/DependencyScanningFilesystem.h
index d12814e7c9253..648158c5287c6 100644
--- a/clang/include/clang/Tooling/DependencyScanning/DependencyScanningFilesystem.h
+++ b/clang/include/clang/Tooling/DependencyScanning/DependencyScanningFilesystem.h
@@ -220,6 +220,14 @@ class DependencyScanningFilesystemSharedCache {
   CacheShard &getShardForFilename(StringRef Filename) const;
   CacheShard &getShardForUID(llvm::sys::fs::UniqueID UID) const;
 
+  /// Visits all cached entries and re-stat an entry using the UnderlyingFS if
+  /// it is negatively stat cached. If the re-stat succeeds, print diagnostics
+  /// information to OS indicating that negative stat caching has been
+  /// invalidated.
+  void
+  diagnoseNegativeStatCachedPaths(llvm::raw_ostream &OS,
+                                  llvm::vfs::FileSystem &UnderlyingFS) const;
+
 private:
   std::unique_ptr<CacheShard[]> CacheShards;
   unsigned NumShards;
diff --git a/clang/lib/Tooling/DependencyScanning/DependencyScanningFilesystem.cpp b/clang/lib/Tooling/DependencyScanning/DependencyScanningFilesystem.cpp
index 4d738e4bea41a..fe72a1a91909e 100644
--- a/clang/lib/Tooling/DependencyScanning/DependencyScanningFilesystem.cpp
+++ b/clang/lib/Tooling/DependencyScanning/DependencyScanningFilesystem.cpp
@@ -108,6 +108,32 @@ DependencyScanningFilesystemSharedCache::getShardForUID(
   return CacheShards[Hash % NumShards];
 }
 
+void DependencyScanningFilesystemSharedCache::diagnoseNegativeStatCachedPaths(
+    llvm::raw_ostream &OS, llvm::vfs::FileSystem &UnderlyingFS) const {
+  // Iterate through all shards and look for cached stat errors.
+  for (unsigned i = 0; i < NumShards; i++) {
+    const CacheShard &Shard = CacheShards[i];
+    for (const auto &Path : Shard.CacheByFilename.keys()) {
+      auto CachedPairIt = Shard.CacheByFilename.find(Path);
+      auto CachedPair = CachedPairIt->getValue();
+      const CachedFileSystemEntry *Entry = CachedPair.first;
+
+      if (Entry->getError()) {
+        // Only examine cached errors.
+        llvm::ErrorOr<llvm::vfs::Status> Stat = UnderlyingFS.status(Path);
+        if (Stat) {
+          // This is the case where we have negatively cached the non-existence
+          // of the file at Path, but the cache is later invalidated. Report
+          // diagnostics.
+          OS << Path << " did not exist when it was first searched "
+             << "but was created later. This may have led to a build failure "
+                "due to missing files.\n";
+        }
+      }
+    }
+  }
+}
+
 const CachedFileSystemEntry *
 DependencyScanningFilesystemSharedCache::CacheShard::findEntryByFilename(
     StringRef Filename) const {
diff --git a/clang/unittests/Tooling/DependencyScanning/DependencyScanningFilesystemTest.cpp b/clang/unittests/Tooling/DependencyScanning/DependencyScanningFilesystemTest.cpp
index 111351fb90cee..02b192f1ba587 100644
--- a/clang/unittests/Tooling/DependencyScanning/DependencyScanningFilesystemTest.cpp
+++ b/clang/unittests/Tooling/DependencyScanning/DependencyScanningFilesystemTest.cpp
@@ -178,3 +178,31 @@ TEST(DependencyScanningFilesystem, CacheStatFailures) {
   DepFS.exists("/cache/a.pcm");
   EXPECT_EQ(InstrumentingFS->NumStatusCalls, 5u);
 }
+
+TEST(DependencyScanningFilesystem, DiagnoseStaleStatFailures) {
+  auto InMemoryFS = llvm::makeIntrusiveRefCnt<llvm::vfs::InMemoryFileSystem>();
+  InMemoryFS->setCurrentWorkingDirectory("/");
+  InMemoryFS->addFile("/dir", 0, llvm::MemoryBuffer::getMemBuffer(""));
+
+  DependencyScanningFilesystemSharedCache SharedCache;
+  DependencyScanningWorkerFilesystem DepFS(SharedCache, InMemoryFS);
+
+  bool Path1Exists = DepFS.exists("/path1");
+  EXPECT_EQ(Path1Exists, false);
+
+  // Adding a file that has been stat-ed,
+  InMemoryFS->addFile("/path1", 0, llvm::MemoryBuffer::getMemBuffer(""));
+  Path1Exists = DepFS.exists("/path1");
+  // Due to caching in SharedCache, path1 should not exist in
+  // DepFS's eyes.
+  EXPECT_EQ(Path1Exists, false);
+
+  std::string Diags;
+  llvm::raw_string_ostream DiagsStream(Diags);
+  SharedCache.diagnoseNegativeStatCachedPaths(DiagsStream, *InMemoryFS.get());
+
+  ASSERT_STREQ(
+      "/path1 did not exist when it was first searched but was created later. "
+      "This may have led to a build failure due to missing files.\n",
+      Diags.c_str());
+}

``````````

</details>

author_association: MEMBER
user: login: qiongsiwu
created_at: 2025-04-15T15:39:50Z
body: Note to reviewers:
I am not quite sure if passing in the underlying VFS is a good design. If I understand correctly, the underlying VFS is a dependency scanning worker property, so on the surface it seems that we will need this diagnostic per worker, instead of per scanning service, and we do need a VFS from somewhere to query. 

I am all ears for a different design. 

Additionally, I can store (pointers to) the cached failed results in a list, so that we do not need to scan the whole cache for the diagnostic.
author_association: CONTRIBUTOR
user: login: qiongsiwu
created_at: 2025-04-16T17:21:20Z
body: > On a second thought, what's the current intent for build system integration? I'm curious why we're emitting a diagnostic here rather than just passing the list of paths back to the build system and letting it decide how to report these.

This is a good point. Additionally, the current API design requires a worker calling it, since it needs the worker's VFS (or whoever owns the "current " VFS) to do the stat again. So this will report a diagnostic per worker, which may not be ideal. 

I can change the API such that we have two new methods instead of one. The first method collects the erroneously negatively cached paths, and the second reports it (in a form that the build system likes). The collection has to be done per worker, but the reporting can now be done once when the scanning service is destroyed. Does this sound reasonable? 
author_association: CONTRIBUTOR
user: login: jansvoboda11
created_at: 2025-04-16T19:54:11Z
body: I don't think workers need to be involved at all - we just need to pass the same VFS to this function that we use to instantiate all workers. For example, downstream `clang_experimental_DependencyScannerWorker_create_v0()` always uses `llvm::vfs::createPhysicalFileSystem()`, so I'm thinking whatever C API we invent for this functionality, it can operate on the service only, it just needs to be consistent in using the physical VFS. That C API can return `CXCStringArray` and let the build system invent its own message format.
author_association: CONTRIBUTOR
user: login: jansvoboda11
created_at: 2025-04-16T19:57:51Z
body: (To be frank, I think it would be better to enforce the consistency of VFSs by having the service own one and the workers just applying overlays as needed, but that's definitely out of scope for this patch.)
author_association: CONTRIBUTOR
user: login: qiongsiwu
created_at: 2025-04-17T20:22:30Z
body: Thanks @jansvoboda11 ! The API is updated so that the end result is a vector of paths (strings), so the caller can create the diagnostics. 

I will add the C-API to https://github.com/swiftlang/llvm-project after this PR lands. 
author_association: CONTRIBUTOR