Revert "Invalidate the file system cache entries for files that may rebuild"

This reverts commit r209910, which is breaking some of the bots.