Repository: python/mypy

Issue:
Rework cache to key on hash of file contents instead of mtime
The key of the cache is currently the mtime (and size) of the source file. A problem here is that it's impossible to share a "pre-warmed" cache between users because the mtimes of files are different for each users. I propose to change the cache to be keyed on a hash instead. It should probably be sha1, or perhaps md5 if it's significantly faster.

Hashing all files in a large tree is pretty efficient. But just to make it more efficient we could add a "meta cache" that is keyed by mtime, where if we've already got a hash for a file and its mtime isn't changed, we bypass computation of the hash. If the mtime changed (but the size matches that in the cache) we re-hash the file and if that hash matches the hash from the cache we update the mtime in the "meta cache".

Once we have all this in place we can think about sharing copies of the cache between users (using a mechanism outside mypy, so mypy won't have to worry about authentication and security).


Discussion:
I'll work on it.
It seems we may also need to calculate the hash of the cache (.json) file because there's a risk that the meta file is no longer in sync with the cache file. Currently, this is ensured by checking the modified time of the data file; this won't work when files are moved between users.

Our options are:

(1) verify that the modified date of the cache file matches the corresponding field of the meta file
(2) assume that meta and cache files are always in sync (by using atomic multiple file writes) and hoping no one finds a way to use a inconsistent cache file
(3) merge meta and cache file into one
Option (1) is ok, but may increase the hashing overhead.

Option (2) is unacceptable because *multiple file* atomic write is unavailable without file locking, and even if it wasn't a problem, other problems might cause meta and cache files to go out of sync.

Option (3) seems the most attractive. We can separate the meta into the header section of the file in order to avoid reading the entire file when the header already tells us that it's not fresh; as a result, there's no performance impact (in fact it may even be slightly faster than the current approach, since we'll open fewer file).
Summarizing the discussion at the sprint:

Currently we use `mtime_data` (the time the cache file was last modified) to do two things:

- validate that the cache file is in sync with the source file
- validate that the cache file is in sync with the dependencies

The first bullet point is solved by using file content hashes. The second bullet point can be solved by using the same approach as git: include the immediate parent hashes inside the file hash. (This is probably a cleaner approach than adding generation ids with its history dependence.)

We need to make sure that our approach addresses cycles (which don't exist in git). Hopefully the fact that we aggregate them into SCCs is enough, but we should double check cases like this: `A(SCC1) <- B(SCC2) <- C(SCC3)`, `C` changed after `A` cache was generated. Note that we do not want to worry about recursive dependencies, everything should work just based on direct ancestors.
@JukkaL  suggested to not merge meta and data files yet, and do everything else as the first PR. Whether or not merging the files is worthwhile can be discussed later.

Also, we need to make sure the optimizations that involve `interface_hash` aren't lost; therefore, `interface_hash` should not be changed to depend on any additional information (that's why `mtime` and `interface_hash` are separated).

Resolving patch:
diff --git a/mypy/build.py b/mypy/build.py
--- a/mypy/build.py
+++ b/mypy/build.py
@@ -294,6 +294,7 @@ def default_lib_path(data_dir: str,
                         ('path', str),
                         ('mtime', float),
                         ('size', int),
+                        ('hash', str),
                         ('dependencies', List[str]),  # names of imported modules
                         ('data_mtime', float),  # mtime of data_json
                         ('data_json', str),  # path of <id>.data.json
@@ -684,8 +685,12 @@ def verify_module(id: str, path: str) -> bool:
     return True
 
 
-def read_with_python_encoding(path: str, pyversion: Tuple[int, int]) -> str:
-    """Read the Python file with while obeying PEP-263 encoding detection"""
+def read_with_python_encoding(path: str, pyversion: Tuple[int, int]) -> Tuple[str, str]:
+    """Read the Python file with while obeying PEP-263 encoding detection.
+
+    Returns:
+      A tuple: the source as a string, and the hash calculated from the binary representation.
+    """
     source_bytearray = bytearray()
     encoding = 'utf8' if pyversion[0] >= 3 else 'ascii'
 
@@ -693,6 +698,7 @@ def read_with_python_encoding(path: str, pyversion: Tuple[int, int]) -> str:
         # read first two lines and check if PEP-263 coding is present
         source_bytearray.extend(f.readline())
         source_bytearray.extend(f.readline())
+        m = hashlib.md5(source_bytearray)
 
         # check for BOM UTF-8 encoding and strip it out if present
         if source_bytearray.startswith(b'\xef\xbb\xbf'):
@@ -705,16 +711,17 @@ def read_with_python_encoding(path: str, pyversion: Tuple[int, int]) -> str:
             if _encoding != 'mypy':
                 encoding = _encoding
 
-        source_bytearray.extend(f.read())
+        remainder = f.read()
+        m.update(remainder)
+        source_bytearray.extend(remainder)
         try:
-            source_bytearray.decode(encoding)
+            source_text = source_bytearray.decode(encoding)
         except LookupError as lookuperr:
             raise DecodeError(str(lookuperr))
-        return source_bytearray.decode(encoding)
+        return source_text, m.hexdigest()
 
 
-def get_cache_names(id: str, path: str, cache_dir: str,
-                    pyversion: Tuple[int, int]) -> Tuple[str, str]:
+def get_cache_names(id: str, path: str, manager: BuildManager) -> Tuple[str, str]:
     """Return the file names for the cache files.
 
     Args:
@@ -727,6 +734,8 @@ def get_cache_names(id: str, path: str, cache_dir: str,
       A tuple with the file names to be used for the meta JSON and the
       data JSON, respectively.
     """
+    cache_dir = manager.options.cache_dir
+    pyversion = manager.options.python_version
     prefix = os.path.join(cache_dir, '%d.%d' % pyversion, *id.split('.'))
     is_package = os.path.basename(path).startswith('__init__.py')
     if is_package:
@@ -747,8 +756,7 @@ def find_cache_meta(id: str, path: str, manager: BuildManager) -> Optional[Cache
       valid; otherwise None.
     """
     # TODO: May need to take more build options into account
-    meta_json, data_json = get_cache_names(
-        id, path, manager.options.cache_dir, manager.options.python_version)
+    meta_json, data_json = get_cache_names(id, path, manager)
     manager.trace('Looking for {} {}'.format(id, data_json))
     if not os.path.exists(meta_json):
         manager.trace('Could not load cache for {}: could not find {}'.format(id, meta_json))
@@ -766,6 +774,7 @@ def find_cache_meta(id: str, path: str, manager: BuildManager) -> Optional[Cache
         meta.get('path'),
         meta.get('mtime'),
         meta.get('size'),
+        meta.get('hash'),
         meta.get('dependencies', []),
         meta.get('data_mtime'),
         data_json,
@@ -802,28 +811,72 @@ def find_cache_meta(id: str, path: str, manager: BuildManager) -> Optional[Cache
     return m
 
 
-def is_meta_fresh(meta: Optional[CacheMeta], id: str, path: str, manager: BuildManager) -> bool:
-    if meta is None:
+def random_string() -> str:
+    return binascii.hexlify(os.urandom(8)).decode('ascii')
+
+
+def atomic_write(filename: str, *lines: str) -> bool:
+    tmp_filename = filename + '.' + random_string()
+    try:
+        with open(tmp_filename, 'w') as f:
+            for line in lines:
+                f.write(line)
+        os.replace(tmp_filename, filename)
+    except os.error as err:
         return False
+    return True
+
+
+def validate_meta(meta: Optional[CacheMeta], id: str, path: str,
+                  manager: BuildManager) -> Optional[CacheMeta]:
+    '''Checks whether the cached AST of this module can be used.
+
+    Return:
+      None, if the cached AST is unusable.
+      Original meta, if mtime/size matched.
+      Meta with mtime updated to match source file, if hash/size matched but mtime didn't.
+    '''
+    # This requires two steps. The first one is obvious: we check that the module source file
+    # contents is the same as it was when the cache data file was created. The second one is not
+    # too obvious: we check that the cache data file mtime has not changed; it is needed because
+    # we use cache data file mtime to propagate information about changes in the dependencies.
+
+    if meta is None:
+        return None
 
     # TODO: Share stat() outcome with find_module()
     st = manager.get_stat(path)  # TODO: Errors
-    if st.st_mtime != meta.mtime or st.st_size != meta.size:
-        manager.log('Metadata abandoned for {}: file {} is modified'.format(id, path))
-        return False
+    if st.st_size != meta.size:
+        manager.log('Metadata abandoned for {}: file {} has different size'.format(id, path))
+        return None
 
-    # It's a match on (id, path, mtime, size).
+    if st.st_mtime != meta.mtime:
+        with open(path, 'rb') as f:
+            source_hash = hashlib.md5(f.read()).hexdigest()
+        if source_hash != meta.hash:
+            manager.log('Metadata abandoned for {}: file {} has different hash'.format(id, path))
+            return None
+        else:
+            manager.log('Metadata ok for {}: file {} (match on size, hash)'.format(id, path))
+            # Optimization: update meta.mtime (otherwise, this mismatch will not disappear).
+            meta = meta._replace(mtime=st.st_mtime)
+            if manager.options.debug_cache:
+                meta_str = json.dumps(meta, indent=2, sort_keys=True)
+            else:
+                meta_str = json.dumps(meta)
+            meta_json, _ = get_cache_names(id, os.path.abspath(path), manager)
+            manager.log('Updating mtime for {}: file {}, meta {}, mtime {}'
+                        .format(id, path, meta_json, meta.mtime))
+            atomic_write(meta_json, meta_str)  # Ignore errors, since this is just an optimization.
+
+    # It's a match on (id, path, mtime/hash, size).
     # Check data_json; assume if its mtime matches it's good.
     # TODO: stat() errors
     if os.path.getmtime(meta.data_json) != meta.data_mtime:
         manager.log('Metadata abandoned for {}: data cache is modified'.format(id))
-        return False
+        return None
     manager.log('Found {} {} (metadata is fresh)'.format(id, meta.data_json))
-    return True
-
-
-def random_string() -> str:
-    return binascii.hexlify(os.urandom(8)).decode('ascii')
+    return meta
 
 
 def compute_hash(text: str) -> str:
@@ -836,7 +889,7 @@ def compute_hash(text: str) -> str:
 def write_cache(id: str, path: str, tree: MypyFile,
                 dependencies: List[str], suppressed: List[str],
                 child_modules: List[str], dep_prios: List[int],
-                old_interface_hash: str, manager: BuildManager) -> str:
+                old_interface_hash: str, source_hash: str, manager: BuildManager) -> str:
     """Write cache files for a module.
 
     Note that this mypy's behavior is still correct when any given
@@ -858,19 +911,13 @@ def write_cache(id: str, path: str, tree: MypyFile,
     """
     # Obtain file paths
     path = os.path.abspath(path)
-    meta_json, data_json = get_cache_names(
-        id, path, manager.options.cache_dir, manager.options.python_version)
+    meta_json, data_json = get_cache_names(id, path, manager)
     manager.log('Writing {} {} {} {}'.format(id, path, meta_json, data_json))
 
     # Make sure directory for cache files exists
     parent = os.path.dirname(data_json)
     assert os.path.dirname(meta_json) == parent
 
-    # Construct temp file names
-    nonce = '.' + random_string()
-    data_json_tmp = data_json + nonce
-    meta_json_tmp = meta_json + nonce
-
     # Serialize data and analyze interface
     data = tree.serialize()
     if manager.options.debug_cache:
@@ -903,16 +950,10 @@ def write_cache(id: str, path: str, tree: MypyFile,
         manager.trace("Interface for {} is unchanged".format(id))
     else:
         manager.trace("Interface for {} has changed".format(id))
-        try:
-            with open(data_json_tmp, 'w') as f:
-                f.write(data_str)
-                f.write('\n')
-            os.replace(data_json_tmp, data_json)
-            data_mtime = os.path.getmtime(data_json)
-        except os.error as err:
+        if not atomic_write(data_json, data_str, '\n'):
             # Most likely the error is the replace() call
             # (see https://github.com/python/mypy/issues/3215).
-            manager.log("Error writing data JSON file {}".format(data_json_tmp))
+            manager.log("Error writing data JSON file {}".format(data_json))
             # Let's continue without writing the meta file.  Analysis:
             # If the replace failed, we've changed nothing except left
             # behind an extraneous temporary file; if the replace
@@ -922,14 +963,17 @@ def write_cache(id: str, path: str, tree: MypyFile,
             # Both have the effect of slowing down the next run a
             # little bit due to an out-of-date cache file.
             return interface_hash
+        data_mtime = os.path.getmtime(data_json)
 
     mtime = st.st_mtime
     size = st.st_size
     options = manager.options.clone_for_module(id)
+    assert source_hash is not None
     meta = {'id': id,
             'path': path,
             'mtime': mtime,
             'size': size,
+            'hash': source_hash,
             'data_mtime': data_mtime,
             'dependencies': dependencies,
             'suppressed': suppressed,
@@ -941,18 +985,15 @@ def write_cache(id: str, path: str, tree: MypyFile,
             }
 
     # Write meta cache file
-    try:
-        with open(meta_json_tmp, 'w') as f:
-            if manager.options.debug_cache:
-                json.dump(meta, f, indent=2, sort_keys=True)
-            else:
-                json.dump(meta, f)
-        os.replace(meta_json_tmp, meta_json)
-    except os.error as err:
+    if manager.options.debug_cache:
+        meta_str = json.dumps(meta, indent=2, sort_keys=True)
+    else:
+        meta_str = json.dumps(meta)
+    if not atomic_write(meta_json, meta_str):
         # Most likely the error is the replace() call
         # (see https://github.com/python/mypy/issues/3215).
         # The next run will simply find the cache entry out of date.
-        manager.log("Error writing meta JSON file {}".format(meta_json_tmp))
+        manager.log("Error writing meta JSON file {}".format(meta_json))
 
     return interface_hash
 
@@ -1116,6 +1157,7 @@ class State:
     path = None  # type: Optional[str]  # Path to module source
     xpath = None  # type: str  # Path or '<string>'
     source = None  # type: Optional[str]  # Module source code
+    source_hash = None  # type: str  # Hash calculated based on the source code
     meta = None  # type: Optional[CacheMeta]
     data = None  # type: Optional[str]
     tree = None  # type: Optional[MypyFile]
@@ -1243,7 +1285,8 @@ def __init__(self,
             if self.meta is not None:
                 self.interface_hash = self.meta.interface_hash
         self.add_ancestors()
-        if is_meta_fresh(self.meta, self.id, self.path, manager):
+        self.meta = validate_meta(self.meta, self.id, self.path, manager)
+        if self.meta:
             # Make copies, since we may modify these and want to
             # compare them to the originals later.
             self.dependencies = list(self.meta.dependencies)
@@ -1255,7 +1298,6 @@ def __init__(self,
             self.dep_line_map = {}
         else:
             # Parse the file (and then some) to get the dependencies.
-            self.meta = None
             self.parse_file()
             self.suppressed = []
             self.child_modules = set()
@@ -1421,7 +1463,8 @@ def parse_file(self) -> None:
             if self.path and source is None:
                 try:
                     path = manager.maybe_swap_for_shadow_path(self.path)
-                    source = read_with_python_encoding(path, self.options.python_version)
+                    source, self.source_hash = read_with_python_encoding(
+                        path, self.options.python_version)
                 except IOError as ioerr:
                     raise CompileError([
                         "mypy: can't read file '{}': {}".format(self.path, ioerr.strerror)])
@@ -1572,7 +1615,7 @@ def write_cache(self) -> None:
         new_interface_hash = write_cache(
             self.id, self.path, self.tree,
             list(self.dependencies), list(self.suppressed), list(self.child_modules),
-            dep_prios, self.interface_hash,
+            dep_prios, self.interface_hash, self.source_hash,
             self.manager)
         if new_interface_hash == self.interface_hash:
             self.manager.log("Cached module {} has same interface".format(self.id))


Test patch:
diff --git a/mypy/test/testcheck.py b/mypy/test/testcheck.py
--- a/mypy/test/testcheck.py
+++ b/mypy/test/testcheck.py
@@ -282,7 +282,7 @@ def find_missing_cache_files(self, modules: Dict[str, str],
         missing = {}
         for id, path in modules.items():
             meta = build.find_cache_meta(id, path, manager)
-            if not build.is_meta_fresh(meta, id, path, manager):
+            if not build.validate_meta(meta, id, path, manager):
                 missing[id] = path
         return set(missing.values())