Cache: only check each path for outside updates once
Make sure only checking for updates once doesn't mess up our tests