Repository: iterative/dvc

Issue:
diff: output after partial pull wrongfully includes top dir
# Bug Report

## Description

After pulling a subset of a dvc tracked dataset, the results of `dvc diff <subpath in dataset>` show wrong (or confusing) results.
Specifically, if only a subdir of a dataset is pulled and modified, the top level item appears as "Removed", which is wrong.

### Reproduce

Using a fresh repo with the `dataset.dvc` from [dvc-bench/data/mnist](https://github.com/iterative/dvc-bench/blob/main/data/mnist/dataset.dvc) at its root.

1. Pull data partially, create 1 new file
```console
$ dvc pull dataset/train/4
$ echo "new" > dataset/train/4/new.txt
```
Note, this uses the newly developed "virtual dir":
- https://github.com/iterative/dvc/pull/9389

2. git diff shows (expected) 1 added file to dataset:
```console
$ git --no-pager diff
diff --git a/dataset.dvc b/dataset.dvc
index ac31d91..79740af 100644
--- a/dataset.dvc
+++ b/dataset.dvc
@@ -1,6 +1,5 @@
 outs:
-- md5: e42412b82dcab425ce9c7e2d0abdfb78.dir
-  size: 19258482
-  nfiles: 70000
+- md5: 881ee47fcae5c4d9625071cfdc5c3991.dir
+  nfiles: 70001
   path: dataset
 ```
3. running dvc diff, here we have the fishy result:

```console
$ dvc diff --targets dataset/train/4
Added:
    dataset/train/4/
    dataset/train/4/new.txt

Deleted:
    dataset/

files summary: 1 added
```

☝️  it shows a the top level dir as "Deleted", and also the containing dir as added (not just the file), looks like a bug


Another questionable behavior is running when running `dvc diff --targets dataset/` - there it would dump all the files not present in workspace (`dataset/test/*` , `dataset/traing/{!4}/*`) - Here I intentionally didn't pull them, but they are not missing from my local `dataset.dvc`


### Expected
```console
$ dvc diff --targets dataset/train/4
Added:
    dataset/train/4/new.txt
files summary: 1 added
```

Questionable, should we also expect to see `dataset/` as "modified" since it's the dataset (data-item?) level, and `dataset.dvc` contents are modified and `dataset.dvc` ~represents~ `datasets/` ? I would argue it's not expected but, not sure

### Environment information

```console
$ dvc doctor
DVC version: 2.57.3.dev6+g3ddd4b87
----------------------------------
Platform: Python 3.8.12 on macOS-13.0-arm64-arm-64bit
Subprojects:
	dvc_data = 0.51.0
	dvc_objects = 0.22.0
	dvc_render = 0.5.3
	dvc_task = 0.2.1
	scmrepo = 1.0.3
Supports:
	http (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
	https (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
	s3 (s3fs = 2023.5.0, boto3 = 1.26.76)
Config:
	Global: /Users/oded/Library/Application Support/dvc
	System: /Library/Application Support/dvc
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk3s1s1
Caches: local
Remotes: local, s3, https
Workspace directory: apfs on /dev/disk3s1s1
Repo: dvc, git
Repo.site_cache_dir: /Library/Caches/dvc/repo/3b754cc9e7b81a867238f983ad8551e1
```

**Additional Information:**

Related:
- #8327


Discussion:
Additional info, `dvc data status --granular` output attached
[data_status_gran.txt](https://github.com/iterative/dvc/files/11559435/data_status_gran.txt)

> it shows a the top level dir as "Deleted", and also the containing dir as added

This may have actually got worse since 3.0. Even without adding a new file, I see that behavior, and now I also see all the other files in `dataset/train/4` as modified:

```bash
git clone git@github.com:iterative/dvc-bench.git
cd dvc-bench/data/mnist
dvc pull dataset/train/4
dvc diff --targets dataset/train/4
```

The output looks like:

```
Added:
    data/mnist/dataset/train/4/

Deleted:
    data/mnist/dataset/

Modified:
    data/mnist/dataset/train/4/00003.png
    data/mnist/dataset/train/4/00010.png
    data/mnist/dataset/train/4/00021.png
    ...
```
New `modified` state for other files is related to the hash changes. md5 != md5-dos2unix, hence hash mismatch and that diff. Taking a closer look.
You can ignore my comment then since it seems like it's just about 2.x->3.x conversion.

Resolving patch:
diff --git a/dvc/repo/diff.py b/dvc/repo/diff.py
--- a/dvc/repo/diff.py
+++ b/dvc/repo/diff.py
@@ -22,7 +22,7 @@ def _hash(entry):
     return None
 
 
-def _diff(old, new, with_missing=False):
+def _diff(old, new, data_keys, with_missing=False):
     from dvc_data.index.diff import ADD, DELETE, MODIFY, RENAME
     from dvc_data.index.diff import diff as idiff
 
@@ -34,12 +34,23 @@ def _diff(old, new, with_missing=False):
         "not in cache": [],
     }
 
+    def meta_cmp_key(meta):
+        if not meta:
+            return meta
+        return meta.isdir
+
     for change in idiff(
         old,
         new,
         with_renames=True,
-        hash_only=True,
+        meta_cmp_key=meta_cmp_key,
+        roots=data_keys,
     ):
+        if (change.old and change.old.isdir and not change.old.hash_info) or (
+            change.new and change.new.isdir and not change.new.hash_info
+        ):
+            continue
+
         if change.typ == ADD:
             ret["added"].append(
                 {
@@ -116,6 +127,7 @@ def diff(
         b_rev = "workspace"
         with_missing = True
 
+    data_keys = set()
     for rev in self.brancher(revs=[a_rev, b_rev]):
         if rev == "workspace" and b_rev != "workspace":
             # brancher always returns workspace, but we only need to compute
@@ -132,6 +144,8 @@ def onerror(target, _exc):
             recursive=recursive,
         )
 
+        data_keys.update(view.data_keys.get("repo", set()))
+
         if rev == "workspace":
             from .index import build_data_index
 
@@ -165,4 +179,4 @@ def onerror(target, _exc):
         new = indexes[b_rev]
 
     with ui.status("Calculating diff"):
-        return _diff(old, new, with_missing=with_missing)
+        return _diff(old, new, data_keys, with_missing=with_missing)
diff --git a/dvc/repo/index.py b/dvc/repo/index.py
--- a/dvc/repo/index.py
+++ b/dvc/repo/index.py
@@ -1,5 +1,6 @@
 import logging
 import time
+from collections import defaultdict
 from functools import partial
 from typing import (
     TYPE_CHECKING,
@@ -319,6 +320,22 @@ def outs(self) -> Iterator["Output"]:
         for stage in self.stages:
             yield from stage.outs
 
+    @cached_property
+    def out_data_keys(self) -> Dict[str, Set["DataIndexKey"]]:
+        by_workspace: Dict[str, Set["DataIndexKey"]] = defaultdict(set)
+
+        by_workspace["repo"] = set()
+        by_workspace["local"] = set()
+
+        for out in self.outs:
+            if not out.use_cache:
+                continue
+
+            ws, key = out.index_key
+            by_workspace[ws].add(key)
+
+        return dict(by_workspace)
+
     @property
     def decorated_outs(self) -> Iterator["Output"]:
         for output in self.outs:
@@ -359,8 +376,6 @@ def _plot_sources(self) -> List[str]:
 
     @cached_property
     def data_keys(self) -> Dict[str, Set["DataIndexKey"]]:
-        from collections import defaultdict
-
         by_workspace: Dict[str, Set["DataIndexKey"]] = defaultdict(set)
 
         by_workspace["repo"] = set()
@@ -377,8 +392,6 @@ def data_keys(self) -> Dict[str, Set["DataIndexKey"]]:
 
     @cached_property
     def metric_keys(self) -> Dict[str, Set["DataIndexKey"]]:
-        from collections import defaultdict
-
         from .metrics.show import _collect_top_level_metrics
 
         by_workspace: Dict[str, Set["DataIndexKey"]] = defaultdict(set)
@@ -400,8 +413,6 @@ def metric_keys(self) -> Dict[str, Set["DataIndexKey"]]:
 
     @cached_property
     def plot_keys(self) -> Dict[str, Set["DataIndexKey"]]:
-        from collections import defaultdict
-
         by_workspace: Dict[str, Set["DataIndexKey"]] = defaultdict(set)
 
         by_workspace["repo"] = set()
@@ -520,8 +531,6 @@ def used_objs(
         jobs: Optional[int] = None,
         push: bool = False,
     ) -> "ObjectContainer":
-        from collections import defaultdict
-
         used: "ObjectContainer" = defaultdict(set)
         pairs = self.collect_targets(targets, recursive=recursive, with_deps=with_deps)
         for stage, filter_info in pairs:
@@ -640,9 +649,23 @@ def outs(self) -> Iterator["Output"]:
         yield from {out for (out, _) in self._filtered_outs}
 
     @cached_property
-    def _data_prefixes(self) -> Dict[str, "_DataPrefixes"]:
-        from collections import defaultdict
+    def out_data_keys(self) -> Dict[str, Set["DataIndexKey"]]:
+        by_workspace: Dict[str, Set["DataIndexKey"]] = defaultdict(set)
+
+        by_workspace["repo"] = set()
+        by_workspace["local"] = set()
+
+        for out in self.outs:
+            if not out.use_cache:
+                continue
+
+            ws, key = out.index_key
+            by_workspace[ws].add(key)
 
+        return dict(by_workspace)
+
+    @cached_property
+    def _data_prefixes(self) -> Dict[str, "_DataPrefixes"]:
         prefixes: Dict[str, "_DataPrefixes"] = defaultdict(
             lambda: _DataPrefixes(set(), set())
         )
@@ -660,8 +683,6 @@ def _data_prefixes(self) -> Dict[str, "_DataPrefixes"]:
 
     @cached_property
     def data_keys(self) -> Dict[str, Set["DataIndexKey"]]:
-        from collections import defaultdict
-
         ret: Dict[str, Set["DataIndexKey"]] = defaultdict(set)
 
         for out, filter_info in self._filtered_outs:
@@ -714,7 +735,7 @@ def key_filter(workspace: str, key: "DataIndexKey"):
         return data
 
 
-def build_data_index(  # noqa: C901
+def build_data_index(  # noqa: C901, PLR0912
     index: Union["Index", "IndexView"],
     path: str,
     fs: "FileSystem",
@@ -777,14 +798,6 @@ def build_data_index(  # noqa: C901
             data.add(entry)
             callback.relative_update(1)
 
-        if compute_hash:
-            tree_meta, tree = build_tree(data, key, name=hash_name)
-            out_entry.meta = tree_meta
-            out_entry.hash_info = tree.hash_info
-            out_entry.loaded = True
-            data.add(out_entry)
-            callback.relative_update(1)
-
     for key in parents:
         parent_path = fs.path.join(path, *key)
         if not fs.exists(parent_path):
@@ -793,6 +806,23 @@ def build_data_index(  # noqa: C901
         data.add(direntry)
         callback.relative_update(1)
 
+    if compute_hash:
+        out_keys = index.out_data_keys.get(workspace, set())
+        data_keys = index.data_keys.get(workspace, set())
+        for key in data_keys.intersection(out_keys):
+            hash_name = _get_entry_hash_name(index, workspace, key)
+
+            out_entry = data.get(key)
+            if not out_entry or not out_entry.isdir:
+                continue
+
+            tree_meta, tree = build_tree(data, key, name=hash_name)
+            out_entry.meta = tree_meta
+            out_entry.hash_info = tree.hash_info
+            out_entry.loaded = True
+            data.add(out_entry)
+            callback.relative_update(1)
+
     return data
 
 
@@ -801,10 +831,14 @@ def _get_entry_hash_name(
 ) -> str:
     from dvc_data.hashfile.hash import DEFAULT_ALGORITHM
 
-    try:
-        src_entry = index.data[workspace][key]
+    for idx in reversed(range(len(key) + 1)):
+        prefix = key[:idx]
+        try:
+            src_entry = index.data[workspace][prefix]
+        except KeyError:
+            continue
+
         if src_entry.hash_info and src_entry.hash_info.name:
             return src_entry.hash_info.name
-    except KeyError:
-        pass
+
     return DEFAULT_ALGORITHM


Test patch:
diff --git a/tests/func/test_diff.py b/tests/func/test_diff.py
--- a/tests/func/test_diff.py
+++ b/tests/func/test_diff.py
@@ -668,3 +668,59 @@ def test_rename_multiple_files_same_hashes(tmp_dir, scm, dvc):
             },
         ],
     }
+
+
+def test_diff_granular(tmp_dir, dvc, scm):
+    tmp_dir.gen(
+        {
+            "dir": {
+                "data": {
+                    "subdir": {"subfoo": "subfoo", "subbar": "subbar"},
+                    "foo": "foo",
+                    "bar": "bar",
+                },
+            },
+        }
+    )
+
+    dvc.add(os.path.join("dir", "data"))
+    scm.add(os.path.join("dir", "data.dvc"))
+    scm.add(os.path.join("dir", ".gitignore"))
+    scm.commit("data")
+
+    assert dvc.diff() == {}
+
+    (tmp_dir / "dir" / "data" / "subdir" / "new").write_text("new")
+
+    assert dvc.diff() == {
+        "added": [
+            {
+                "hash": "22af645d1859cb5ca6da0c484f1f37ea",
+                "path": os.path.join("dir", "data", "subdir", "new"),
+            }
+        ],
+        "deleted": [],
+        "modified": [
+            {
+                "hash": {
+                    "new": "efa5b20d5f935dcc5555b26db6e19b76.dir",
+                    "old": "1aca2c799df82929bbdd976557975546.dir",
+                },
+                "path": os.path.join("dir", "data", ""),
+            }
+        ],
+        "not in cache": [],
+        "renamed": [],
+    }
+    assert dvc.diff(targets=[os.path.join("dir", "data", "subdir")]) == {
+        "added": [
+            {
+                "hash": "22af645d1859cb5ca6da0c484f1f37ea",
+                "path": os.path.join("dir", "data", "subdir", "new"),
+            }
+        ],
+        "deleted": [],
+        "modified": [],
+        "not in cache": [],
+        "renamed": [],
+    }