Repository: iterative/dvc

Issue:
status: "recalculating" hashes each call
When calling `status` on modified dir, DVC shows message `Computing file/dir hashes (only done once)` for each status call.
Reproduction script:

```bash
#!/bin/bash

set -exu

TMPDIR=issue-7390
mkdir $TMPDIR
pushd $TMPDIR

wsp=test_wspace
rp=test_repo

rm -rf $wsp && mkdir $wsp && pushd $wsp

mkdir $rp && pushd $rp

git init --quiet
dvc init --quiet
dvc config core.checksum_jobs 1

git add -A
git commit -am "init"

mkdir data

for i in {1..2000}
do
	dd if=/dev/random of=data/file$i bs=1k count=10 >& /dev/null
done

dvc add data
time dvc status
echo modification >> data/another_file

time dvc status 
time dvc status 
```
It might not necessarily be an issue with DVC, as "calculation" for subsequent status calls seems to be faster than the original one. Maybe it's just a problem with a message.

EDIT:
Running previous script for much bigger files shows that the recalculation does not happen. So its probably a message that is the problem.


Discussion:
@pared Are you sure the recalculation doesn't happen? I see that subsequent calls take just as long.

This seems related to known regressions from the current refactoring. @efiop Is this expected?

```
$ time dvc status
wiki.dvc:
        changed outs:
                modified:           wiki
dvc status  47.80s user 11.55s system 104% cpu 56.869 total
$ time dvc status
wiki.dvc:
        changed outs:
                modified:           wiki
dvc status  47.78s user 11.07s system 102% cpu 57.451 total
```

A smaller dataset to reproduce:

```
$ dvc get-url http://images.cocodataset.org/zips/val2017.zip
$ unzip val2017.zip
$ dvc add val2017
$ echo foo > val2017/foo
$ time dvc status
val2017.dvc:
        changed outs:
                modified:           val2017
dvc status  5.62s user 1.18s system 98% cpu 6.896 total
$ time dvc status
val2017.dvc:
        changed outs:
                modified:           val2017
dvc status  5.61s user 1.19s system 95% cpu 7.105 total
$ dvc add val2017
$ time dvc status
Data and pipelines are up to date.
dvc status  0.42s user 0.13s system 95% cpu 0.575 total
```
Can confirm:
```
+ dvc add data
100% Adding...|██████████|1/1 [00:07,  7.09s/file]

To track the changes with git, run:

    git add .gitignore data.dvc

To enable auto staging, run:

	dvc config core.autostage true
+ dvc status
Data and pipelines are up to date.

real	0m0.569s
user	0m0.454s
sys	0m0.077s
+ echo modification
+ dvc status
data.dvc:
	changed outs:
		modified:           data

real	0m2.941s
user	0m2.494s
sys	0m0.243s
+ dvc status
data.dvc:
	changed outs:
		modified:           data

real	0m3.053s
user	0m2.555s
sys	0m0.301s```
The hashes are not recalculated, they are retrieved from the `md5s` db, so messaging is somehow misleading. 
It's true that "computing" it's only done once but it should not say 'computing hashes' if we are not computing, IMO.

Times for subsequent runs are the same, though.

I run with viztracer as follows (same snippet from @pawel but I just used 50 files instead of 2000 for clearer debug):

```
dvc status --viztracer --viztracer-depth 10
dvc status --viztracer --viztracer-depth 10
```

And visualized with:

```
vizviewer viztracer.dvc-20220218_093001.json
```


It appears that https://github.com/iterative/dvc/commit/114a07e7a448b667682fbe331ac06efbf4453494 introduced a performance downgrade for `status`:

```
# Previous commit
# @34697ef2983cd4fac7cc66aefc8577fdde8980b4
dvc status  0,31s user 0,11s system 64% cpu 0,654 total
# @114a07e7a448b667682fbe331ac06efbf4453494
dvc status  1,46s user 0,32s system 98% cpu 1,807 total
```

After https://github.com/iterative/dvc/commit/114a07e7a448b667682fbe331ac06efbf4453494, when there is a new untracked file, we are building a new `Tree` object (`_get_tree_obj`->`_build_tree`) on each `status` call. 

Before https://github.com/iterative/dvc/commit/114a07e7a448b667682fbe331ac06efbf4453494, we were loading from state (`_load_from_state`) from the stagging DB.

This is causing the `Computing file/dir hashes (only done once)` to appear and the performance downgrade.

I suspect this affects more places beyond `status`.

cc @efiop @pmrowla 
@dberenbaum strange, what `dvc version` you used? For me on latest master, (500 files, 10Mb each):
```bash
+ dvc add data
100% Adding...|██████████████████████████████████████████████████████████████████████████████████████████████████|1/1 [00:12, 12.75s/file]
                                                                                                                                          
To track the changes with git, run:                                                                                                       

    git add .gitignore data.dvc

To enable auto staging, run:

        dvc config core.autostage true

real    0m13.364s
user    0m8.684s
sys     0m1.900s
+ echo modification
+ tee data/another_file
+ dvc status
data.dvc:                                                                                                                                 
        changed outs:                                                                                                                     
                modified:           data

real    0m0.959s
user    0m0.759s
sys     0m0.163s
+ dvc status
data.dvc:                                                                                                                                 
        changed outs:                                                                                                                     
                modified:           data

real    0m0.984s
user    0m0.789s
sys     0m0.162s
```
@pared I also used latest master. Maybe I misinterpreted your concern. It seems like the two `status` calls after modification took the same time, which I thought was the problem?
Sorry @dberenbaum, my concern was that we get `Computing file/dir hashes (only done once)` each time we call `dvc status` - so its clearly not done only once. I am yet not sure if recalculation actually happens, but the message seems misleading.
We have been looking into that with @daavoo and it seems that before https://github.com/iterative/dvc/commit/114a07e7a448b667682fbe331ac06efbf4453494

`staging` DB, under `.dvc/tmp/staging` was able to locate hash_info for the target:

https://github.com/iterative/dvc/blob/114a07e7a448b667682fbe331ac06efbf4453494/dvc/objects/stage.py#L231-L232

however, after the change, staging db became a ReferenceObjectDB, and underlying MemoryFileSystem does not seem to find staged dir, `_load_from_state` fails with `FileNotFoundError` and the tree gets build as @daavoo mentioned in [previous comment](https://github.com/iterative/dvc/issues/7390#issuecomment-1044208951) . We did not manage to find exactly why. 
That's great @pared, thanks! So hashes are being recomputed each time, right?

> I suspect this affects more places beyond `status`.

Do you know if this is true?
@pmrowla Any insight into this?
I'll take a look into this
So this occurs for `status` because computed .dir hashes are not saved to disk after the status call (the actual .dir json data only exists in the memory fs for the duration of the `dvc status` call). But this is expected behavior, and IMO the issue here is a UI problem. We display the progress bar but are not actually computing file hashes, @daavoo's initial comment was correct: https://github.com/iterative/dvc/issues/7390#issuecomment-1044149274.

---

This was an intentional decision when we moved to using objects, since the previous behavior was to save the .dir file to `.dvc/cache`, even though none of the actual file data is committed to cache at that time. With this behavior we were actually corrupting `.dvc/cache` (since having .dir files present without the committed file data breaks the "if dir hash exists in ODB assume that the entire dir contents also exists in ODB performance optimization).

The actual md5 for the modified directory (`abc123.dir`) is saved and retrieved from the state DB, but we cannot load the actual directory tree (since the json data that tells us `abc123.dir` contains files `def456, abc789, ...` is unavailable). So in this case, we rebuild the tree each time (instead of loading it from a .dir file).

However, when we rebuild the tree, we don't recalculate file MD5s more than once. Even though the .dir file is not saved to cache, computed hashes are saved in the state db. So when we rebuild the tree, each hash will be loaded from state instead of recomputed entirely:
https://github.com/iterative/dvc/blob/6c04d0efd106e1987cfc4c4252c542a013dcaf51/dvc/data/stage.py#L97-L99
https://github.com/iterative/dvc/blob/6c04d0efd106e1987cfc4c4252c542a013dcaf51/dvc/data/stage.py#L80-L86

Some performance regression is expected here since we have to walking the filesystem directory each for each `dvc status` call to rebuild the ODB tree object, but the hit is only due to walking the fs tree, it's not from recomputing MD5s for every file in that tree. Computing MD5s still only happens on the first `dvc status` call. (Note that this isnt necessarily specific to `dvc status`, it applies to any other command that checks stage/output status without committing to cache)

---

IMO the main problem in this issue is that the progress bar message for when we build the tree is misleading, since it is hard coded to just say that DVC is recomputing hashes and that this is only supposed to happen once

https://github.com/iterative/dvc/blob/6c04d0efd106e1987cfc4c4252c542a013dcaf51/dvc/data/stage.py#L126-L130

It should probably just say something like `Checking status of <dir>` and drop the `only done once` part of the message.

---

The actual ODB/dvc data fix for this is is for us to separate the performance optimization check from .dir file existence in dvc data, but this isn't really something we can do until 3.0 cache changes are made. Basically what we want is to be able to save the object containing the .dir json data to cache, and use a different file (something like abc123.dir.index) as the indicator of whether or not the rest of that directory's files have also been committed to an ODB.

One temporary workaround would be to store these loose .dir files somewhere other than `.dvc/cache` (like in `.dvc/tmp/...`) which we discussed doing back when these changes were first made, but decided not to (based on the assumption that the performance hit here is still acceptable until it is fixed properly in 3.0)
> Some performance regression is expected here since we have to walking the filesystem directory each for each dvc status call to rebuild the ODB tree object, but the hit is only due to walking the fs tree, it's not from recomputing MD5s for every file in that tree.

The problem I see is that the performance gap increases with the number of existing files tracked in directories. 

I don't know how critical the scenario is (it might somehow overlap with LDB scope) but it has a significant impact in datasets with many "small" files, for example, with 10k files:

```
# Previous commit
# @34697ef2983cd4fac7cc66aefc8577fdde8980b4
dvc status  0,80s user 0,29s system 82% cpu 1,329 total
# >=114a07e7a448b667682fbe331ac06efbf4453494
dvc status  28,47s user 6,14s system 99% cpu 34,687 total                                                                                                 
```
Where are jumping from 0.8s to 28s.

Increasing the number of threads doesn't help, probably a different issue and related to
https://github.com/iterative/dvc/discussions/7296#discussioncomment-2038086.

> The actual ODB/dvc data fix for this is is for us to separate the performance optimization check from .dir file existence in dvc data, but this isn't really something we can do until 3.0 cache changes are made. Basically what we want is to be able to save the object containing the .dir json data to cache, and use a different file (something like abc123.dir.index) as the indicator of whether or not the rest of that directory's files have also been committed to an ODB.

Apart from the UI fix, I would then add this to a checklist for 3.0 changes to ensure it gets revisited.
> Note that this isnt necessarily specific to `dvc status`, it applies to any other command that checks stage/output status without committing to cache

Besides any `--no-commit` operations, are there any other commands where this would apply?

> This was an intentional decision when we moved to using objects, since the previous behavior was to save the .dir file to `.dvc/cache`, even though none of the actual file data is committed to cache at that time. With this behavior we were actually corrupting `.dvc/cache` (since having .dir files present without the committed file data breaks the "if dir hash exists in ODB assume that the entire dir contents also exists in ODB performance optimization).

In the previous behavior, how was the `.dir` json file used to optimize getting the status of a modified directory?

> One temporary workaround would be to store these loose .dir files somewhere other than `.dvc/cache` (like in `.dvc/tmp/...`) which we discussed doing back when these changes were first made, but decided not to (based on the assumption that the performance hit here is still acceptable until it is fixed properly in 3.0)

I think we should at least reassess some of these tradeoffs/temporary regressions until we have a clear plan for when they will be "properly" fixed. @pmrowla Do you have thoughts on the level of effort and other pros/cons?
> Besides any `--no-commit` operations, are there any other commands where this would apply?

Commands like `dvc diff` that check workspace status


> In the previous behavior, how was the `.dir` json file used to optimize getting the status of a modified directory?

We know what the hash of the modified dir will be after the first `dvc status` call, since we save it to the state db (assuming that the dir has not been modified again since that `status` call).

Previously, since we saved the `.dir` file to cache we could then just `json.load()` it to get the rest of the information about "what files and hashes are inside this directory". Since we no longer actually have the `.dir` file in cache, we have to walk the filesystem instead.

> I think we should at least reassess some of these tradeoffs/temporary regressions until we have a clear plan for when they will be "properly" fixed. @pmrowla Do you have thoughts on the level of effort and other pros/cons?

The `.dvc/tmp` fix isn't too complicated, it's just a bit of a hack.
Action point for start is to change the message, since it clearly does not correspond to what is going under the hood. 
Also related benchmark issue: https://github.com/iterative/dvc-bench/issues/315.
For the record: added a reproducer in https://github.com/iterative/dvc-bench/pull/341

The issue is in workspace_status branch. Looks like we need to save raw `.dir` object (e.g. if `12345.dir` is proper typed dir object, then `12345` is raw) to local cache somewhere during staging and try to load it during staging next time if we can't find proper `12345.dir`.

Resolving patch:
diff --git a/dvc/data/stage.py b/dvc/data/stage.py
--- a/dvc/data/stage.py
+++ b/dvc/data/stage.py
@@ -1,3 +1,4 @@
+import errno
 import hashlib
 import logging
 import os
@@ -19,6 +20,9 @@
     from dvc.fs.base import AnyFSPath, FileSystem
     from dvc.objects.db import ObjectDB
 
+    from .tree import Tree
+
+
 logger = logging.getLogger(__name__)
 
 
@@ -233,7 +237,7 @@ def _make_staging_url(
     return url
 
 
-def _get_staging(odb: "ObjectDB") -> "ObjectDB":
+def _get_staging(odb: "ObjectDB") -> "ReferenceObjectDB":
     """Return an ODB that can be used for staging objects.
 
     Staging will be a reference ODB stored in the the global memfs.
@@ -247,7 +251,33 @@ def _get_staging(odb: "ObjectDB") -> "ObjectDB":
     return ReferenceObjectDB(fs, fs_path, state=state)
 
 
-def _load_from_state(odb, staging, fs_path, fs, name):
+def _load_raw_dir_obj(odb: "ObjectDB", hash_info: "HashInfo") -> "Tree":
+    from dvc.objects.errors import ObjectFormatError
+
+    from .tree import Tree
+
+    try:
+        tree = Tree.load(odb, hash_info.as_raw())
+        tree.check(odb)
+        tree.hash_info = hash_info
+    except ObjectFormatError as exc:
+        raise FileNotFoundError(
+            errno.ENOENT,
+            "No such object",
+            odb.hash_to_path(hash_info.as_raw().value),
+        ) from exc
+
+    return tree
+
+
+def _load_from_state(
+    odb: "ObjectDB",
+    staging: "ReferenceObjectDB",
+    fs_path: "AnyFSPath",
+    fs: "FileSystem",
+    name: str,
+    dry_run: bool,
+) -> Tuple["ObjectDB", "Meta", "HashFile"]:
     from dvc.objects.errors import ObjectFormatError
 
     from . import check, load
@@ -255,19 +285,56 @@ def _load_from_state(odb, staging, fs_path, fs, name):
 
     state = odb.state
     meta, hash_info = state.get(fs_path, fs)
-    if hash_info:
-        for odb_ in (odb, staging):
-            if odb_.exists(hash_info):
-                try:
-                    obj = load(odb_, hash_info)
-                    check(odb_, obj, check_hash=False)
-                    if isinstance(obj, Tree):
-                        meta.nfiles = len(obj)
-                    assert obj.hash_info.name == name
-                    return odb_, meta, obj
-                except (ObjectFormatError, FileNotFoundError):
-                    pass
-    raise FileNotFoundError
+    if not hash_info:
+        raise FileNotFoundError
+
+    for odb_ in (odb, staging):
+        if not odb_.exists(hash_info):
+            continue
+
+        try:
+            obj = load(odb, hash_info)
+            check(odb, obj, check_hash=False)
+        except (ObjectFormatError, FileNotFoundError):
+            continue
+
+        if isinstance(obj, Tree):
+            meta.nfiles = len(obj)
+        assert obj.hash_info.name == name
+        return odb_, meta, obj
+
+    if not hash_info.isdir:
+        raise FileNotFoundError
+
+    # Try loading the raw dir object saved by `stage`, see below and #7390
+    tree = _load_raw_dir_obj(odb, hash_info)
+    meta.nfiles = len(tree)
+    assert tree.hash_info.name == name
+
+    if not dry_run:
+        assert tree.fs
+        for key, _, oid in tree:
+            staging.add(
+                fs.path.join(fs_path, *key),
+                fs,
+                oid,
+                hardlink=False,
+                verify=False,
+            )
+
+        staging.add(
+            tree.fs_path,
+            tree.fs,
+            hash_info,
+            hardlink=False,
+        )
+
+        raw = staging.get(hash_info)
+        tree.fs = raw.fs
+        tree.fs_path = raw.fs_path
+
+    logger.debug("loaded tree '%s' from raw dir obj", tree)
+    return staging, meta, tree
 
 
 def _stage_external_tree_info(odb, tree, name):
@@ -318,7 +385,7 @@ def stage(
     staging = _get_staging(odb)
     if odb:
         try:
-            return _load_from_state(odb, staging, fs_path, fs, name)
+            return _load_from_state(odb, staging, fs_path, fs, name, dry_run)
         except FileNotFoundError:
             pass
 
@@ -336,6 +403,12 @@ def stage(
         logger.debug("staged tree '%s'", obj)
         if name != "md5":
             obj = _stage_external_tree_info(odb, obj, name)
+
+        # In order to avoid re-building the tree when it is not committed to
+        # the local odb (e.g. for a status call), we save it as a raw object.
+        # Loading this instead of building the tree can speed up `dvc status`
+        # for modified directories, see #7390
+        odb.add(obj.fs_path, obj.fs, obj.hash_info.as_raw())
     else:
         _, meta, obj = _stage_file(
             fs_path,
diff --git a/dvc/data/tree.py b/dvc/data/tree.py
--- a/dvc/data/tree.py
+++ b/dvc/data/tree.py
@@ -140,7 +140,7 @@ def from_list(cls, lst):
         return tree
 
     @classmethod
-    def load(cls, odb, hash_info):
+    def load(cls, odb, hash_info) -> "Tree":
         obj = odb.get(hash_info)
 
         try:
diff --git a/dvc/objects/hash_info.py b/dvc/objects/hash_info.py
--- a/dvc/objects/hash_info.py
+++ b/dvc/objects/hash_info.py
@@ -41,3 +41,9 @@ def isdir(self):
         if not self:
             return False
         return self.value.endswith(HASH_DIR_SUFFIX)
+
+    def as_raw(self) -> "HashInfo":
+        assert self.value
+        return HashInfo(
+            self.name, self.value.rsplit(HASH_DIR_SUFFIX)[0], self.obj_name
+        )


Test patch:
diff --git a/tests/func/test_commit.py b/tests/func/test_commit.py
--- a/tests/func/test_commit.py
+++ b/tests/func/test_commit.py
@@ -177,10 +177,13 @@ def test_commit_granular_dir(tmp_dir, dvc):
 
     cache = tmp_dir / ".dvc" / "cache"
 
-    assert set(cache.glob("*/*")) == set()
+    assert set(cache.glob("*/*")) == {
+        cache / "1a" / "ca2c799df82929bbdd976557975546",
+    }
 
     dvc.commit(os.path.join("data", "foo"))
     assert set(cache.glob("*/*")) == {
+        cache / "1a" / "ca2c799df82929bbdd976557975546",
         cache / "1a" / "ca2c799df82929bbdd976557975546.dir",
         cache / "ac" / "bd18db4cc2f85cedef654fccc4a4d8",
     }
@@ -188,6 +191,8 @@ def test_commit_granular_dir(tmp_dir, dvc):
 
     dvc.commit(os.path.join("data", "subdir"))
     assert set(cache.glob("*/*")) == {
+        cache / "26" / "d6b64d96a660707412f523e8184b5f",
+        cache / "1a" / "ca2c799df82929bbdd976557975546",
         cache / "1a" / "ca2c799df82929bbdd976557975546.dir",
         cache / "ac" / "bd18db4cc2f85cedef654fccc4a4d8",
         cache / "4c" / "e8d2a2cf314a52fa7f315ca37ca445",
@@ -197,6 +202,8 @@ def test_commit_granular_dir(tmp_dir, dvc):
 
     dvc.commit(os.path.join("data"))
     assert set(cache.glob("*/*")) == {
+        cache / "26" / "d6b64d96a660707412f523e8184b5f",
+        cache / "1a" / "ca2c799df82929bbdd976557975546",
         cache / "1a" / "ca2c799df82929bbdd976557975546.dir",
         cache / "ac" / "bd18db4cc2f85cedef654fccc4a4d8",
         cache / "4c" / "e8d2a2cf314a52fa7f315ca37ca445",
diff --git a/tests/func/test_external_repo.py b/tests/func/test_external_repo.py
--- a/tests/func/test_external_repo.py
+++ b/tests/func/test_external_repo.py
@@ -219,6 +219,7 @@ def test_subrepos_are_ignored(tmp_dir, erepo_dir):
             hardlink=True,
         )
         assert set(cache_dir.glob("??/*")) == {
+            cache_dir / "e1" / "d9e8eae5374860ae025ec84cfd85c7",
             cache_dir / "e1" / "d9e8eae5374860ae025ec84cfd85c7.dir",
             cache_dir / "37" / "b51d194a7513e45b56f6524f2d51f2",
             cache_dir / "94" / "7d2b84e5aa88170e80dff467a5bfb6",
diff --git a/tests/func/test_gc.py b/tests/func/test_gc.py
--- a/tests/func/test_gc.py
+++ b/tests/func/test_gc.py
@@ -20,13 +20,16 @@ def setUp(self):
         super().setUp()
 
         self.dvc.add(self.FOO)
-        self.dvc.add(self.DATA_DIR)
+        stages = self.dvc.add(self.DATA_DIR)
+        raw_dir_hash = stages[0].outs[0].hash_info.as_raw().value
+
         self.good_cache = [
             self.dvc.odb.local.hash_to_path(md5)
             for md5 in self.dvc.odb.local.all()
+            if md5 != raw_dir_hash
         ]
 
-        self.bad_cache = []
+        self.bad_cache = [self.dvc.odb.local.hash_to_path(raw_dir_hash)]
         for i in ["123", "234", "345"]:
             path = os.path.join(self.dvc.odb.local.cache_dir, i[0:2], i[2:])
             self.create(path, i)
@@ -203,7 +206,7 @@ def test_gc_no_dir_cache(tmp_dir, dvc):
     with pytest.raises(CollectCacheError):
         dvc.gc(workspace=True)
 
-    assert _count_files(dvc.odb.local.cache_dir) == 4
+    assert _count_files(dvc.odb.local.cache_dir) == 5
     dvc.gc(force=True, workspace=True)
     assert _count_files(dvc.odb.local.cache_dir) == 2
 
diff --git a/tests/func/test_odb.py b/tests/func/test_odb.py
--- a/tests/func/test_odb.py
+++ b/tests/func/test_odb.py
@@ -205,6 +205,9 @@ def test_shared_cache(tmp_dir, dvc, group):
 
     expected = {
         os.path.join(cache_dir, "17"): dir_mode,
+        os.path.join(
+            cache_dir, "17", "4eaa1dd94050255b7b98a7e1924b31"
+        ): file_mode,
         os.path.join(
             cache_dir, "17", "4eaa1dd94050255b7b98a7e1924b31.dir"
         ): file_mode,
diff --git a/tests/func/test_remote.py b/tests/func/test_remote.py
--- a/tests/func/test_remote.py
+++ b/tests/func/test_remote.py
@@ -154,6 +154,9 @@ def test_dir_hash_should_be_key_order_agnostic(tmp_dir, dvc):
         _, _, obj = stage(dvc.odb.local, path, dvc.odb.local.fs, "md5")
         hash1 = obj.hash_info
 
+    # remove the raw dir obj to force building the tree on the next stage call
+    dvc.odb.local.fs.remove(dvc.odb.local.hash_to_path(hash1.as_raw().value))
+
     tree = Tree.from_list(
         [{"md5": "1", "relpath": "1"}, {"md5": "2", "relpath": "2"}]
     )
diff --git a/tests/func/test_stage.py b/tests/func/test_stage.py
--- a/tests/func/test_stage.py
+++ b/tests/func/test_stage.py
@@ -13,6 +13,7 @@
 from dvc.utils.serialize import dump_yaml, load_yaml
 from dvc.utils.strictyaml import YAMLValidationError
 from tests.basic_env import TestDvc
+from tests.utils import clean_staging
 
 
 def test_cmd_obj():
@@ -321,3 +322,62 @@ def test_stage_run_checkpoint(tmp_dir, dvc, mocker, checkpoint):
     mock_cmd_run.assert_called_with(
         stage, checkpoint_func=callback, dry=False, run_env=None
     )
+
+
+@pytest.mark.parametrize(
+    "dry_run, expected_staging_contents",
+    [
+        (True, set()),
+        (
+            False,
+            {
+                "37b51d194a7513e45b56f6524f2d51f2",
+                "568f3dd88592a68ef99459a5491011cd",
+                "68dde2c3c4e7953c2290f176bbdc9a54",
+                "fd4034d9514d6e875538422c8b0dbeb2.dir",
+            },
+        ),
+    ],
+)
+def test_stage_dir_optimization(
+    tmp_dir, dvc, mocker, dry_run, expected_staging_contents
+):
+    from dvc.data import stage
+    from dvc.data.tree import Tree
+
+    tmp_dir.dvc_gen(
+        {
+            "data": {
+                "foo": "bar",
+                "subdir": {"subfoo": "subbar"},
+            }
+        }
+    )
+    odb = dvc.odb.local
+
+    objs = set(odb.all())
+    clean_staging()
+
+    tmp_dir.gen({"data": {"baz": "quz"}})
+
+    stage_spy = mocker.spy(stage, "_stage_tree")
+    _, _, tree = stage.stage(odb, "data", odb.fs, odb.fs.PARAM_CHECKSUM)
+
+    assert stage_spy.called
+    assert set(odb.all()) - objs == {tree.hash_info.as_raw().value}
+    stage_spy.reset_mock()
+    clean_staging()
+
+    load_spy = mocker.spy(Tree, "load")
+    build_tree_spy = mocker.spy(stage, "_build_tree")
+
+    staging, _, tree = stage.stage(
+        odb, "data", odb.fs, odb.fs.PARAM_CHECKSUM, dry_run=dry_run
+    )
+    assert not stage_spy.called
+    assert not build_tree_spy.called
+
+    load_args, _ = load_spy.call_args
+    assert load_args[1].value == tree.hash_info.as_raw().value
+
+    assert set(staging.all()) == expected_staging_contents
diff --git a/tests/unit/test_hashinfo.py b/tests/unit/test_hashinfo.py
new file mode 100644
--- /dev/null
+++ b/tests/unit/test_hashinfo.py
@@ -0,0 +1,17 @@
+from dvc.objects.hash_info import HashInfo
+
+
+def test_as_raw():
+    hash_info = HashInfo(
+        "md5", "a1d0c6e83f027327d8461063f4ac58a6.dir", "objname"
+    )
+
+    raw = hash_info.as_raw()
+
+    assert hash_info.name == "md5"
+    assert hash_info.value == "a1d0c6e83f027327d8461063f4ac58a6.dir"
+    assert hash_info.obj_name == "objname"
+
+    assert raw.name == "md5"
+    assert raw.value == "a1d0c6e83f027327d8461063f4ac58a6"
+    assert raw.obj_name == "objname"