Repository: iterative/dvc

Issue:
gc: remove cache files that were already pushed to remote
It would be useful to be able to clear up disk space on one's local machine by removing the cache files of specific targets.


Discussion:
Hi @shaunirwin !

The problem with applying `gc` to specific dvc files is that the same cache files might be used by other dvc files, so it is not trivial to solve in general case. Maybe you could describe your scenario so we could try to see if there are better approaches in your case?

Thanks,
Ruslan
I think I have possible use case for this request.

**TL;DR:** there should be a way to `dvc gc` cache files which are already pushed to remote DVC storage.

**Detailed use case**

I have two dirs under dvc control: `data1` and `data2`. Here is the file structure:

```
├── data1
├── data1.dvc
├── data2
└── data2.dvc
```

At first I worked with `data1`:

```
$dvc add data1
$git add .
$git commit
$dvc push
$git push
```

Since I commited and pushed `data1` it hasn't changed. Now I'm working with `data2`. The problem is that `data1` is big and I'm running out of disk space on my local machine. I want to delete `data1` both from my workspace and cache. Deleting `data1` cache is safe because it is already pushed to remote DVC storage. I can delete `data1` from workspace by `dvc remove` but I can't clear my cache because `dvc gc` considers `data1` cache as "current" and does nothing.

**P.S.:**

I'm not sure that `dvc gc` for specific targets is the best option. Probably `dvc gc` should provide an option like "clear all cache files which are already synchronized with specific remote storage"

My use case:
I'm working on multiple projects in parallel that all have large datasets tracked by dvc. They are located on a (small) SSD so that I can't have all datasets on disk simultaneously. DVC tracked files are backed up on an SSH-remote. I want to quickly be able to clear the space of files from one project so that i can dvc checkout the files from another.
In terms of implementation, it would look like this:
1) Add a new flag to https://github.com/iterative/dvc/blob/master/dvc/command/gc.py . Not sure how to call it though, maybe anyone has any ideas about it? 🙂 Seems like it should be an additional flag for `-c|--cloud` though.
2) Add support for that new flag to https://github.com/iterative/dvc/blob/master/dvc/repo/gc.py . To support it, if the flag is specified, in the last `if cloud` block, instead of `used` we should supply all local cache. E.g. POC would look something like
```
from dvc.cache import NamedCache
...
if cloud:
    if our_new_flag:
        used = [NamedCache.make("local", checksum, None) for checksum in self.cache.local.all()]
    _do_gc("remote", self.cloud.get_remote(remote, "gc -c").gc, used)
```
That last "used =" is needed to transform str checksums returned by `all()` to NamedCache, which is expected by `gc()`. Might be a nicer way to organize this, but it def gets the job done 🙂 And that is pretty much it in terms of implementation.



@efiop I'd like to take a swing at it :)

@kaiogu Please let us know if you need any help 🙂 
Hi, I was on long vacation, but am back now and will start this today.
@kaiogu Sounds good, let us know if you'll have any questions :) Btw, we have a dev-general channel in our discord, please feel free to join :slightly_smiling_face: 
@efiop, I was thinking of calling it "safe" because it would only clear the cache if there was a backup safe in a remote. I'll let you know if I get stuck :)
@kaiogu `safe` is too generic for my personal taste :slightly_smiling_face: , maybe @iterative/engineering have any thoughts/ideas about it?
`local`, `pushed`, `synced`?
Another option is to include this into push somehow:
```bash
dvc push --remove
```
Or `dvc remote gc` as it was suggested during the `gc` discussion.
I would expect `dvc remote gc` to gc on remote not in local cache.
Any updates on this feature?
@Viktor2k Not actively working on this directly yet 🙁 
Me and my company would also be interested in this! It seems this feature would be quite useful to be able to use DVC for sharing locally downloaded datasets across multiple projects on remote servers. For this, we need a way to clean up the local cache from currently no longer needed data, though the data is still required to reproduce results in the future (hence they are still referenced in `dvc.yaml` files and a backup stored in the remote storage).
@efiop this comment:

> In terms of implementation, it would look like this:
> 
> ... in the last `if cloud` block, instead of `used` we should supply all local cache. E.g. POC would look something like
> 
> ```
> from dvc.cache import NamedCache
> ...
> if cloud:
>     if our_new_flag:
>         used = [NamedCache.make("local", checksum, None) for checksum in self.cache.local.all()]
>     _do_gc("remote", self.cloud.get_remote(remote, "gc -c").gc, used)
> ```

and this one:

> Or `dvc remote gc` as it was suggested during the gc discussion.

seem to suggest that you're thinking about removing remote files, not local ones. Isn't that the opposite of the requested feature? Am I misunderstanding something?
So is there currently no supported command to clear the local cache (keeping files backed up in remote)? And as a workaround could you just delete the cache folder and run `dvc pull` to repopulate it when needed?
That all sounds right @mvkvc 
I came here looking for an equivalent of "git annex drop" for dvc. Is manually deleting the cache and the dvc-pulled file really the only option to free disk space?
It seems the primary blocker here is UI. Let's make a decision so we could move forward. How about `dvc gc --not-in-remote`?

Patch:
diff --git a/dvc/commands/gc.py b/dvc/commands/gc.py
--- a/dvc/commands/gc.py
+++ b/dvc/commands/gc.py
@@ -10,7 +10,7 @@
 
 
 class CmdGC(CmdBase):
-    def run(self):  # noqa: C901
+    def run(self):  # noqa: C901, PLR0912
         from dvc.repo.gc import _validate_args
 
         _validate_args(
@@ -22,6 +22,8 @@ def run(self):  # noqa: C901
             workspace=self.args.workspace,
             rev=self.args.rev,
             num=self.args.num,
+            cloud=self.args.cloud,
+            not_in_remote=self.args.not_in_remote,
         )
 
         if self.args.rev:
@@ -46,6 +48,10 @@ def run(self):  # noqa: C901
 
         if self.args.all_experiments:
             msg += " and all experiments"
+
+        if self.args.not_in_remote:
+            msg += " that are present in the DVC remote"
+
         if self.args.repos:
             msg += " of the current and the following repos:"
 
@@ -74,6 +80,7 @@ def run(self):  # noqa: C901
             workspace=self.args.workspace,
             rev=self.args.rev,
             num=self.args.num,
+            not_in_remote=self.args.not_in_remote,
         )
         return 0
 
@@ -157,6 +164,12 @@ def add_parser(subparsers, parent_parser):
         default=False,
         help="Keep data files for all experiments.",
     )
+    gc_parser.add_argument(
+        "--not-in-remote",
+        action="store_true",
+        default=False,
+        help="Keep data files that are not present in the remote.",
+    )
     gc_parser.add_argument(
         "-c",
         "--cloud",
diff --git a/dvc/repo/gc.py b/dvc/repo/gc.py
--- a/dvc/repo/gc.py
+++ b/dvc/repo/gc.py
@@ -1,5 +1,5 @@
 import logging
-from typing import TYPE_CHECKING, List, Optional
+from typing import TYPE_CHECKING, List, Optional, Set, Tuple
 
 from dvc.exceptions import InvalidArgumentError
 
@@ -7,11 +7,19 @@
 
 if TYPE_CHECKING:
     from dvc.repo import Repo
+    from dvc_data.hashfile.hash_info import HashInfo
+    from dvc_objects.db import ObjectDB
 
 logger = logging.getLogger(__name__)
 
 
 def _validate_args(**kwargs):
+    not_in_remote = kwargs.pop("not_in_remote", None)
+    cloud = kwargs.pop("cloud", None)
+    if not_in_remote and cloud:
+        raise InvalidArgumentError(
+            "`--not-in-remote` and `--cloud` are mutually exclusive"
+        )
     if not any(kwargs.values()):
         raise InvalidArgumentError(
             "Either of `-w|--workspace`, `-a|--all-branches`, `-T|--all-tags` "
@@ -22,8 +30,22 @@ def _validate_args(**kwargs):
         raise InvalidArgumentError("`--num` can only be used alongside `--rev`")
 
 
+def _used_obj_ids_not_in_remote(repo, default_remote, jobs, remote_odb_to_obj_ids):
+    used_obj_ids = set()
+    remote_oids = set()
+    for remote_odb, obj_ids in remote_odb_to_obj_ids:
+        if remote_odb is None:
+            remote_odb = repo.cloud.get_remote_odb(default_remote, "gc --not-in-remote")
+
+        remote_oids.update(
+            remote_odb.list_oids_exists({x.value for x in obj_ids}, jobs=jobs)
+        )
+        used_obj_ids.update(obj_ids)
+    return {obj for obj in used_obj_ids if obj.value not in remote_oids}
+
+
 @locked
-def gc(  # noqa: PLR0913
+def gc(  # noqa: PLR0912, PLR0913, C901
     self: "Repo",
     all_branches: bool = False,
     cloud: bool = False,
@@ -39,6 +61,7 @@ def gc(  # noqa: PLR0913
     commit_date: Optional[str] = None,
     rev: Optional[str] = None,
     num: Optional[int] = None,
+    not_in_remote: bool = False,
 ):
     # require `workspace` to be true to come into effect.
     # assume `workspace` to be enabled if any of `all_tags`, `all_commits`,
@@ -52,6 +75,8 @@ def gc(  # noqa: PLR0913
         commit_date=commit_date,
         rev=rev,
         num=num,
+        cloud=cloud,
+        not_in_remote=not_in_remote,
     )
 
     from contextlib import ExitStack
@@ -64,13 +89,14 @@ def gc(  # noqa: PLR0913
         repos = []
     all_repos = [Repo(path) for path in repos]
 
-    used_obj_ids = set()
+    odb_to_obj_ids: List[Tuple[Optional["ObjectDB"], Set["HashInfo"]]] = []
+
     with ExitStack() as stack:
         for repo in all_repos:
             stack.enter_context(repo.lock)
 
         for repo in [*all_repos, self]:
-            for obj_ids in repo.used_objs(
+            for odb, obj_ids in repo.used_objs(
                 all_branches=all_branches,
                 with_deps=with_deps,
                 all_tags=all_tags,
@@ -82,13 +108,22 @@ def gc(  # noqa: PLR0913
                 jobs=jobs,
                 revs=[rev] if rev else None,
                 num=num or 1,
-            ).values():
-                used_obj_ids.update(obj_ids)
+            ).items():
+                odb_to_obj_ids.append((odb, obj_ids))
+
+    if not odb_to_obj_ids:
+        odb_to_obj_ids = [(None, set())]
+
+    if not_in_remote:
+        used_obj_ids = _used_obj_ids_not_in_remote(self, remote, jobs, odb_to_obj_ids)
+    else:
+        used_obj_ids = set()
+        for _, obj_ids in odb_to_obj_ids:
+            used_obj_ids.update(obj_ids)
 
     for scheme, odb in self.cache.by_scheme():
         if not odb:
             continue
-
         removed = ogc(odb, used_obj_ids, jobs=jobs)
         if not removed:
             logger.info("No unused '%s' cache to remove.", scheme)
@@ -96,9 +131,11 @@ def gc(  # noqa: PLR0913
     if not cloud:
         return
 
-    odb = self.cloud.get_remote_odb(remote, "gc -c")
-    removed = ogc(odb, used_obj_ids, jobs=jobs)
-    if removed:
-        get_index(odb).clear()
-    else:
-        logger.info("No unused cache to remove from remote.")
+    for remote_odb, obj_ids in odb_to_obj_ids:
+        if remote_odb is None:
+            remote_odb = repo.cloud.get_remote_odb(remote, "gc -c")
+        removed = ogc(remote_odb, obj_ids, jobs=jobs)
+        if removed:
+            get_index(remote_odb).clear()
+        else:
+            logger.info("No unused cache to remove from remote.")


Test patch:
diff --git a/tests/func/test_gc.py b/tests/func/test_gc.py
--- a/tests/func/test_gc.py
+++ b/tests/func/test_gc.py
@@ -2,11 +2,12 @@
 import logging
 import os
 import shutil
+import textwrap
 
 import pytest
 
 from dvc.cli import main
-from dvc.exceptions import CollectCacheError
+from dvc.exceptions import CollectCacheError, InvalidArgumentError
 from dvc.fs import LocalFileSystem
 from dvc.utils.fs import remove
 from dvc_data.hashfile.db.local import LocalHashFileDB
@@ -334,3 +335,125 @@ def test_date(tmp_dir, scm, dvc):
     assert dvc.cache.local.exists(
         "3bcf3b1be3e794a97a5a6b93a005784c"
     )  # "modified, again"
+
+
+def test_gc_not_in_remote(tmp_dir, scm, dvc, tmp_path_factory, mocker):
+    storage = os.fspath(tmp_path_factory.mktemp("test_remote_base"))
+    dvc.config["remote"]["local_remote"] = {"url": storage}
+    dvc.config["core"]["remote"] = "local_remote"
+
+    (standalone, dir1, dir2) = tmp_dir.dvc_gen(
+        {
+            "file1": "standalone",
+            "dir1": {"file2": "file2"},
+            "dir2": {"file3": "file3", "file4": "file4"},
+        }
+    )
+    mocked_remove = mocker.spy(LocalFileSystem, "remove")
+    dvc.gc(workspace=True)
+    assert not mocked_remove.call_args_list
+
+    dvc.push(["file1", "dir1"])
+
+    dvc.gc(workspace=True, not_in_remote=True)
+
+    assert len(mocked_remove.mock_calls) == 3
+
+    arg_list = mocked_remove.call_args_list
+
+    standalone_hash = standalone.outs[0].hash_info.value
+    dir1_hash = dir1.outs[0].hash_info.value
+    assert f"{dir1_hash[2:]}.unpacked" in arg_list[0][0][1]
+    assert f"{dir1_hash[2:]}" in arg_list[1][0][1][0]
+    # We expect 2 calls: standalone_hash and dir1/file2/file2
+    assert len(arg_list[2][0][1]) == 2
+    # Order is not guaranteed here.
+    assert (
+        f"{standalone_hash[2:]}" in arg_list[2][0][1][0]
+        or f"{standalone_hash[2:]}" in arg_list[2][0][1][1]
+    )
+
+
+def test_gc_not_in_remote_remote_arg(tmp_dir, scm, dvc, tmp_path_factory, mocker):
+    storage = os.fspath(tmp_path_factory.mktemp("test_remote_base"))
+    dvc.config["remote"]["local_remote"] = {"url": storage}
+    dvc.config["core"]["remote"] = "local_remote"
+    other_storage = os.fspath(tmp_path_factory.mktemp("test_remote_other"))
+    dvc.config["remote"]["other_remote"] = {"url": other_storage}
+
+    tmp_dir.dvc_gen(
+        {
+            "file1": "standalone",
+            "dir1": {"file2": "file2"},
+            "dir2": {"file3": "file3", "file4": "file4"},
+        }
+    )
+    mocked_remove = mocker.spy(LocalFileSystem, "remove")
+
+    dvc.push(["file1", "dir1"], remote="other_remote")
+
+    dvc.gc(workspace=True, not_in_remote=True)
+
+    assert not mocked_remove.mock_calls
+
+    dvc.gc(workspace=True, not_in_remote=True, remote="other_remote")
+
+    assert len(mocked_remove.mock_calls) == 3
+
+
+def test_gc_not_in_remote_with_remote_field(
+    tmp_dir, scm, dvc, tmp_path_factory, mocker
+):
+    storage = os.fspath(tmp_path_factory.mktemp("test_remote_base"))
+    dvc.config["remote"]["local_remote"] = {"url": storage}
+    dvc.config["core"]["remote"] = "local_remote"
+
+    other_storage = os.fspath(tmp_path_factory.mktemp("test_remote_other"))
+    dvc.config["remote"]["other_remote"] = {"url": other_storage}
+
+    text = textwrap.dedent(
+        """\
+        outs:
+        - path: foo
+          remote: other_remote
+    """
+    )
+    tmp_dir.gen("foo.dvc", text)
+    tmp_dir.dvc_gen("foo", "foo")
+    dvc.push()
+
+    mocked_remove = mocker.spy(LocalFileSystem, "remove")
+    dvc.gc(workspace=True, not_in_remote=True)
+    assert len(mocked_remove.mock_calls) == 1
+
+
+def test_gc_not_in_remote_cloud(tmp_dir, scm, dvc):
+    with pytest.raises(
+        InvalidArgumentError,
+        match="`--not-in-remote` and `--cloud` are mutually exclusive",
+    ):
+        dvc.gc(workspace=True, not_in_remote=True, cloud=True)
+
+
+def test_gc_cloud_remote_field(tmp_dir, scm, dvc, tmp_path_factory, mocker):
+    storage = os.fspath(tmp_path_factory.mktemp("test_remote_base"))
+    dvc.config["remote"]["local_remote"] = {"url": storage}
+    dvc.config["core"]["remote"] = "local_remote"
+    other_storage = os.fspath(tmp_path_factory.mktemp("test_remote_other"))
+    dvc.config["remote"]["other_remote"] = {"url": other_storage}
+
+    text = textwrap.dedent(
+        """\
+        outs:
+        - path: foo
+          remote: other_remote
+    """
+    )
+    tmp_dir.gen("foo.dvc", text)
+    tmp_dir.dvc_gen("foo", "foo")
+    dvc.push()
+    tmp_dir.dvc_gen("foo", "bar")
+
+    mocked_remove = mocker.spy(LocalFileSystem, "remove")
+    dvc.gc(workspace=True, cloud=True)
+    assert len(mocked_remove.mock_calls) == 2  # local and other_remote
diff --git a/tests/unit/command/test_gc.py b/tests/unit/command/test_gc.py
--- a/tests/unit/command/test_gc.py
+++ b/tests/unit/command/test_gc.py
@@ -48,6 +48,7 @@ def test_(dvc, scm, mocker):
         repos=["project1", "project2"],
         rev=None,
         num=None,
+        not_in_remote=False,
     )
 
     cli_args = parse_args(["gc"])