Repository: iterative/dvc

Issue:
`dvc gc --time <datetime>`
#669 

@dmpetrov : 

> the data time could be used for the garbage collector - very natural way to reduce size. dvc gc --time month - clean everything not from the current branch and older than a month.


Comments:
Not sure about this, as not mtime on data files nor mtime on cache files is really indicative of the usage or lack of it. Plus, I don't see a good way that we will not break a particular pipeline with older files. Need to reconsider it later. Moving to 0.9.8 milestone.
More organic way would be to use git commits as a limiter. I.e. gc --time month will save data for all pipelines up to a week.
Moved to 0.11.0. 
A user mentioned this idea on Discord today.
BTW, another "natural way to reduce size" could be **by size** e.g. `dvc gc --size-over 1G` - removes all cached files larger than a GB.
BTW, this (`--time` feature) can be done currently via Git. It may be a little complicated e.g. requiring rewriting the repo history but we could write a guide or how-to about it instead of implementing the feature. If so we could move this issue over to dvc.org.

And the `--size` feature is doable via regular GNU tools like `du ... | xargs rm ...` etc. which we could also document.
We had another user request this feature today, in the context of shared caches ([see chat](https://discord.com/channels/485586884165107732/563406153334128681/861989689891880971)).
@jorgeorpinel I would also be interested in this feature. Could you elaborate on how the `--time` feature could be achieved with git?
@lassepe sure, by deleting all the commits before that date (see [this Q](https://stackoverflow.com/questions/29042783)) and then using `dvc gc -A`.
Ah okay, this seems like a bit too aggressive then because I still need to keep the code history. I'll wait for the actual feature then.
You could do it in a temporary repo clone and then discard it (don't rewrite the upstream or your main working repo).
Any updates? this feature would be very important to me.. I currently have some models that are retrained weekly, and I was using dvc to version the models and datasets, but the bucket size was getting too big and I don't need the old models (I need keep only the last 3 or 4 models in case rollback is needed).
as currently this feature is not available it is an impediment for me to use dvc.
> Any updates? this feature would be very important to me.. I currently have some models that are retrained weekly, and I was using dvc to version the models and datasets, but the bucket size was getting too big and I don't need the old models (I need keep only the last 3 or 4 models in case rollback is needed).
as currently this feature is not available it is an impediment for me to use dvc.

@rafaelsdellama Do you need the `datetime` feature specifically or would some alternative approach like `gc -n <N_COMMITS>` (https://github.com/iterative/dvc/issues/7695) work for your use case?



Patch:
diff --git a/dvc/commands/experiments/gc.py b/dvc/commands/experiments/gc.py
--- a/dvc/commands/experiments/gc.py
+++ b/dvc/commands/experiments/gc.py
@@ -12,8 +12,8 @@
 def _raise_error_if_all_disabled(**kwargs):
     if not any(kwargs.values()):
         raise InvalidArgumentError(
-            "Either of `-w|--workspace`, `-a|--all-branches`, `-T|--all-tags` "
-            "or `--all-commits` needs to be set."
+            "Either of `-w|--workspace`, `-a|--all-branches`, `-T|--all-tags`,"
+            " `--all-commits` or `--date` needs to be set."
         )
 
 
@@ -23,6 +23,7 @@ def run(self):
             all_branches=self.args.all_branches,
             all_tags=self.args.all_tags,
             all_commits=self.args.all_commits,
+            commit_date=self.args.commit_date,
             workspace=self.args.workspace,
         )
 
@@ -31,13 +32,21 @@ def run(self):
         msg += "the workspace"
         if self.args.all_commits:
             msg += " and all git commits"
-        elif self.args.all_branches and self.args.all_tags:
-            msg += " and all git branches and tags"
-        elif self.args.all_branches:
-            msg += " and all git branches"
-        elif self.args.all_tags:
-            msg += " and all git tags"
+        else:
+            if self.args.all_branches and self.args.all_tags:
+                msg += " and all git branches and tags"
+            elif self.args.all_branches:
+                msg += " and all git branches"
+            elif self.args.all_tags:
+                msg += " and all git tags"
+
+            if self.args.commit_date:
+                msg += (
+                    " and all git commits before date "
+                    f"{self.args.commit_date}"
+                )
         msg += " of the current repo."
+
         if self.args.queued:
             msg += " Run queued experiments will be preserved."
         else:
@@ -54,6 +63,7 @@ def run(self):
             all_tags=self.args.all_tags,
             all_commits=self.args.all_commits,
             workspace=self.args.workspace,
+            commit_date=self.args.commit_date,
             queued=self.args.queued,
         )
 
@@ -110,6 +120,17 @@ def add_parser(experiments_subparsers, parent_parser):
         default=False,
         help="Keep experiments derived from all Git commits.",
     )
+    experiments_gc_parser.add_argument(
+        "--date",
+        type=str,
+        dest="commit_date",
+        metavar="<YYYY-MM-DD>",
+        default=None,
+        help=(
+            "Keep experiments from the commits after (inclusive) a certain "
+            "date. Date must match the extended ISO 8601 format (YYYY-MM-DD)."
+        ),
+    )
     experiments_gc_parser.add_argument(
         "--queued",
         action="store_true",
diff --git a/dvc/commands/gc.py b/dvc/commands/gc.py
--- a/dvc/commands/gc.py
+++ b/dvc/commands/gc.py
@@ -17,6 +17,8 @@ def run(self):
             all_branches=self.args.all_branches,
             all_tags=self.args.all_tags,
             all_commits=self.args.all_commits,
+            all_experiments=self.args.all_experiments,
+            commit_date=self.args.commit_date,
             workspace=self.args.workspace,
         )
 
@@ -25,13 +27,20 @@ def run(self):
         msg += "the workspace"
         if self.args.all_commits:
             msg += " and all git commits"
-        elif self.args.all_branches and self.args.all_tags:
-            msg += " and all git branches and tags"
-        elif self.args.all_branches:
-            msg += " and all git branches"
-        elif self.args.all_tags:
-            msg += " and all git tags"
-        elif self.args.all_experiments:
+        else:
+            if self.args.all_branches and self.args.all_tags:
+                msg += " and all git branches and tags"
+            elif self.args.all_branches:
+                msg += " and all git branches"
+            elif self.args.all_tags:
+                msg += " and all git tags"
+            if self.args.commit_date:
+                msg += (
+                    " and all git commits before date "
+                    f"{self.args.commit_date}"
+                )
+
+        if self.args.all_experiments:
             msg += " and all experiments"
 
         if self.args.repos:
@@ -53,6 +62,7 @@ def run(self):
             all_tags=self.args.all_tags,
             all_commits=self.args.all_commits,
             all_experiments=self.args.all_experiments,
+            commit_date=self.args.commit_date,
             cloud=self.args.cloud,
             remote=self.args.remote,
             force=self.args.force,
@@ -104,6 +114,18 @@ def add_parser(subparsers, parent_parser):
         default=False,
         help="Keep data files for all Git commits.",
     )
+    gc_parser.add_argument(
+        "--date",
+        type=str,
+        dest="commit_date",
+        metavar="<YYYY-MM-DD>",
+        default=None,
+        help=(
+            "Keep cached data referenced in the commits after ( inclusive )"
+            " a certain time. Date must match the extended ISO 8601 format "
+            "(YYYY-MM-DD)."
+        ),
+    )
     gc_parser.add_argument(
         "--all-experiments",
         action="store_true",
diff --git a/dvc/repo/__init__.py b/dvc/repo/__init__.py
--- a/dvc/repo/__init__.py
+++ b/dvc/repo/__init__.py
@@ -386,6 +386,7 @@ def used_objs(
         all_tags=False,
         all_commits=False,
         all_experiments=False,
+        commit_date: Optional[str] = None,
         remote=None,
         force=False,
         jobs=None,
@@ -416,6 +417,7 @@ def used_objs(
             all_tags=all_tags,
             all_commits=all_commits,
             all_experiments=all_experiments,
+            commit_date=commit_date,
         ):
             for odb, objs in self.index.used_objs(
                 targets,
diff --git a/dvc/repo/brancher.py b/dvc/repo/brancher.py
--- a/dvc/repo/brancher.py
+++ b/dvc/repo/brancher.py
@@ -1,3 +1,5 @@
+from typing import Optional
+
 from dvc.scm import iter_revs
 
 
@@ -8,6 +10,7 @@ def brancher(  # noqa: E302
     all_tags=False,
     all_commits=False,
     all_experiments=False,
+    commit_date: Optional[str] = None,
     sha_only=False,
 ):
     """Generator that iterates over specified revisions.
@@ -17,6 +20,9 @@ def brancher(  # noqa: E302
         all_branches (bool): iterate over all available branches.
         all_commits (bool): iterate over all commits.
         all_tags (bool): iterate over all available tags.
+        commit_date (str): Keep experiments from the commits after(include)
+                            a certain date. Date must match the extended
+                            ISO 8601 format (YYYY-MM-DD).
         sha_only (bool): only return git SHA for a revision.
 
     Yields:
@@ -62,6 +68,7 @@ def brancher(  # noqa: E302
         all_tags=all_tags,
         all_commits=all_commits,
         all_experiments=all_experiments,
+        commit_date=commit_date,
     )
 
     try:
@@ -72,7 +79,9 @@ def brancher(  # noqa: E302
             self.root_dir = self.fs.path.join("/", *repo_root_parts)
 
             if cwd_parts:
-                cwd = self.fs.path.join("/", *cwd_parts)
+                cwd = self.fs.path.join(  # type: ignore[unreachable]
+                    "/", *cwd_parts
+                )
                 self.fs.path.chdir(cwd)
 
             # ignore revs that don't contain repo root
diff --git a/dvc/repo/experiments/gc.py b/dvc/repo/experiments/gc.py
--- a/dvc/repo/experiments/gc.py
+++ b/dvc/repo/experiments/gc.py
@@ -15,6 +15,7 @@ def gc(
     all_tags: Optional[bool] = False,
     all_commits: Optional[bool] = False,
     workspace: Optional[bool] = False,
+    commit_date: Optional[str] = None,
     queued: Optional[bool] = False,
 ):
     keep_revs = set(
@@ -22,6 +23,7 @@ def gc(
             all_branches=all_branches,
             all_tags=all_tags,
             all_commits=all_commits,
+            commit_date=commit_date,
             sha_only=True,
         )
     )
diff --git a/dvc/repo/gc.py b/dvc/repo/gc.py
--- a/dvc/repo/gc.py
+++ b/dvc/repo/gc.py
@@ -1,4 +1,5 @@
 import logging
+from typing import Optional
 
 from dvc.exceptions import InvalidArgumentError
 
@@ -11,7 +12,7 @@ def _raise_error_if_all_disabled(**kwargs):
     if not any(kwargs.values()):
         raise InvalidArgumentError(
             "Either of `-w|--workspace`, `-a|--all-branches`, `-T|--all-tags` "
-            "`--all-experiments` or `--all-commits` needs to be set."
+            "`--all-experiments`, `--all-commits` or `--date` needs to be set."
         )
 
 
@@ -25,6 +26,7 @@ def gc(
     all_tags=False,
     all_commits=False,
     all_experiments=False,
+    commit_date: Optional[str] = None,
     force=False,
     jobs=None,
     repos=None,
@@ -40,6 +42,7 @@ def gc(
         all_commits=all_commits,
         all_branches=all_branches,
         all_experiments=all_experiments,
+        commit_date=commit_date,
     )
 
     from contextlib import ExitStack
@@ -64,6 +67,7 @@ def gc(
                 all_tags=all_tags,
                 all_commits=all_commits,
                 all_experiments=all_experiments,
+                commit_date=commit_date,
                 remote=remote,
                 force=force,
                 jobs=jobs,
diff --git a/dvc/scm.py b/dvc/scm.py
--- a/dvc/scm.py
+++ b/dvc/scm.py
@@ -176,20 +176,7 @@ def resolve_rev(scm: "Git", rev: str) -> str:
         raise RevError(str(exc))
 
 
-def iter_revs(
-    scm: "Git",
-    revs: Optional[List[str]] = None,
-    num: int = 1,
-    all_branches: bool = False,
-    all_tags: bool = False,
-    all_commits: bool = False,
-    all_experiments: bool = False,
-) -> Mapping[str, List[str]]:
-
-    if not any([revs, all_branches, all_tags, all_commits, all_experiments]):
-        return {}
-
-    revs = revs or []
+def _get_n_commits(scm: "Git", revs: List[str], num: int) -> List[str]:
     results = []
     for rev in revs:
         if num == 0:
@@ -205,6 +192,37 @@ def iter_revs(
             except RevError:
                 break
             n += 1
+    return results
+
+
+def iter_revs(
+    scm: "Git",
+    revs: Optional[List[str]] = None,
+    num: int = 1,
+    all_branches: bool = False,
+    all_tags: bool = False,
+    all_commits: bool = False,
+    all_experiments: bool = False,
+    commit_date: Optional[str] = None,
+) -> Mapping[str, List[str]]:
+    from scmrepo.exceptions import SCMError as _SCMError
+
+    from dvc.repo.experiments.utils import exp_commits
+
+    if not any(
+        [
+            revs,
+            all_branches,
+            all_tags,
+            all_commits,
+            all_experiments,
+            commit_date,
+        ]
+    ):
+        return {}
+
+    revs = revs or []
+    results: List[str] = _get_n_commits(scm, revs, num)
 
     if all_commits:
         results.extend(scm.list_all_commits())
@@ -215,9 +233,24 @@ def iter_revs(
         if all_tags:
             results.extend(scm.list_tags())
 
-    if all_experiments:
-        from dvc.repo.experiments.utils import exp_commits
+        if commit_date:
+            from datetime import datetime
 
+            commit_datestamp = datetime.strptime(
+                commit_date, "%Y-%m-%d"
+            ).timestamp()
+
+            def _time_filter(rev):
+                try:
+                    if scm.resolve_commit(rev).commit_date >= commit_datestamp:
+                        return True
+                    return False
+                except _SCMError:
+                    return True
+
+            results.extend(filter(_time_filter, scm.list_all_commits()))
+
+    if all_experiments:
         results.extend(exp_commits(scm))
 
     rev_resolver = partial(resolve_rev, scm)


Test patch:
diff --git a/tests/func/test_cli.py b/tests/func/test_cli.py
--- a/tests/func/test_cli.py
+++ b/tests/func/test_cli.py
@@ -6,7 +6,6 @@
 from dvc.commands.checkout import CmdCheckout
 from dvc.commands.config import CmdConfig
 from dvc.commands.data_sync import CmdDataPull, CmdDataPush
-from dvc.commands.gc import CmdGC
 from dvc.commands.init import CmdInit
 from dvc.commands.remove import CmdRemove
 from dvc.commands.repro import CmdRepro
@@ -144,27 +143,6 @@ def test(self):
         cmd.repo.close()
 
 
-class TestGC(TestDvc):
-    def test(self):
-        args = parse_args(["gc"])
-        cmd = args.func(args)
-        self.assertIsInstance(cmd, CmdGC)
-
-        cmd.repo.close()
-
-
-class TestGCMultipleDvcRepos(TestDvc):
-    def test(self):
-        args = parse_args(["gc", "-p", "/tmp/asdf", "/tmp/xyz"])
-
-        cmd = args.func(args)
-        self.assertIsInstance(cmd, CmdGC)
-
-        self.assertEqual(args.repos, ["/tmp/asdf", "/tmp/xyz"])
-
-        cmd.repo.close()
-
-
 class TestConfig(TestDvc):
     def test(self):
         name = "section.option"
diff --git a/tests/func/test_gc.py b/tests/func/test_gc.py
--- a/tests/func/test_gc.py
+++ b/tests/func/test_gc.py
@@ -268,23 +268,14 @@ def test_gc_without_workspace_on_tags_branches_commits(tmp_dir, dvc):
     dvc.gc(force=False, all_branches=True, all_commits=False, workspace=False)
 
 
-def test_gc_without_workspace(tmp_dir, dvc, caplog):
+@pytest.mark.parametrize("cloud", ["c", ""])
+def test_gc_without_workspace(tmp_dir, dvc, caplog, cloud):
     with caplog.at_level(logging.WARNING, logger="dvc"):
-        assert main(["gc", "-vf"]) == 255
+        assert main(["gc", f"-{cloud}vf"]) == 255
 
     assert (
         "Either of `-w|--workspace`, `-a|--all-branches`, `-T|--all-tags` "
-        "`--all-experiments` or `--all-commits` needs to be set."
-    ) in caplog.text
-
-
-def test_gc_cloud_without_any_specifier(tmp_dir, dvc, caplog):
-    with caplog.at_level(logging.WARNING, logger="dvc"):
-        assert main(["gc", "-cvf"]) == 255
-
-    assert (
-        "Either of `-w|--workspace`, `-a|--all-branches`, `-T|--all-tags` "
-        "`--all-experiments` or `--all-commits` needs to be set."
+        "`--all-experiments`, `--all-commits` or `--date` needs to be set."
     ) in caplog.text
 
 
diff --git a/tests/unit/command/test_experiments.py b/tests/unit/command/test_experiments.py
--- a/tests/unit/command/test_experiments.py
+++ b/tests/unit/command/test_experiments.py
@@ -155,6 +155,8 @@ def test_experiments_gc(dvc, scm, mocker):
             "--all-branches",
             "--all-commits",
             "--queued",
+            "--date",
+            "2022-07-04",
             "--force",
         ]
     )
@@ -171,6 +173,7 @@ def test_experiments_gc(dvc, scm, mocker):
         all_tags=True,
         all_branches=True,
         all_commits=True,
+        commit_date="2022-07-04",
         queued=True,
     )
 
diff --git a/tests/unit/command/test_gc.py b/tests/unit/command/test_gc.py
new file mode 100644
--- /dev/null
+++ b/tests/unit/command/test_gc.py
@@ -0,0 +1,54 @@
+import pytest
+
+from dvc.cli import parse_args
+from dvc.commands.gc import CmdGC
+from dvc.exceptions import InvalidArgumentError
+
+
+def test_(dvc, scm, mocker):
+    cli_args = parse_args(
+        [
+            "gc",
+            "--workspace",
+            "--all-tags",
+            "--all-branches",
+            "--all-commits",
+            "--all-experiments",
+            "--date",
+            "2022-06-30",
+            "--cloud",
+            "--remote",
+            "origin",
+            "--force",
+            "--jobs",
+            "3",
+            "--projects",
+            "project1",
+            "project2",
+        ]
+    )
+    assert cli_args.func == CmdGC
+
+    cmd = cli_args.func(cli_args)
+    m = mocker.patch("dvc.repo.Repo.gc", return_value={})
+
+    assert cmd.run() == 0
+
+    m.assert_called_once_with(
+        workspace=True,
+        all_tags=True,
+        all_branches=True,
+        all_commits=True,
+        all_experiments=True,
+        commit_date="2022-06-30",
+        cloud=True,
+        remote="origin",
+        force=True,
+        jobs=3,
+        repos=["project1", "project2"],
+    )
+
+    cli_args = parse_args(["gc"])
+    cmd = cli_args.func(cli_args)
+    with pytest.raises(InvalidArgumentError):
+        cmd.run()
diff --git a/tests/unit/scm/test_scm.py b/tests/unit/scm/test_scm.py
--- a/tests/unit/scm/test_scm.py
+++ b/tests/unit/scm/test_scm.py
@@ -1,3 +1,7 @@
+from datetime import datetime
+
+from scmrepo.exceptions import SCMError
+
 from dvc.repo.experiments import ExpRefInfo
 from dvc.scm import GitMergeError, iter_revs
 
@@ -5,6 +9,7 @@
 def test_iter_revs(
     tmp_dir,
     scm,
+    mocker,
 ):
     """
     new         other
@@ -62,6 +67,25 @@ def test_iter_revs(
         rev_root: [rev_root],
     }
 
+    def _func(rev):
+
+        if rev == rev_root:
+            return mocker.Mock(commit_date=datetime(2022, 6, 28).timestamp())
+        if rev == rev_old:
+            raise SCMError
+        return mocker.Mock(commit_date=datetime(2022, 6, 30).timestamp())
+
+    mocker.patch(
+        "scmrepo.git.Git.resolve_commit", mocker.MagicMock(side_effect=_func)
+    )
+
+    gen = iter_revs(scm, commit_date="2022-06-29")
+    assert gen == {
+        rev_new: [rev_new],
+        rev_old: [rev_old],
+        rev_other: [rev_other],
+    }
+
 
 def test_merge_error(tmp_dir, scm):
     exc = GitMergeError("Merge failed")