Repository: iterative/dvc

Issue:
dvc: provide granularity for commands that could target specific tracked files
> This can be seen as revisiting feature request #1026

UPDATE: Please scroll down to https://github.com/iterative/dvc/issues/2458#issuecomment-566649020 for most recent, summarized requirement.
Here is the original context also (still relevant):

---

There's different scenarios in which _being able to manipulate files granularly independently of how they were committed/pushed to DVC_ could be useful. The problem with using `dvc add -R` now is that it can generate lots of `.dvc` files, but what if a directory could be added without `-R` (producing a single DVC-file) and yet other commands (lock, update, get, etc) could be applied to individual files inside the added directory tree?

**Example** (from https://github.com/iterative/dataset-registry/commit/7476a858f6200864b5755863c729bff41d0fb045)

Project 1:
```console
$ tree
.
└── tutorial
    └── nlp
        ├── Posts.xml.zip
        └── pipeline.zip
$ dvc add tutorial
...
$ dvc push
...
```

Project 2:
```console
$ dvc import {project-1-url} tutorial/nlp/pipeline.zip
...
$ tree
.
├── tutorial
│   └── nlp
│       └── pipeline.zip
└── tutorial.dvc
```
> Not sure about where the `.dvc` would have to be placed in this example though.

And also this is how Git works, I believe. Files are tracked individually (in fact it doesn't even recognize empty dirs).


Discussion:
@jorgeorpinel so, you propose storing particular files metadata inside stage file, instead of storing metadata for whole directory? 
I didn't think on the implementation details but that sounds reasonable... Unless there's thousand of files in there (which is possible): this could make it hard for Git to handle the DVC-file produced.

I would pull a "how does Git do it?" card here ♠️ 
A dir as a single entity is a desirable thing as I see it.
Well I think that is a bit dangerous ground:
 
big directory == a lot of metadata entries

which can cause problem when loading it, and also, big metadata file could be problematic to handle by git.

@jorgeorpinel I agree with @Suor on this. In most cases the way ML projects are organized the current default behavior is desirable choice. And we have -R in case someone needs to override it. 

It might make sense though to implement -R in a different way that allows for more granular control - may be repurpose the ticket for some research on this?
What kind of granularity (other than the already existing -R) are you guys talking about?
@efiop It reminds me the same discussion we had with some other users and with the team about partial checkouts, for example. Basically, there is a feeling that people are using -R in some cases when `dvc add` and a single DVC files it creates are not granular enough. But on the other hand -R is cumbersome with all these DVC-files it is creating - it's another painful issue.
OK I repurposed this issue for exploring about providing more granular control of files in added dirs. (No longer related to `-R` I think)

@efiop 

> What kind of granularity... are you guys talking about?

Partial syncs like Ivan mentioned. Please see the description of this issue for an example.
> I would pull a "how does Git do it?" card here ￼

My impression is that the analogy between GIT and DVC is not very good, because they try to do different things (code management vs. data management). So, we should not strive to imitate git on everything.
I did not say we should imitate Git on everything. What is your suggestion?
> @dashohoxha what is your suggestion?

I don't have any suggestion. I cannot even understand the problem. But if people are complaining about it, maybe there is some problem.
@shcheklein @jorgeorpinel Thanks for the explanation guys. Yeah, I remember those users that were asking about checking out or pulling only a particular number of files out of a directory. One of the big concerns at that time was that dvc was pretty slow with directories, which was significantly improved since then (still pretty long way to go though), so maybe it is acceptable now. But I can still see cases where someone would only want to pull one file out of the directory, and it seems to me that we could support that for most of the operations, we just need to do a few improvements to our logic.

For example, first step would be to support using paths instead of dvc files as arguments. E.g. `dvc pull dir` would be the same as `dvc pull dir.dvc`. After that, we will need to adjust the commands to understand subpaths, so that `dvc pull dir/file` would only download cache for `file` and checkout it.

So to summarize, in terms of the current architecture, granular operations are possible with `dvc add dir`(without -R), but we need to adjust particular dvc commands to understand granular arguments.

Related https://github.com/iterative/dvc/issues/2180 

I like the idea, and glad to hear it doesn't seem to difficult to enable this. Using existing `<stage>.dvc` files as command targets (as is done now) could also be left as an option (for backward compatibility).
@jorgeorpinel sure, we will support both. It is also useful to have `stage.dvc` support when you have more than 1 output in it.
Related: https://discordapp.com/channels/485586884165107732/485596304961962003/634088447858180108:

```
    ... 
    I sometime want to have a peak of the data on remote without download the entire dataset.
   ...
```
> From Dmitry ([on Slack](https://iterativeai.slack.com/archives/CB41NAL8H/p1576575241020000)):

There are two general scenarios:

1) - [x] (Higher priority) Make `dvc pull <file_name>` work without specifying the DVC-file (where `file_name` is tracked) – similar to `dvc get <file_name>` I guess.
2) - [x] Get files from tracked data directory e.g. `dvc pull data-dir/file.txt`. (Requires 1.)

- [x] It should work for any command but these have a higher priority: pull/push/fetch/get/status.

> Further rationale/motivation:
> There are quite a lot of requests to this problem: a guy from ScribbleData asked personally, a request from Discord a couple months ago, I asked for this recently for `dvc get` command.
> Also, this problem blocks automation scenarios like CD4ML: you cannot push only changed files programmatically to another remote since `dvc status` returns changed file names that cannot be used as an input of `dvc push` which requires DVC-files. 

Next step: support for get and import.

Resolving patch:
diff --git a/dvc/external_repo.py b/dvc/external_repo.py
--- a/dvc/external_repo.py
+++ b/dvc/external_repo.py
@@ -7,13 +7,15 @@
 
 from funcy import retry, suppress, wrap_with, cached_property
 
+from dvc.path_info import PathInfo
 from dvc.compat import fspath
 from dvc.repo import Repo
 from dvc.config import NoRemoteError, NotDvcRepoError
 from dvc.exceptions import NoRemoteInExternalRepoError
 from dvc.exceptions import OutputNotFoundError, NoOutputInExternalRepoError
 from dvc.exceptions import FileMissingError, PathMissingError
-from dvc.utils.fs import remove, fs_copy
+from dvc.utils.fs import remove, fs_copy, move
+from dvc.utils import tmp_fname
 from dvc.scm.git import Git
 
 
@@ -67,32 +69,49 @@ def __init__(self, root_dir, url):
         self._set_upstream()
 
     def pull_to(self, path, to_info):
-        try:
-            out = None
-            with suppress(OutputNotFoundError):
-                out = self.find_out_by_relpath(path)
+        """
+        Pull the corresponding file or directory specified by `path` and
+        checkout it into `to_info`.
+
+        It works with files tracked by Git and DVC, and also local files
+        outside the repository.
+        """
+        out = None
+        path_info = PathInfo(self.root_dir) / path
+
+        with suppress(OutputNotFoundError):
+            (out,) = self.find_outs_by_path(fspath(path_info), strict=False)
 
+        try:
             if out and out.use_cache:
-                self._pull_cached(out, to_info)
+                self._pull_cached(out, path, to_info)
                 return
 
-            # Git handled files can't have absolute path
+            # Check if it is handled by Git (it can't have an absolute path)
             if os.path.isabs(path):
                 raise FileNotFoundError
 
-            fs_copy(os.path.join(self.root_dir, path), fspath(to_info))
+            fs_copy(fspath(path_info), fspath(to_info))
         except FileNotFoundError:
             raise PathMissingError(path, self.url)
 
-    def _pull_cached(self, out, to_info):
+    def _pull_cached(self, out, src, dest):
         with self.state:
+            tmp = PathInfo(tmp_fname(dest))
+            target = (out.path_info.parent / src).relative_to(out.path_info)
+            src = tmp / target
+
+            out.path_info = tmp
+
             # Only pull unless all needed cache is present
-            if out.changed_cache():
-                self.cloud.pull(out.get_used_cache())
+            if out.changed_cache(filter_info=src):
+                self.cloud.pull(out.get_used_cache(filter_info=src))
+
+            failed = out.checkout(filter_info=src)
+
+            move(src, dest)
+            remove(tmp)
 
-            out.path_info = to_info
-            failed = out.checkout()
-            # This might happen when pull haven't really pulled all the files
             if failed:
                 raise FileNotFoundError
 
diff --git a/dvc/output/base.py b/dvc/output/base.py
--- a/dvc/output/base.py
+++ b/dvc/output/base.py
@@ -162,11 +162,11 @@ def exists(self):
     def changed_checksum(self):
         return self.checksum != self.remote.get_checksum(self.path_info)
 
-    def changed_cache(self):
+    def changed_cache(self, filter_info=None):
         if not self.use_cache or not self.checksum:
             return True
 
-        return self.cache.changed_cache(self.checksum)
+        return self.cache.changed_cache(self.checksum, filter_info=filter_info)
 
     def status(self):
         if self.checksum and self.use_cache and self.changed_cache():
diff --git a/dvc/utils/fs.py b/dvc/utils/fs.py
--- a/dvc/utils/fs.py
+++ b/dvc/utils/fs.py
@@ -97,8 +97,8 @@ def move(src, dst, mode=None):
     of data is happening.
     """
 
-    src = fspath_py35(src)
-    dst = fspath_py35(dst)
+    src = fspath(src)
+    dst = fspath(dst)
 
     dst = os.path.abspath(dst)
     tmp = "{}.{}".format(dst, uuid())


Test patch:
diff --git a/tests/func/test_get.py b/tests/func/test_get.py
--- a/tests/func/test_get.py
+++ b/tests/func/test_get.py
@@ -168,6 +168,33 @@ def test_get_from_non_dvc_master(tmp_dir, git_dir, caplog):
     assert (tmp_dir / "some_dst").read_text() == "some text"
 
 
+def test_get_file_from_dir(tmp_dir, erepo_dir):
+    with erepo_dir.chdir():
+        erepo_dir.dvc_gen(
+            {
+                "dir": {
+                    "1": "1",
+                    "2": "2",
+                    "subdir": {"foo": "foo", "bar": "bar"},
+                }
+            },
+            commit="create dir",
+        )
+
+    Repo.get(fspath(erepo_dir), os.path.join("dir", "1"))
+    assert (tmp_dir / "1").read_text() == "1"
+
+    Repo.get(fspath(erepo_dir), os.path.join("dir", "2"), out="file")
+    assert (tmp_dir / "file").read_text() == "2"
+
+    Repo.get(fspath(erepo_dir), os.path.join("dir", "subdir"))
+    assert (tmp_dir / "subdir" / "foo").read_text() == "foo"
+    assert (tmp_dir / "subdir" / "bar").read_text() == "bar"
+
+    Repo.get(fspath(erepo_dir), os.path.join("dir", "subdir", "foo"), out="X")
+    assert (tmp_dir / "X").read_text() == "foo"
+
+
 def test_get_url_positive(tmp_dir, erepo_dir, caplog):
     with erepo_dir.chdir():
         erepo_dir.dvc_gen("foo", "foo")
diff --git a/tests/func/test_import.py b/tests/func/test_import.py
--- a/tests/func/test_import.py
+++ b/tests/func/test_import.py
@@ -101,6 +101,42 @@ def test_import_dir(tmp_dir, scm, dvc, erepo_dir):
     }
 
 
+def test_import_file_from_dir(tmp_dir, scm, dvc, erepo_dir):
+    with erepo_dir.chdir():
+        erepo_dir.dvc_gen(
+            {
+                "dir": {
+                    "1": "1",
+                    "2": "2",
+                    "subdir": {"foo": "foo", "bar": "bar"},
+                }
+            },
+            commit="create dir",
+        )
+
+    stage = dvc.imp(fspath(erepo_dir), os.path.join("dir", "1"))
+
+    assert (tmp_dir / "1").read_text() == "1"
+    assert scm.repo.git.check_ignore("1")
+    assert stage.deps[0].def_repo == {
+        "url": fspath(erepo_dir),
+        "rev_lock": erepo_dir.scm.get_rev(),
+    }
+
+    dvc.imp(fspath(erepo_dir), os.path.join("dir", "2"), out="file")
+    assert (tmp_dir / "file").read_text() == "2"
+    assert (tmp_dir / "file.dvc").exists()
+
+    dvc.imp(fspath(erepo_dir), os.path.join("dir", "subdir"))
+    assert (tmp_dir / "subdir" / "foo").read_text() == "foo"
+    assert (tmp_dir / "subdir" / "bar").read_text() == "bar"
+    assert (tmp_dir / "subdir.dvc").exists()
+
+    dvc.imp(fspath(erepo_dir), os.path.join("dir", "subdir", "foo"), out="X")
+    assert (tmp_dir / "X").read_text() == "foo"
+    assert (tmp_dir / "X.dvc").exists()
+
+
 def test_import_non_cached(erepo_dir, tmp_dir, dvc, scm):
     src = "non_cached_output"
     dst = src + "_imported"