Repository: iterative/dvc

Issue:
get: load performance is significantly (many times) lower than for pull (reproduced on dvc-bench dataset)
# Bug Report
`get` command for external repo works many times slower than clone + `pull` command for the same repo (checked on local remote). The same issue happens for the `import` command. 
For reproducible toy dataset (1Gb, 10^4 files) there is a 5x difference, measured by cProfile (archive attached). 
For my real data (which actually raised the issue in the first place) with 20Gb and 150k files the downgrade is huge: `pull` takes 30min, while `get` estimation was 350+hours!

## Reproduce
I have created a test [setup](https://github.com/zimka/example-get-started) to create toy dataset with random binary files and measure the difference on them, but I think it can be reproduced from scratch on any data. 
1. Create dataset, it should contain enough data and files (1Gb and 10^4 file in my case)
2. Set `local` remote, `push` data there, commit and push your repo on github
3. Clone your repo, use `pull` to load data (measure)
4. Use `get` to load data (measure)

I measured both ways with cProfile and `get` is 5x slower than `pull`.
[cProfiles2.zip](https://github.com/iterative/dvc/files/6489530/cProfiles2.zip)

The problem is, that it is probably not a constant ratio, but depends on the amount of files or data. For my real data with 160k files and 20Gb data, the `get` after 8 hours of exectuion(which by its own 10x longer) the ETA was 350+ hours, while `pull` took only 30min.
In my view, even if it is expected for `get` to work slow, such decrease is too bad to be bearable.

I have also noticed that CPU is used intensively during the `pull` and is not during the `get` execution. I tried to increase number of jobs for `get` in several times, but it has not affected the ETA anyhow.

### Environment information
**Output of `dvc doctor`:**

```console
$ dvc doctor
DVC version: 2.1.0 (pip)
---------------------------------
Platform: Python 3.7.6 on Windows-10-10.0.19041-SP0
Supports: http, https
Cache types: hardlink
Cache directory: NTFS on C:\
Caches: local
Remotes: local
Workspace directory: NTFS on C:\
Repo: dvc, git
```


Discussion:
Most likely this is happening because of https://github.com/iterative/dvc/issues/5546. `cProfile` log is not being very helpful here to point out the cause.
Should get be included in dvc-bench?
@dberenbaum it looks like it is [included](https://github.com/iterative/dvc-bench/blob/master/benchmarks/imports.py) (afaik `import` and `get` use similar code) and I have found some [graphs](https://iterative.github.io/dvc-bench/#imports.ImportBench.time_imports).
However, on my PC execution time is drastically different from bench graphs:
1. git clone + dvc pull data/cats_dogs.dvc: 5-10 mins (33 mins on the bench graph)
2. dvc get https://github.com/iterative/dvc-bench data/cats_dogs.dvc: more 60 (5 mins on the bench graph)
I'll check later if results are the same on Linux. 
I have got the same results on Linux and dvc 2.0.17 with https://github.com/iterative/dvc-bench data/cats_dogs.dvc - which means that the problem is not specific to Windows or local storage only, and can be reproduced on already existent dataset in dvc-bench.
Also in my case `dvc get` used 4 cpus according to htop and the average load was similar to `dvc pull`, but File/s was much lower.
Not sure why my results are different from dvc-bench graphs.
For the record: pre-requisites: https://github.com/iterative/dvc/pull/6109 and the following `save()/transfer()` unification. We'll double check this issue after those changes are merged, will likely be fixed automatically though. ETA for those so far is Jul ~1st,
As noted in #6109, `dvc import` performance is roughly equivalent to `dvc pull` in master now, but `dvc get` is still very slow. The issue with `get` performance is because we directly call `repo_fs.download(path)` which will just `fs.open()` and stream/copy all the files in a directory (whether the file ends up coming from git or from a DVC out).

`repo_fs.download` will need to be updated to do use the same object collection and save/transfer optimizations as import so that we don't do the individual stream/copy for everything inside DVC outs

Resolving patch:
diff --git a/dvc/fs/dvc.py b/dvc/fs/dvc.py
--- a/dvc/fs/dvc.py
+++ b/dvc/fs/dvc.py
@@ -62,9 +62,7 @@ def _get_granular_hash(
             return obj.hash_info
         raise FileNotFoundError
 
-    def open(  # type: ignore
-        self, path: PathInfo, mode="r", encoding=None, remote=None, **kwargs
-    ):  # pylint: disable=arguments-differ
+    def _get_fs_path(self, path: PathInfo, remote=None):
         try:
             outs = self._find_outs(path, strict=False)
         except OutputNotFoundError as exc:
@@ -92,16 +90,20 @@ def open(  # type: ignore
             else:
                 checksum = out.hash_info.value
             remote_info = remote_odb.hash_to_path_info(checksum)
-            return remote_odb.fs.open(
-                remote_info, mode=mode, encoding=encoding
-            )
+            return remote_odb.fs, remote_info
 
         if out.is_dir_checksum:
             checksum = self._get_granular_hash(path, out).value
             cache_path = out.odb.hash_to_path_info(checksum).url
         else:
             cache_path = out.cache_path
-        return open(cache_path, mode=mode, encoding=encoding)
+        return out.odb.fs, cache_path
+
+    def open(  # type: ignore
+        self, path: PathInfo, mode="r", encoding=None, **kwargs
+    ):  # pylint: disable=arguments-renamed
+        fs, fspath = self._get_fs_path(path, **kwargs)
+        return fs.open(fspath, mode=mode, encoding=encoding)
 
     def exists(self, path):  # pylint: disable=arguments-renamed
         try:
@@ -253,3 +255,9 @@ def info(self, path_info):
                 ret[obj.hash_info.name] = obj.hash_info.value
 
         return ret
+
+    def _download(self, from_info, to_file, **kwargs):
+        fs, path = self._get_fs_path(from_info)
+        fs._download(  # pylint: disable=protected-access
+            path, to_file, **kwargs
+        )
diff --git a/dvc/fs/git.py b/dvc/fs/git.py
--- a/dvc/fs/git.py
+++ b/dvc/fs/git.py
@@ -126,3 +126,17 @@ def walk_files(self, path_info, **kwargs):
             for file in files:
                 # NOTE: os.path.join is ~5.5 times slower
                 yield f"{root}{os.sep}{file}"
+
+    def _download(
+        self, from_info, to_file, name=None, no_progress_bar=False, **kwargs
+    ):
+        import shutil
+
+        from dvc.progress import Tqdm
+
+        with open(to_file, "wb+") as to_fobj:
+            with Tqdm.wrapattr(
+                to_fobj, "write", desc=name, disable=no_progress_bar
+            ) as wrapped:
+                with self.open(from_info, "rb", **kwargs) as from_fobj:
+                    shutil.copyfileobj(from_fobj, wrapped)
diff --git a/dvc/fs/repo.py b/dvc/fs/repo.py
--- a/dvc/fs/repo.py
+++ b/dvc/fs/repo.py
@@ -453,19 +453,20 @@ def walk_files(self, path_info, **kwargs):
             for fname in files:
                 yield PathInfo(root) / fname
 
-    def _download(
-        self, from_info, to_file, name=None, no_progress_bar=False, **kwargs
-    ):
-        import shutil
-
-        from dvc.progress import Tqdm
+    def _download(self, from_info, to_file, **kwargs):
+        fs, dvc_fs = self._get_fs_pair(from_info)
+        try:
+            fs._download(  # pylint: disable=protected-access
+                from_info, to_file, **kwargs
+            )
+            return
+        except FileNotFoundError:
+            if not dvc_fs:
+                raise
 
-        with open(to_file, "wb+") as to_fobj:
-            with Tqdm.wrapattr(
-                to_fobj, "write", desc=name, disable=no_progress_bar
-            ) as wrapped:
-                with self.open(from_info, "rb", **kwargs) as from_fobj:
-                    shutil.copyfileobj(from_fobj, wrapped)
+        dvc_fs._download(  # pylint: disable=protected-access
+            from_info, to_file, **kwargs
+        )
 
     def metadata(self, path):
         abspath = os.path.abspath(path)


Test patch: