Repository: iterative/dvc

Issue:
cloud versioning: slow
Part of #7995

### Description

Cloud versioning is working much slower than traditional remotes

### Reproduce

```bash
$ git clone git@github.com:iterative/example-get-started.git
$ cd example-get-started
$ dvc pull
$ dvc remote add -f -d cloud s3://mybucket/remote
$ time dvc push
8 files pushed
dvc push  0.71s user 0.17s system 13% cpu 6.626 total
$ dvc remote modify cloud version_aware true
$ dvc remote modify cloud worktree true
$ time dvc push
Updating lock file 'dvc.lock'
Updating lock file 'dvc.lock'
9 files pushed
dvc push  0.98s user 0.21s system 6% cpu 17.443 total
```


Discussion:
In the time it takes to complete a regular `dvc push` or `aws s3 sync` of a large directory, `dvc push` to a cloud-versioned remote hangs and shows no output (even with `-v`) and makes no progress that I can tell. Tested with the contents of s3://dave-sandbox-versioning/coco/test2014.
Seconded, it would be nice to at least see some logging output with `-v` when pushing while `worktree = true`
Another concern here is whether the current `.dvc` yaml structure is too slow when it contains large directories (tens or hundreds of thousands of files). 
Tested with 2800 images taken from https://github.com/iterative/dataset-registry/blob/master/use-cases/cats-dogs.dvc. 

Results with cloud versioning:

```
$ time dvc push
2800 files pushed
dvc push  40.71s user 6.08s system 12% cpu 6:13.09 total
```

```
$ time dvc pull
A       cats-dogs/
1 file added and 2800 files fetched
dvc pull  24.05s user 6.45s system 15% cpu 3:17.54 total
```

Results without cloud versioning:

```
$ time dvc push 
2801 files pushed
dvc push -r docs/dvc use-cases/cats-dogs.dvc  24.75s user 3.50s system 37% cpu 1:15.45 total
```

```
$ time dvc pull
A       use-cases/cats-dogs/
1 file added and 2600 files fetched
dvc pull -r docs/dvc use-cases/cats-dogs.dvc  6.82s user 2.49s system 45% cpu 20.355 total
```
Results for a slightly larger dataset.

With cloud versioning:

```
$ time dvc push
6946 files pushed
dvc push  97.23s user 16.91s system 8% cpu 23:39.43 total
```

```
$ time dvc pull
M       test2014/
1 file modified and 6946 files fetched
dvc pull  66.84s user 21.57s system 11% cpu 13:00.08 total
```

Without cloud versioning:

```
$ time dvc push
6947 files pushed
dvc push  50.29s user 8.22s system 21% cpu 4:35.87 total
```

```
$ time dvc pull
M       test2014/
1 file modified and 6946 files fetched
dvc pull  22.32s user 8.36s system 33% cpu 1:30.41 total
```

I can follow up with profiling for each.
Some profiling done with the data from s3://dave-sandbox-versioning/registry-cloud-versioned/cats-dogs/.

For regular cache, here's the zip of the yappi cachegrind output per thread:

[cache.zip](https://github.com/iterative/dvc/files/10290671/cache.zip)

Here's the same for `version-aware`:

[version.zip](https://github.com/iterative/dvc/files/10290677/version.zip)

Edit: And here's `worktree`:

[worktree.zip](https://github.com/iterative/dvc/files/10290736/worktree.zip)

Haven't looked through the output at all, but I notice that there are 45 threads/outputs from the regular cache but only 5 from the cloud-versioned ones. Is that expected?

After https://github.com/iterative/dvc-data/pull/246, the only blocker should be pushing incremental changes to a `version_aware` remote (not an issue with `worktree`). For example, pushing when nothing has changed:

```
$ time dvc push -r cache
Everything is up to date.
dvc push -r cache  1.74s user 0.16s system 83% cpu 2.290 total
$ time dvc push -r worktree
Everything is up to date.
dvc push -r worktree  1.95s user 0.14s system 75% cpu 2.788 total
$ time dvc push -r versioned
Everything is up to date.
dvc push -r versioned  5.23s user 0.38s system 50% cpu 11.095 total
```

The `version_aware` remote hangs with a status bar stuck at 0% for most of the runtime. I'm guessing DVC is looking up each file by version and this takes a long time. Is it needed? I think we discussed that it might not be necessary to validate that the data exists on cloud if there is a `version_id` in the .dvc file. That might be a strong assumption to make, but it feels painfully slow compared to other remote types to check the status.


Resolving patch:
diff --git a/dvc/repo/fetch.py b/dvc/repo/fetch.py
--- a/dvc/repo/fetch.py
+++ b/dvc/repo/fetch.py
@@ -78,6 +78,7 @@ def fetch(
                 all_tags=all_tags,
                 all_commits=all_commits,
                 targets=targets,
+                jobs=jobs,
                 with_deps=with_deps,
                 recursive=recursive,
             )
@@ -181,6 +182,7 @@ def _fetch_worktree(
     all_tags: bool = False,
     all_commits: bool = False,
     targets: Optional["TargetType"] = None,
+    jobs: Optional[int] = None,
     **kwargs,
 ) -> int:
     from dvc.repo.worktree import fetch_worktree
@@ -192,5 +194,7 @@ def _fetch_worktree(
         all_tags=all_tags,
         all_commits=all_commits,
     ):
-        downloaded += fetch_worktree(repo, remote, targets=targets, **kwargs)
+        downloaded += fetch_worktree(
+            repo, remote, targets=targets, jobs=jobs, **kwargs
+        )
     return downloaded
diff --git a/dvc/repo/push.py b/dvc/repo/push.py
--- a/dvc/repo/push.py
+++ b/dvc/repo/push.py
@@ -55,6 +55,7 @@ def push(
             all_tags=all_tags,
             all_commits=all_commits,
             targets=expanded_targets,
+            jobs=jobs,
             with_deps=with_deps,
             recursive=recursive,
         )
@@ -105,6 +106,7 @@ def _push_worktree(
     all_tags: bool = False,
     all_commits: bool = False,
     targets: Optional["TargetType"] = None,
+    jobs: Optional[int] = None,
     **kwargs,
 ) -> int:
     from dvc.repo.worktree import push_worktree
@@ -114,4 +116,4 @@ def _push_worktree(
             "Multiple rev push is unsupported for cloud versioned remotes"
         )
 
-    return push_worktree(repo, remote, targets=targets, **kwargs)
+    return push_worktree(repo, remote, targets=targets, jobs=jobs, **kwargs)
diff --git a/dvc/repo/worktree.py b/dvc/repo/worktree.py
--- a/dvc/repo/worktree.py
+++ b/dvc/repo/worktree.py
@@ -32,7 +32,6 @@ def worktree_view(
     index: "Index",
     targets: Optional["TargetType"] = None,
     push: bool = False,
-    latest_only: bool = True,
     **kwargs,
 ) -> "IndexView":
     """Return view of data that can be stored in worktree remotes.
@@ -56,10 +55,6 @@ def outs_filter(out: "Output") -> bool:
             or (push and not out.can_push)
         ):
             return False
-        # If we are not enforcing push to latest version and have a version
-        # for this out, we assume it still exists and can skip pushing it
-        if push and not latest_only and out.meta.version_id is not None:
-            return False
         return True
 
     return index.targets_view(
@@ -74,6 +69,7 @@ def fetch_worktree(
     repo: "Repo",
     remote: "Remote",
     targets: Optional["TargetType"] = None,
+    jobs: Optional[int] = None,
     **kwargs,
 ) -> int:
     from dvc_data.index import save
@@ -91,13 +87,14 @@ def fetch_worktree(
         unit="file", desc="Fetch", disable=total == 0
     ) as cb:
         cb.set_size(total)
-        return save(index, callback=cb)
+        return save(index, callback=cb, jobs=jobs)
 
 
 def push_worktree(
     repo: "Repo",
     remote: "Remote",
     targets: Optional["TargetType"] = None,
+    jobs: Optional[int] = None,
     **kwargs,
 ) -> int:
     from dvc_data.index import checkout
@@ -106,7 +103,6 @@ def push_worktree(
         repo.index,
         push=True,
         targets=targets,
-        latest_only=remote.worktree,
         **kwargs,
     )
     new_index = view.data["repo"]
@@ -139,6 +135,7 @@ def push_worktree(
             delete=remote.worktree,
             callback=cb,
             latest_only=remote.worktree,
+            jobs=jobs,
             **diff_kwargs,
         )
     if pushed:


Test patch: