Repository: iterative/dvc

Issue:
pull: --jobs not honored for imported data
## DVC pull from not honoring --jobs or -j

from git clone of an existing repository that includes an imported dvc resource, after cloning on windows, the dvc pull is not honoring the --jobs or -j flag.

dvc pull --jobs 2

(30 or 40 downloads.....)

Nothing in special in the repo's .dvc/config 
```bash
$ cat .dvc/config
[cache]
    type = "symlink,copy"
    shared = group
```

no .dvc/config.local for the repo

the remote resource that we are pulling from has a .dvc/config
```bash
[core]
    remote = origin
[cache]
    type = "symlink,copy"
    shared = group
['remote "origin"']
    url = s3://dvc-bucket/datasets/foo
    endpointurl = https://endpoint.com
    profile = my_profile
```

### Please provide information about your setup

**Output of `dvc version`:**

Windows: from installer

```console
1.8.1
```



Discussion:
> -j <number>, --jobs <number> - parallelism level for DVC to download data from remote storage.Using more jobs may improve the overall transfer speed.

Seems it isn't configurable in the config file, it can be a new feature, and ought to be a local config(depends on both the local machine and the remote storage) I think. 


Was able to get around this by building a local cache from data repo  (where pull -j works) - then perform pull on imported resource (where -j doesn’t work).... the 48 concurrent requests on large file - caused timeouts on the s3 resource - leading to multiple failures to pull... repeating the pull multiple times - was incrementally getting more of the data - before I gave up - and went the two step process of creating the local cache from the data repo first....

> > -j , --jobs  - parallelism level for DVC to download data from remote storage.Using more jobs may improve the overall transfer speed.
> 
> Seems it isn't configurable in the config file, it can be a new feature, and ought to be a local config(depends on both the local machine and the remote storage) I think.

@efiop 
Oh sorry, this is my own thought not about this issue.  I'll create a new ticket for it later. 

@wdixon 
`dvc pull` creates more connections than what `-j` config in your cases and that means it didn't work. Besides this, it even causes some other problems and can't get the files downloaded? Can you please give more details and help us reproduce it?
reproducing seems straight forward.....  If you have an remote  .dvc resource committed to a repo.... 

```bash
git clone http://github.com/repo_with_imported_dvc_resource.git
cd repo_with_imported_dvc_resource
dvc pull -j <limit>
```

Assuming no shared cache configured - you will see that it fetches the remote resources - but does not limit the number of concurrent downloads to what is specified by the -j flag.

This repo happened to have several hundred large files (~1GB each) - lead to timeouts and failures with so many concurrent downloads (think its defaults to 4*number of cores streams - which is really not practical).  

But, the large files are irrelevant to reproducing more concurrent downloads than specified by the -j limit.


Seems OK on Linux with HDFS remote in version 1.8.1, I'll try it on Windows later.


![BB06097C-CD6B-45E8-B78A-7391DEA4647B](https://user-images.githubusercontent.com/6745454/95703005-8adcb780-0c80-11eb-9b36-0eeab7b7971e.png)
![B70978FB-4713-4F05-ACFF-1FA2A2BD56A9](https://user-images.githubusercontent.com/6745454/95702969-77315100-0c80-11eb-8a0f-33057c4682de.png)

I'm sorry I can not reproduce it on the Windows platform with a local cache. I haven't got an s3 server now.
![image](https://user-images.githubusercontent.com/6745454/95759129-e6885e80-0cdb-11eb-9067-8b4772148515.png)
![image](https://user-images.githubusercontent.com/6745454/95759144-eab47c00-0cdb-11eb-80df-200f8f332b5a.png)
Just to be clear - the issue I face is when the dvc resource is imported from another repo. A dvc pull for resources directly within your repo does obey the jobs limit.

I setup a dvc data repo... and a dvc training repot. The training repo defined a dvc import for subset of the data repo...  after  a fresh git clone of the training repo, the dvc pull of the training repo’s dvc data resource - doesn’t honor the -j or —jobs flag 
here is what I get when I clone a repo with the remote resource.  
You will see I limit to 1 job - but it kicks off multiple downloads...
If the resource is within the repo (not import from another) - the limit works.


![pull_example](https://user-images.githubusercontent.com/1872776/95878670-7ad9dc00-0d43-11eb-953b-04119455ab84.png)

![image](https://user-images.githubusercontent.com/6745454/95883320-5dd9f300-0dad-11eb-8461-7ec5bac2a1b9.png)
@wdixon 
You are right, thanks for your report. It is a bug I think.
Seems that it is because although we pass arguments `job` to the `fetch_external` in 
https://github.com/iterative/dvc/blob/68220cbea07fd445901f7f124d900676d2d536e1/dvc/repo/fetch.py#L70
But it takes no effect.
https://github.com/iterative/dvc/blob/68220cbea07fd445901f7f124d900676d2d536e1/dvc/external_repo.py#L149-L176

Resolving patch:
diff --git a/dvc/external_repo.py b/dvc/external_repo.py
--- a/dvc/external_repo.py
+++ b/dvc/external_repo.py
@@ -130,13 +130,15 @@ def get_rev(self):
             return self.scm.get_rev()
         return self.tree.rev
 
-    def _fetch_to_cache(self, path_info, repo, callback):
+    def _fetch_to_cache(self, path_info, repo, callback, **kwargs):
         # don't support subrepo traversal as it might fail due to difference
         # in remotes
         tree = self._get_tree_for(repo)
         cache = repo.cache.local
 
-        hash_info = tree.get_hash(path_info, download_callback=callback)
+        hash_info = tree.get_hash(
+            path_info, download_callback=callback, **kwargs
+        )
         cache.save(
             path_info,
             tree,
@@ -170,7 +172,9 @@ def download_update(result):
 
             self._check_repo(path, metadata.repo)
             repo = metadata.repo
-            hash_info = self._fetch_to_cache(path, repo, download_update)
+            hash_info = self._fetch_to_cache(
+                path, repo, download_update, **kwargs
+            )
             hash_infos.append(hash_info)
 
         return sum(download_results), failed, hash_infos


Test patch:
diff --git a/tests/unit/test_external_repo.py b/tests/unit/test_external_repo.py
--- a/tests/unit/test_external_repo.py
+++ b/tests/unit/test_external_repo.py
@@ -77,3 +77,25 @@ def test_subrepo_is_constructed_properly(
             assert repo.config["remote"]["auto-generated-upstream"][
                 "url"
             ] == str(main_cache)
+
+
+def test_fetch_external_repo_jobs(tmp_dir, scm, mocker, dvc, local_remote):
+    tmp_dir.dvc_gen(
+        {
+            "dir1": {
+                "file1": "file1",
+                "file2": "file2",
+                "file3": "file3",
+                "file4": "file4",
+            },
+        },
+        commit="init",
+    )
+
+    dvc.push()
+
+    with external_repo(str(tmp_dir)) as repo:
+        spy = mocker.spy(repo.cloud, "pull")
+        repo.fetch_external(["dir1"], jobs=3)
+        run_jobs = tuple(spy.call_args_list[0])[1].get("jobs")
+        assert run_jobs == 3