Repository: iterative/dvc

Issue:
pull from external repo: support --j to limit the number of parallel connections
We are introducing `DVC` in our company and were quite happy until we started using it on a large project containing few hundred of thousands of files representing approximatively 300 Gb.
We use S3 as storage.
When someone from our team did a `dvc pull` of this project, it sucked the whole internet bandwidth of our office. 

We tried to mitigate the issue by limiting the number of concurrent jobs to 1 (option `-j 1`) but it was not enough.
Our IT Ops team told us that `dvc` has opened hundred of concurrent connections to download files from our `S3` bucket, and that it explains why we have been able to suck most of the bandwidth.

Is there other option than `--jobs` to limit the number of parallel connections we should take care of?
Is there some existing workaround for this situation?
pull from external repo: support --j to limit the number of parallel connections
We are introducing `DVC` in our company and were quite happy until we started using it on a large project containing few hundred of thousands of files representing approximatively 300 Gb.
We use S3 as storage.
When someone from our team did a `dvc pull` of this project, it sucked the whole internet bandwidth of our office. 

We tried to mitigate the issue by limiting the number of concurrent jobs to 1 (option `-j 1`) but it was not enough.
Our IT Ops team told us that `dvc` has opened hundred of concurrent connections to download files from our `S3` bucket, and that it explains why we have been able to suck most of the bandwidth.

Is there other option than `--jobs` to limit the number of parallel connections we should take care of?
Is there some existing workaround for this situation?


Discussion:
@pommedeterresautee I was not able to reproduce it 🤔 

could you please run `dvc version`? 

also, when you run `dvc pull -j 1` and it starts downloading, how many progress bars do you see?

```
DVC version: 0.86.2
Python version: 3.6.9
Platform: Linux-5.3.0-40-generic-x86_64-with-Ubuntu-19.10-eoan
Binary: False
Package: snap
Cache: reflink - not supported, hardlink - supported, symlink - supported
```

I see quite a lot of progress bars, too many for my terminal which crazily scroll the output:
```bash
dvc pull -j 1
```

![image](https://user-images.githubusercontent.com/1029874/75245104-5c7f9a00-57cd-11ea-8d6a-dea7ad4029c5.png)

@iterative/engineering someone who is using Linux, could you please check really quick that `j` is being propagated properly?

@pommedeterresautee could you if you have anything in you DVC config file related to the number of jobs? It's `.dvc/config` and `.dvc/config.local`?
Had a quick look; not sure if this is the issue but `repo.fetch._fetch_external()` doesn't get a `jobs` argument.
@casperdcl great catch! I just realized that it `pull` (fetches) from external repos! So, yes it looks like we definitely need to pass `j` to it down to it.

@efiop it looks like the reason for this is clear, can we prioritize and add this? 
> great catch!

Never underestimate the debugging power of debian on a phone :)
@pommedeterresautee I was not able to reproduce it 🤔 

could you please run `dvc version`? 

also, when you run `dvc pull -j 1` and it starts downloading, how many progress bars do you see?

```
DVC version: 0.86.2
Python version: 3.6.9
Platform: Linux-5.3.0-40-generic-x86_64-with-Ubuntu-19.10-eoan
Binary: False
Package: snap
Cache: reflink - not supported, hardlink - supported, symlink - supported
```

I see quite a lot of progress bars, too many for my terminal which crazily scroll the output:
```bash
dvc pull -j 1
```

![image](https://user-images.githubusercontent.com/1029874/75245104-5c7f9a00-57cd-11ea-8d6a-dea7ad4029c5.png)

@iterative/engineering someone who is using Linux, could you please check really quick that `j` is being propagated properly?

@pommedeterresautee could you if you have anything in you DVC config file related to the number of jobs? It's `.dvc/config` and `.dvc/config.local`?
Had a quick look; not sure if this is the issue but `repo.fetch._fetch_external()` doesn't get a `jobs` argument.
@casperdcl great catch! I just realized that it `pull` (fetches) from external repos! So, yes it looks like we definitely need to pass `j` to it down to it.

@efiop it looks like the reason for this is clear, can we prioritize and add this? 
> great catch!

Never underestimate the debugging power of debian on a phone :)

Resolving patch:
diff --git a/dvc/repo/fetch.py b/dvc/repo/fetch.py
--- a/dvc/repo/fetch.py
+++ b/dvc/repo/fetch.py
@@ -58,7 +58,7 @@ def _fetch(
         failed += exc.amount
 
     for (repo_url, repo_rev), files in used.external.items():
-        d, f = _fetch_external(self, repo_url, repo_rev, files)
+        d, f = _fetch_external(self, repo_url, repo_rev, files, jobs)
         downloaded += d
         failed += f
 
@@ -68,7 +68,7 @@ def _fetch(
     return downloaded
 
 
-def _fetch_external(self, repo_url, repo_rev, files):
+def _fetch_external(self, repo_url, repo_rev, files, jobs):
     from dvc.external_repo import external_repo
 
     failed = 0
@@ -91,7 +91,7 @@ def _fetch_external(self, repo_url, repo_rev, files):
                         cache.update(out.get_used_cache())
 
                 try:
-                    return repo.cloud.pull(cache), failed
+                    return repo.cloud.pull(cache, jobs=jobs), failed
                 except DownloadError as exc:
                     failed += exc.amount
     except CloneError:


Test patch: