Repository: iterative/dvc

Issue:
pull: going completely crazy on large dataset
**Version:**

```
DVC version: 0.73.0
Python version: 3.6.8
Platform: Linux-4.15.0-1056-aws-x86_64-with-Ubuntu-18.04-bionic
Binary: False
Package: None
Filesystem type (cache directory): ('xfs', '/dev/nvme1n1')
Filesystem type (workspace): ('xfs', '/dev/nvme1n1')
```

**Reproduce:**

```
git clone git@github.com:iterative/dataset-registry-private.git
cd dataset-registry-private
dvc pull -j 100 ILSVRC.dvc
```

**Output:**

(sorry for the video in this format, was the fastest way to capture it while it was running)

https://www.dropbox.com/s/m821sgc1flk770o/dvc-pull-going-crazy.mov?dl=0

[WIP] pull: display a lot of jobs
## Disclaimer
I would like to change the implementation completely but shared for collaboration

## How to try
There is [a script](https://gist.github.com/JIoJIaJIu/05f3b72982373771da88f313bb5cad9f) for simulating behavior
[Before](https://asciinema.org/a/0JYsWtiSuOkQ1xVGkmNpVBKng)
[After](https://asciinema.org/a/j36cX6XYPTLBbduKs9KZJBUOY)


## TLDR
I tried an approach with limiting amount of visible pbars but would like to change the solution to the single bar that contains information about all files.
The idea was show only **N** pbars, and when one is finished - add new from the background. It allows to avoid flickering, but it has problems:
1. Pbars hierarchy (main/nested). At this moment main pbar is defined as pbar with [0 position](https://github.com/iterative/dvc/blob/f05e6e0529ae147e82e3c392951f03c01b6b0c64/dvc/progress.py#L96), but the order is not guaranteed. Can be addressed by more smart main bar definition, relates https://github.com/iterative/dvc/issues/3452
2. When files are small - there is flickering
3. the solution interacts with a lot of internal `Tqdm` stuff 

Close #2905

<hr/>

* [ ] ❗ Have you followed the guidelines in the [Contributing to DVC](https://dvc.org/doc/user-guide/contributing/core) list?

* [ ] 📖 Check this box if this PR **does not** require [documentation](https://dvc.org/doc) updates, or if it does **and** you have created a separate PR in [dvc.org](https://github.com/iterative/dvc.org) with such updates (or at least opened an issue about it in that repo). Please link below to your PR (or issue) in the [dvc.org](https://github.com/iterative/dvc.org) repo.

* [ ] ❌ Have you checked DeepSource, CodeClimate, and other sanity checks below? We consider their findings recommendatory and don't expect everything to be addressed. Please review them carefully and fix those that actually improve code or fix bugs.

Thank you for the contribution - we'll try to review it as soon as possible. 🙏

pull: going completely crazy on large dataset
**Version:**

```
DVC version: 0.73.0
Python version: 3.6.8
Platform: Linux-4.15.0-1056-aws-x86_64-with-Ubuntu-18.04-bionic
Binary: False
Package: None
Filesystem type (cache directory): ('xfs', '/dev/nvme1n1')
Filesystem type (workspace): ('xfs', '/dev/nvme1n1')
```

**Reproduce:**

```
git clone git@github.com:iterative/dataset-registry-private.git
cd dataset-registry-private
dvc pull -j 100 ILSVRC.dvc
```

**Output:**

(sorry for the video in this format, was the fastest way to capture it while it was running)

https://www.dropbox.com/s/m821sgc1flk770o/dvc-pull-going-crazy.mov?dl=0

[WIP] pull: display a lot of jobs
## Disclaimer
I would like to change the implementation completely but shared for collaboration

## How to try
There is [a script](https://gist.github.com/JIoJIaJIu/05f3b72982373771da88f313bb5cad9f) for simulating behavior
[Before](https://asciinema.org/a/0JYsWtiSuOkQ1xVGkmNpVBKng)
[After](https://asciinema.org/a/j36cX6XYPTLBbduKs9KZJBUOY)


## TLDR
I tried an approach with limiting amount of visible pbars but would like to change the solution to the single bar that contains information about all files.
The idea was show only **N** pbars, and when one is finished - add new from the background. It allows to avoid flickering, but it has problems:
1. Pbars hierarchy (main/nested). At this moment main pbar is defined as pbar with [0 position](https://github.com/iterative/dvc/blob/f05e6e0529ae147e82e3c392951f03c01b6b0c64/dvc/progress.py#L96), but the order is not guaranteed. Can be addressed by more smart main bar definition, relates https://github.com/iterative/dvc/issues/3452
2. When files are small - there is flickering
3. the solution interacts with a lot of internal `Tqdm` stuff 

Close #2905

<hr/>

* [ ] ❗ Have you followed the guidelines in the [Contributing to DVC](https://dvc.org/doc/user-guide/contributing/core) list?

* [ ] 📖 Check this box if this PR **does not** require [documentation](https://dvc.org/doc) updates, or if it does **and** you have created a separate PR in [dvc.org](https://github.com/iterative/dvc.org) with such updates (or at least opened an issue about it in that repo). Please link below to your PR (or issue) in the [dvc.org](https://github.com/iterative/dvc.org) repo.

* [ ] ❌ Have you checked DeepSource, CodeClimate, and other sanity checks below? We consider their findings recommendatory and don't expect everything to be addressed. Please review them carefully and fix those that actually improve code or fix bugs.

Thank you for the contribution - we'll try to review it as soon as possible. 🙏



Discussion:
Should be fixed by combining pbars. @casperdcl 
`-j 100` would be the issue (nested bars scrolling off the screen). I should be able to come up with a temp patch for now. P.S. what happened with the old method? No scrolling but lots of flickering and very slow?
@casperdcl what's the point? Let's go for the actual fix, i.e. combining into a single pbar.
The point would be I can downgrade a `p1` to `p2`/`p3` today, but don't think I'll have time for a full fix soon.
@casperdcl p1 is not p0 (fix asap), p1 can wait. TBH, I see combining threads as top priority in pbars (maybe even UI as a whole) now since it's the highest value, so no point doing something else instead/in the meantime unless it makes dvc unusable.
Totally agree with @Suor .
Btw `dvc push` (without any explicit multithread args) also has this issue.
The problem appears when amount of the bars is more then rows in the terminal.
We use [threads](https://github.com/iterative/dvc/blob/e69b44d1ed15eb2ff1b01adc088d20c5d51fff20/dvc/remote/base.py#L607) where [Tqdm instance](https://github.com/iterative/dvc/blob/e69b44d1ed15eb2ff1b01adc088d20c5d51fff20/dvc/remote/base.py#L607) is created. So when we have amount of jobs (threads) are more then rows in the terminal [Tqdm draws](https://github.com/tqdm/tqdm/blob/80be780ab988c20048ade293ef84d2d8fd9e922a/tqdm/std.py#L1438) bars on top each other

* I saw the solution that pointed in the PR https://github.com/iterative/dvc/pull/3453, with hidden bars:  
limit visible bars to some number, and replace finished one with some from background. Unfortunately solution is not ideal, and interacts with Tqdm internals, I think if needs to apply something like this then better to do it on the library side.
* Another way is to return from [self.display](https://github.com/tqdm/tqdm/blob/80be780ab988c20048ade293ef84d2d8fd9e922a/tqdm/std.py#L1421) when `self.pos >= ROWS`, but there are also problems with internals

Hi @casperdcl, I know that you are an author of the Tqdm library (awesome tool :+1:  :1st_place_medal: ), could you take a look on my [PR](https://github.com/iterative/dvc/pull/3453) and share your thoughts please?

Another approach is to limit bars to only to one and show accumulated progress. It requires to rework few Tqdm calls in `dvc pull`, `dvc push`, `dvc add`([on adding >1Gb files](https://github.com/iterative/dvc/blob/e69b44d1ed15eb2ff1b01adc088d20c5d51fff20/dvc/utils/__init__.py#L44)), provide wrapper that can retrieve updates through **Queue** and draws single bar. I like that approach but it will affect a lot of places of the system
I should ask - are there any opetations where a user would ever benefit from having more than about 5 threads? Surely I/O is a bottleneck for anything more? Maybe we can just set an upper limit to number of threads?
It's definitely a good first step. The main intention is too show that something is happening and probably it's fine to limit the number of `pbars` if we can somehow show the total progress as well and make sure that it's clear from the UI that number of pbars is artificially limited.

> I tried an approach with limiting amount of visible pbars but would like to change the solution to the single bar that contains information about all files.

how would it look like? do we show names, or only number of files?

can we take a look at how other tools like `lftp`, `rclone` show this? just to see if we like some ideas?
@skshetry I have plans to change implementation, but would like to get your opinion about the current one. Maybe things change I will keep/improve this one
> I see what you're trying to do here, but it's not really a full fix (it's just the temporary patch I was thinking of in [#2905 (comment)](https://github.com/iterative/dvc/issues/2905#issuecomment-562516143)).
> 
> There's a more meta comment which maybe needs to be addressed first: [#2905 (comment)](https://github.com/iterative/dvc/issues/2905#issuecomment-599503500)

Hi @casperdcl , you mentioned that it's not a full fix, but how do you see it should work at the latest stage?
I see that `MAX_LINES` should be calculated dynamically based on terminal rows, what else?   
Maybe there should be completely different approach you see

>  from having more than about 5 threads?

As I got for network operations user gets benefit with multiple open connections, it would be great if you can share some info about it @efiop. Another point that it's just an interface - we provide option to point `jobs` parameter, that can be custom and is not limited to cpu count
@JIoJIaJIu there are a few small bugs I can see with this PR but it's not really worth getting into if we can avoid it altogether by say setting `jobs = min(10, args.jobs)`.

Apart from that (in a general use case/if we decide there should be no upper bound on threads) there probably should be a PR on the upstream https://github.com/tqdm/tqdm repo...

Anyway for DVC purposes, the best way to go about this is to collect info about all tasks first (e.g. total number of bytes) and have them all report back to just one total bar if there are more than say 10 tasks.

@shcheklein / @efiop lemme know if actually you would prefer some per-task info printed at least after each task completes. If you do, this may be fairly easy to solve using something similar to this PR.
@casperdcl there are definitely cases where 10 or even 20 jobs is not enough. Since some workloads are IO bound it might be beneficial to have 100 jobs, so I doubt that it's a good solution to limit to some upper bound.

Could we check some other tools that do parallel processing? sync tools, package managers, etc .. to see some ideas and how people solve this problem.
Should be fixed by combining pbars. @casperdcl 
`-j 100` would be the issue (nested bars scrolling off the screen). I should be able to come up with a temp patch for now. P.S. what happened with the old method? No scrolling but lots of flickering and very slow?
@casperdcl what's the point? Let's go for the actual fix, i.e. combining into a single pbar.
The point would be I can downgrade a `p1` to `p2`/`p3` today, but don't think I'll have time for a full fix soon.
@casperdcl p1 is not p0 (fix asap), p1 can wait. TBH, I see combining threads as top priority in pbars (maybe even UI as a whole) now since it's the highest value, so no point doing something else instead/in the meantime unless it makes dvc unusable.
Totally agree with @Suor .
Btw `dvc push` (without any explicit multithread args) also has this issue.
The problem appears when amount of the bars is more then rows in the terminal.
We use [threads](https://github.com/iterative/dvc/blob/e69b44d1ed15eb2ff1b01adc088d20c5d51fff20/dvc/remote/base.py#L607) where [Tqdm instance](https://github.com/iterative/dvc/blob/e69b44d1ed15eb2ff1b01adc088d20c5d51fff20/dvc/remote/base.py#L607) is created. So when we have amount of jobs (threads) are more then rows in the terminal [Tqdm draws](https://github.com/tqdm/tqdm/blob/80be780ab988c20048ade293ef84d2d8fd9e922a/tqdm/std.py#L1438) bars on top each other

* I saw the solution that pointed in the PR https://github.com/iterative/dvc/pull/3453, with hidden bars:  
limit visible bars to some number, and replace finished one with some from background. Unfortunately solution is not ideal, and interacts with Tqdm internals, I think if needs to apply something like this then better to do it on the library side.
* Another way is to return from [self.display](https://github.com/tqdm/tqdm/blob/80be780ab988c20048ade293ef84d2d8fd9e922a/tqdm/std.py#L1421) when `self.pos >= ROWS`, but there are also problems with internals

Hi @casperdcl, I know that you are an author of the Tqdm library (awesome tool :+1:  :1st_place_medal: ), could you take a look on my [PR](https://github.com/iterative/dvc/pull/3453) and share your thoughts please?

Another approach is to limit bars to only to one and show accumulated progress. It requires to rework few Tqdm calls in `dvc pull`, `dvc push`, `dvc add`([on adding >1Gb files](https://github.com/iterative/dvc/blob/e69b44d1ed15eb2ff1b01adc088d20c5d51fff20/dvc/utils/__init__.py#L44)), provide wrapper that can retrieve updates through **Queue** and draws single bar. I like that approach but it will affect a lot of places of the system
I should ask - are there any opetations where a user would ever benefit from having more than about 5 threads? Surely I/O is a bottleneck for anything more? Maybe we can just set an upper limit to number of threads?
It's definitely a good first step. The main intention is too show that something is happening and probably it's fine to limit the number of `pbars` if we can somehow show the total progress as well and make sure that it's clear from the UI that number of pbars is artificially limited.

> I tried an approach with limiting amount of visible pbars but would like to change the solution to the single bar that contains information about all files.

how would it look like? do we show names, or only number of files?

can we take a look at how other tools like `lftp`, `rclone` show this? just to see if we like some ideas?
@skshetry I have plans to change implementation, but would like to get your opinion about the current one. Maybe things change I will keep/improve this one
> I see what you're trying to do here, but it's not really a full fix (it's just the temporary patch I was thinking of in [#2905 (comment)](https://github.com/iterative/dvc/issues/2905#issuecomment-562516143)).
> 
> There's a more meta comment which maybe needs to be addressed first: [#2905 (comment)](https://github.com/iterative/dvc/issues/2905#issuecomment-599503500)

Hi @casperdcl , you mentioned that it's not a full fix, but how do you see it should work at the latest stage?
I see that `MAX_LINES` should be calculated dynamically based on terminal rows, what else?   
Maybe there should be completely different approach you see

>  from having more than about 5 threads?

As I got for network operations user gets benefit with multiple open connections, it would be great if you can share some info about it @efiop. Another point that it's just an interface - we provide option to point `jobs` parameter, that can be custom and is not limited to cpu count
@JIoJIaJIu there are a few small bugs I can see with this PR but it's not really worth getting into if we can avoid it altogether by say setting `jobs = min(10, args.jobs)`.

Apart from that (in a general use case/if we decide there should be no upper bound on threads) there probably should be a PR on the upstream https://github.com/tqdm/tqdm repo...

Anyway for DVC purposes, the best way to go about this is to collect info about all tasks first (e.g. total number of bytes) and have them all report back to just one total bar if there are more than say 10 tasks.

@shcheklein / @efiop lemme know if actually you would prefer some per-task info printed at least after each task completes. If you do, this may be fairly easy to solve using something similar to this PR.
@casperdcl there are definitely cases where 10 or even 20 jobs is not enough. Since some workloads are IO bound it might be beneficial to have 100 jobs, so I doubt that it's a good solution to limit to some upper bound.

Could we check some other tools that do parallel processing? sync tools, package managers, etc .. to see some ideas and how people solve this problem.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -69,7 +69,7 @@ def run(self):
     "funcy>=1.14",
     "pathspec>=0.6.0",
     "shortuuid>=0.5.0",
-    "tqdm>=4.40.0,<5",
+    "tqdm>=4.44.0,<5",
     "packaging>=19.0",
     "zc.lockfile>=1.2.1",
     "flufl.lock>=3.2",


Test patch: