Repository: iterative/dvc

Issue:
pull/fetch: progress bar is all over the console output
DVC version: 

```
DVC version: 0.91.1
Python version: 3.7.7
Platform: Darwin-19.4.0-x86_64-i386-64bit
Binary: False
Package: brew
Supported remotes: azure, gdrive, gs, http, https, s3, ssh, oss
Cache: reflink - supported, hardlink - supported, symlink - supported
Repo: dvc, git
```

When I run `dvc pull`, dvc tries to show the progress bar for downloading data files, but it looks like there's a bug in rendering it. 

Here's the video:  https://share.getcloudapp.com/jkuKG6rp

Please let me know if I can provide more details on this or help in any way. I know this reproduces for one of my teammates, too, on Ubuntu with DVC 0.91.0. 

pull/fetch: progress bar is all over the console output
DVC version: 

```
DVC version: 0.91.1
Python version: 3.7.7
Platform: Darwin-19.4.0-x86_64-i386-64bit
Binary: False
Package: brew
Supported remotes: azure, gdrive, gs, http, https, s3, ssh, oss
Cache: reflink - supported, hardlink - supported, symlink - supported
Repo: dvc, git
```

When I run `dvc pull`, dvc tries to show the progress bar for downloading data files, but it looks like there's a bug in rendering it. 

Here's the video:  https://share.getcloudapp.com/jkuKG6rp

Please let me know if I can provide more details on this or help in any way. I know this reproduces for one of my teammates, too, on Ubuntu with DVC 0.91.0. 



Discussion:
Hi, @xnutsive. Thanks for opening this issue.

The files that you are downloading are small, and hence looks flickering, and is a problem on the
`fetch` part.

#3500 set a threshold for showing maximum number of bars and hides them (it was more crazier before), but we still need to unify multiple bars into a single bar. This was done on upstream tqdm/tqdm#924.

Related/part-of #2905.

@casperdcl, I assume this is going to be a bit nicer with tqdm/tqdm#930. I notice that the bars are getting placed randomly on each render making it look flickery?
The ordering isn't random - the screen is actually "full" of bars "in order." To maintain the order, the whole screen below a bar is cleared when said bar completes, and the cleared bars are moved upwards. However, some (most) are randomly skipping display for speed purposes.

https://github.com/tqdm/tqdm/issues/930 will actually fix this by changing to random ordering. This means finished bars get replaced without clearing others.

(disclaimer: this is a simplified explanation; in practice it's a little more complicated because concurrency + UI + cross-platform support + speed optimisation = 4 x ~~hell~~ more fun)
Experience-wise (please take this with a grain of salt, I'm not by any means a serious ML practitioner), a single pbar would be nicer — I don't really care how much of a specific image / text is downloaded, I just care about the whole data downloading / uploading process.

I've read another issue where you discuss this and I imagine doing a single pbar is a ton of work though ;)

Let me know if I can be useful in the meantime!

On Tue, Mar 31, 2020 at 11:52 PM, Casper da Costa-Luis < notifications@github.com > wrote:

> 
> 
> 
> The ordering isn't random - the screen is actually "full" of bars "in
> order." To maintain the order, the whole screen below a bar is cleared
> when said bar completes, and the cleared bars are moved upwards. However,
> some (most) are randomly skipping display for speed purposes.
> 
> 
> 
> tqdm/ tqdm#930 ( https://github.com/tqdm/tqdm/pull/930 ) will actually fix
> this by changing to random ordering. This means finished bars get replaced
> without clearing others.
> 
> 
> 
> (disclaimer: this is a simplified explanation; in practice it's a little
> more complicated because concurrency + UI + cross-platform support + speed
> optimisation = 4 x hell hard)
> 
> 
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub (
> https://github.com/iterative/dvc/issues/3565#issuecomment-607068855 ) , or
> unsubscribe (
> https://github.com/notifications/unsubscribe-auth/AAAKU4IJX6UYETDADQ3QVLDRKLQDXANCNFSM4LYGK3HA
> ).
> 
> 
>
Hi, @xnutsive. Thanks for opening this issue.

The files that you are downloading are small, and hence looks flickering, and is a problem on the
`fetch` part.

#3500 set a threshold for showing maximum number of bars and hides them (it was more crazier before), but we still need to unify multiple bars into a single bar. This was done on upstream tqdm/tqdm#924.

Related/part-of #2905.

@casperdcl, I assume this is going to be a bit nicer with tqdm/tqdm#930. I notice that the bars are getting placed randomly on each render making it look flickery?
The ordering isn't random - the screen is actually "full" of bars "in order." To maintain the order, the whole screen below a bar is cleared when said bar completes, and the cleared bars are moved upwards. However, some (most) are randomly skipping display for speed purposes.

https://github.com/tqdm/tqdm/issues/930 will actually fix this by changing to random ordering. This means finished bars get replaced without clearing others.

(disclaimer: this is a simplified explanation; in practice it's a little more complicated because concurrency + UI + cross-platform support + speed optimisation = 4 x ~~hell~~ more fun)
Experience-wise (please take this with a grain of salt, I'm not by any means a serious ML practitioner), a single pbar would be nicer — I don't really care how much of a specific image / text is downloaded, I just care about the whole data downloading / uploading process.

I've read another issue where you discuss this and I imagine doing a single pbar is a ton of work though ;)

Let me know if I can be useful in the meantime!

On Tue, Mar 31, 2020 at 11:52 PM, Casper da Costa-Luis < notifications@github.com > wrote:

> 
> 
> 
> The ordering isn't random - the screen is actually "full" of bars "in
> order." To maintain the order, the whole screen below a bar is cleared
> when said bar completes, and the cleared bars are moved upwards. However,
> some (most) are randomly skipping display for speed purposes.
> 
> 
> 
> tqdm/ tqdm#930 ( https://github.com/tqdm/tqdm/pull/930 ) will actually fix
> this by changing to random ordering. This means finished bars get replaced
> without clearing others.
> 
> 
> 
> (disclaimer: this is a simplified explanation; in practice it's a little
> more complicated because concurrency + UI + cross-platform support + speed
> optimisation = 4 x hell hard)
> 
> 
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub (
> https://github.com/iterative/dvc/issues/3565#issuecomment-607068855 ) , or
> unsubscribe (
> https://github.com/notifications/unsubscribe-auth/AAAKU4IJX6UYETDADQ3QVLDRKLQDXANCNFSM4LYGK3HA
> ).
> 
> 
>

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -69,7 +69,7 @@ def run(self):
     "funcy>=1.14",
     "pathspec>=0.6.0",
     "shortuuid>=0.5.0",
-    "tqdm>=4.44.0,<5",
+    "tqdm>=4.45.0,<5",
     "packaging>=19.0",
     "zc.lockfile>=1.2.1",
     "flufl.lock>=3.2",


Test patch: