Repository: iterative/dvc

Issue:
dvc push -q is not completely quiet, shows file transfer tqdm progress bars
Hey guys, love what you've done with DVC.

Had a quick bug that's causing me a little issue. When I use 'dvc push -q' I'm still seeing tqdm progress bars. Wouldn't be a huge issue, but I'm probably pushing 100K 250kb files. This is a local remote, so the transfer speeds are quick. I know in some of my other scripts where I use tqdm, if the iteration time is very small, the tqdm overhead of writing to std::out actually starts to contribute to performance.

dvc version: 0.83.0
os: Windows 10

![image](https://user-images.githubusercontent.com/45394583/74362981-a1500d80-4d97-11ea-9df8-c56fbadea69b.png)



dvc push -q is not completely quiet, shows file transfer tqdm progress bars
Hey guys, love what you've done with DVC.

Had a quick bug that's causing me a little issue. When I use 'dvc push -q' I'm still seeing tqdm progress bars. Wouldn't be a huge issue, but I'm probably pushing 100K 250kb files. This is a local remote, so the transfer speeds are quick. I know in some of my other scripts where I use tqdm, if the iteration time is very small, the tqdm overhead of writing to std::out actually starts to contribute to performance.

dvc version: 0.83.0
os: Windows 10

![image](https://user-images.githubusercontent.com/45394583/74362981-a1500d80-4d97-11ea-9df8-c56fbadea69b.png)





Discussion:
@casperdcl could you take a look please?
@WillNichols726 

> I know in some of my other scripts where I use tqdm, if the iteration time is very small, the tqdm overhead of writing to std::out actually starts to contribute to performance

That should really not be the case as `tqdm` is designed to be negligible overhead in all cases - could you elaborate a bit more? I know this isn't actually the issue here, though.

In terms of this issue, the progress output should be suppressed completely by `-q`. Will take a look & fix soon.
@casperdcl - Appreciate everything you've done with tqdm! Recently, I had something that was running at 200 iterations a second, when I set mininterval to 1 sec, jumped up to 300 iterations a sec.  Didn't spend a lot of time determining root cause for the speedup, it could've been caching behavior or something else. I've also seen some minor speedups on other rapidly-iterating scripts when increasing mininterval. Not a huge issue. I'm typically running in Powershell or cmd on Windows 10. 
Hmm interesting as the default `mininterval=0.1` should be fine on most consoles.

No real measurable difference for me (`it/s`)... Note that it's under `200it/s` because `sleep` isn't accurate.

```
$ python -c 'from time import sleep; from tqdm import trange
[sleep(1/200) for _ in trange(100)]'
100%|██████████| 100/100 [00:00<00:00, 192.77it/s]
$ python -c 'from time import sleep; from tqdm import trange
[sleep(1/200) for _ in trange(100, mininterval=1)]'
100%|█████████| 100/100 [00:00<00:00, 192.62it/s]
```
ah also (back to this issue) `--quiet` is actually currently ignored (i.e. not implemented) by `push`
nvm looks like this is a bug
@casperdcl could you take a look please?
@WillNichols726 

> I know in some of my other scripts where I use tqdm, if the iteration time is very small, the tqdm overhead of writing to std::out actually starts to contribute to performance

That should really not be the case as `tqdm` is designed to be negligible overhead in all cases - could you elaborate a bit more? I know this isn't actually the issue here, though.

In terms of this issue, the progress output should be suppressed completely by `-q`. Will take a look & fix soon.
@casperdcl - Appreciate everything you've done with tqdm! Recently, I had something that was running at 200 iterations a second, when I set mininterval to 1 sec, jumped up to 300 iterations a sec.  Didn't spend a lot of time determining root cause for the speedup, it could've been caching behavior or something else. I've also seen some minor speedups on other rapidly-iterating scripts when increasing mininterval. Not a huge issue. I'm typically running in Powershell or cmd on Windows 10. 
Hmm interesting as the default `mininterval=0.1` should be fine on most consoles.

No real measurable difference for me (`it/s`)... Note that it's under `200it/s` because `sleep` isn't accurate.

```
$ python -c 'from time import sleep; from tqdm import trange
[sleep(1/200) for _ in trange(100)]'
100%|██████████| 100/100 [00:00<00:00, 192.77it/s]
$ python -c 'from time import sleep; from tqdm import trange
[sleep(1/200) for _ in trange(100, mininterval=1)]'
100%|█████████| 100/100 [00:00<00:00, 192.62it/s]
```
ah also (back to this issue) `--quiet` is actually currently ignored (i.e. not implemented) by `push`
nvm looks like this is a bug

Resolving patch:
diff --git a/dvc/progress.py b/dvc/progress.py
--- a/dvc/progress.py
+++ b/dvc/progress.py
@@ -55,7 +55,8 @@ def __init__(
         desc  : persists after `close()`
         level  : effective logging level for determining `disable`;
             used only if `disable` is unspecified
-        disable  : If (default: None), will be determined by logging level.
+        disable  : If (default: None) or False,
+            will be determined by logging level.
             May be overridden to `True` due to non-TTY status.
             Skip override by specifying env var `DVC_IGNORE_ISATTY`.
         kwargs  : anything accepted by `tqdm.tqdm()`
@@ -69,7 +70,7 @@ def __init__(
             file = sys.stderr
         self.desc_persist = desc
         # auto-disable based on `logger.level`
-        if disable is None:
+        if not disable:
             disable = logger.getEffectiveLevel() > level
         # auto-disable based on TTY
         if (
diff --git a/dvc/repo/add.py b/dvc/repo/add.py
--- a/dvc/repo/add.py
+++ b/dvc/repo/add.py
@@ -55,7 +55,7 @@ def add(repo, targets, recursive=False, no_commit=False, fname=None):
                 total=len(stages),
                 desc="Processing",
                 unit="file",
-                disable=True if len(stages) == 1 else None,
+                disable=len(stages) == 1,
             ) as pbar_stages:
                 for stage in stages:
                     try:
@@ -102,7 +102,7 @@ def _create_stages(repo, targets, fname, pbar=None):
     for out in Tqdm(
         targets,
         desc="Creating DVC-files",
-        disable=True if len(targets) < LARGE_DIR_SIZE else None,
+        disable=len(targets) < LARGE_DIR_SIZE,
         unit="file",
     ):
         stage = Stage.create(repo, outs=[out], add=True, fname=fname)


Test patch:
diff --git a/tests/unit/test_progress.py b/tests/unit/test_progress.py
--- a/tests/unit/test_progress.py
+++ b/tests/unit/test_progress.py
@@ -16,6 +16,17 @@ def test_quiet_logging(caplog, capsys):
         assert out_err.err == ""
 
 
+def test_quiet_logging_disable_false(caplog, capsys):
+    with caplog.at_level(logging.CRITICAL, logger="dvc"):
+        # simulate interactive terminal
+        with mock.patch.object(sys.stderr, "isatty", return_value=True):
+            for _ in Tqdm(range(10), disable=False):
+                pass
+            out_err = capsys.readouterr()
+            assert out_err.out == ""
+            assert out_err.err == ""
+
+
 def test_quiet_notty(caplog, capsys):
     with caplog.at_level(logging.INFO, logger="dvc"):
         for _ in Tqdm(range(10)):