Repository: dask/dask

Issue:
Minimum time threshold before showing progress bars
Just started playing around with dask progress bars. These are very handy -- thanks!

I especially like the ability to register a progress bar and see it for all operations.

One annoyance when using this with xray datasets is that xray operations often involve some very small calculations (e.g., to verify aligned indexes) before doing calculations. These take a fraction of a second, so showing a progress bar is not very useful:

```
[########################################] | 100% Completed |  0.1s
[########################################] | 100% Completed |  0.1s
[########################################] | 100% Completed |  0.1s
[########################################] | 100% Completed |  0.1s
[########################################] | 100% Completed |  0.1s
[########################################] | 100% Completed |  0.1s
```

I wonder if we could add some optional minimum time threshold before displaying the progress bar?



Discussion:
Could definitely do that. If you know an operation is cheap, you could also turn it off (unregister it), or just use it in the contextmanager form. I'm happy to add this option if desired though, as long as it's not the default.

This seems reasonable to me for any computation that takes less than, say 500ms.

I only actually value the progressbar for longer running computations.  In @shoyer's case he has lots of small computations that are getting in the way.  These are in library code and so it's not convenient to wrap or unwrap them.  

We all agree that an option with a minimum time before visible is good.  We potentially disagree on what that default should be (potentially zero).  Fortunately most of the work is independent of the choice of this default.

Actually @jcrist 's suggestion of zero for the default minimum time might be a sane default just because it is a natural choice (not at all subjective).  I don't mind saying the following when I start up.

``` Python
ProgressBar(minimum=0.500).register()
```

I'll throw this in with the other progressbar changes in #581.

Agreed, I don't mind specifying the minimum time explicitly.


Resolving patch:
diff --git a/dask/diagnostics/progress.py b/dask/diagnostics/progress.py
--- a/dask/diagnostics/progress.py
+++ b/dask/diagnostics/progress.py
@@ -29,16 +29,42 @@ def format_time(t):
 class ProgressBar(Callback):
     """A progress bar for dask.
 
-    Can be used as a context manager around dask computations.
+    Parameters
+    ----------
+    minimum : int, optional
+        Minimum time threshold in seconds before displaying a progress bar.
+        Default is 0 (always display)
+    width : int, optional
+        Width of the bar
+    dt : float, optional
+        Update resolution in seconds, default is 0.1 seconds
 
     Examples
     --------
-    >>> with ProgressBar():    # doctest: +SKIP
-    ...     out = res.compute()
+
+    Below we create a progress bar with a minimum threshold of 1 second before
+    displaying. For cheap computations nothing is shown:
+
+    >>> with ProgressBar(minimum=1.0):      # doctest: +SKIP
+    ...     out = some_fast_computation.compute()
+
+    But for expensive computations a full progress bar is displayed:
+
+    >>> with ProgressBar(minimum=1.0):      # doctest: +SKIP
+    ...     out = some_slow_computation.compute()
+    [########################################] | 100% Completed | 10.4 s
+
+    You can also register a progress bar so that it displays for all
+    computations:
+
+    >>> pbar = ProgressBar()                # doctest: +SKIP
+    >>> pbar.register()                     # doctest: +SKIP
+    >>> some_slow_computation.compute()     # doctest: +SKIP
     [########################################] | 100% Completed | 10.4 s
     """
 
-    def __init__(self, width=40, dt=0.1):
+    def __init__(self, minimum=0, width=40, dt=0.1):
+        self._minimum = minimum
         self._width = width
         self._dt = dt
 
@@ -57,8 +83,11 @@ def _pretask(self, key, dsk, state):
     def _finish(self, dsk, state, errored):
         self._running = False
         self._timer.join()
+        elapsed = default_timer() - self._start_time
+        if elapsed < self._minimum:
+            return
         if not errored:
-            self._draw_bar(1)
+            self._draw_bar(1, elapsed)
         else:
             self._update_bar()
         sys.stdout.write('\n')
@@ -67,22 +96,24 @@ def _finish(self, dsk, state, errored):
     def _timer_func(self):
         """Background thread for updating the progress bar"""
         while self._running:
-            self._update_bar()
+            elapsed = default_timer() - self._start_time
+            if elapsed > self._minimum:
+                self._update_bar(elapsed)
             time.sleep(self._dt)
 
-    def _update_bar(self):
+    def _update_bar(self, elapsed):
         s = self._state
         if not s:
-            self._draw_bar(0)
+            self._draw_bar(0, elapsed)
             return
         ndone = len(s['finished'])
         ntasks = sum(len(s[k]) for k in ['ready', 'waiting', 'running']) + ndone
-        self._draw_bar(ndone / ntasks if ntasks else 0)
+        self._draw_bar(ndone / ntasks if ntasks else 0, elapsed)
 
-    def _draw_bar(self, frac):
+    def _draw_bar(self, frac, elapsed):
         bar = '#' * int(self._width * frac)
         percent = int(100 * frac)
-        elapsed = format_time(default_timer() - self._start_time)
+        elapsed = format_time(elapsed)
         msg = '\r[{0:<{1}}] | {2}% Completed | {3}'.format(bar, self._width,
                                                            percent, elapsed)
         with ignoring(ValueError):


Test patch:
diff --git a/dask/diagnostics/tests/test_progress.py b/dask/diagnostics/tests/test_progress.py
--- a/dask/diagnostics/tests/test_progress.py
+++ b/dask/diagnostics/tests/test_progress.py
@@ -30,6 +30,13 @@ def test_progressbar(capsys):
     check_bar_completed(capsys, 20)
 
 
+def test_minimum_time(capsys):
+    with ProgressBar(1.0):
+        out = get(dsk, 'e')
+    out, err = capsys.readouterr()
+    assert out == '' and err == ''
+
+
 def test_clean_exit():
     dsk = {'a': (lambda: 1/0,)}
     try: