Repository: tqdm/tqdm (Python)

Issue:
tqdm hangs up with highly irregular iterations speed
This is basically the same issue as #54, but that one is closed. The problem still persists, consider this example:

``` python
for i in tqdm.trange(1000):
    if i > 500:
        sleep(10)
```

It updates first time after the first 10-sec iteration, and the second time after more than 5 minutes (!) - I didn't wait for the second update. You may say this is a highly artificial case - but it doesn't seem so: for example, when you process items, some of which are skipped (e.g. already processed, or marked to skip, ...) and others require a lot of time. I myself routinely use tqdm with iterations in the order of minutes (with another faster inner progressbar, btw) and was really surprised to see the outer progressbar not moving for hours.

I expected the example at the beginning to show `501` almost immediately and then increase by one each ten seconds.

P.S.: yes, it works correctly with `miniters=1`, but I could find that this is required only reading #54 (there is a paragraph saying like 'recommended for downloads' in the readme, but it doesn't say that the progress bar will work extremely incorrectly otherwise). And anyway, even if there is no way to make it work without additional overhead, I think `miniters=1` must be the default option, as correctness is more important than speed. And then those who need to remove every possible overhead would specify `None` or something, but as for now this case is really confusing.



Hints:
This is not how `maxinterval` works (even if I would like it very much!): `maxinterval` kicks in only after the last refresh being longer than the `maxinterval` threshold and then readjust the internal parameters so that next refresh will be every 10 seconds (hopefully).

In your particular example, yes at iteration 500, you will need to basically wait 500*10 seconds, but after that, `maxinterval` will kick in and the next refresh will be 10 seconds later (at iteration 1001, 1002, 1003, etc.).

There is no way around but by setting `miniters=1`. This could be made the default behavior but I am against as this is still debatable if such extreme cases can happen in real applications. It can of course happen that some refresh gets longer than `maxinterval`, but I expect these cases to be at most a few minutes, not a few hours like in your example.

Also, I disagree that correctness in all scenarios is more important than performance: noone would use a progress bar that significantly slows down their computations. `tqdm` is meant to be usable in tight loops by default.

But we could add a better note in the readme, it's clear that the current note isn't descriptive enough.

Yes, I agree with @lrq3000

On 27 August 2016 at 22:36, Stephen L. notifications@github.com wrote:

> This is not how maxinterval works (even if I would like it very much!):
> maxinterval kicks in only after the last refresh being longer than the
> maxinterval threshold and then readjust the internal parameters so that
> next refresh will be every 10 seconds (hopefully).
> 
> In your particular example, yes at iteration 500, you will need to
> basically wait 500*10 seconds, but after that, maxinterval will kick in
> and the next refresh will be 10 seconds later (at iteration 1001, 1002,
> 1003, etc.).
> 
> There is no way around but by setting miniters=1. This could be made the
> default behavior but I am against as this is still debatable if such
> extreme cases can happen in real applications. It can of course happen that
> some refresh gets longer than maxinterval, but I expect these cases to be
> at most a few minutes, not a few hours like in your example.
> 
> Also, I disagree that correctness in all scenarios is more important than
> performance: noone would use a progress bar that significantly slow down
> their computations. tqdm is meant to be usable in tight loops by default.
> 
> But we could add a better note in the readme, it's clear that the current
> note isn't descriptive enough.
> 
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> https://github.com/tqdm/tqdm/issues/249#issuecomment-242942221, or mute
> the thread
> https://github.com/notifications/unsubscribe-auth/AKR9mxChFUR6CbNA3fkyoRLHFmKgCC7dks5qkK3bgaJpZM4Jux7F
> .

Maybe we can overcome this issue by using a monitoring thread tied to the class: it would basically sleep and wake up every `monitor_interval` (default to 10 second like default maxinterval) and check whether all the bars that have `miniters > 1` have printed in the last `maxinterval` time (just check `self.last_print_t`), if not, then `miniters = 1`.

/EDIT: I mistakenly typed `mininterval` instead of `miniters` initially in the paragraph above. This is fixed now.

If we spawn only one monitoring thread, the impact on CPU wouldn't be much (considering it would sleep much of the time except when it wakes up every `monitor_interval`).

The only issue is: is it possible to access `tqdm` class variables from a thread? If yes, then the plan should work.

I'm not very good with threading, so if someone would like to give it a try...

/EDIT: here is a [tutorial about monitoring threads in python](http://notemagnet.blogspot.fr/2009/10/writing-monitoring-threads-in-python.html).

@lrq3000 well, I understand the problem and tradeoffs here, but something needs to be done.

I think I have an idea which may help dealing with this issue and still have little overhead. If I understand correctly, the problem is calling `time()` or `clock()` on each iteration as they take about 100 ns (i.e. rate is 10 M/sec) and this is important for progress bars with a lot of iterations. So, I suggest that `tqdm` default to `miniters=1` for e.g. `total` < 10k iterations: this way the overhead from getting the time on each iteration will be less than 1 ms for the whole duration of progressbar (or maybe set an even higher threshold than 10k; I personally would choose 100k, as even for a short one-second-long progressbar the overhead would be 1%). This should get rid of the hanging issue for the huge majority of cases, because it's really unlikely to have e.g. 10k iterations each lasting for a minute (that's a whole week!).

Such a solution will still show `0` before the first "long" iteration completes, but this is relatively minor inconvenience and I think can't be solved without another thread.


Patch:
diff --git a/examples/simple_examples.py b/examples/simple_examples.py
--- a/examples/simple_examples.py
+++ b/examples/simple_examples.py
@@ -28,8 +28,22 @@
     pass
 
 # Comparison to https://code.google.com/p/python-progressbar/
-from progressbar.progressbar import ProgressBar
-for i in ProgressBar()(_range(int(1e8))):
+try:
+    from progressbar.progressbar import ProgressBar
+except ImportError:
+    pass
+else:
+    for i in ProgressBar()(_range(int(1e8))):
+        pass
+
+# Dynamic miniters benchmark
+from tqdm import trange
+for i in trange(int(1e8), miniters=None, mininterval=0.1, smoothing=0):
+    pass
+
+# Fixed miniters benchmark
+from tqdm import trange
+for i in trange(int(1e8), miniters=4500000, mininterval=0.1, smoothing=0):
     pass
 """
 
diff --git a/tqdm/_tqdm.py b/tqdm/_tqdm.py
--- a/tqdm/_tqdm.py
+++ b/tqdm/_tqdm.py
@@ -570,12 +570,20 @@ def __init__(self, iterable=None, desc=None, total=None, leave=True,
             fallback is a meter width of 10 and no limit for the counter and
             statistics. If 0, will not print any meter (only stats).
         mininterval  : float, optional
-            Minimum progress update interval, in seconds [default: 0.1].
+            Minimum progress display update interval, in seconds [default: 0.1].
         maxinterval  : float, optional
-            Maximum progress update interval, in seconds [default: 10.0].
+            Maximum progress display update interval, in seconds [default: 10].
+            Automatically adjusts `miniters` to correspond to `mininterval`
+            after long display update lag. Only works if `dynamic_miniters`
+            or monitor thread is enabled.
         miniters  : int, optional
-            Minimum progress update interval, in iterations.
-            If specified, will set `mininterval` to 0.
+            Minimum progress display update interval, in iterations.
+            If 0 and `dynamic_miniters`, will automatically adjust to equal
+            `mininterval` (more CPU efficient, good for tight loops).
+            If > 0, will skip display of specified number of iterations.
+            Tweak this and `mininterval` to get very efficient loops.
+            If your progress is erratic with both fast and slow iterations
+            (network, skipping items, etc) you should set miniters=1.
         ascii  : bool, optional
             If unspecified or False, use unicode (smooth blocks) to fill
             the meter. The fallback is to use ASCII characters `1-9 #`.
@@ -782,6 +790,7 @@ def __iter__(self):
             ncols = self.ncols
             mininterval = self.mininterval
             maxinterval = self.maxinterval
+            miniters = self.miniters
             dynamic_miniters = self.dynamic_miniters
             unit = self.unit
             unit_scale = self.unit_scale
@@ -839,19 +848,27 @@ def __iter__(self):
                             self.moveto(-self.pos)
 
                         # If no `miniters` was specified, adjust automatically
-                        # to the maximum iteration rate seen so far.
+                        # to the max iteration rate seen so far between 2 prints
                         if dynamic_miniters:
-                            if maxinterval and delta_t > maxinterval:
-                                # Set miniters to correspond to maxinterval
-                                miniters = delta_it * maxinterval / delta_t
-                            elif mininterval and delta_t:
+                            if maxinterval and delta_t >= maxinterval:
+                                # Adjust miniters to time interval by rule of 3
+                                if mininterval:
+                                    # Set miniters to correspond to mininterval
+                                    miniters = delta_it * mininterval / delta_t
+                                else:
+                                    # Set miniters to correspond to maxinterval
+                                    miniters = delta_it * maxinterval / delta_t
+                            elif smoothing:
                                 # EMA-weight miniters to converge
                                 # towards the timeframe of mininterval
-                                miniters = smoothing * delta_it * mininterval \
-                                    / delta_t + (1 - smoothing) * miniters
+                                miniters = smoothing * delta_it * \
+                                              (mininterval / delta_t
+                                               if mininterval and delta_t
+                                               else 1) + \
+                                              (1 - smoothing) * miniters
                             else:
-                                miniters = smoothing * delta_it + \
-                                           (1 - smoothing) * miniters
+                                # Maximum nb of iterations between 2 prints
+                                miniters = max(miniters, delta_it)
 
                         # Store old values for next call
                         self.n = self.last_print_n = last_print_n = n
@@ -862,6 +879,7 @@ def __iter__(self):
             # Update some internal variables for close().
             self.last_print_n = last_print_n
             self.n = n
+            self.miniters = miniters
             self.close()
 
     def update(self, n=1):
@@ -926,21 +944,26 @@ def update(self, n=1):
                     self.moveto(-self.pos)
 
                 # If no `miniters` was specified, adjust automatically to the
-                # maximum iteration rate seen so far.
+                # maximum iteration rate seen so far between two prints.
                 # e.g.: After running `tqdm.update(5)`, subsequent
                 # calls to `tqdm.update()` will only cause an update after
                 # at least 5 more iterations.
                 if self.dynamic_miniters:
-                    if self.maxinterval and delta_t > self.maxinterval:
-                        self.miniters = self.miniters * self.maxinterval \
+                    if self.maxinterval and delta_t >= self.maxinterval:
+                        if self.mininterval:
+                            self.miniters = delta_it * self.mininterval \
                                         / delta_t
-                    elif self.mininterval and delta_t:
-                        self.miniters = self.smoothing * delta_it \
-                                        * self.mininterval / delta_t + \
+                        else:
+                            self.miniters = delta_it * self.maxinterval \
+                                        / delta_t
+                    elif self.smoothing:
+                        self.miniters = self.smoothing * delta_it * \
+                                        (self.mininterval / delta_t
+                                         if self.mininterval and delta_t
+                                         else 1) + \
                                         (1 - self.smoothing) * self.miniters
                     else:
-                        self.miniters = self.smoothing * delta_it + \
-                                        (1 - self.smoothing) * self.miniters
+                        self.miniters = max(self.miniters, delta_it)
 
                 # Store old values for next call
                 self.last_print_n = self.n


Test patch:
diff --git a/tqdm/tests/tests_tqdm.py b/tqdm/tests/tests_tqdm.py
--- a/tqdm/tests/tests_tqdm.py
+++ b/tqdm/tests/tests_tqdm.py
@@ -382,9 +382,9 @@ def test_max_interval():
     total = 100
     bigstep = 10
     smallstep = 5
-    timer = DiscreteTimer()
 
     # Test without maxinterval
+    timer = DiscreteTimer()
     with closing(StringIO()) as our_file:
         with closing(StringIO()) as our_file2:
             # with maxinterval but higher than loop sleep time
@@ -417,6 +417,7 @@ def test_max_interval():
         assert "25%" not in our_file.read()
 
     # Test with maxinterval effect
+    timer = DiscreteTimer()
     with closing(StringIO()) as our_file:
         with tqdm(total=total, file=our_file, miniters=None, mininterval=0,
                   smoothing=1, maxinterval=1e-4) as t:
@@ -433,6 +434,7 @@ def test_max_interval():
             assert "25%" in our_file.read()
 
     # Test iteration based tqdm with maxinterval effect
+    timer = DiscreteTimer()
     with closing(StringIO()) as our_file:
         with tqdm(_range(total), file=our_file, miniters=None,
                   mininterval=1e-5, smoothing=1, maxinterval=1e-4) as t2:
@@ -448,6 +450,69 @@ def test_max_interval():
         our_file.seek(0)
         assert "15%" in our_file.read()
 
+    # Test different behavior with and without mininterval
+    timer = DiscreteTimer()
+    total = 1000
+    mininterval = 0.1
+    maxinterval = 10
+    with closing(StringIO()) as our_file:
+        with tqdm(total=total, file=our_file, miniters=None, smoothing=1,
+                  mininterval=mininterval, maxinterval=maxinterval) as tm1:
+            with tqdm(total=total, file=our_file, miniters=None, smoothing=1,
+                      mininterval=0, maxinterval=maxinterval) as tm2:
+
+                cpu_timify(tm1, timer)
+                cpu_timify(tm2, timer)
+
+                # Fast iterations, check if dynamic_miniters triggers
+                timer.sleep(mininterval)  # to force update for t1
+                tm1.update(total/2)
+                tm2.update(total/2)
+                assert int(tm1.miniters) == tm2.miniters == total/2
+
+                # Slow iterations, check different miniters if mininterval
+                timer.sleep(maxinterval*2)
+                tm1.update(total/2)
+                tm2.update(total/2)
+                res = [tm1.miniters, tm2.miniters]
+                assert res == [
+                               (total/2)*mininterval/(maxinterval*2),
+                               (total/2)*maxinterval/(maxinterval*2)
+                               ]
+
+    # Same with iterable based tqdm
+    timer1 = DiscreteTimer()  # need 2 timers for each bar because zip not work
+    timer2 = DiscreteTimer()
+    total = 100
+    mininterval = 0.1
+    maxinterval = 10
+    with closing(StringIO()) as our_file:
+        t1 = tqdm(_range(total), file=our_file, miniters=None, smoothing=1,
+                  mininterval=mininterval, maxinterval=maxinterval)
+        t2 = tqdm(_range(total), file=our_file, miniters=None, smoothing=1,
+                  mininterval=0, maxinterval=maxinterval)
+
+        cpu_timify(t1, timer1)
+        cpu_timify(t2, timer2)
+
+        for i in t1:
+            if i == ((total/2)-2):
+                timer1.sleep(mininterval)
+            if i == (total-1):
+                timer1.sleep(maxinterval*2)
+
+        for i in t2:
+            if i == ((total/2)-2):
+                timer2.sleep(mininterval)
+            if i == (total-1):
+                timer2.sleep(maxinterval*2)
+
+        assert t1.miniters == 0.255
+        assert t2.miniters == 0.5
+
+        t1.close()
+        t2.close()
+
 
 @with_setup(pretest, posttest)
 def test_min_iters():
@@ -494,12 +559,40 @@ def test_dynamic_min_iters():
     assert '60%' not in out
     assert '70%' in out
 
+    # Check with smoothing=0, miniters should be set to max update seen so far
+    with closing(StringIO()) as our_file:
+        total = 10
+        t = tqdm(total=total, file=our_file, miniters=None, mininterval=0,
+                 smoothing=0)
+
+        t.update()
+        t.update(2)
+        t.update(5)  # this should be stored as miniters
+        t.update(1)
+
+        our_file.seek(0)
+        out = our_file.read()
+        assert t.dynamic_miniters and not t.smoothing
+        assert t.miniters == 5
+        t.close()
+
+    # Check iterable based tqdm
+    with closing(StringIO()) as our_file:
+        t = tqdm(_range(10), file=our_file, miniters=None, mininterval=None,
+                 smoothing=0.5)
+        for _ in t:
+            pass
+        assert t.dynamic_miniters
+
+    # No smoothing
     with closing(StringIO()) as our_file:
-        t = tqdm(_range(10), file=our_file, miniters=None, mininterval=None)
+        t = tqdm(_range(10), file=our_file, miniters=None, mininterval=None,
+                 smoothing=0)
         for _ in t:
             pass
         assert t.dynamic_miniters
 
+    # No dynamic_miniters (miniters is fixed manually)
     with closing(StringIO()) as our_file:
         t = tqdm(_range(10), file=our_file, miniters=1, mininterval=None)
         for _ in t: