Repository: tqdm/tqdm (Python)

Issue:
Remove support for *args in pandas wrappers
Fix total computation for pandas apply

migrated from #244, and rebased

Wrong output in Windows console
Standard Windows 10 console (cmd.exe). Python 3.6.2. Test code:

```
import tqdm
import time

for i in tqdm.trange(10):
    time.sleep(0.1)
```
 Output: 
![clipboard01](https://user-images.githubusercontent.com/3974855/31056859-1f1f3730-a6d9-11e7-9300-41f89d02a298.png)

Is this expected? I'm sure it worked correctly in some older version.


Hints:
@aplavin @CrazyPython not sure what you wanted to achieve with this. seems like once all the commits from #244 were squashed there's just the one change (drop support of `*args`). did i miss something?

# [Codecov](https://codecov.io/gh/tqdm/tqdm/pull/299?src=pr&el=h1) Report
> :exclamation: No coverage uploaded for pull request base (`master@a379e33`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
> The diff coverage is `0%`.



```diff
@@            Coverage Diff            @@
##             master     #299   +/-   ##
=========================================
  Coverage          ?   91.02%           
=========================================
  Files             ?        7           
  Lines             ?      546           
  Branches          ?      100           
=========================================
  Hits              ?      497           
  Misses            ?       48           
  Partials          ?        1
```

Ok I agree, then let's keep this PR open until we get someone competent to
review and contribute to this.

2016-10-29 23:36 GMT+02:00 Casper da Costa-Luis notifications@github.com:

> ## _@casperdcl_ commented on this pull request.
> 
> In tqdm/_tqdm.py https://github.com/tqdm/tqdm/pull/299:
> 
> > ```
> >              # Precompute total iterations
> >              total = getattr(df, 'ngroups', None)
> >              if total is None:  # not grouped
> > ```
> > -                    total = len(df) if isinstance(df, Series) \
> > -                        else df.size // len(df)
> > -                    if df_function == 'applymap':
> > -                        total = df.size
> > -                    else:
> > -                        axis = kwargs.get('axis', 0)
> > -                        total = df.shape[axis]
> > -                        if isinstance(df, DataFrame):
> > -                            total += 1  # pandas calls update once too many
> 
> I think we need to get someone who develops similar things in pandas to
> take a look. Don't really have the time to investigate seeing as I don't
> even use this feature.
> 
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> https://github.com/tqdm/tqdm/pull/299, or mute the thread
> https://github.com/notifications/unsubscribe-auth/ABES3pDRAhMknflKJztHAt2xg0YwxD-Gks5q47xfgaJpZM4KkJ18
> .

@casperdcl huh?

Casper does not frequently use pandas, he already said so. He just
implemented the tqdm_pandas submodule using only his great insights into
the pandas API. But we are reaching a state where we need someone that is
more expert than us in pandas. I used pandas for a whole project in the
past, so you can say that I am a casual user, and this is getting too
complex for me too.

2016-10-30 0:19 GMT+02:00 CrazyPython notifications@github.com:

> @casperdcl https://github.com/casperdcl huh?
> 
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> https://github.com/tqdm/tqdm/pull/299#issuecomment-257119371, or mute
> the thread
> https://github.com/notifications/unsubscribe-auth/ABES3h9dQ9LditGk54sqxePJ52a2s7oIks5q48ZsgaJpZM4KkJ18
> .

add help-wanted.

@aplavin review this

@CrazyPython well, this is my patch - how can I review it? :) To best of my knowledge it's correct, but I'm not a pandas developer either.
I myself use tqdm with this modification and it works ok.

Thank's @aplavin. Well, then we either should just merge it in and see if it fixes issues (or if someone reports incorrect behavior), or add additional unit tests (to check correct length detection on various pandas datatypes and axis argument, if that's possible).

this fails unit tests on my machine `FAIL: Test pandas.DataFrame[.series].progress_apply` in tests_pandas.py line 60:

```
AssertionError: 
Expected:
100% at least twice
```

Ok I'll have a look, the tests passed last time I tried.

2016-11-12 17:21 GMT+01:00 Casper da Costa-Luis notifications@github.com:

> this fails unit tests on my machine FAIL: Test pandas.DataFrame[.series].
> progress_apply in tests_pandas.py line 60:
> 
> AssertionError:
> Expected:
> 100% at least twice
> 
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> https://github.com/tqdm/tqdm/pull/299#issuecomment-260131509, or mute
> the thread
> https://github.com/notifications/unsubscribe-auth/ABES3s5w1KFok3ssO8118s9k_R3WxDtkks5q9eeAgaJpZM4KkJ18
> .

@casperdcl Ah sorry I confused this PR with #271. I have no idea why this fails, but probably because of the different axis handling. We should put this PR aside until someone with more experience can provide a better fix...

from #322:
``` python
# Precompute total iterations
if kwargs.get('axis', 0)==1:
    total = len(df)
        else:
            total = getattr(df, 'ngroups', None)
                if total is None:  # not grouped
                    total = len(df) if isinstance(df, Series) \
                        else df.size // len(df)
                else:
                    total += 1  # pandas calls update once too many
```
Just rebased onto master. But CI now failing.
any updates on this?
well if anyone (@cancan101?) would like to fix the conflicts and unit tests, we'd be happy to merge this in...
@"People with write access", can anyone please review and merge the changes. 🙏 
I am using pandas frequently. Maybe I could help. 
thanks @chengs 
even if we exclude args, still there is a problem to get a correct "total". 

pandas apply calls func twice on the first column/row to decide whether it can take a fast or slow code path.

so, sometimes apply_func will run one more time, but maybe not if the func is slow.

still trying to find a way to determine if total = total + 1 or not.
please check #524  @casperdcl @lrq3000 
I cannot commit to this #299
which version of `tqdm` are you using?
tqdm==4.18.0

Edit: I use a high dpi display with 150% scaling, if that matters.

Edit 2: when I set the font size to 14, it works correctly. But I need to set the font size to 16 (because of the high dpi), and with this font, the effect is as I shown above.
interesting. so windows is incorrectly reporting the terminal width when you mess with fonts and scaling. you could always use `trange(..., ncols=75)` or whatever to fix the width
Setting `ncols` does not help. The progress bar is shorter but the text still wraps if the number of iterations is sufficiently large. Interestingly, I see the cursor positioned beyond the line (see screenshot) and it moves further with each iteration, eventually skipping to a new line.
![image](https://user-images.githubusercontent.com/3974855/31194196-aad12832-a946-11e7-91d2-41c0d0f86a8e.png)

Progress bar works fine with `ascii=True`. There is obviously an error with unicode characters in Windows console.
yes looks like it. I just tried win10 + py27, and couldn't get unicode working at all. it could be that windows thinks the unicode characters are "widechars"
Unicode support for Windows console was added in python 3.6. So it should work for me. Maybe the number of characters going backwards when updating the bar is calculated incorrectly? It's strange that the result depends on the font size. BTW, I believe that Unicode characters *are* widechars.
well, the true widths of many unicode chars are not wide, which some environments don't seem to understand
I cant confirm the experience above inside the windows comand line. But i have this problems when i use the PyCharm console.

---

With mutlithreading support, like i already mentioned in #329.

I get this output with unicode:
```
D:\Iceflower\programming\tqdm>python test.py
#0, est. 2.50s: 100%|█████████████████████████████████████████████████████████████| 5000/5000 [00:05<00:00, 840.30it/s]
#1, est. 1.70s:  99%|███████████████████████████████████████████████████████████? | 4936/5000 [00:05<00:00, 838.44it/s]
#1, est. 1.70s: 100%|█████████████████████████████████████████████████████████████| 5000/5000 [00:05<00:00, 838.20it/s]
#2, est. 1.20s: 100%|█████████████████████████████████████████████████████████████| 5000/5000 [00:05<00:00, 836.62it/s]
#3, est. 1.00s: 100%|█████████████████████████████████████████████████████████████| 5000/5000 [00:06<00:00, 833.22it/s]
#4, est. 0.83s: 100%|█████████████████████████████████████████████████████████████| 5000/5000 [00:06<00:00, 823.69it/s]
#5, est. 0.71s: 100%|█████████████████████████████████████████████████████████████| 5000/5000 [00:06<00:00, 830.42it/s]
#6, est. 0.62s: 100%|█████████████████████████████████████████████████████████████| 5000/5000 [00:06<00:00, 821.60it/s]
#7, est. 0.56s: 100%|█████████████████████████████████████████████████████████████| 5000/5000 [00:06<00:00, 816.37it/s]
#8, est. 0.50s: 100%|█████████████████████████████████████████████████████████████| 5000/5000 [00:06<00:00, 823.41it/s]
D:\Iceflower\programming\tqdm>
```

and this with `ascii=True`

```
D:\Iceflower\programming\tqdm>python test.py
#0, est. 2.50s: 100%|###########################################| 5000/5000 [00:05<00:00, 853.46it/s]
#1, est. 1.70s:  98%|##########################################2| 4917/5000 [00:05<00:00, 847.49it/s]
#1, est. 1.70s: 100%|###########################################| 5000/5000 [00:05<00:00, 848.74it/s]
#2, est. 1.20s: 100%|###########################################| 5000/5000 [00:05<00:00, 849.17it/s]
#3, est. 1.00s: 100%|###########################################| 5000/5000 [00:05<00:00, 844.14it/s]
#4, est. 0.83s: 100%|###########################################| 5000/5000 [00:05<00:00, 844.86it/s]
#5, est. 0.71s: 100%|###########################################| 5000/5000 [00:05<00:00, 841.36it/s]
#6, est. 0.62s: 100%|###########################################| 5000/5000 [00:05<00:00, 842.14it/s]
#7, est. 0.56s: 100%|###########################################| 5000/5000 [00:05<00:00, 842.64it/s]
#8, est. 0.50s: 100%|###########################################| 5000/5000 [00:05<00:00, 839.17it/s]
D:\Iceflower\programming\tqdm>
```
Hey guys, thanks for your answers. 
I tried with setting `ascii = True`, then it works well on my windows.
On windows 10 development in PTVS, python dos console output looks as follows with simple range. Notice the odd \uXXXX chars at the end. This also causes status on each newline unless ncols reduced:
```
for x in tqdm(range(10000)):
        sleep(0.001)
```
![capture](https://user-images.githubusercontent.com/15924596/38055253-16c3b2f8-32d1-11e8-91e1-a4dacc7a23d0.JPG)

That's partial unicode support. You can probably fix it with `tqdm(..., ascii=True)`
@IceflowRE your PyCharm issue is mentioned in https://github.com/tqdm/tqdm#faq-and-known-issues
Might add to the FAQs that environments which lie about unicode support will also have problems without `ascii=True`
ta, yup, tried ascii=True after following an earlier comment thread and it did the trick.

Patch:
diff --git a/tqdm/__init__.py b/tqdm/__init__.py
--- a/tqdm/__init__.py
+++ b/tqdm/__init__.py
@@ -4,7 +4,7 @@
 from ._tqdm_gui import tgrange
 from ._tqdm_pandas import tqdm_pandas
 from ._main import main
-from ._monitor import TMonitor
+from ._monitor import TMonitor, TqdmSynchronisationWarning
 from ._version import __version__  # NOQA
 from ._tqdm import TqdmTypeError, TqdmKeyError, TqdmDeprecationWarning, \
     TqdmMonitorWarning
@@ -12,7 +12,7 @@
 __all__ = ['tqdm', 'tqdm_gui', 'trange', 'tgrange', 'tqdm_pandas',
            'tqdm_notebook', 'tnrange', 'main', 'TMonitor',
            'TqdmTypeError', 'TqdmKeyError', 'TqdmDeprecationWarning',
-           'TqdmMonitorWarning',
+           'TqdmMonitorWarning', 'TqdmSynchronisationWarning',
            '__version__']
 
 
diff --git a/tqdm/_monitor.py b/tqdm/_monitor.py
--- a/tqdm/_monitor.py
+++ b/tqdm/_monitor.py
@@ -1,6 +1,13 @@
 from threading import Event, Thread
 from time import time
-__all__ = ["TMonitor"]
+from warnings import warn
+__all__ = ["TMonitor", "TqdmSynchronisationWarning"]
+
+
+class TqdmSynchronisationWarning(RuntimeWarning):
+    """tqdm multi-thread/-process errors which may cause incorrect nesting
+    but otherwise no adverse effects"""
+    pass
 
 
 class TMonitor(Thread):
@@ -59,7 +66,8 @@ def run(self):
             with self.tqdm_cls.get_lock():
                 cur_t = self._time()
                 # Check tqdm instances are waiting too long to print
-                for instance in self.tqdm_cls._instances:
+                instances = self.tqdm_cls._instances.copy()
+                for instance in instances:
                     # Check event in loop to reduce blocking time on exit
                     if self.was_killed.is_set():
                         return
@@ -76,6 +84,10 @@ def run(self):
                         instance.miniters = 1
                         # Refresh now! (works only for manual tqdm)
                         instance.refresh(nolock=True)
+                if instances != self.tqdm_cls._instances:  # pragma: nocover
+                    warn("Set changed size during iteration" +
+                         " (see https://github.com/tqdm/tqdm/issues/481)",
+                         TqdmSynchronisationWarning)
 
     def report(self):
         return not self.was_killed.is_set()
diff --git a/tqdm/_tqdm.py b/tqdm/_tqdm.py
--- a/tqdm/_tqdm.py
+++ b/tqdm/_tqdm.py
@@ -539,21 +539,21 @@ def inner(df, func, *args, **kwargs):
                     Data (may be grouped).
                 func  : function
                     To be applied on the (grouped) data.
-                *args, *kwargs  : optional
+                **kwargs  : optional
                     Transmitted to `df.apply()`.
                 """
+
                 # Precompute total iterations
                 total = getattr(df, 'ngroups', None)
                 if total is None:  # not grouped
-                    if isinstance(df, Series):
+                    if df_function == 'applymap':
+                        total = df.size
+                    elif isinstance(df, Series):
                         total = len(df)
-                    else:
-                        if kwargs.get('axis') == 1:
-                            total = len(df)
-                        else:
-                            total = df.size // len(df)
-                else:
-                    total += 1  # pandas calls update once too many
+                    else:  # DataFrame or Panel
+                        axis = kwargs.get('axis', 0)
+                        # when axis=0, total is shape[axis1]
+                        total = df.size // df.shape[axis]
 
                 # Init bar
                 if deprecated_t[0] is not None:
@@ -562,14 +562,27 @@ def inner(df, func, *args, **kwargs):
                 else:
                     t = tclass(*targs, total=total, **tkwargs)
 
+                if len(args) > 0:
+                    # *args intentionally not supported (see #244, #299)
+                    TqdmDeprecationWarning(
+                        "Except func, normal arguments are intentionally" +
+                        " not supported by" +
+                        " `(DataFrame|Series|GroupBy).progress_apply`." +
+                        " Use keyword arguments instead.",
+                        fp_write=getattr(t.fp, 'write', sys.stderr.write))
+
                 # Define bar updating wrapper
                 def wrapper(*args, **kwargs):
-                    t.update()
+                    # update tbar correctly
+                    # it seems `pandas apply` calls `func` twice
+                    # on the first column/row to decide whether it can
+                    # take a fast or slow code path; so stop when t.total==t.n
+                    t.update(n=1 if t.total and t.n < t.total else 0)
                     return func(*args, **kwargs)
 
-                # Apply the provided function (in *args and **kwargs)
+                # Apply the provided function (in **kwargs)
                 # on the df using our wrapper (which provides bar updating)
-                result = getattr(df, df_function)(wrapper, *args, **kwargs)
+                result = getattr(df, df_function)(wrapper, **kwargs)
 
                 # Close bar and return pandas calculation result
                 t.close()


Test patch:
diff --git a/tqdm/tests/tests_pandas.py b/tqdm/tests/tests_pandas.py
--- a/tqdm/tests/tests_pandas.py
+++ b/tqdm/tests/tests_pandas.py
@@ -5,76 +5,134 @@
 
 
 @with_setup(pretest, posttest)
-def test_pandas_groupby_apply():
-    """Test pandas.DataFrame.groupby(...).progress_apply"""
+def test_pandas_series():
+    """Test pandas.Series.progress_apply and .progress_map"""
     try:
         from numpy.random import randint
         import pandas as pd
-    except:
+    except ImportError:
         raise SkipTest
 
     with closing(StringIO()) as our_file:
-        tqdm.pandas(file=our_file, leave=False, ascii=True)
-
-        df = pd.DataFrame(randint(0, 50, (500, 3)))
-        df.groupby(0).progress_apply(lambda x: None)
+        tqdm.pandas(file=our_file, leave=True, ascii=True)
 
-        dfs = pd.DataFrame(randint(0, 50, (500, 3)), columns=list('abc'))
-        dfs.groupby(['a']).progress_apply(lambda x: None)
+        series = pd.Series(randint(0, 50, (123,)))
+        res1 = series.progress_apply(lambda x: x + 10)
+        res2 = series.apply(lambda x: x + 10)
+        assert res1.equals(res2)
 
-        our_file.seek(0)
+        res3 = series.progress_map(lambda x: x + 10)
+        res4 = series.map(lambda x: x + 10)
+        assert res3.equals(res4)
 
-        # don't expect final output since no `leave` and
-        # high dynamic `miniters`
-        nexres = '100%|##########|'
-        if nexres in our_file.read():
+        expects = ['100%', '123/123']
+        for exres in expects:
             our_file.seek(0)
-            raise AssertionError("\nDid not expect:\n{0}\nIn:{1}\n".format(
-                nexres, our_file.read()))
+            if our_file.getvalue().count(exres) < 2:
+                our_file.seek(0)
+                raise AssertionError(
+                    "\nExpected:\n{0}\nIn:\n{1}\n".format(
+                        exres + " at least twice.", our_file.read()))
 
 
 @with_setup(pretest, posttest)
-def test_pandas_apply():
-    """Test pandas.DataFrame[.series].progress_apply"""
+def test_pandas_data_frame():
+    """Test pandas.DataFrame.progress_apply and .progress_applymap"""
     try:
         from numpy.random import randint
         import pandas as pd
-    except:
+    except ImportError:
         raise SkipTest
 
     with closing(StringIO()) as our_file:
         tqdm.pandas(file=our_file, leave=True, ascii=True)
-        df = pd.DataFrame(randint(0, 50, (500, 3)))
-        df.progress_apply(lambda x: None)
+        df = pd.DataFrame(randint(0, 50, (100, 200)))
 
-        dfs = pd.DataFrame(randint(0, 50, (500, 3)), columns=list('abc'))
-        dfs.a.progress_apply(lambda x: None)
+        def task_func(x):
+            return x + 1
+
+        # applymap
+        res1 = df.progress_applymap(task_func)
+        res2 = df.applymap(task_func)
+        assert res1.equals(res2)
+
+        # apply
+        for axis in [0, 1]:
+            res3 = df.progress_apply(task_func, axis=axis)
+            res4 = df.apply(task_func, axis=axis)
+            assert res3.equals(res4)
 
         our_file.seek(0)
+        if our_file.read().count('100%') < 3:
+            our_file.seek(0)
+            raise AssertionError("\nExpected:\n{0}\nIn:\n{1}\n".format(
+                '100% at least three times', our_file.read()))
 
-        if our_file.read().count('100%') < 2:
+        # apply_map, apply axis=0, apply axis=1
+        expects = ['20000/20000', '200/200', '100/100']
+        for exres in expects:
             our_file.seek(0)
-            raise AssertionError("\nExpected:\n{0}\nIn:{1}\n".format(
-                '100% at least twice', our_file.read()))
+            if our_file.getvalue().count(exres) < 1:
+                our_file.seek(0)
+                raise AssertionError(
+                    "\nExpected:\n{0}\nIn:\n {1}\n".format(
+                        exres + " at least once.", our_file.read()))
 
 
 @with_setup(pretest, posttest)
-def test_pandas_map():
-    """Test pandas.Series.progress_map"""
+def test_pandas_groupby_apply():
+    """Test pandas.DataFrame.groupby(...).progress_apply"""
     try:
         from numpy.random import randint
         import pandas as pd
-    except:
+    except ImportError:
         raise SkipTest
 
+    with closing(StringIO()) as our_file:
+        tqdm.pandas(file=our_file, leave=False, ascii=True)
+
+        df = pd.DataFrame(randint(0, 50, (500, 3)))
+        df.groupby(0).progress_apply(lambda x: None)
+
+        dfs = pd.DataFrame(randint(0, 50, (500, 3)), columns=list('abc'))
+        dfs.groupby(['a']).progress_apply(lambda x: None)
+
+        our_file.seek(0)
+
+        # don't expect final output since no `leave` and
+        # high dynamic `miniters`
+        nexres = '100%|##########|'
+        if nexres in our_file.read():
+            our_file.seek(0)
+            raise AssertionError("\nDid not expect:\n{0}\nIn:{1}\n".format(
+                nexres, our_file.read()))
+
     with closing(StringIO()) as our_file:
         tqdm.pandas(file=our_file, leave=True, ascii=True)
+
         dfs = pd.DataFrame(randint(0, 50, (500, 3)), columns=list('abc'))
-        dfs.a.progress_map(lambda x: None)
+        dfs.loc[0] = [2, 1, 1]
+        dfs['d'] = 100
+
+        expects = ['500/500', '1/1', '4/4', '2/2']
+        dfs.groupby(dfs.index).progress_apply(lambda x: None)
+        dfs.groupby('d').progress_apply(lambda x: None)
+        dfs.groupby(dfs.columns, axis=1).progress_apply(lambda x: None)
+        dfs.groupby([2, 2, 1, 1], axis=1).progress_apply(lambda x: None)
+
+        our_file.seek(0)
+        if our_file.read().count('100%') < 4:
+            our_file.seek(0)
+            raise AssertionError("\nExpected:\n{0}\nIn:\n{1}\n".format(
+                '100% at least four times', our_file.read()))
 
-        if our_file.getvalue().count('100%') < 1:
-            raise AssertionError("\nExpected:\n{0}\nIn:{1}\n".format(
-                '100% at least twice', our_file.getvalue()))
+        for exres in expects:
+            our_file.seek(0)
+            if our_file.getvalue().count(exres) < 1:
+                our_file.seek(0)
+                raise AssertionError(
+                    "\nExpected:\n{0}\nIn:\n {1}\n".format(
+                        exres + " at least once.", our_file.read()))
 
 
 @with_setup(pretest, posttest)
@@ -83,7 +141,7 @@ def test_pandas_leave():
     try:
         from numpy.random import randint
         import pandas as pd
-    except:
+    except ImportError:
         raise SkipTest
 
     with closing(StringIO()) as our_file:
@@ -93,13 +151,35 @@ def test_pandas_leave():
 
         our_file.seek(0)
 
-        exres = '100%|##########| 101/101'
+        exres = '100%|##########| 100/100'
         if exres not in our_file.read():
             our_file.seek(0)
             raise AssertionError(
                 "\nExpected:\n{0}\nIn:{1}\n".format(exres, our_file.read()))
 
 
+@with_setup(pretest, posttest)
+def test_pandas_apply_args_deprecation():
+    """Test warning info in
+    `pandas.Dataframe(Series).progress_apply(func, *args)`"""
+    try:
+        from numpy.random import randint
+        from tqdm import tqdm_pandas
+        import pandas as pd
+    except ImportError:
+        raise SkipTest
+
+    with closing(StringIO()) as our_file:
+        tqdm_pandas(tqdm(file=our_file, leave=False, ascii=True, ncols=20))
+        df = pd.DataFrame(randint(0, 50, (500, 3)))
+        df.progress_apply(lambda x: None, 1)  # 1 shall cause a warning
+        # Check deprecation message
+        res = our_file.getvalue()
+        assert all([i in res for i in (
+            "TqdmDeprecationWarning", "not supported",
+            "keyword arguments instead")])
+
+
 @with_setup(pretest, posttest)
 def test_pandas_deprecation():
     """Test bar object instance as argument deprecation"""
@@ -107,7 +187,7 @@ def test_pandas_deprecation():
         from numpy.random import randint
         from tqdm import tqdm_pandas
         import pandas as pd
-    except:
+    except ImportError:
         raise SkipTest
 
     with closing(StringIO()) as our_file: