BrandonFors/pandas-github-issues huggingface dataset:
BUG: rowLabels and colLabels args in pandas.plotting.table have no effect (#37811)
State: closed

- [ ] closes #37811
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] Pass `rowLabels` and `colLabels` args to `plot_backend.table`. Docs improved: the descriptions of `rowLables` and `colLabels` added to their own entries in the docstring.


Labels:
name: Visualization
description: plotting
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
closing as stale. if you want to continue, pls ping and can re-open.
Its a shame that such a trivial fix to a modestly common bug stalled out like this, one that just bit me hard today. @EvsanDlg if you don't object, I'd be willing to add a test, cherrypick `4fb57b2` as a starting point (so you get commit credit), and add types to everything plus any other housekeeping (changelog entry, etc), presumably in a new PR. 

A few pandas-specific questions on the finer details of your conventions and style:

@ivanovmg @jreback Is there a reason `test_table` uses the `_check_plot_works` helper for `DataFrame.plot(table=True)` but not `pandas.plotting.table` (as @EvsanDlg demonstrates in their example)? Also, just to confirm since I couldn't find it in the docs, is `unittest.mock.patch` the canonical pandas way of mocking/patching (vs. e.g. monkeypath, pytest-mocker, etc), and from `from unittest.mock import patch` the standard pandas convention for importing it (vs import `unittest.mock` and referring to it explicitly)?

Also, @jreback should this get a versionchanged marker, so users understand these arguments are non-functional in prior versions of pandas? If so, any particular guidance on phrasing? Thanks!
@CAM-Gerlach, ``_check_plot_works`` does more than simply run a plotting function: it ensures that the figure is closed afterwards, that it is saveable and that the returned object is of the correct type.

Since my previous comment here on the use of ``unittest.mock.patch`` I attempted to submit a PR, where I used mocking. It got rejected by the CI. If I remember correctly, it seemed that the CI did not like the use of ``patch`` (I'm not sure whether it was because of importing it in particular way, or just ``patch`` was banned, period).

Throughout the code base I saw the use of ``monkeypatch`` in multiple places, though.
Thus, my comment on ``unittest.mock.patch`` should be reconsidered.
I am not sure if using ``monkeypatch`` one can check whether ``matplotlib.pyplot.table`` was called correctly.
TST: parametrize test_info
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Precursor for #37320
- Parametrize tests
- Rename some of test functions


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
Can anyone suggest on dealing with pre-commit check?
It complains that I used non-standard imports. In particular I imported fixtures from ``pandas/conftest.py``.
Without this import the parametrization I used will not work (import error).
@jreback @jbrockmendel 
I think we need to abandon the idea of using the indirect fixture parametrization
> 
> 
> I think we need to abandon the idea of using the indirect fixture parametrization

So I should go ahead and split ``test_info_smoke_test`` into four different test functions (for ``int_frame``, ``float_frame``, ``datatime_frame`` and ``duplicate_columns_frame``)? Much like it was there before.
bottom line i dont think this is worth futzing over all that much.  if you really want, you could try something like

```
@pytest.mark.parametrize("tmfunc", [tm.getSeriesData, tm.getTimeSeriesData, lambda: tm.getSeriesData()).astype("int64"),  duplicate_columns_frame]
def test_thing(tmfunc):
    frame = DataFrame(tmfunc())
    [...]
```
It's greenish.
thanks @ivanovmg
CI: DataFrame.apply(np.any)
State: closed

Should fix these failures https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=47831&view=logs&j=f016abb9-7827-5fa2-935a-22bd9b1477b6&t=c86edbe1-8c1d-5e5c-5b6f-d970fa4acf6d

Labels:
name: CI
description: Continuous Integration

Comments:
great merge on greenish
ENH: Support groupby.ewm operations
State: closed

- [x] closes #16037
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Additionally enables a Numba engine for `groupby(...).ewm(...).mean(...)`


```
In [1]: df = pd.DataFrame({"A": range(10_000), "B": range(10_000)})

In [2]: gb_ewm = df.groupby("A").ewm(com=1.0)

--cache time first
In [3]: %timeit -r 1 -n 1 gb_ewm.mean(engine='numba')
1.02 s ± 0 ns per loop (mean ± std. dev. of 1 run, 1 loop each)

In [4]: %timeit gb_ewm.mean(engine='numba')
578 ms ± 28.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [5]: %timeit df.groupby('A').apply(lambda x: x.ewm(com=1.0).mean())
3.43 s ± 208 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [6]: %timeit gb_ewm.mean(engine='cython')
4.19 s ± 204 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

Labels:
name: Enhancement
name: Window
description: rolling, ewma, expanding

Comments:
Hello @mroeschke! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

























There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-11-17 21:58:52 UTC
can you show a timeit result of the new groupby.ewm vs groupby.apply(.....ewm) for posterity in the top of this PR
Posted the timings. I think there is a slight slowdown using the cython engine with the convenience wrapper since we have to reconstruct the EWM object in the apply loop.
what is the timing for master (before this PR for the groupby apply)

i would have expected the numba one to be way after here
Updated the snippet above; the numba timing including a caching run which skewed the benchmark.

The groupby apply timing is similar on master.
thanks @mroeschke really nice
DEPR: Index.asi8
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Deprecate
description: Functionality to remove in pandas
name: Index
description: Related to the Index class or subclasses

Comments:
(whatsnew conflict)
ok resolved, will merge on greenish
thanks @jbrockmendel
BUG in reindex raises IndexingError when df is empty sometimes
State: closed

- [x] closes #27315
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Build on top of #27326

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
thanks @phofl
ENH: Add support for min_count keyword for Resample and Groupby functions
State: closed

- [x] closes #37768
- [x] closes #37821
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

This would add support for the documented keyword ```min_count``. If we decide to move forward with this, we have to decide what to do in the following case:
```
df = DataFrame({"a": [1] * 3, "b": [np.nan] * 3})
result = df.groupby("a").func(min_count=0)
```

``first`` and ``last`` return ``0.0`` which is fine I think and consistent with ``sum``, but ``min`` returns ``inf`` while ``max`` returns ``-inf``.

Default parameter is set to ``1`` to avoid backwards incompatible changes. We should keep this at least until 2.0

If we decide to not implement ``min_count`` here, we should adjust the docs and the function signatures for the groupby functions.

cc @rhshadrach 

Labels:
name: Groupby
name: Resample
description: resample method

Comments:
Thanks for the PR @phofl. It seems to me that first, last, min, and max should all return nan even when `min_count` is 0. Sum behaves differently because the empty sum is 0 (likewise, the empty product is 1).
> Default parameter is set to 1 to avoid backwards incompatible changes. We should keep this at least until 2.0

Do you perhaps mean -1? Otherwise, I don't follow.
My 2 cents: min_count=0 is the same as not specifying a min_count. Min_count of x says you need to have at least x non-nan values in the bin to show a result, otherwise show nan. If you have a min_count of 0 isnt that like saying show me the sum, first etc regardless how many nans are there? Maybe I am missing something? 

But while it is important to decide what min_count=0 should show it is less important for the implementation, you would not specify that imo, because as mentioned it is like not specifying it. I am ok with any decision.
cc @rhshadrach  When we want to return nan in this case, we can keep -1 as default.
lgtm. ping on green (resolved conflict)
@jreback greenish. Failure is unrelated
thanks @phofl
ENH/POC: ExtensionIndex for arbitrary EAs
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Currently 4 tests failing locally bc lib.infer_dtype raises on DecimalArray and JsonArray (xref #37367)

Not _remotely_ enough testing.

cc @jorisvandenbossche 

Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: Index
description: Related to the Index class or subclasses

Comments:
As I have explained before (https://github.com/pandas-dev/pandas/pull/34159), I am not a fan of exposing an ExtensionIndex class (IMO it's bad user experience + it's not needed). So can you motivate this PR a bit more compared to https://github.com/pandas-dev/pandas/pull/34159
> As I have explained before (#34159), I am not a fan of exposing an ExtensionIndex class (IMO it's bad user experience + it's not needed). So can you motivate this PR a bit more compared to #34159

I put the question back to you @jorisvandenbossche why would you NOT have an EA Index Class? its makes so much sense
@jorisvandenbossche I'm aware of your preference.  For now I'm just trying to get one fully-working implementation and figure out what testing we need.
mothballing pending #37367
rebased + reopened, but there are a few new test failures locally i still need to track down
From test logs https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=54246&view=logs&j=f016abb9-7827-5fa2-935a-22bd9b1477b6&t=c86edbe1-8c1d-5e5c-5b6f-d970fa4acf6d&l=428

```
self = <ArrowStringArray>
['A', 'B', 'C']
Length: 3, dtype: arrow_string
item = 0

    def __getitem__(self, item: Any) -> Any:
        [...]
    
        # We are not an array indexer, so maybe e.g. a slice or integer
        # indexer. We dispatch to pyarrow.
>       value = self._data[item]

pandas/core/arrays/string_arrow.py:324: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   TypeError: key must either be a slice or integer

pyarrow/table.pxi:164: TypeError
```

This looks like the key is an int, but arrow seems to think otherwise.  Thoughts @jorisvandenbossche ?
The problem is that `item` is a numpy.int64 scalar, and not a python int. That's a bug in pyarrow, though (it should work for both), and also already fixed in pyarrow 2.0+ (https://issues.apache.org/jira/browse/ARROW-9879), but so we will need to work around it for now (or set pyarrow 2.0 as the minimum required version for the string array, now it is 1.0)
Makes sense, thanks.  im guessing for this POC your suggestion would be requiring 2.0 for these particular tests?
re-mothballing to clear the queue.  needs signficant work in adding tests.
REF: prepare dataframe info for series info
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Precursor for #37320

Refactor ``pandas/io/formats/info.py`` to simplify further introduction of ``Series.info``.
Basically what is done here:
 - Move docstring to ``BaseInfo`` from ``DataFrameInfo`` - supposedly will be shared with ``SeriesInfo``
 - Move ``col_count``, ``ids`` from ``BaseInfo`` to ``DataFrameInfo`` (will not be relevant to ``SeriesInfo``)
 - Move ``memory_usage_bytes`` to ``DataFrameInfo`` (will have different implementation for ``SeriesInfo``)
 - Extract ``InfoPrinterAbstract``. Concrete classes will differ by the ``_create_table_builder``
 - Move common methods and properties from ``DataFrameTableBuilder`` to ``TableBuilderAbstract``
 - Extract ``TableBuilderVerboseMixin`` from ``DataFrameTableBuilderVerbose`` as the generator functions there will be used for verbose series info builders

I suggest moving the substitution decorator for the docstring from ``pandas/core/frame.py`` to ``pandas/io/formats/info.py`` later on in a separate PR.

Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: Clean

Comments:
Got some irrelevant error in CI Check.

```
mypy --version
mypy 0.782
Performing static analysis using mypy
pandas/core/indexes/datetimelike.py:776: error: Argument 1 to "_simple_new" of "DatetimeIndexOpsMixin" has incompatible type "Union[ExtensionArray, Any]"; expected "Union[DatetimeArray, TimedeltaArray, PeriodArray]"  [arg-type]
Found 1 error in 1 file (checked 1119 source files)
```
> Got some irrelevant error in CI Check.

fixed on master
ENH: Implement cross method for Merge Operations
State: closed

- [x] closes #5401
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

This is a first draft of a ``cross`` method for merge operations. As suggested by @jreback I added a new column and dispatched to ``inner`` as a first naive implementation. I am currently wondering, if there would be a better place to adjust the arguments than the places I selected. I wanted to avoid to modify ``self`` in some method called within the constructor, so i modified the inputs before adding the parameters to ``self``. This currently does not support the ``cross`` method for join. If we get a consensus where to put the modifications, I would add support for ``join``. Also have to add more tests probably and something to the userguide


Have to look into the performance afterwards. Copies of the Left and Right frames may be a big performance hit for bigger frames?

Labels:
name: Enhancement
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Thx for the feedback, will start to implement the changes in the evening probably
Hello @phofl! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:











There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-11-25 22:31:42 UTC
Sorry for the confusion. I did not get what you meant with *assert is* for the unmodified input frames, if you meant a regular assert_frame_equal my I was just confused. Thanks for the clarification
I added a few tests covering mixed dtypes, nulls, more columns and different lengths
@jreback greenish. Failure unrelated
Good point, deleted the column before. Now we are raising a ``MergeError``
ok i think your version before the last commit is good. ping on green.
thanks @phofl very nice!
Great job, and thanks for implementing this! I've given your solution the advertisement it deserves on [stack](https://stackoverflow.com/a/65017552/4909087). 

Excited to see this in 1.2.
@phofl created an issue for benchmarks 

it might make sense to change to use a broadcasting impl (like refs on SO) or use pandas cartesian product function - if the benchmarks show this - can create an issue or better to do benchmarks first
@jreback If you are referring to asvs, I have added two benchmarks through this pr
oh right u did! ok
Will try to look into the numpy or cartesian function when I've got a bit more time uninterrupted
CLN/TYP: pandas/io/formats/excel.py
State: closed

- [ ] xref #37715
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

- Handle two type ignore items in ``pandas/io/formats/excel.py``
- De-duplicate ``has_aliases`` variable by extracting property
- Add typing for iterators yielding ``ExcelCell``
- Use named arguments for ``ExcelCell`` on some occasions (in two places de-duplicate by updating kwargs inplace)

Labels:
name: IO Excel
description: read_excel, to_excel
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
It's green.
lgtm. @simonjayhawkins generally these typing PRs will defer to you so merge when ready.
@simonjayhawkins is there anything else expected from me on this PR?
thanks @ivanovmg
BUG: Prevent Series.isin from unwantedly casting isin values from float to integer (GH21804)
State: closed

- [x] closes #21804 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
Ran some perf benchmarks for tests that were troublesome last time (#21893): `series_methods.IsIn.time_isin`, `categoricals.Isin.time_isin_categorical` and `indexing.MultiIndexing.time_index_slice` did not significantly change

Will run entire test suite once we agree on the imp, since the run takes quite some time.
note that this is duplicating: https://github.com/pandas-dev/pandas/pull/37770 (though that has issues as well)
CLN: use putmask_simple instead of putmask
State: closed


Labels:
name: Refactor
description: Internal refactoring of code
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
looks fine, i assume this doesn't have any perf differences as no tests broke (so for example we are not coercing to object any differnt than current).
> (so for example we are not coercing to object any differnt than current).

definitely not, the point of this is to use the method that _never_ coerces
> > (so for example we are not coercing to object any differnt than current).
> 
> definitely not, the point of this is to use the method that _never_ coerces

sure, benchmarks should show that something unintential didn't change
am sure you are keeping track of perf
CLN: remove something.xlsx
State: closed

`test_register_writer` in `pandas/tests/io/excel/test_writers.py` wasn't expecting that `something.xls` and `something.xlsx` will ever be created.

Since #37639, `ExcelWriter` opens the file which will create it.

Labels:

Comments:
LGTM ping on green
@jbrockmendel green'ish
thanks @twoertwein
DEP: update Versioneer
State: closed

bump versioneer from 0.15 to 0.19

```
(pandas-dev) fangchenli@Fangchens-MacBook-Pro pandas-fangchenli % python setup.py version
running version
keywords are unexpanded, not using
got version from VCS {'version': '1.2.0.dev0+1185.g82894a4b6', 'full-revisionid': '82894a4b6329b8af74bb9c593858453a01b91bf6', 'dirty': False, 'error': None, 'date': '2020-11-14T16:24:02-0600'}
Version: 1.2.0.dev0+1185.g82894a4b6
 full-revisionid: 82894a4b6329b8af74bb9c593858453a01b91bf6
 dirty: False
 date: 2020-11-14T16:24:02-0600
```

Labels:
name: Dependencies
description: Required and optional dependencies

Comments:
i suppose this doesn't really change much.
thanks @fangchenli
CLN: Another unused script
State: closed

xref https://github.com/pandas-dev/pandas/pull/37825

Another script that looks unused and undocumented


Labels:
name: Windows
description: Windows OS
name: Clean

Comments:
> LGTM, maybe should get an opinion from a windows user

@simonjayhawkins maybe?
i tend to use git bash as the terminal rather than anaconda prompt so never use this (or test_fast.bat). Have just tested in anaconda prompt and the scripts both work.

if the scripts are undocumented then I guess no harm in removing.
I kept the `test_fast.*` scripts since they were documented in `contributing.rst`. I'll move forward with removing this one, thanks
making namespace usage more consistent
State: closed

- [x] closes #37838
- [x] tests added / passed
- [x] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
small comment, ping on green
I've made the suggested changes. 
thanks @christopherhadley
Backport PR #37787 on branch 1.1.x (Fix regression for loc and __setitem__ when one-dimensional tuple was given for MultiIndex)
State: closed

Backport #37787

``frame_or_series`` not know here.

cc @simonjayhawkins 

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: MultiIndex

Comments:
Thanks @phofl
Updated script to check inconsistent pandas namespace
State: closed

- [ ] closes #37838 
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Updates the script for inconsistent namespace usage. To be merged after all the errors are fixed with the updated script. The script is described here : https://github.com/MarcoGorelli/PyDataGlobal2020-sprint/issues/1 by @MarcoGorelli 

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Hello @mohdkashif93! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

















There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-11-18 17:28:47 UTC
we will likely need to merge master to make sure after some other merges (but wait till this is green)
can you merge master and see if you can fixup and remaining issues ( pre-commits are failing). ping on green.
this is pending on #37852 , once that's in I'd expect this to pass
Hi @mohdkashif93 - can you try fetching and merging upstream/master now please and see if this passes? If not, do you want to fix up the remaining files as well?
yep this needs a rebase and merge on green.
Yes @MarcoGorelli , I will rebase and check if there are any pending errors with the script 😄 
can you merge master
@jreback I merged master and fixed up the few remaining files - good to go?
great thanks @mohdkashif93
DOC: add unqiue value counts example to groupyby guide
State: closed

Co-authored-by: Abdulelah Almesfer <28743265+abdulelahsm@users.noreply.github.com>
Co-authoerd-by: Abdulellah Alnumay <33042538+Abo7atm@users.noreply.github.com>

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Adding an example to the groupby user guide. The example showcases the usage of ```nunique``` aggregate function. This is from the PyData Global sprint, from [this issue](https://github.com/MarcoGorelli/PyDataGlobal2020-sprint/issues/5). The example is taken from [StackOverflow](https://stackoverflow.com/questions/15411158/pandas-countdistinct-equivalent).

Labels:
name: Docs
name: Groupby

Comments:
![Screen Shot 2020-11-15 at 18 40 51](https://user-images.githubusercontent.com/33042538/99189442-38585480-2772-11eb-9ad1-505dd028addc.png)

@jreback done 👍
can you merge master @Abo7atm the precommit checks are failing
@jreback merged master and ran precommit locally with `pre-commit run --files doc/source/user_guide/groupby.rst` and everything passed. 
thanks @Abo7atm