BrandonFors/pandas-github-issues huggingface dataset:
BUG: Fix __ne__ comparison for Categorical
State: closed

- [x] closes #32276
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Categorical
description: Categorical Data Type

Comments:
thanks @dsaxton
TYP: Groupby sum|prod|min|max|first|last methods
State: closed

This makes two things:
* makes mypy know the signatures of the agg methods (``sum``, ``prod``, etc.) on Groupby
* gives the agg methods on Groupby objects a return type

This ensues that mypy and  IDE's always know that the return type of e.g. ``df.groupby('a').sum()`` is a DataFrame, which is nice when chaining.

Labels:
name: Groupby
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
there's got to be a less verbose way of doing this, isnt there?
> why are you not generating these like we do in genetic for the stat functions ?

Doing e.g. ``reveal_type(df.var)``on master gives a revealed type of ``Any``. I don't think it's possible to programatically add methods and also have mypy infer their types, parameters or return values. Mypy isn't advance enough to do that.

It's true that this is a bit repetitive, but always with slight variations. Getting correct parameter and return types is something I value highly in a library as complex as Pandas, so IMO this is worth it.

> I don't think it's possible to programatically add methods and also have mypy infer their types, parameters or return values. Mypy isn't advance enough to do that.

cc @simonjayhawkins is this likely to improve at any point?  would something like stub files help?

I find this pattern annoying at debug-time, when it is an extra layer of calls to look through (since I have to double-check whether the super().foo(...) call is changing any kwargs).

(core.window.expanding is another place where we do something like this, but in that case for docstrings)
FWIW I don't find stepping through the functions as is necessarily any clearer than inheriting as done here. This is more verbose, but I don't think factory-generating these really adds all that much value and I do think would be nice to support annotations on these
> there's got to be a less verbose way of doing this, isnt there?

@topper-123 commit making _Groupby a generic class. https://github.com/pandas-dev/pandas/commit/dccbd00a75f5f39c2979e1970635ce00d26c3a11
@topper-123 still active? I think this is a nice change and helpful when we think about bumping our mypy pin, so would love if there's an easy way to address comments
I've been a bit busy lately. I can see if I can make something work over the coming days.
can you merge master ping on green.
I've updated the PR and removed the return types, so this is only refactoring ``_add_numeric_operations`` and make mypy discover the methods.

So, this does:

```python
>>> meth = df.groupby('x').sum
>>> reveal_type(meth)
Any  # master
Revealed type is 'def (numeric_only: builtins.bool =, min_count: builtins.int =) -> Any'  # this PR
```

The Travis failure looks unrelated.
I've made a new commit that simplifies the PR wrt. the static method.
Superceded by #34200.
REF/TST: misplaced tests in test_timeseries, test_timezones
State: closed


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Clean

Comments:
> pandas/tests/frame/test_timezones.py, pandas/tests/series/test_timeseries.py and pandas/tests/series/test_timezones.py will eventually disappear?

Seems likely
CLN: Using sum instead of looping
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
@momisbestfriend just as a general comment Python idioms may not always be applicable for Cython. 

Changes like this definitely need benchmarks to go with them
I bench marked it (hopefully I did it well)

The test I use to check both is the following:

```python
import numpy as np

from pandas._libs.lib import memory_usage_of_objects

arr = np.random.rand(1, 100)

%timeit memory_usage_of_objects(arr)
```

---

master:
```
In [4]: %timeit memory_usage_of_objects(arr)                                                                  
268 ns ± 16.9 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
```

this PR:
```
In [4]: %timeit memory_usage_of_objects(arr)                                                                  
463 ns ± 9.7 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
```
Yea that's a pretty large slowdown. Still ns which is good and I doubt anyone is calling this function in a tight loop so probably invisible to the end user, but in any case I don't think worth changing.

Thanks in any case!
ENH: Move corrwith from transformation to reduction kernels in groupby.base
State: closed

- [x] closes #31270
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Labels:
name: Groupby

Comments:
this looks correct, but @TomAugspurger if you'd have a look.
What's the user-facing change here? Can you add that to the release note?
> What's the user-facing change here? Can you add that to the release note?

There change doesn't affect users. However this allows us to make subsequent changes that have benefit users.
ping
Thanks @fujiaxiang
TST: Use `sort` fixture in more places
State: closed

- [x] closes #32183
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`



Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Thanks @SaturnFromTitan 

In order to close the issue can you do test_union_noncomparable in pandas\tests\indexes\ranges\test_setops.py as well.

There are several cases of @pytest.mark.parametrize("sort", [True, False]) as well. so having this fixture in top-level conftest maybe confusing.

for instance groupby where the default is True, unlike intersection where the default is False.

so is the None value in the fixture effectively True.

would tests break if the fixture was params=[None, True, False]?
Yes, the tests break when `True` is added to the fixture params:
```
ValueError: The 'sort' keyword only takes the values of None or False; True was passed.
```

I'll move the fixture to pandas/tests/indexes/conftest.py instead and adjust the docstring
@simonjayhawkins I updated the docstring again, please have another look
BUG: groupby with sort=False creates buggy MultiIndex 
State: closed

- [ ] closes #32259 
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

EDIT
----
probably better/less invasive to make sure .groupby(, sort=False) returns an object where the index has codes which reflect the ordering

Labels:

Comments:
can you set the names of PRs to be the similar to the issue
harder than I thought, closing for now, will return to it at later stage
TST: Split and simplify test_value_counts_unique_nunique
State: closed

closes #32205, closes #32220

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`



Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
The failing CI is unrelated to these changes
@jreback merged master to resolve conflicts
thanks @SaturnFromTitan 

one other followup on this. I think we should split out the tests you just changed (e.g. unique, value_counts) into a test_unique.py module; happy to do this generally as well (though should be separate from a refactorting PR).
Will do. I'll wait until #32311 is merged though as I think it should go into the test_uniques.py as well
CI: Temporary fix to the docs build while we fix the ssh problems
State: closed

Looks like the ssh key we have in the settings is not working as expected in #32074.

While we fix it (I don't have access to the settings, so can't check much), this prevents master from failing for the publishing of the dev docs to GitHub pages.

Labels:
name: Docs
name: CI
description: Continuous Integration
name: Web
description: pandas website

Comments:
wait. so the *new* docs are failing but we didn't notice? just revert #32074 
I can revert the failing changes, but it'll be hard to fix the problem if we're not trying to push the docs in the CI. The problem is that github secrets can't be accessed from PRs, so we can't test things using secrets before merging.

Not sure why github actions doesn't notify us when a build fails, but that's a good point, I'll open a ticket and investigate. 
ok fair

merge away then
Thanks @datapythonista do we need/have an open issue for this?
@datapythonista not sure this is working as intended. revert?
@datapythonista nvm we now have green.
Yes, looks like this work as expected. The pushing to GitHub pages still failed, but the CI result is green:
- https://github.com/pandas-dev/pandas/actions?query=event%3Apush
- https://github.com/pandas-dev/pandas/runs/472871471?check_suite_focus=true#step:14:11

I created #32301 for the problem with the notifications. @jreback feel free to edit if that's not exactly what we should have. I also created #32303 for the original problem with the ssh key.
CLN: Use defaultdict for minor optimization
State: closed

Edit `_from_nested_dict()` by using defaultdict for optimization in performance

- [ ] closes #32209 
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Performance
description: Memory or execution speed performance

Comments:
@WillAyd @jreback Should this fix the problem? Thanks again for your input, and apologies for achieving this in such a roundabout fashion. 
Yea we typically ask that you keep updates to the change all in one PR, but OK as is 
this is ok

pls add labels
> Yea we typically ask that you keep updates to the change all in one PR, but OK as is

@WillAyd I will make sure to follow that guideline for any future PRs. Thanks for the understanding.
thanks @jaketae
TST: implement test_first
State: closed

cc @MomIsBestFriend there are three things about these tests that I think would make for good (separate) follow-ups if you're interested:

1) They are just begging to be parametrized
2) We don't have a systematic way of naming/organizing tests that are for two specific methods rather than one specific method, which is the idea behind tests.(frame|series).methods.  (see also: head/tail, first_valid_index/last_valid_index)
3) These tests could/should also be shared/parametrized over Series vs DataFrame, but we dont have a systematic home or naming convention for these.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Clean

Comments:
@jbrockmendel I would love to open a PR parameterising these tests.

Choosing a naming convention seems too difficult for me, tbh
Thanks @jbrockmendel I think naming the module test_first as you originally did is more consistent. Maybe we could have a test_last.py file which is just a comment that directs the contributor/reader to test_first.py
TST: method-specific files for droplevel
State: closed


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Maybe missed on a previous PR but is the plan to do a separate module for every method? I don’t think its current location is necessarily problematic so just curious of the motivation for this
> but is the plan to do a separate module for every method?

Not quite.  Some criteria that make a method a candidate for this treatment:

- is X a top-level method that isnt relied on by any/many other methods?
  - e.g. #32274 I could easily go months at a time without remembering that `first` or `last` exist
- is it easy to grep to determine that we have collected all tests for X?
   - e.g. `first_valid_index`
- is it not-obvious where tests for X would go?
   - e.g. `DataFrame.insert` tests could go in a number of places based on filenames in tests/frame/
- does X _not_ fit into one of the better-defined buckets of arithmetic/indexing/reductions/cumulative?
- are the tests for X candidates to be shared between Series/DataFrame
   - e.g. `droplevel`, `to_timestamp`, `to_period`, see also discussion in #32274.
- is X a method that exists on multiple classes where we can't necessary share tests, but it is helpful to have a parallel test structure
   - e.g. searchsorted

Hope this helps.
thanks @jbrockmendel 

@WillAyd as a general rule having more fine-grained test files is a goal.
TST: broken off from #32187
State: closed

troubleshooting an apparently-unrelated Travis failure there

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
@simonjayhawkins if you're interested in a change of pace from test organization, the Travis failure here is a mystery ive had no luck resolving.  any ideas?
hey neat!  any guesses why using the fixture instead of the generator worked?
I guess just changed the order/randomness of the tests. I couldn't reproduce the failure locally, but we had a similar issue once before.
thanks
PERF: pass through to numpy validation for iloc setitem
State: closed

We lose a little bit of ground on the range and slice (not really sure why), pick up a bigger amount of ground on list or ndarray.

```
In [3]: ser = pd.Series(range(10**5))                                                                                      
In [4]: key = range(100, 200) 
In [5]: key2 = list(key)
In [6]: key3 = slice(100, 200)                                                                                             
In [7]: key4 = np.array(key2)                                                                                                 

In [16]: %timeit ser.iloc[key] = 1
56.3 µs ± 1.19 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)  # <-- master
62.9 µs ± 1.77 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)  # <-- PR

In [17]: %timeit ser.iloc[key2] = 1
95.6 µs ± 2.69 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)  # <-- master
55.6 µs ± 922 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)  # <-- PR

In [20]: %timeit ser.iloc[key3] = 1
49 µs ± 756 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)  # <-- master
50.6 µs ± 1.01 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)  # <-- PR

In [21]: %timeit ser.iloc[key4] = 1
71.6 µs ± 1.98 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)  # <-- master
45.7 µs ± 427 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)  # <-- PR
```

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Performance
description: Memory or execution speed performance

Comments:
Removed the tuple length validation, picked up on the order of a microsecond compared to the posts from the OP.
thanks
Implement __array__ on ExtensionIndex
State: closed


Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: Index
description: Related to the Index class or subclasses

Comments:
looks ok. @TomAugspurger if you'd have a look.
do we have an issue for adding ``__array_ufunc__`` to index? 
Thanks @jbrockmendel
(wip) BUG: itertuples was retuning empty list if frame had no columns
State: closed

(currently not right)

- [ ] closes #25408 
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
Hmm I guess does this make any practical difference? I’m not sure that this is necessarily more correct than what is happening now
I don't know that this PR has any traction so let's close for now to keep the queue down, but can pick back up if something you want to move forward for sure
REF: method-specific file for to_timestamp
State: closed

cc @MomIsBestFriend this is the only test we have for DataFrame.to_timestamp, but it was apparently written before the convention of "small, focused tests" took hold.  It would be helpful if this could be split/modernized.

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
thanks, love to have split big tests when you can. maybe a way to run a linter to see what are long-ish tests
BUG: Fixed bug, where pandas._libs.lib.maybe_convert_objects function improperly handled arrays with bools and NaNs
State: closed

- [x] closes #32146
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
Hello @AnnaDaglis! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-03-04 13:58:45 UTC
@jreback All tests passed.
@TomAugspurger @jreback All tests passed.
Ok, let's merge this. Having extra tests probably isn't a bad thing :)

Thanks @AnnaDaglis, and congrats on your first PR to pandas!
Thanks @AnnaDaglis !
CI: troubleshoot 32bit build
State: closed

closes #32229

Labels:
name: CI
description: Continuous Integration

Comments:
cc @jreback @WillAyd looks like this fixed azure, travis failure is orthogonal (and i think intermittent)
cc @jreback @WillAyd looks like this fixed azure, travis failure is orthogonal (and i think intermittent)
@jbrockmendel does this close #32229?
I think so, yes
Merging to get the CI going, can revert later if there are objections.
travis: enable bionic & multi-cpu testing
State: closed

- [x] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: CI
description: Continuous Integration

Comments:
what does this actually do? (the arch directive)
> what does this actually do? (the arch directive)

spins up Ubuntu LXD containers on arm64, s390x (big endian, 64bit mainframe), ppc64el (POWER8 6bit little endian), amd64 metal and lets one build & run code on those architectures, thanks to hardware that Travis got from partners.
@jreback  you can see from https://travis-ci.org/pandas-dev/pandas/builds/654862395?utm_source=github_status&utm_medium=notification that all tests on non-x86 are currently failing.

Similarly this is the reason we can't update pandas in Ubuntu 20.04 LTS for the release in april at the moment - https://launchpad.net/ubuntu/+source/pandas/0.25.3+dfsg-5
@xnox we also don't have the capacity to actually have multi arch on *every* build.
@xnox is there also infrastructure for building wheels on these platforms?
@WillAyd I had a very good experience with https://github.com/joerick/cibuildwheel
we would *only* need a single build on each arch, need a PR more like: https://github.com/pandas-dev/pandas/pull/30641
Added message to pytest raises for test_constructor_dict
State: closed

- [x] ref #30999 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x]  added message to pytest.raises for test_constructor_dict


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Thanks @simonjayhawkins  @MarcoGorelli  :)

Merged and pushed.
thanks @DominicG88
Fix exception causes in 14 modules
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
This covers around half of the project. We could do the rest later.
Any idea why some checks are failing? I'm seeing "ImportError: No module named conda.cli" but I don't see how that's related to my PR.
> Any idea why some checks are failing? I'm seeing "ImportError: No module named conda.cli" but I don't see how that's related to my PR.

failing CI should be fixed now. can you merge master.
thanks @cool-RR 

in theory we could have a code_checks rule that fails if we *don't* have ```from err```, but might be non-trivial to do via bash, maybe a linter could do this? (flake plugin)
REF/TST: method-specific files for Series timeseries methods
State: closed

cc @WillAyd @simonjayhawkins got the [go ahead](https://github.com/pandas-dev/pandas/pull/32110#issuecomment-589974610) the other day to take these off Jeff's plate

Labels:
name: Refactor
description: Internal refactoring of code
name: Testing
description: pandas testing functions or related to the test suite

Comments:
@jbrockmendel running `pytest pandas/tests/series/`

this PR
=========== 2620 passed, 23 skipped, 7 xfailed, 1 warning in 19.32s ===========
master 
=========== 2617 passed, 23 skipped, 7 xfailed, 1 warning in 22.35s ===========


test_localized_at_time_between_time gets split into test_localized_at_time and test_localize_between_time.  Since that is parametrized over two values, that accounts for 2 extra tests.  not clear where the third is coming from.  ill see if i can narrow it down
The third extra test was a test_between in test_period.py that was left behind, just updated.
Thanks @jbrockmendel
BUG: Fix DateFrameGroupBy.mean error for Int64 dtype
State: closed

- [x] closes #32219
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

It looks like this was due to the `TypeError` not being caught

Labels:
name: Groupby

Comments:
also merge master
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.0.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 9e7cb7c102655d0ba92d2561c178da9254d5cef5
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #32223: BUG: Fix DateFrameGroupBy.mean error for Int64 dtype'
```

4. Push to a named branch :

```
git push YOURFORK 1.0.x:auto-backport-of-pr-32223-on-1.0.x
```

5. Create a PR against branch 1.0.x, I would have named this PR:

> "Backport PR #32223 on branch 1.0.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
thanks @dsaxton 
@dsaxton if you can try a manual backport would be great :->
Thanks @dsaxton. FYI, in the future you can remove the "still needs manual backport" label after the backport is merged.