BrandonFors/pandas-github-issues huggingface dataset:
Fixes Issues#44132, #40148, #29033, #22275, #18869: groupby
State: closed

- [x] closes #44132
- [x] closes #40148 
- [x] closes #29033
- [x] closes #22275
- [x] closes #18869
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Groupby
name: Stale
name: Closing Candidate
description: May be closeable, needs more eyeballs
name: Nuisance Columns
description: Identifying/Dropping nuisance columns in reductions, groupby.add, DataFrame.apply

Comments:
Hello @ikramersh! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

























There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-11-10 04:53:05 UTC
Of note: it appears you are modifying a lot of existing tests / docs which is a sign that this fix is probably not the right approach. Usually a fix should only address the example in the associated issue.
> Of note: it appears you are modifying a lot of existing tests / docs which is a sign that this fix is probably not the right approach. Usually a fix should only address the example in the associated issue.

I agree and would like some confirmation that the fix is desirable. It is a minute core change with a broad impact. To me it opens up a lot of opportunity because it's more pure Python where anything that can be summed is allowed, for example strings, tupples, lists, dictionaries etc are all included in results now. Previously strings for example were automatically excluded and not expected by the existing tests as results.

Please can I get some direction as to the desirability of this approach?
> -1 on this change
> 
> it is unexpected from a user pov as all operations know ally to numeric columns
> 
> this can easily be done in nested columns with an .apply

That makes sense too but the fix also makes use of the numeric_only=True argument more applicable now as well as the existing warnings that non numeric columns should be filtered out prior to calling numeric operations.

Happy to abort just explaining what I have discovered on this journey.
The reported issue #44132 shows that the inclusion of not only numeric but all summable types in the result of .sum() is already part of dataframe behaviour. The complaint is that behaviour is not carried through after calling groupby() which is what this fix attempts to resolve.
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
I think we already have a deprecation path in place for numeric_only=None, can close this.
closing as stale and https://github.com/pandas-dev/pandas/pull/44142#issuecomment-1001265072 comment
Move moments/test_moments_rolling_apply.py to window/test_apply.py
State: closed

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them

Since some tests are not running on `windows/moments` (https://github.com/pandas-dev/pandas/issues/37535), moving some lighter apply tests to `test_apply.py`


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Window
description: rolling, ewma, expanding

Comments:
Flakey test - restarting CI. https://github.com/pandas-dev/pandas/runs/3981972547?check_suite_focus=true

```
def assert_equal_zip_safe(result: bytes, expected: bytes):
        """
        We would like to assert these are equal, but the 10th and 11th bytes are a
        last-modified timestamp, which in some builds is off-by-one, so we check around
        that.
    
        See https://en.wikipedia.org/wiki/ZIP_(file_format)#File_headers
        """
        assert result[:9] == expected[:9]
>       assert result[11:] == expected[11:]
E       AssertionError: assert b'\x11WS\x83\...0\x00\x00\x00' == b'\x11WS\x83\...0\x00\x00\x00'
E         At index 1589 diff: b'\xd7' != b'\xd6'
E         Use -v to get the full diff
```
BUG: yaml.dump(DataFrame)
State: closed

- [x] closes #42748
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
Looks good overall. I'm not sure if the test is being run in CI, though(I'm not seeing pyyaml in the CI dependencies). 

Also, it would be nice to backport to 1.3.x, but its fine if it can't be.
> Looks good overall. I'm not sure if the test is being run in CI, though(I'm not seeing pyyaml in the CI dependencies).
> 
> Also, it would be nice to backport to 1.3.x, but its fine if it can't be.

Yeah agree, will be being run in the Data Manager Tests (since that just uses environment.yml) https://github.com/pandas-dev/pandas/runs/3969868404?check_suite_focus=true -> but might be better to move elsewhere since these are experimental.. 
thanks @jbrockmendel if you want to attempt to backport ok by me (would involve also moving the note in master).
CI: Pin pre-commit python version
State: closed

- [x] closes #44116

Looks like GH actions/setup-python seems to have randomly jumped from python 3.9.7 to python 3.10.0

Working:
https://github.com/pandas-dev/pandas/runs/3947277622?check_suite_focus=true#step:3:5

Failing:
https://github.com/pandas-dev/pandas/runs/3960125988?check_suite_focus=true#step:3:5

We can pin to a specific python version as per:
https://github.com/actions/setup-python#usage


Needs backport to 1.3.x also failing there to: https://github.com/pandas-dev/pandas/runs/3959049188?check_suite_focus=true

Labels:
name: CI
description: Continuous Integration

Comments:
thanks @alimcmaster1 merge on green
> thanks @alimcmaster1 merge on green

Thanks will do and will backport
REF: simplify EA.astype
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Refactor
description: Internal refactoring of code
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Thanks @jbrockmendel do we need to include any additional tests to the base extension tests for the benefit of 3rd party EA devs?
> do we need to include any additional tests to the base extension tests for the benefit of 3rd party EA devs?

I don't think so, doesnt change what authors are expected to implement

unless they have already implemented their own astype method I assume?
I don't understand the question
I'm thinking that at the moment we don't support astype(EADType) except StringDType (for which IIRC we added tests to the base extension tests) unless we have implemented specifically it in the child classes.

Adding support in the base class for any EADType.astype(EADType) now means that we don't generate errors doing this from the main codebase and therefore all EA subclasses will now need to support this. EA authors that have implemented astype before this requirement may not support this?
OK I see.  I don't think this changes what we expect from EA authors.  I guess we could test that doing an .astype to `PandasDtype[object]` works, Categorical following #43930 would be more meaningful.  Did you have something else in mind?
disambiguate documentation for pandas.Series replace
State: closed

Currently, the `pandas.Series` that is used for the example for `pandas.Series.replace` has an index which is identical to its values (i.e. `pandas.Series([0, 1, 2, 3, 4])`). This can be quite confusing at a first glance (as it was for me), as I assumed that `pandas.Series.replace` was using the index of the series (whereas it is actually using the values). Updated the documentation to use `pandas.Series([1, 2, 3, 4, 5])` to remove this ambiguity.


Labels:
name: Docs

Comments:
Can you merge master code-checks should now be fixed? 
pre commit checks are failing
@jreback @alimcmaster1 apologies for the delay - made a very silly mistake and it took me longer than it should have to track it down 👍 Looks like one of the tests timed-out, but the rest are okay? 
thanks @el-iot
REF: use concat_compat in union_with_duplicates
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Clean

Comments:
wow

any perf implications?
> any perf implications?

we're going to take a small hit bc isinstance(obj, ABCFoo) is slower than isinstance(obj, Foo), and the runtime import of ensure_wrapped_if_datetimelike (i guess could move that to dtypes.cast).  the union cases we'll take a hit in non-EA cases bc concat_compat is slower than np.concatenate, but for EA  cases we ought to do better
> > any perf implications?
> 
> we're going to take a small hit bc isinstance(obj, ABCFoo) is slower than isinstance(obj, Foo), and the runtime import of ensure_wrapped_if_datetimelike (i guess could move that to dtypes.cast). the union cases we'll take a hit in non-EA cases bc concat_compat is slower than np.concatenate, but for EA cases we ought to do better

sounds good about moving that, but this looks fine.
CI: Use Windows and MacOS Numpy Wheels
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: CI
description: Continuous Integration
name: Windows
description: Windows OS
name: OS X
description: Related to Mac OS & hardware issues (M1)

Comments:
ahh this fixed #44126 ?
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
Not really, this really just there to save time(don't compile numpy from source which takes ~5 minutes, and probably makes CI occasionally timeout). It "passes" because numpy nightly wheels are weekly, but will probably fail at the end of this week. 
got it makes sense
BUG: DataFrame index name no longer resets after appending a list of series
State: closed

- [x] closes #44109
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Hello @robbertjan94! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-11-03 21:06:57 UTC
thanks @robbertjan94 very nice!
REF: reuse self.grouper.result_index
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Refactor
description: Internal refactoring of code
name: Groupby

Comments:
do we know where the 3.10 dev fails are coming from? can you open an issue about this
BUG: ignore EAs when counting blocks to raise fragmentation PerformanceWarning
State: closed

Closes #44098


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
sure. the 3.10 build is failing but look unrelated. but pls merge master  anyhow.
nvm these 3.10 are failing all over.
thanks @jorisvandenbossche
TST: Adding test for checking that plotting grid param by default is mpl.rcParams['axes.grid']
State: closed

- [ ] Related to #3188
- [ ] tests added to pandas/tests/plotting/test_style.py and passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] Seems the issue is obsolete. As I can see now the tested behavior is by design with matplotlib backend for visualization, but more tests is better ;)


Labels:

Comments:
Issue #3188 is a duplicate of #9792.
Already tested so this pr is redundant.
DOC: ewm constructor
State: closed

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them


* Made language more consistent in the Parameters
* Organized and added more Examples


Labels:
name: Docs
name: Window
description: rolling, ewma, expanding

Comments:
@mroeschke I would suggest to use this as an opportunity to add an emphasis in the documentation about the difference in behaviour between `.rolling()` and `.ewm()` in their use of the `min_periods` argument default, this is something that has caught me out before:
````python
import pandas as pd

df = pd.DataFrame(range(10), columns=["spot"])

df["rolling"] = df["spot"].rolling(5).mean()
df["ewm"] = df["spot"].ewm(5).mean()
df
````
Output:
````console
   spot  rolling       ewm
0     0      NaN  0.000000
1     1      NaN  0.545455
2     2      NaN  1.120879
3     3      NaN  1.725782
4     4      2.0  2.359493
5     5      3.0  3.021172
6     6      4.0  3.709837
7     7      5.0  4.424377
8     8      6.0  5.163576
9     9      7.0  5.926138
````
@sdementen sure would take a PR for that. I think this should be in the docs themselves with references (but wouldn't object to also in the doc-strings).
> @sdementen sure would take a PR for that. I think this should be in the docs themselves with references (but wouldn't object to also in the doc-strings).

@jreback for what would you take a PR? I am bit lost in the conversation as I can't find back the context from this PR
sorry meant to tag @sdfordham
BUG: PeriodIndex[B].to_timestamp inferring "D" instead of "B".
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Datetime
description: Datetime data dtype
name: Period
description: Period data type

Comments:
looks user facing? needs a note
> looks user facing? needs a note

sure, will update.
> > looks user facing? needs a note
> 
> sure, will update.

ping when ready
ping
hmm are these failures in master?
> hmm are these failures in master?

yes.  i think #44381 handles all but one of them
ok can you rebase just to check then. what is the one that is not handedl?
> what is the one that is not handedl?

theres an sdist one that's failing in 52 seconds
ENH: Negative Reference Percent Changes
State: closed

- [x] closes #xxxx
- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

I recently came across a problem at work that looks like it should be solved in `pct_change`. To make the problem apperent, here is a quote from the [relative change and difference Wikipedia page](https://en.wikipedia.org/wiki/Relative_change_and_difference#Definitions):
> If we are calibrating a thermometer which reads −6 °C when it should read −10 °C, this formula for relative change (which would be called relative error in this application) gives ((−6) − (−10)) / (−10) = 4 / −10 = −0.4, yet the reading is too high To fix this problem we alter the definition of relative change so that it works correctly for all nonzero values of x<sub>reference</sub>:
![alt text](https://wikimedia.org/api/rest_v1/media/math/render/svg/5c819461a566c88ed96dc839638581a875c7fe31)

The example I list in the updated docstrings is also a good example:
> -10 -> 10

Clearly it seems weird to say that the pct change is -200% for this. Dividing by the abs of the reference value fixes this to +200%.

Labels:

Comments:
Hello @HomelessSandwich! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-10-19 20:04:11 UTC
-1 on expanding the API for this
@jreback 
> a keyword to control this doesn't make sense

What would be a more appropriate system to control this? 

> a) this would need tests

Could you direct me to the location of tests I would need to include? I am a bit lost with the process of all this.

> b) does changing the divisor just work?

Yes, the issue is (take the example of -10 -> 10) the divisor's sign affects the sign of percentage change. Taking the absolute fixes that.

Obviously you could just do it yourself like this:
`.apply(lambda x: (x - x.shift()) / abs(x.shift()))`

But I think it boils down to expanding the API, or just keeping it simple and letting people do it themselves.

You can achieve this with `s.diff()/s.shift().abs()`

I agree with the sentiment of just letting users do this

Thanks for the PR, but in the future please discuss new features first by opening an issue
Agreed @MarcoGorelli, thanks for the info!
DOC: Perf notes
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Docs
name: Performance
description: Memory or execution speed performance

Comments:
OT

```
Falsifying example: test_range_difference(
    start1=1, stop1=-3, step1=-1, start2=0, stop2=-3, step2=-2,
)
Traceback (most recent call last):
  File "/home/runner/work/pandas/pandas/pandas/tests/indexes/ranges/test_setops.py", line 472, in test_range_difference
    tm.assert_index_equal(result, alt, exact="equiv")
  File "/home/runner/work/pandas/pandas/pandas/_testing/asserters.py", line 408, in assert_index_equal
    raise_assert_detail(obj, msg, left, right)
  File "/home/runner/work/pandas/pandas/pandas/_testing/asserters.py", line 679, in raise_assert_detail
    raise AssertionError(msg)
AssertionError: Index are different

Index values are different (100.0 %)
[left]:  RangeIndex(start=0, stop=-3, step=-2)
[right]: Int64Index([1, -1], dtype='int64')


You can reproduce this example by temporarily adding @reproduce_failure('6.23.3', b'AAABAQMBAQAAAQMBAg==') as a decorator on your test case
Falsifying example: test_range_difference(
    start1=1, stop1=-3, step1=-1, start2=0, stop2=-2, step2=-1,
)
Traceback (most recent call last):
  File "/home/runner/work/pandas/pandas/pandas/tests/indexes/ranges/test_setops.py", line 463, in test_range_difference
    assert_range_or_not_is_rangelike(result)
  File "/home/runner/work/pandas/pandas/pandas/tests/indexes/ranges/test_setops.py", line 441, in assert_range_or_not_is_rangelike
    assert not (diff == diff[0]).all()
AssertionError: assert not True
 +  where True = <built-in method all of numpy.ndarray object at 0x7f0d01937390>()
 +    where <built-in method all of numpy.ndarray object at 0x7f0d01937390> = Int64Index([-3], dtype='int64') == -3.all
```

looks like hypothesis found an issue :->
> looks like hypothesis found an issue :->

yah just started a branch for this...
testing broadcast on multiindex
State: closed

- [x] closes #34388
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: MultiIndex

Comments:
can you merge master and update to comments
Yes, I was working on other stuff, will work on it now!
@gabrieldi95 can you merge once again and ping on green
thanks @gabrieldi95
BUG: crosstab fails with lists/tuples
State: closed

- [x] closes #44076
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
thanks @rhshadrach
DOC: Fix cheatsheet automatic uploading
State: closed

Follow up of #44018.

I didn't realize before merging that we were first uploading the web directory, and then copying the cheatsheets to it, so they won't be uploaded. The cheatsheets will be removed from the website in the CI of hte previous PR, so would be good to merge this asap, so the cheatsheets are restored.


Labels:
name: Docs
name: CI
description: Continuous Integration

Comments:
Seems like this worked, the cheat sheets were correctly uploaded to the server, but for whatever reason (I guess a memory cache) nginx is still serving the old file. I tried to restart nginx, but I don't think I've got permissions to do it. Let's see if the new file starts being served eventually, or I'll check with NumFOCUS if they can restart nginx.
I think the cache issue may have been on your browser.  I am now getting the latest version of the cheat sheet.  But now https://pandas.pydata.org/docs/ is a mess - the formatting is gone.
![image](https://user-images.githubusercontent.com/15113894/137781625-c269a07d-3ea4-40a4-b293-2ded0078f10e.png)

Thanks for the info. I created #44087, where I guessed the problem is cloudflare or similar. If it works for you, that may be the case, cloudflare usually caches depending on your location.

https://pandas.pydata.org/docs/ renders correctly here, I guess the request to a css file fail in your end. Can you reload and see if it's fixed?
> https://pandas.pydata.org/docs/ renders correctly here, I guess the request to a css file fail in your end. Can you reload and see if it's fixed?

Yes, reloading worked.  Should have tried that.
Ok, and I'm getting the new cheatsheet now too, all good :)