BrandonFors/pandas-github-issues huggingface dataset:
REGR: fix reset_index (Index.insert) regression with custom Index subclasses
State: closed

- [x] closes #47071
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

I added a new test file with a minimal custom Index subclass, as I don't know if it can otherwise be tested with out own indexes (and added a new file since I didn't find existing tests for Index subclasses)

Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Index
description: Related to the Index class or subclasses

Comments:
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 1.4.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 009d0ad155f0a75669a333513c015436325db56f
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #48058: REGR: fix reset_index (Index.insert) regression with custom Index subclasses'
```

4. Push to a named branch:

```
git push YOURFORK 1.4.x:auto-backport-of-pr-48058-on-1.4.x
```

5. Create a PR against branch 1.4.x, I would have named this PR:

> "Backport PR #48058 on branch 1.4.x (REGR: fix reset_index (Index.insert) regression with custom Index subclasses)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
Fixed docstring error with style.py, unknown parameter: {'text_color_threshold'}
State: closed

When you run python` ./scripts/validate_docstrings.py --errors=PR02`, one of the errors is `/home/pandas/pandas/io/formats/style.py:3144:PR02:pandas.io.formats.style.Styler.text_gradient:Unknown parameters {'text_color_threshold'}`

The error I believe was because the function text_gradient was using the doc-string for background gradient, which had text_color_threshold as one of the parameters, so validate doc-string expected text_gradient to also have text_color_threshold, even though that parameter was not intended for text_gradient.

- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Labels:
name: Docs
name: Styler
description: conditional formatting using DataFrame.style

Comments:
this looks good pending the checks and also have you rendered the docs, does it all display OK?
Hi, I don't believe the changes I have made, contributed to the errors in the checks. Could you please clarify?
Thanks @joshuabello2550
DOC: add source link to properties
State: closed

The current version of `linkcode_resolve` is unable to handle properties and ignore them. Hence there are no links to the source code for properties unlike for everything else. This add support to properties I did in https://github.com/geopandas/geopandas/pull/2526#issuecomment-1212926545 which is based on pandas code, so I thought it is worth contributing the enhancement back.

Labels:
name: Docs

Comments:
Sounds reasonable. Could you show a before/after example of a docstring that is positively impacted?
Stuff like `GeoSeries.size` or `GeoDataFrame.axes` currently don't have a `[source]` link shown in the API docs:
<img width="796" alt="Screenshot 2022-08-12 at 20 32 41" src="https://user-images.githubusercontent.com/36797143/184421935-759b7c6d-ebe5-4dc3-bae9-76d0fe307750.png">

Docs rendered in this PR have the link to Github source code as all methods and functions already have.

 
<img width="790" alt="Screenshot 2022-08-12 at 20 33 37" src="https://user-images.githubusercontent.com/36797143/184422056-21f895c5-7803-4591-8076-2b8b04785652.png">

The link for this specific example points [here](https://github.com/pandas-dev/pandas/blob/main/pandas/core/frame.py#L851-L866).

Thanks @martinfleis
WEB: Update list of maintainers and improve inactive maintainers format
State: closed

- Adding @lithomas1 and @mzeitlin11 to the maintainers list in the website (sorry it wasn't done when you were promoted)
- Moving @shoyer, @chris-b1 and @toobaz to inactive, renaming the section "Inactive maintainers", and using the same logic to display the names with links as we have for active maintainers


Labels:
name: Admin
description: Administrative tasks related to the pandas project
name: Web
description: pandas website

Comments:
Thanks @datapythonista
REF: dont alter self in _validate_specification
State: closed

Besides being better-practice, this will make it easier to annotate.  Done separately bc im tracking down what appear to be incorrect annotations that need to be fixed in order to annotate these methods.

Labels:
name: Refactor
description: Internal refactoring of code
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
good here?  follow-up on deck
Thanks @jbrockmendel
ENH: allow user to infer SAS file encoding; add correct encoding names
State: closed

- [x] closes #48048 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Labels:
name: IO SAS
description: SAS: read_sas

Comments:
I have included release information; this would fit into an enhancement section, right?

Additionally: If you think that this issue/PR is worthwile, I'd include all the other [encoding options from the SAS documentation](https://support.sas.com/documentation/onlinedoc/dfdmstudio/2.6/dmpdmsug/Content/dfU_Encodings_SAS.html) as well to be complete with all use cases.
> Additionally: If you think that this issue/PR is worthwile, I'd include all the other [encoding options from the SAS documentation](https://support.sas.com/documentation/onlinedoc/dfdmstudio/2.6/dmpdmsug/Content/dfU_Encodings_SAS.html) as well to be complete with all use cases.

I have now included the other encoding names, though I don't know how to _factually_ test them; I just know that using these within Python's `encode(...)` does not throw an error :)
@mroeschke Is this done in your opinion? :-) The `Data Manager` check seems to have failed for unrelated reasons; do you mind restarting?
> @mroeschke Is this done in your opinion? :-) The `Data Manager` check seems to have failed for unrelated reasons; do you mind restarting?

Would you mind merging in main and moving the release note to `1.6.0.rst`? We are cutting a 1.5.0 fairly soon so we are freezing changes like these into the 1.5 release
Does this not make the 1.5 release because of the SAS7BDAT reader refactoring here (#47403)  (and others)?

But of course, will do once the 1.6.0 doc is merged / started (#48221)
> Does this not make the 1.5 release because of the SAS7BDAT reader refactoring here (#47403) (and others)?
> 
> But of course, will do once the 1.6.0 doc is merged / started (#48221)

The team is releasing the 1.5rc in the next few days so we want to make minimal changes. It may have a change to make it though depending if other team members find it appropriate. 
Have rebased (and uncluttered the PR from the other commits) and moved into the `1.6.0` release notes. ✅ 

Sorry for the delay here. Just needs to resolve the merge conflict otherwise LGTM
@mroeschke It has been done!
Thanks @YYYasin19!
CI: Pin setuptools
State: closed

- [ ] xref #48046 (Replace xxxx with the Github issue number)



Labels:
name: CI
description: Continuous Integration

Comments:
The docker build can not pass on this branch, because we are using the main branch when building the dockerfile instead of the pr branch. Was this the intended behavior?
> The docker build can not pass on this branch, because we are using the main branch when building the dockerfile instead of the pr branch. Was this the intended behavior?

Yeah that build should test that any code change should still be able to build the DockerFile, but the file is supposed to be for local development which should start from main.

The `docker build` command should maybe include a `--build-arg` with the target branch in the GHA
ENH: set_index copy kwd
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Enhancement

Comments:
Thanks @jbrockmendel
TST: GH39984 Addition to tests
State: closed

- [x] closes #39984 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: MultiIndex
name: Arrow
description: pyarrow functionality

Comments:
Hello @BarkotBeyene! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-08-23 19:24:18 UTC
Thanks @BarkotBeyene 
@mroeschke 
Thanks for the comment and all.
CI: Include FutureWarning from numpy in npdev build
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature

xref https://github.com/pandas-dev/pandas/pull/46767#discussion_r849950834

cc @jbrockmendel 


Labels:
name: CI
description: Continuous Integration

Comments:
@jbrockmendel just to confirm, do we want to check `FutureWarning`s that pandas issues or numpy? I don't think I fully understand the use case here yet.
>  just to confirm, do we want to check FutureWarnings that pandas issues or numpy?

pandas

> I don't think I fully understand the use case here yet.

based on test_fillna_datelike:

```
df = DataFrame(
            {
                "Date": [NaT, Timestamp("2014-1-1")],
                "Date2": [Timestamp("2013-1-1"), NaT],
            }
        )

>>> result = df.fillna(value={"Date": df["Date2"]})
<stdin>:1: FutureWarning: In a future version, `df.iloc[:, i] = newvals` will attempt to set the values inplace instead of always setting a new array. To retain the old behavior, use either `df[df.columns[i]] = newvals` or, if columns are non-unique, `df.isetitem(i, newvals)`
```

The warning issued here isn't because the user needs to update their code, but because _we_ haven't updated our internal usage.  i.e. it indicates something we need to change on our end, and shouldn't be shown to the user.  So getting it should fail the CI.
Ah gotcha, that makes sense. I think failing the CI for pandas warning should happen for all builds not just the npdev one so I'll work on adding that.
I'll try to add separate PRs addressing these warnings too (I see some pyarrow ones as well). This PR may be quite large if we tackle them all in this PR. I think running pytest with `-W error:::pandas` should catch them.

This PR though independently should be a positive change; we will also now fail the npdev build if there are `FutureWarning`s from numpy
CLN: Remove mangle_dupe_cols argument
State: closed

- [X] xref #47718
- [X] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [X] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [X] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Addressing #47718 step by step. Here I remove the `mangle_dupe_cols`, which was never implemented. I directly remove it, instead of showing deprecation, because I don't think in practice this should be breaking code. For example, the next code:

```python
pandas.read_csv(fname, mangle_dupe_cols=False)
```
Would raise an exception in all cases. So, I don't think making it showing a warning before raising the exception should be very useful. There could be the case of someone using:

```python
pandas.read_csv(fname, mangle_dupe_cols=True)
```
Where the `mangle_dupe_cols=True` literally does nothing as it's the default. If people were doing this it'd make sense to show a `FutureWarning`, but I don't see any reason why anyone would do that. So, in practice, I think we'd just be overcomplicating things with the deprecation in this case, and I think it's better to simply delete it.


Labels:
name: API Design
name: IO CSV
description: read_csv, to_csv
name: IO Excel
description: read_excel, to_excel
name: Clean

Comments:
We have deprecation in 1.5 for an unused argument in a function, so there's precedence for also adding a deprecation here too https://github.com/pandas-dev/pandas/pull/47728. It would make this PR a little more complicated as you said (change default arg to `lib.no_default` and raise warning if True was passed)

Would this close https://github.com/pandas-dev/pandas/issues/13262 and https://github.com/pandas-dev/pandas/issues/8908 as well?
Thanks for the feedback Matthew. Feels it may be a bit too much conservative in my opinion, but if this is what was agreed and what we've been doing, I'll better add the deprecation then. This should close the issues you mention too, thanks for the reminder.

I'll implement the deprecation in a separate branch, as there is not much that can be reused here. So, closing this PR.
Doc: Series.rename parameter documentation with Tests
State: closed

- [x] closes #46889
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Docs
name: Series
description: Series data structure

Comments:
Hello @Th3nn3ss! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-08-11 14:09:12 UTC
Thanks @Th3nn3ss
TST: catch some test warnings
State: closed

I get a hundred and change warnings locally when I run the test suite, am surprised the npdev build isn't failing on these.  This gets a handful of them.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Warnings
description: Warnings that appear or should be added to pandas

Comments:
Thanks @jbrockmendel (npdev build only raises on DeprecationWarning not FutureWarning)
> npdev build only raises on DeprecationWarning not FutureWarning

Has it always been like this?  I thought we checked that we checked for FutureWarnings to make sure we weren't issuing any accidentally (e.g. ATM we issue some in fillna that we shouldn't bc we are using deprecated usage)
Ah looks like I made the filter more narrow because numpy was propagating a distutils warning too? https://github.com/pandas-dev/pandas/pull/46767
BUG: use uuid4 instead of md5 for fake column in cross merge
State: closed

- [x] closes #48024 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

cc @mroeschke Any idea how to test this? Did not add it for the test, do you think thats necessary?

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
@szelenka, could you confirm using uuid4 would be a sufficient replacement for this use case (if you can test locally)?

From what I am seeing online, I don't see a really a convenient way of testing this in our CI system
@mroeschke yes this seems to work okay:
```
>>> ssl.FIPS_mode_set(0)
>>> ssl.FIPS_mode_set(1)

# Old method
>>> cross_col = f"_cross_{hashlib.md5().hexdigest()}"
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: [digital envelope routines] disabled for fips

# New method
>>> cross_col = f"_cross_{uuid.uuid4()}"
>>> cross_col
'_cross_0a42418d-46db-40c9-8cbf-72d672d9863a'
```
Thanks @phofl and @szelenka for confirming.
ENH: Support masks in groupby prod
State: closed

- [x] xref #37493 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

cc @jorisvandenbossche 

This will have conflicts with #48018, will resolve after this is merged. Gives us the opportunity to simplify tests too

Labels:
name: Groupby
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
Adressed the overflow concerns here to, behaves the same as the sum case now
cc @mroeschke  gentle ping
Thanks @phofl 
Works for me, thx.

i will update the other prs after the ci is green again, makes reviewing easier
DEPR fixup warning message of MultiIndex.lexsort_depth deprecation
State: closed

xref https://github.com/pandas-dev/pandas/pull/38701/files#r942749562

Labels:
name: Deprecate
description: Functionality to remove in pandas
name: Warnings
description: Warnings that appear or should be added to pandas

Comments:
Thanks - TBH I doubt anyone's using this method anyway (searches I did for this in code only brought up the pandas test suite). Unless we've previously been strict about backporting warning messages, I'd suggest not bothering with this one
ENH: change get_dummies default dtype to bool
State: closed

Added a future warning when no dtype is passed to `get_dummies` stating the the default dtype will change to `bool` from `np.uint8`
- [x] Closes #45848 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Awesome, thanks for sticking with this @kianelbo 
FWIW, this does cause some confusion for downstream consumers--for example the common approach of using `pd.get_dummies(pd.cut(...))` to get one-hot encoded data (common for ML applications) might more naturally be expected to continue to return a numeric type, which I believe is also the default for `sklearn.preprocessing.OneHotEncoder`. I actually plucked that approach right out of Wes' book I think.

Well, I'm not really that annoyed minus 90 minutes of debugging, and the fix is trivial for the consuming code (just specify the old default `dtype` to `get_dummies`), but I'll just place this comment here in case it helps others adapt their downstream code.
It also lead to some not obvious errors in statsmodels when testing against the pre-prelease.  It wasn't that hard to change since I know about this change, but would have been hard to determine had one not. When making the changes I half wondered if the default should have become `double` which would have always prevented the math issues that this was desired to fix, albeit at the cost of 8x storage (although one could always choose `bool` if storage was more important than simplicity of use).