BrandonFors/pandas-github-issues huggingface dataset:
BUG: Fix bug in pd.eval when UnaryOp in function
State: closed

- [x] closes #46471
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

I ran into a similar issue, and added it in test. They both produce `AttributeError: 'UnaryOp' object has no attribute 'value'`.

Labels:
name: expressions
description: pd.eval, query

Comments:
> Could you add a whatsnew not in `1.6.0.rst` (in `Conversion` section)?

Added.
Thanks @yuanx749
TYP: type all arguments with str default values
State: closed

After this PR autotyping will gently nudge people to add type annotations for arguments with a string default value.

This PR is bigger than expected - there were many untyped arguments that have a string default value.

Typing arguments that have default values is important, as pyright assumes that untyped arguments that have default values are of their default value type (except for None).

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
> After this PR autotyping will gently nudge people to add type annotations for arguments with a string default value.

The suggested type by autotyping might not be correct: an argument might accept only specific strings (`Literal[...]`) or might also accept other types.  If the type is incorrect, the PR author who adds a new argument should know how to adjust the type if needed (much easier than someone else trying to add annotations for code they are not 100% sure about).

My goal is to also enable autotyping's suggestion for `int` and `bool` but that requires first annotating all arguments with int/bool default values.
Looked fairly good. Just a merge conflict
some network timeouts on windows and mac but otherwise all green
Thanks @twoertwein!
SAS7BDAT parser: Speed up blank_missing
State: closed

Move `blank_missing` to Cython.

ASV:

```
# Old
io.sas.SAS.time_read_sas7bdat		10.6±0.05ms

# New
io.sas.SAS.time_read_sas7bdat		5.47±0.02ms
```

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Performance
description: Memory or execution speed performance
name: IO SAS
description: SAS: read_sas

Comments:
Thanks @jonashaag
DOC: Add SeriesGroupBy and DataFrameGroupBy reference pages
State: closed

- [x] closes #48399  (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

I grepped py / rst files for any references to `GroupBy.`:

<details>
<summary>grep results</summary>

    grep -rl --include \*.rst --include \*.py "[^se]GroupBy\."

pandas/core/sample.py
pandas/core/resample.py
pandas/core/apply.py
pandas/core/groupby/grouper.py
pandas/core/groupby/groupby.py
pandas/core/groupby/indexing.py
pandas/tests/plotting/test_groupby.py
pandas/tests/groupby/test_indexing.py
pandas/tests/groupby/test_pipe.py
doc/source/whatsnew/v1.0.1.rst
doc/source/whatsnew/v0.18.0.rst
doc/source/whatsnew/v0.25.2.rst
doc/source/whatsnew/v1.0.4.rst
doc/source/whatsnew/v0.21.0.rst
doc/source/whatsnew/v0.14.0.rst
doc/source/whatsnew/v1.4.0.rst
doc/source/whatsnew/v0.20.0.rst
doc/source/whatsnew/v1.3.3.rst
doc/source/whatsnew/v1.5.0.rst
doc/source/whatsnew/v0.24.2.rst
doc/source/whatsnew/v1.1.5.rst
doc/source/whatsnew/v1.2.1.rst
doc/source/whatsnew/v0.16.1.rst
doc/source/whatsnew/v1.1.4.rst
doc/source/whatsnew/v0.16.2.rst
doc/source/whatsnew/v0.20.2.rst
doc/source/whatsnew/v0.23.1.rst
doc/source/whatsnew/v1.3.0.rst
doc/source/whatsnew/v1.0.0.rst
doc/source/whatsnew/v0.25.0.rst
doc/source/whatsnew/v0.24.0.rst
doc/source/whatsnew/v1.1.0.rst
doc/source/whatsnew/v0.25.1.rst
doc/source/whatsnew/v1.3.5.rst
doc/source/whatsnew/v1.3.4.rst
doc/source/whatsnew/v1.6.0.rst
doc/source/whatsnew/v0.23.0.rst
doc/source/whatsnew/v1.2.0.rst
doc/source/whatsnew/v0.14.1.rst
doc/source/whatsnew/v0.15.0.rst
doc/source/whatsnew/v0.18.1.rst
doc/source/whatsnew/v1.0.2.rst
</details>

Looking through the py files, all remaining references are internal. This only leaves what to do with the whatsnew files. I could  

 - replace all occurrences;
 - replace all occurrences in 1.x files;
 - replace all occurrences in 1.4/1.5/1.6 files;
 - leave any past whatsnews alone

Any thoughts @mroeschke?

Labels:
name: Enhancement
name: Docs
name: Groupby

Comments:
If there's not too many occurrences, it would be great to replace all the occurrences in the whatsnew files (I've wanted to do this for other references too but low priority). Otherwise, might be good just to do the recent ones (1.4/1.5/1.6).
Nice! Thanks @rhshadrach
BUG: MultiIndex.union losing extension array dtype
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


This covers some cases only, will work on the remaining ones as a follow up

Labels:
name: MultiIndex
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
Ah no, sorry. This was not explicit enough. It does not work if we run into the ``algos.union_with_duplicates(lvals, rvals)`` code branch in ``_union()``. This is something for a follow up
Thanks @phofl
Bug: raise TypeError when doing infer_freq using RangeIndex
State: closed

- [x] closes #48439 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Frequency
description: DateOffsets

Comments:
LGTM pending @phofl review.
thx @ambujpawar
ENH: Support mask properly in GroupBy.quantile
State: closed

- [x] xref #37493 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Will investigate in a follow up, if we can send this through the regular groupby ops code paths


Labels:
name: Groupby
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
Thanks @phofl
BUG/PERF: use lexsort_indexer in MultiIndex.argsort
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v1.6.0.rst` file if fixing a bug or adding a new feature.

cc: @phofl, @mroeschke - follow on to #48406

Had I been aware of lexsort_indexer I would have used that in #48406. Using lexsort_indexer handles `pd.NA` (np.lexsort raises) and has better perf for nullable types (Int64 on par with int64).

While the perf for non-nullable types (e.g. int64) suffers a bit, this is still much faster than prior to #48406. 

```
       before           after         ratio
     [4e72340d]       [ead3c00d]
     <main>           <multiindex-argsort>
+     1.78±0.03ms       3.59±0.4ms     2.01  multiindex_object.SortValues.time_sort_values('int64')
-      26.7±0.5ms       3.51±0.5ms     0.13  multiindex_object.SortValues.time_sort_values('Int64')
``` 

Labels:
name: Bug
name: Performance
description: Memory or execution speed performance
name: MultiIndex
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
> Special casing only numpy dtypes makes it ugly, correct?

That was my thinking. DataFrame.sort_index uses the same approach with lexsort_indexer and does not special case numpy dtypes.
Ok, sounds reasonable
Can you merge main to get ci green?
> Would be good to have a test that calls `MultiIndex.argsort` directly

added an explicit argsort test
Just merged the other pr. Could you maybe run asvs once more? Union has a bunch of code paths, want to see if this is affected 
> Just merged the other pr. Could you maybe run asvs once more? Union has a bunch of code paths, want to see if this is affected

I don’t think this touches union. Were you thinking of #48504 ?
@phofl, @mroeschke 

This is now failing following the merge of #48505. The test failing is `test_union_sort_other_incomparable` which expects to see a RunTimeWarning from `MultiIndex._union`. However, this uncovers some inconsistency in the main branch.

Here is an example:

```
import pandas as pd
idx = pd.MultiIndex.from_product([[pd.Timestamp("2000"), 1], ["a", "b"]])
df = pd.DataFrame({"A": 1}, index=idx)

# on main, this returns without warning/error as it goes through lexsort_indexer
df.sort_index()

# on main, this raises a TypeError as it goes through np.lexsort
df.index.sort_values()
```

With this PR the behavior will be consistent as we'll be sending both through lexsort_indexer. I've removed the expectation for a RunTimeWarning from the test. Let me know if that doesn't make sense. 
@phofl - gentle ping. I think this is ready.
thx. @lukemanley
WEB: Update Python for Data Analysis book
State: closed


Labels:
name: Web
description: pandas website

Comments:
Sounds reasonable to convert the jpg to gif to reduce size. Not too familiar how to do that so any suggestions? 
This is the original 30 kb file:

![pydata_book](https://user-images.githubusercontent.com/10058240/189479393-934cd9d3-faa7-4e84-9f8c-0ea654173c23.jpg)

I'd use this version, it's 12kb:

![pydata_book_16colors](https://user-images.githubusercontent.com/10058240/189479268-724d5459-9be3-4868-95c4-ebe249b9877c.gif)

 I can reduce it more, but the new cover has more colors and it's more sensitive than the previous to compression. While probably no big deal, the quality of a smaller version starts to be clearly worse. This is a 8kb image:

![pydata_book_8colors](https://user-images.githubusercontent.com/10058240/189479434-56b3233b-3375-43d4-9669-ebe9b88f7911.gif)

Thanks, any thoughts whether we should include this for 1.5?
CI: Use -j1 for python-dev build to avoid flaky build error
State: closed

xref #47305

e.g. https://github.com/pandas-dev/pandas/runs/8280474438?check_suite_focus=true


Labels:
name: CI
description: Continuous Integration

Comments:
Thanks @datapythonista. I think the closest explanation of the issue can be found in https://github.com/pandas-dev/pandas/pull/47310#issuecomment-1232024906
DOC: A blank line is needed at the beginning of list.
State: closed

Fixes not displaying as a list in keep parameter of ``DataFrame.drop_duplicates``.
Added a blank line.

![image](https://user-images.githubusercontent.com/30496251/189386107-1421391f-22bf-461f-9cd2-0f8d87471272.png)


Labels:
name: Docs

Comments:
Yes!
![image](https://user-images.githubusercontent.com/30496251/189389959-5756fca3-fd60-45ad-ab91-73955e0a8751.png)

Thanks @15b3
BUG: fix valueError for groupby with empty inputs when sample method is called
State: closed

- [x] closes ##48459
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).


Labels:
name: Bug
name: Groupby

Comments:
Looks pretty good. Could use a whatsnew note in `1.6.0.rst` under the groupby section.
Thanks @Th3nn3ss
BUG: GroupBy equates different types of null values
State: closed

- [x] closes #48476 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Groupby

Comments:
Given the feedback in #48476, there needs to be lengthy consideration on how nulls of different types should be treated, and what the path for implementing this will be. Taking this off the queue.
DOC: Remove storage options from doc of Styler.to_excel
State: closed

Remove parameter from doc string that is not in function.  `Styler.to_excel` only supports `storage_options` in 1.5.x+

The PR is only for the 1.4.x branch.

Labels:
name: Docs
name: IO Excel
description: read_excel, to_excel
name: Styler
description: conditional formatting using DataFrame.style
name: Stale

Comments:
Not sure if there is an easy fix for this.  Getting a double line break issues since I think omitting the storage options added a line break.
You need to remove the import of `_shared_docs` which is now unused.

The way this was originally implemented seems a bit cumbersome, but one problem seems to be the version added, which expect `storage_options` to not be empty as it is. What I'd do is:

- Remove the `.. versionadded:: ...` from the original docstring (NDFrame.to_excel docstring)
- Add it to the storage parameter injected with `doc`, something like `storage_options=_shared_docs["storage_options"] + '  ..versionadded:: ...'`

If too many blank lines exist in that block after this (I don't think it'll happen), you can remove them from the docstring with `{storage_options}\` and add them to the injected values if any misses them.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
I think we can close this? There is currently no release planned for the 1.4.x branch
Makes sense.
REGR: .describe on unsigned dtypes results in object
State: closed

- [x] closes #48340 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

No whatsnew because this regression was introduced in 1.5. ~This is slightly a hack - we can't always specify the dtype as float because dtype of results could be TimeDelta.~ Implementation improved thanks to @phofl.

Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
Yes - complex worked before, looks like there wasn't tests.
Thx @rhshadrach
PERF: keep using ObjectEngine for ExtensionArrays for 1.5
State: closed

Closes https://github.com/pandas-dev/pandas/issues/45652, see https://github.com/pandas-dev/pandas/pull/45514#issuecomment-1025637651 for context.

@jbrockmendel I think we discussed this at some point a few months ago, but I forgot about it, that if we didn't improve ExtensionEngine for nullable dtypes before 1.5, we could continue falling back to ObjectEngine for now, as we did before. @phofl is working on that now, but so not for 1.5. 

This PR just does that for all EAs, but maybe we could also only do this for our built-in ones? (where we know that conversion to object dtype is expensive, but not _super_ expensive, while for external EAs we don't really know the cost of that)

```
idx_ea = pd.Index(np.arange(1_000_000), dtype="Int64")
indexer = np.arange(500, 1000)

In [2]: %timeit idx_ea.get_indexer(indexer)
25.3 ms ± 3.51 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)    # <-- main
211 µs ± 9.8 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)    # <-- PR
```

This PR might need to bring over some additional changes from https://github.com/pandas-dev/pandas/pull/45514

Labels:
name: Index
description: Related to the Index class or subclasses
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
No objection here.
Thanks @jorisvandenbossche
BUG: merge with CategoricalIndex for left_on/right_on
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Categorical
description: Categorical Data Type

Comments:
Thanks @jbrockmendel
TST: Fix unsigned pyarrow types in SIGNED_INT_PYARROW_DTYPES
State: closed

- [x] closes #48471 48471
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Arrow
description: pyarrow functionality

Comments:
Thanks @gsheni, is there an issue for this? Can you please provide context about this?
Thanks for the pull request, but when implementing this I did not specifically include testing `pa.string()` because pyarrow string functionality still fully exists in `StringDtype("pyarrow")` and `ArrowStringArray`. This note will be in the 1.5 docs: https://pandas.pydata.org/docs/dev/reference/arrays.html#pyarrow

What would be helpful is to start an issue to discuss whether there's interest to have `ArrowDtype(pa.string())` replace `StringDtype("pyarrow")` in the future.
@mroeschke sure, issue created here: https://github.com/pandas-dev/pandas/issues/48469

It might be worth reviewing this MR, as the `SIGNED_INT_PYARROW_DTYPES` list has incorrect data types.
> It might be worth reviewing this MR, as the SIGNED_INT_PYARROW_DTYPES list has incorrect data types.

Yes those fixes would be welcome!
Hello @gsheni! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-09-12 22:05:40 UTC
@gsheni mind merging main? Don't remember if these failures were real or unrelated
`test_add_series_with_extension_array` might need an xfail
Thanks @gsheni
TYP: remove mypy ignores from io/formats/printing.py
State: closed

- [x] Related to https://github.com/pandas-dev/pandas/issues/37715
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

The [mypy issue](https://github.com/python/mypy/issues/4975) that required this ignore statement has been solved and included in a release.


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
One of the mypy CI check failed. I cannot reproduce this with mypy version `0.971` which is the version in `environment.yml`.
Thanks @Moisan
DOC: Add missing API item to reference docs
State: closed

- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

* `Float32/64Dtype` & `arrays.FloatingArray`
* `NaT` & `NA`
* `lreshape`
* `set_eng_float_format`
* `NamedAgg`


Labels:
name: Docs

Comments:
Will merge in a few days if no further comments.