BrandonFors/pandas-github-issues huggingface dataset:
TYP: Add typing.overload signatures to DataFrame.drop_duplicates.
State: closed

This adds overloads so that a type checker can determine whether drop_duplicates returns DataFrame or None based on the value of the `inplace` argument.

The motivation for this is that we're using a type checker that looks at the pandas source code for type information. I realize that we should be using the stubs instead, but this will unblock us in the meantime and seems like an improvement to the type annotations anyway.

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Thank you @rchen152!
Thanks for the quick review and merge :)
Added AWS_S3_HOST environment variable workaround to user_guide for i…
State: closed

…ssue #26195

- [ ] closes #26195(Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks @kvn4
DOC: Fixing EX01 - Added examples
State: closed

- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Towards https://github.com/pandas-dev/pandas/issues/37875

Example by itself because it was not raising the warning every time locally. 

Labels:
name: Docs

Comments:
```
Error: /home/runner/work/pandas/pandas/pandas/errors/__init__.py:46:GL03:pandas.errors.PerformanceWarning:Double line break found; please use only one blank line to separate sections or paragraphs, and do not leave blank lines at the end of docstrings
Error: /home/runner/work/pandas/pandas/pandas/errors/__init__.py:46:EX02:pandas.errors.PerformanceWarning:Examples do not pass tests:
**********************************************************************
Line 18, in pandas.errors.PerformanceWarning
Failed example:
    df.loc[(1, 'z')]
Expected:
    Traceback (most recent call last):
    PerformanceWarning: indexing past lexsort depth may impact performance.
    df.loc[(1, 'z')])
              jolie
    jim  joe
    1    z        3
Got:
             jolie
    jim joe       
    1   z        3
```
updated CODEOWNERS file
State: closed

https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners#codeowners-syntax

Labels:

Comments:
what are you trying to do?
closing as this file is fine as-is
> what are you trying to do?
I saw that  CODEOWNERS file contains errors.
I was trying to solve errors.
> closing as this file is fine as-is

ya fine
> I saw that CODEOWNERS file contains errors.

which errors?
Remove test_groupby_agg_extension (marked xfail)
State: closed

Remove special `test_groupby_agg_extension` in `test_json.py` as we can now pass the test generally with these fixes to `GroupBy.first` and `GroupBy.last`.

- [x] closes #39098
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Groupby

Comments:
In the course of winnowing away unnecessary changes in a PR related to extension arrays, I found that this was one useful result of that work.  I'm re-submitting as a fresh PR that specifically fixes a single reported bug, following guidance in that bug report, and what I've learned along the way.
Thanks @MichaelTiemannOSC
Wrote test for Issue#29576 with resampling in non-unitary quarters
State: closed

- [ ] closes #29576
- [ ] Added test to `tests/resample/test_datetime_index.py` following expected and actual output of original commenter
- [ ] All [code checks passed] (https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).


Labels:

Comments:
Noticed the tests failed, I don't think the issue was fixed. Is the the expected output given in the issue description what the test should anticipate?
I don't think `expected` is correctly constructed to match https://github.com/pandas-dev/pandas/issues/29576#issue-521751237
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
DEPS: Update reprs with numpy NEP51 scalar repr
State: closed

xref https://github.com/numpy/numpy/pull/22449

Labels:
name: CI
description: Continuous Integration
name: Dependencies
description: Required and optional dependencies

Comments:
This will get the CI back to green?
> This will get the CI back to green?

Yup. I guess we can merge to get back to green and follow up if needed.
DOC: Fixing EX01 - Added examples
State: closed

- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Towards https://github.com/pandas-dev/pandas/issues/37875

Example by itself because I have doubts if it's good enough

Labels:
name: Docs

Comments:
Thanks @DeaMariaLeon
DOC: Fixing EX01 - Added examples
State: closed

- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Towards https://github.com/pandas-dev/pandas/issues/37875



Labels:
name: Docs

Comments:
It looks like failing tests are unrelated
Could you merge in main once more?
Thanks @DeaMariaLeon
BUG: pivot_table mean of integer input casted back to int
State: closed

- [x] closes #16676
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Remove downcasting and update tests to expect floats according to https://github.com/pandas-dev/pandas/issues/16676#issuecomment-1636827617.


Labels:
name: Bug
name: Groupby
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: API - Consistency
description: Internal Consistency of API/Behavior

Comments:
Thanks @austinauyeung!
Thanks for the code review @rhshadrach!
DEPR: fillna downcasting from object dtype
State: closed

- [x] closes #11537 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Deprecate
description: Functionality to remove in pandas

Comments:
```
2023-07-26T01:20:55.4868437Z RuntimeError: Non Expected warning in `/home/runner/work/pandas/pandas/doc/source/user_guide/missing_data.rst` line 709
```
updated+green
Thanks @jbrockmendel
STYL: replace autotyping with ruff's ANN autofixes
State: closed

ruff's ANN has many of the autofixes from autotyping but it doesn't have all of them. Ruff doesn't annotate parameters with simple default values (autotyping would change `def foo(x=1)` to `def foo(x: int = 1)`).

Labels:
name: Code Style
description: Code style, linting, code_checks
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Thanks @twoertwein
REF: de-duplicate ensure_np_dtype
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Refactor
description: Internal refactoring of code

Comments:
Thanks @jbrockmendel
BUG: Series.argsort validate axis
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
Thanks @jbrockmendel
DOC: Fixing EX01 - Added examples
State: closed

- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Towards https://github.com/pandas-dev/pandas/issues/37875

Notes:
On the web there are only 3 custom rolling windows: `BaseIndexer`, `FixedForwardWindowIndexer` and `VariableOffsetWindowIndexer`. But there are more of them in the [code](https://github.com/pandas-dev/pandas/blob/main/pandas/core/indexers/objects.py). Like `ExpandingIndexer` etc.. 

~~Added `# doctest: +SKIP` to the last line of `BaseIndexer` because `validate_docstrings.py` was complaining.~~

I need to do \preview here as I was having a lot of issues with the html file made with Sphinx.

I'll open a separate PR to correct the User Guide on `BaseIndexer`.


Labels:
name: Docs

Comments:
/preview
No preview found for PR #54254. Did the docs build complete?
/preview
Website preview of this PR available at: https://pandas.pydata.org/preview/54254/
It looks like failing tests are unrelated
Thanks @DeaMariaLeon
TST: Don't cast in test_cython_agg_nullable_int
State: closed

xref https://github.com/pandas-dev/pandas/pull/54129/files#r1272774080

Nice find @rhshadrach!

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
Thanks @mroeschke
ENH: Support ffill/bfill on IntervalArray
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Also avoids the need for a PerformanceWarning for SparseArray.

Labels:
name: Interval
description: Interval data type

Comments:
Thanks @jbrockmendel
ENH: Implement interchange protocol for DatetimeTZDtype
State: closed

- [x] closes #54239 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Interchange
description: Dataframe Interchange Protocol

Comments:
> There should probably be a test for interchanging from non-pandas tz-aware, such as

Yes definitely. Would be good to have that as a follow up
cool, I'll do that tomorrow
TST: add a test on mixed offsets for read_csv
State: closed

xref #54014

added a test on mixed offsets to increase coverage for `read_csv`

Labels:
name: Needs Tests
description: Unit test(s) needed to prevent regressions

Comments:
test looks good, but I think `pandas/tests/io/parser/test_parse_dates.py` might be a better location
> test looks good, but I think `pandas/tests/io/parser/test_parse_dates.py` might be a better location

Thanks, I moved the test as you suggested.
Thank you for reviewing this PR.
BLD: don't require ninja to come from PyPI
State: closed

In pyproject.toml, the baseline requirements for building are supposed to be listed.

Many end users have a recent enough system-provided ninja, and don't need to install a new copy into every pandas isolated build virtual environment. This is handled inside meson-python itself, which dynamically adds an additional build requirement on ninja via the backend hook `get_requires_for_build_sdist`, but only if a suitable system version is not found.

This is especially helpful on systems where an OS ninja is installed, but the platform is obscure enough that there are no PyPI "ninja" wheels, thus forcing `pip install pandas` to download an additional copy of the ninja source code and build it via cmake before beginning to install pandas itself. This is not guaranteed to work...

The ninja entry in various CI or developer-recommended workflows is left alone because they are opt-in. In such contexts, it may make sense to be fairly specific about which providers you want to use for any given dependency, and that's a workflow preference encoded as a workflow build command. Since it doesn't prevent people from using the baseline install command, this commit makes no value judgment regarding it.

Labels:
name: Build
description: Library building on various platforms

Comments:
thanks @eli-schwartz.
DEPR offsets: rename 'Y' to 'YE'
State: closed

xref #52064

Labels:
name: Frequency
description: DateOffsets
name: Deprecate
description: Functionality to remove in pandas
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
DOC: Imporve `json_normalize` record_prefix and mate_prefix
State: closed

- [ ] closes #54121  (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
That's not how it works. For example:

```python
data = {"A": [1, 2]}
pd.json_normalize(data, "A", record_prefix=True)
```

```none
   True0
0      1
1      2
```

See also the example lower in the docs.
> That's not how it works. For example:
> 
> ```python
> data = {"A": [1, 2]}
> pd.json_normalize(data, "A", record_prefix=True)
> ```
> 
> ```
>    True0
> 0      1
> 1      2
> ```
> 
> See also the example lower in the docs.

You are right. I ignored this and just changed the type. 
@wjandrea How about this time? 
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
@mroeschke Sorry for later reply.

I have merged the branch with main, should we reopen the pr, or I need to open a new one?
Fixed groupby error with nullable types
State: closed

- [x] closes #54206 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
DEPR: idxmin/idxmax with all-NA
State: closed

- [x] closes #51276 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

As discussed in last week's dev call.

Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
Thanks @jbrockmendel 
Am I understanding this correctly that I'm supposed to write this now each and every time I use `idxmax`?:

```python
s = pd.Series([None])

if not s.isnull().all():
    m = s.idxmax()
else:
    m = np.nan
```

instead of just this with the originally promised behavior of "If the entire Series is NA, the result will be NA."

```python
s = pd.Series([None])
m = s.idxmax()
```

this is annoyingly verbose. Or what is the correct way to approach this?

> this is annoyingly verbose. Or what is the correct way to approach this?

Not only is it verbose, I find it less intuitive especially considering the parameter skipna.

`pd.Series([None]).idxmax(skipna=True)` resulting in `np.nan` (and similarly for `pd.DataFrame.idxmax`) is (for me) the expected behavior.
PERF: DataFrame.transpose for pyarrow-backed
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.1.0.rst` file if fixing a bug or adding a new feature.

Similar to #52836, but for pyarrow-backed frames. 

```
import pandas as pd
import pyarrow as pa
import numpy as np

values = np.random.randn(1000, 1000)
df = pd.DataFrame(values, dtype=pd.ArrowDtype(pa.float64()))

%timeit df.T

# 346 ms ± 67.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)    -> main
# 42.5 ms ± 2.12 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) -> PR
```

Renamed the masked implementation from `homogenous` -> `homogeneous`. Both may be valid(?), but `homogeneous` is used elsewhere.

Labels:
name: Performance
description: Memory or execution speed performance
name: Arrow
description: pyarrow functionality

Comments:
Thanks @lukemanley
PERF: ArrowDtype.__eq__
State: closed

Micro-perf improvement:

```
import pandas as pd
import pyarrow as pa

a = pd.ArrowDtype(pa.float64())
b = pd.ArrowDtype(pa.float64())

%timeit a == b

# 2.86 µs ± 214 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)   -> main
# 473 ns ± 15.2 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each) -> PR
```


Labels:
name: Performance
description: Memory or execution speed performance
name: Arrow
description: pyarrow functionality

Comments:
Thanks @lukemanley
DOC: Simplify contributing intro
State: closed

* Reduced some preamble
* Updated some links
* Use `code-block:: shell` where appropriate

Labels:
name: Docs

Comments:
nice! @noatamir fancy taking a look? interested in adding yourself to https://github.com/pandas-dev/pandas/blob/main/.github/CODEOWNERS for `doc/source/development` (and anything else)?
Hmm, @mroeschke the doc build is failing and it looks like an extensive change. Any chance you have screenshots from a local build or can you please fix the build issue? 
/preview
Website preview of this PR available at: https://pandas.pydata.org/preview/54220/
@noatamir was able to fix the issue and generate the preview
This is great! I made a few small suggestions. 
Thanks all the suggestions looked great