BrandonFors/pandas-github-issues huggingface dataset:
BUG: Retain index when merging on an index
State: closed

- [x] closes #57291
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v3.0.0.rst` file if fixing a bug or adding a new feature.

Since `join` uses a `merge` internally, changing the index returned by `merge` would cause problems for `join`. To prevent this, I introduced a new argument that would control which index is returned by a merge operation. I also separated out the logic that determined what index to use from the code that created the indexers. Not sure this was the best way to fix this; apologies if I bit off more than I could chew.

Labels:
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
PERF: Improve efficiency of `BlockValuesRefs._clear_dead_references(...)`
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Improves efficiency of `BlockValuesRefs._clear_dead_references(...)` by allowing for more efficient C code-gen as a function lookup and FastCall is eliminated from a tight loop.

## Before
![image](https://github.com/user-attachments/assets/558a9123-a2ae-467e-a1ed-e9e3c3057b07)
## After
![image](https://github.com/user-attachments/assets/cf622bbc-73c0-4c08-ba02-a89a7c1ea72e)


Labels:
name: Performance
description: Memory or execution speed performance
name: Copy / view semantics

Comments:
Thanks @Tolker-KU
TST (string) fix xfailed groupby tests (3)
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Strings
description: String extension data type and string data
name: backported

Comments:
Thanks @jbrockmendel
DOC: fix docstring validation errors for pandas.PeriodIndex
State: closed

Addresses https://github.com/pandas-dev/pandas/issues/59637.

Fixes:
https://github.com/pandas-dev/pandas/blob/306385b0c157da62595a4b3efb23799b7d136c62/ci/code_checks.sh#L84-L104

Labels:
name: Docs

Comments:
Thanks @hlakams
Fix binary operations on attrs for Series and DataFrame
State: closed

- [ ] closes #51607
- [Test](https://github.com/pandas-dev/pandas/blob/main/pandas/tests/series/test_api.py)
- [Test](https://github.com/pandas-dev/pandas/blob/main/pandas/tests/frame/test_api.py)


Labels:
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: metadata
description: _metadata, .attrs

Comments:
@mroeschke should everything be rewritten using `finalize` then?
Yes, or `__finalize__` needs to be probably be called somewhere
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
@mroeschke, @WillAyd, I tried using `__finalize__` instead. What do you think?
I think it looks good but will defer to @mroeschke 
I don't think the CI failures are related. This still lgtm - @mroeschke can you take another look?
@WillAyd @mroeschke, any chance I could get an update?
@mroeschke I suggested something. I kept some checks for attrs in `_cmp_method` and `_arithm_method` in `pandas/core/frame.py`
This looks like a good job. However, I noticed some conditional logic added in `frame.py` (`_cmp_method`, `_arith_method`, `_construct_result`) and `series.py` (`_construct_result`):
```python
if not getattr(self, "attrs", None) and getattr(other, "attrs", None):
    self.__finalize__(other)
```
This logic seems to attempt copying attributes from the `other` (right) operand if `self` (left) doesn't initially have attributes. This appears potentially incorrect, as standard pandas behavior (and the new test) suggests the result should only inherit attributes from the *left* operand (`self`). The final `out.__finalize__(self)` call in `_construct_result` should already handle this correctly.

Could we simplify the implementation by removing these conditional `self.__finalize__(other)` calls and relying solely on `out.__finalize__(self)`? This seems like it would align better with the intended behavior and the added test case. However I might be wrong, just wanted to point this out in case something there was unintentional.
Thanks for sticking with this @fbourgey! 
Thanks for the help @mroeschke!
DOC: fix docstring validation errors for pandas.core.window
State: closed

Partially addresses https://github.com/pandas-dev/pandas/issues/58063.

Fixes:
https://github.com/pandas-dev/pandas/blob/306385b0c157da62595a4b3efb23799b7d136c62/ci/code_checks.sh#L256-L260

Labels:
name: Docs

Comments:
Thanks @hlakams
DOC: fix docstring validation errors for pandas.Series
State: closed

Partially addresses https://github.com/pandas-dev/pandas/issues/59592

Fixes:
```
-i "pandas.Series.sparse PR01,SA01" \
-i "pandas.Series.sparse.to_coo PR07,RT03,SA01" \
```


Labels:
name: Docs

Comments:
Thanks @pol-rius
TST (string): fix xfailed groupby value_counts tests
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Strings
description: String extension data type and string data
name: backported

Comments:
Thanks @jbrockmendel
PERF: avoid unnecessary copy
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Performance
description: Memory or execution speed performance
name: Strings
description: String extension data type and string data

Comments:
Should this go into 2.3?
No

On Tue, Aug 27, 2024 at 3:56 PM Matthew Roeschke ***@***.***>
wrote:

> Should this go into 2.3?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/59630#issuecomment-2313709535>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AB5UM6ALVBSJDQFWESNWLUDZTT7YBAVCNFSM6AAAAABNG6LSAOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMJTG4YDSNJTGU>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>

Thanks @jbrockmendel
BUG: Fix inconsistent pivot table subaggregation when index is None
State: closed

- [x] closes #58722 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Thanks @matiaslindgren!
BUG (string): str.replace with negative n
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Strings
description: String extension data type and string data
name: backported

Comments:
Thanks @jbrockmendel
REF (string): rename result converter methods
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Strings
description: String extension data type and string data
name: backported

Comments:
Thanks @jbrockmendel
DOC: Add whatsnew for 2.3.0
State: closed

We are probably not going to need all sections for the 2.3.0 release (and need some custom sections for the upcoming changes), but just adding our standard template and can clean up later.

Labels:
name: Docs
name: backported

Comments:
```bash
 /home/runner/work/pandas/pandas/doc/source/whatsnew/v3.0.0.rst:4: WARNING: duplicate label whatsnew_230, other instance in /home/runner/work/pandas/pandas/doc/source/whatsnew/v2.3.0.rst
```
Thanks @jorisvandenbossche 
(cc @WillAyd this needs to be backported as well)
API (string): str.center with pyarrow-backed string dtype
State: closed

- [x] closes #54807 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Two bad options for older-pyarrow: different behavior or object fallback.  I guessed that @mroeschke would prefer different behavior here, would be happy to be proved wrong.

Labels:
name: Strings
description: String extension data type and string data
name: backported

Comments:
> Two bad options for older-pyarrow: different behavior or object fallback. I guessed that @mroeschke would prefer different behavior here, would be happy to be proved wrong.

I won't argue for ArrowDtype if there is a preference to not do object fallback there, but for StringDtype, we definitely want to fallback to object dtype, I think (as I proposed in https://github.com/pandas-dev/pandas/issues/54807#issuecomment-2256041345). Otherwise this gives a breaking change for people with older pyarrow, and inconsistent behaviour depending on the pyarrow version you have installed. While we can very easily avoid this breaking change, so then we should just do that IMO
Using the `lean_left_on_odd_padding=False` breaks `test_string_array`:

```
import numpy as np
import pandas as pd

data = ["a", "bb", np.nan, "ccc"]
ser = pd.Series(data, dtype=pd.StringDtype(storage="pyarrow"))
alt = pd.Series(data, dtype=object)

res = ser.str.center(10)
exp = alt.str.center(10).astype(ser.dtype)

>>> res[0]
'     a    '
>>> exp[0]
'    a     '
```

The first explanation that comes to mind is that I misunderstood something and should be passing `lean_left_on_odd_padding=True`, but that in turn breaks test_center_ljust_rjust_fillchar:

```
import pandas as pd

ser = pd.Series(["a", "bb", "cccc", "ddddd", "eeeeee"], dtype=pd.StringDtype(storage="pyarrow"))
res = ser.str.center(5, fillchar="X")
exp = ser.astype(object).str.center(5, fillchar="X").astype(ser.dtype)

>>> res[2]
'ccccX'
>>> exp[2]
'Xcccc'
```
DOC: Enforce Numpy Docstring Validation (Issue #59458)
State: closed

- [x] Addresses some tasks of  #59458 
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Adding docstring for for `Timestamp.month, Timestamp.hour, Timestamp.minute, Timestamp.second, Timestamp.microsecond`. 

Reference PR: #59590 
Thanks @ankit-dhokariya
BUG: Resampling to "B" frequency with closed="right" and label="right" adds empty bins
State: closed

- [x] closes #59495
- [x] Tests added and passed
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/v3.0.0.rst`



fixed bug: Resampling to frequency='B' with closed='right'a nd label='right'


Labels:
name: Bug
name: Resample
description: resample method
name: Stale

Comments:
@MarcoGorelli, seems like the reproducible example was affected by the changes in your PR #55283. I corrected the definition of `_get_time_bins` to get the expected behavior in the example. However, this caused some of your tests to fail, so I updated the expected output. Would you mind reviewing my changes?
thanks for the ping, will take a look next week
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
REF: Minimize operations in recode_for_groupby
State: closed

* `.categories` should already be unique, so no need to compute the operation on them
* Don't pre-generate `np.arange` if it's not used

Labels:
name: Refactor
description: Internal refactoring of code
name: Groupby
name: Categorical
description: Categorical Data Type

Comments:
Thanks @mroeschke
DEPR (string): non-bool na for obj.str.contains
State: closed

- [x] closes #59561 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

No 2.3 whatsnew file yet.  Not sure if the plan is to introduce one in #59513

Labels:
name: Strings
description: String extension data type and string data
name: Deprecate
description: Functionality to remove in pandas
name: backported

Comments:
I think startswith/endswith have the same issue? If so, can you add the same deprecation message there as well?

> No 2.3 whatsnew file yet. Not sure if the plan is to introduce one in https://github.com/pandas-dev/pandas/pull/59513

Yeah, we should start a 2.3 whatsnew file. We should add it to main, though (not in the backport PR), so things can be added to that file in PRs like this one.

> I think startswith/endswith have the same issue? If so, can you add the same deprecation message there as well?

Sure, updated.
Actually, you could maybe move the warning to core/string/accessors.py, then you don't have to repeat it for the various implementations, and then also ArrowDtype(string) will get it (assuming we want to be consistent and also deprecate it for ArrowDtype, and not only for StringDtype?)
> assuming we want to be consistent and also deprecate it for ArrowDtype, and not only for StringDtype?

Of course, that one already errors :) 
> then you don't have to repeat it for the various implementations

Because it already errors for the ArrowEA version (but with a pyarrow exception), patching it in the StringAccessor method would be bulky.  I think the thing to do is wait until the deprecation is enforced, then do the validation in the StringAccessor method and get consistent exception/message.
> Because it already errors for the ArrowEA version (but with a pyarrow exception), patching it in the StringAccessor method would be bulky. I think the thing to do is wait until the deprecation is enforced, then do the validation in the StringAccessor method and get consistent exception/message.

Sounds good!
Doc/59592/fix docstring validation errors
State: closed

Partially addresses [#59592](https://github.com/pandas-dev/pandas/issues/59592)

Fixes:

-i "pandas.TimedeltaIndex.as_unit RT03,SA01"
-i "pandas.MultiIndex.to_frame RT03"

Labels:
name: Docs

Comments:
Heads up, RT03 for `pandas.Series.str.zfill` was already addressed in https://github.com/pandas-dev/pandas/pull/59597 (merged).
Thank you sir. I will amend my PR accordingly!
Thanks @specialkapa
String dtype: avoid surfacing pyarrow exception in binary operations
State: closed

For the pyarrow storage, in the generic ArrowExtensionArray implementation, we currently just call the pyarrow compute kernel, regardless of the input types are supported or not, potentially resulting in pyarrow errors surfacing to the user (typically `pyarrow.lib.ArrowNotImplementedError`).

This has several issues: 1) it's a bit annoying for testing (the tests currently import pyarrow to test the expected error message, which then fails if pyarrow is not installed), but 2) moreover this is a bad user experience IMO: it gives error messages that can be confusing (including "implementation details" from pyarrow) and inconsistent (compared to non-pyarrow backed dtypes), and it actually also gives a wrong exception type (as many of those operations are simply not supported for strings, and are expected to raise a TypeError, instead of a NotIplementedError which gives the impression it might be implemented in the future).

I am currently doing this for all arrow types, but I could also limit this to only those cases where it is used by the ArrowStringArray, and not for other ArrowDtype cases.

xref https://github.com/pandas-dev/pandas/issues/54792

Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Strings
description: String extension data type and string data
name: Arrow
description: pyarrow functionality
name: backported

Comments:
OK, this should be fully ready for review now
thanks @jorisvandenbossche