BrandonFors/pandas-github-issues huggingface dataset:
BUG: fix convert_dtypes to preserve timezone from tz-aware pyarrow timestamp dtype
State: closed

Removed if statement blocks that were causing timezone to be removed, added a test to confirm that bug is fixed, and added BUG fix to 3.0.0 documentation.
Fixes issue in #60237 


Labels:
name: Stale

Comments:
pre-commit.ci autofix
Hi all, apologies for so many commits to this PR, this is my first issue and I'm not that familiar with GitHub so I was learning as I went. I believe I've fixed the timezone preservation issue and my test case passes for the minimum version unit test as well now. Let me know if you see any issues that need to be addressed though! Thank you for your help!

Fix for #60237 
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
BUG (string dtype): let fillna with invalid value upcast to object dtype
State: closed

Like I did for `replace()` (issue https://github.com/pandas-dev/pandas/issues/60282 / PR https://github.com/pandas-dev/pandas/pull/60285), it turns out that also for `fillna()` we generally upcast to object dtype if the fill value does not fit the dtype:

```python
>>> ser = pd.Series([1, 2, None], dtype="float")
>>> ser.fillna("a")
Out[4]: 
0    1.0
1    2.0
2      a
dtype: object
```

And similarly as for `replace()`, this was raising an error in the case of `StringDtype`.  
The reason is that for generic (external) ExtensionArrays, we are actually strict and `fillna` is just dispatched to the EA without fallback to object dtype; while for all our own _default_ extension dtypes we do have custom code to ensure to do a fallback (but _not_ for the masked dtypes and ArrowDtype ..).

Given this is the default behaviour right now for other dtypes (and so we also haven't deprecated this for current usage of strings in object dtype), I would preserve this fallback for now to avoid this as another breaking change in 3.0.

xref https://github.com/pandas-dev/pandas/issues/54792


Labels:
name: Strings
description: String extension data type and string data
name: API - Consistency
description: Internal Consistency of API/Behavior

Comments:
Thanks @jorisvandenbossche 
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.3.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 34c39e9078ea8af12871a92bdcea2058553c9869
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #60296: BUG (string dtype): let fillna with invalid value upcast to object dtype'
```

4. Push to a named branch:

```
git push YOURFORK 2.3.x:auto-backport-of-pr-60296-on-2.3.x
```

5. Create a PR against branch 2.3.x, I would have named this PR:

> "Backport PR #60296 on branch 2.3.x (BUG (string dtype): let fillna with invalid value upcast to object dtype)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Manual backport -> https://github.com/pandas-dev/pandas/pull/60316
TST (string dtype): resolve xfails for frame fillna and replace tests + fix bug in replace for string
State: closed

Follow-up on https://github.com/pandas-dev/pandas/pull/60285, fixing one additional bug in the string-fallback-to-object implementation, uncovered by enabling all other replace/fillna related tests.

xref https://github.com/pandas-dev/pandas/issues/54792

Labels:
name: Strings
description: String extension data type and string data
name: replace
description: replace method

Comments:
Thanks @jorisvandenbossche 
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.3.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 fae3e8034faf66eb8ef00bcbed73d48e4ef791d3
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #60295: TST (string dtype): resolve xfails for frame fillna and replace tests + fix bug in replace for string'
```

4. Push to a named branch:

```
git push YOURFORK 2.3.x:auto-backport-of-pr-60295-on-2.3.x
```

5. Create a PR against branch 2.3.x, I would have named this PR:

> "Backport PR #60295 on branch 2.3.x (TST (string dtype): resolve xfails for frame fillna and replace tests + fix bug in replace for string)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Manual backport -> https://github.com/pandas-dev/pandas/pull/60331
TST: Add test for x-axis labels in subplot with `secondary_y=True`
State: closed

- [x] closes #14102  (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

This PR adds a unit test to `pandas/tests/plotting/test_series.py` to verify that X-axis labels are preserved when using `secondary_y=True` and creaitng a new subplot. Adds a test for the solved issue described in https://github.com/pandas-dev/pandas/issues/14102.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Visualization
description: plotting

Comments:
Thanks @jasonmokk
BUG: Don't merge Excel cells to a single row with merge_cells=False
State: closed

- [x] closes #60274 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: IO Excel
description: read_excel, to_excel
name: MultiIndex

Comments:
Thanks @rhshadrach
TST: Add test for `pd.read_csv` date parsing not working with `dtype_backend="pyarrow"` and missing values
State: closed

- [x] closes #59904 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: IO CSV
description: read_csv, to_csv
name: Arrow
description: pyarrow functionality

Comments:
I tried using 
```python
    assert pd.api.types.is_datetime64_any_dtype(df["date"])
```
but it seems that the code checks doesn't allow it. Not sure if the assertion is correct:
```python
    assert (df["date"].dtype) == "datetime64[s]"
```
Thanks @KevsterAmp
BUG (string dtype): replace with non-string to fall back to object dtype
State: closed

- [x] closes #60282
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

xref https://github.com/pandas-dev/pandas/issues/54792


Labels:
name: Bug
name: Strings
description: String extension data type and string data
name: replace
description: replace method

Comments:
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.3.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 938832ba325c6efc1710e002c0d3d4d9b3a6c8ba
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #60285: BUG (string dtype): replace with non-string to fall back to object dtype'
```

4. Push to a named branch:

```
git push YOURFORK 2.3.x:auto-backport-of-pr-60285-on-2.3.x
```

5. Create a PR against branch 2.3.x, I would have named this PR:

> "Backport PR #60285 on branch 2.3.x (BUG (string dtype): replace with non-string to fall back to object dtype)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Manual backport -> https://github.com/pandas-dev/pandas/pull/60292
Fix minor typos and remove examples in PR #60171
State: closed

Part of #59698 

Addresses:

pandas.Timedelta

I've made some minor updates to the original pull request (#60171) based on the feedback. The changes include:
Removed unnecessary examples as per the request.
Made small formatting improvements for better clarity.



Labels:
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
BUG: Fix #54763 to_julian_date does not consider timezone
State: closed

- [x] closes #54763
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Stale

Comments:
The merge conflict in `pandas/_libs/tslibs/timestamps.pyx` is because I changed how the additions/subtractions were styled to improve readability. I had to insert a minus for the timezone_offset and it did not make sense to have it on the previous line. This format is also more consistent with the other files I was working in.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
Title: BUG: Fix Incorrect Logical Operation Between Pandas DataFrame and Series (#60204)
State: closed

This PR fixes issue #60204, where logical operations (`&`, `|`, `^`) between a `DataFrame` and a `Series` were not correctly broadcasting, leading to unexpected results. The issue was observed when attempting operations like `(df >= 10) | (df['A'] >= 10)`, where `df` is a `DataFrame` and `df['A']` is a `Series`.

#### Changes Made
- **Adjusted `logical_op` function**: Added logic to handle cases where `left` is a `DataFrame` and `right` is a `Series`. The function now converts `right` to a NumPy array to allow proper broadcasting.
- **Type Check for List-like Objects**: Added error handling to raise a `TypeError` if `right` is a list-like object (e.g., list, tuple) without a defined dtype, as these are unsupported in logical operations with DataFrames.
- **Enhanced Boolean Casting with `fill_bool`**: Modified `fill_bool` to ensure proper casting to boolean, accommodating non-boolean types and handling missing values correctly.

### Code Adjustments
- **Imports and Compatibility**: Ensured correct imports for `DataFrame` and `Series` types, applied `lib.item_from_zerodim` for handling zero-dimensional `right` objects.
- **Broadcasting Conversion**: Converted `right` to `right.values` when it's a Series, enabling proper broadcasting across DataFrame columns in logical operations.

### Testing and Validation
- **New Unit Tests**: Added specific tests in `pandas/tests/arithmetic/test_ops.py` to verify:
  - **Correct Broadcasting**: Tested logical operations between `DataFrame` and `Series` to ensure correct broadcasting behavior.
  - **Error Handling**: Confirmed `TypeError` is raised for unsupported list-like objects without dtype.
  - **Boolean Casting with NaNs**: Verified the behavior when logical operations involve NaNs or non-boolean compatible types.

### Additional Notes
- **Backward Compatibility**: This fix is backward-compatible and applies only to scenarios involving DataFrame and Series logical operations.
- **Minimal Performance Impact**: The change is isolated to broadcasting adjustments, so any performance impact should be minimal.

### Checklist
- [x] Code changes are consistent with Pandas guidelines.
- [x] New tests have been added and pass successfully.
- [x] Documentation and comments are added where necessary.
- [x] Changes are backward-compatible.

### Related Issue
Closes #60204

Thank you for reviewing this PR! Please let me know if further adjustments are needed.

---

Labels:

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
ENH: set __module__ on isna and notna
State: closed

- [ ] Part of #55178 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: Sprints
description: Sprint Pull Requests

Comments:
`pd.notna` is top-level too and is defined in missing.py, so maybe could be included in this PR
https://github.com/pandas-dev/pandas/blob/217a0ed7f1e04dd675d0887ead001cbf2e91d880/pandas/core/dtypes/missing.py#L316
> Thanks @SiemBerhane

Glad it finally works. Thanks for your help!
@jorisvandenbossche 

`isnull` and `notnull` are aliases to `isna` and `notna` respectively that are also defined in `pandas/core/dtypes/missing.py`

now the documentation for isnull is the same as isna (because it's just an alias to isna).

should the docs additionally mention that is an alias? are `isnull` and `notnull` recommended uses?

in IPython the repr now for pd.isnull is `<function pandas.isna(obj: 'object') -> 'bool | npt.NDArray[np.bool_] | NDFrame'>`. While we're improving the output formatting is there any value in making this show `<function pandas.isnull....` instead?

would we also want an assert in the tests for pd.isnull and pd.notnull too?



> `isnull` and `notnull` are aliases to `isna` and `notna` respectively that are also defined in `pandas/core/dtypes/missing.py`

I think for those aliases it is fine that the repr points to the isna/notna instead (I mean, I wouldn't do specific effort to fix that, although addressing the docstring might also address this).

Ideally those docstrings of isnull/notnull should mention they are aliases, though
TST: parametrize the tests checking __module__
State: closed

Parametrize the test part of #55178 

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Sprints
description: Sprint Pull Requests

Comments:
Thanks @espoirMur for working on this. Now that I've had a closer look at the api testing, I see that the tests are class based and not using fixtures so parameterizing the tests probably would not be beneficial as we won't be able to re-use fixtures to ensure that all of the public api is covered.

I suspect that we would need to add the checking of the `__module__` attribute to the class based testing.

@jorisvandenbossche what's you thoughts here?
> Thanks @espoirMur for working on this. Now that I've had a closer look at the api testing, I see that the tests are class based and not using fixtures so parameterizing the tests probably would not be beneficial as we won't be able to re-use fixtures to ensure that all of the public api is covered.
> 
> I suspect that we would need to add the checking of the `__module__` attribute to the class based testing.
> 
> @jorisvandenbossche what's you thoughts here?

Ah no worries, feel free to close this then.
ENH: set __module__ on IndexSlice
State: closed

- [ ] Part of #55178
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: Stale
name: Sprints
description: Sprint Pull Requests

Comments:
@jorisvandenbossche this one is a bit different?
i'll try something that does give the correct output and is technically correct but we'll see what else breaks!

```
>>> pd.IndexSlice
<pandas.IndexSlice object at 0x7f2528c70d60>
```
> i'll try something that does give the correct output and is technically correct

in jupyter notebook the output is `<pandas.IndexSlice at 0x7f2d681cd870>`

> but we'll see what else breaks!

seems fine so far except for the pre-commit error
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
ENH: set __module__ on NamedAgg / SeriesGroupBy / DataFrameGroupBy (`pandas.core.groupby.generic` classes)
State: closed

xref #55178

Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: Sprints
description: Sprint Pull Requests

Comments:
It appears that `DataFrameGroupBy` and `SeriesGroupBy` are not top-level classes/functions:

```python
[ins] In [1]: import pandas as pd

[ins] In [2]: pd.NamedAgg
Out[2]: pandas.core.groupby.generic.NamedAgg

[ins] In [3]: pd.SeriesGroupBy
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[3], line 1
----> 1 pd.SeriesGroupBy

AttributeError: module 'pandas' has no attribute 'SeriesGroupBy'

[ins] In [4]: pd.DataFrameGroupBy
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[4], line 1
----> 1 pd.DataFrameGroupBy

AttributeError: module 'pandas' has no attribute 'DataFrameGroupBy'

```
> It appears that `DataFrameGroupBy` and `SeriesGroupBy` are not top-level classes/functions:

Correct, they are exposed in `pandas.api.typing` as they should not be instantiated by users directly. This is the path that `__module__` should be set to.
> > It appears that `DataFrameGroupBy` and `SeriesGroupBy` are not top-level classes/functions:
> 
> Correct, they are exposed in `pandas.api.typing` as they should not be instantiated by users directly. This is the path that `__module__` should be set to.

Thanks, I'll work on getting this implemented
Thanks @ofsouzap for the updates.

Because of merging some of the related PRs, there is now a merge conflict in the tests file. Could you merge the latest main to resolve that conflict? (and let us know if you need any help with that)
ENH: set __module__ on `timedelta_range`
State: closed

- [ ] Part of https://github.com/pandas-dev/pandas/issues/55178
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This decorates `timedelta_range` with `@set_module('pandas')` (similar to https://github.com/pandas-dev/pandas/pull/55171)



Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: Sprints
description: Sprint Pull Requests

Comments:
@jorisvandenbossche Thanks for reviewing! I did the same and resolved conflicts for this now.