BrandonFors/pandas-github-issues huggingface dataset:
ERR (string dtype): harmonize setitem error message for python and pyarrow storage
State: closed


Related to https://github.com/pandas-dev/pandas/issues/60218, but just already making sure we have the same error message for the string dtype for pyarrow vs python storage.

Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Strings
description: String extension data type and string data

Comments:
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.3.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 692ea6f9d4b05187a05f0811d3241211855d6efb
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #60219: ERR (string dtype): harmonize setitem error message for python and pyarrow storage'
```

4. Push to a named branch:

```
git push YOURFORK 2.3.x:auto-backport-of-pr-60219-on-2.3.x
```

5. Create a PR against branch 2.3.x, I would have named this PR:

> "Backport PR #60219 on branch 2.3.x (ERR (string dtype): harmonize setitem error message for python and pyarrow storage)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Manual backport -> https://github.com/pandas-dev/pandas/pull/60232
TST (string dtype): avoid explicit object dtype Index in fixture data
State: closed

(opening this to see which tests are failing on a full CI test run)

xref https://github.com/pandas-dev/pandas/issues/54792

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Strings
description: String extension data type and string data

Comments:
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.3.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 4b04a2f0043ad04b5546750a8947dfeef68cdb75
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #60217: TST (string dtype): avoid explicit object dtype Index in fixture data'
```

4. Push to a named branch:

```
git push YOURFORK 2.3.x:auto-backport-of-pr-60217-on-2.3.x
```

5. Create a PR against branch 2.3.x, I would have named this PR:

> "Backport PR #60217 on branch 2.3.x (TST (string dtype): avoid explicit object dtype Index in fixture data)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Thanks @jorisvandenbossche 
Manual backport -> https://github.com/pandas-dev/pandas/pull/60225
add warning about setting max_rows/max_columns to 'None'
State: closed

[I teach pandas to students who are new to coding](https://python-public-policy.afeld.me/en/nyu/syllabus.html), and [when I introduce them to `display.max_rows`](https://python-public-policy.afeld.me/en/nyu/lecture_1.html#excluding-bad-records-from-the-dataframe), their instinct is (reasonably) "why can't I just display everything?" Adding this caution to hopefully help clarify.

- [ ] ~~closes #xxxx~~
- [ ] ~~[Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature~~
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] ~~Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.~~
- [ ] ~~Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.~~


Labels:
name: Docs

Comments:
Thanks @afeld
BUG (string dtype): fix escaping of newline/tab characters in the repr
State: closed

For strings in object dtype, we escape `\n`, `\t`, `\r` characters in the series/dataframe repr. Doing the same here for the string dtype.

xref https://github.com/pandas-dev/pandas/issues/54792

Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: Strings
description: String extension data type and string data

Comments:
Thanks @jorisvandenbossche
BUG: Creation of UInt64 column with 18446744073709551615 throws RuntimeWarning
State: closed

- [ ] #60050 
- [ ] Tests added and passed
- [ ] All code checks passed


Labels:
name: Bug
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
BUG: Fix #59429 stacked bar label position
State: closed

- [x] closes #59429
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Visualization
description: plotting

Comments:
Thank you! I made the requested changes and synced the branch. This is ready for re-review after the checks complete.
[x] removed the errant documentation from `doc/source/whatsnew/v2.3.0.rst`
[x] added documentation to `doc/source/whatsnew/v3.0.0.rst`
[x] added the github issue tracking number to the test in `pandas/tests/plotting/frame/test_frame.py`
Thanks @tev-dixon!
Backport PR #60186: TST: Skip flaky offset test case on WASM
State: closed

xref https://github.com/pandas-dev/pandas/pull/60186

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Frequency
description: DateOffsets
name: Timezones
description: Timezone data dtype

Comments:
Oh I forgot we don't have our Pyodide build on the 2.3 branch so I don't think we need to backport this
TYP/COMPAT: don't use Literal for Series.ndim to avoid tab completion bug in IPython
State: closed

Not entirely sure if it is up to us to "fix" this, but given this has not been fixed upstream for several months, and it is quite annoying that tab completion doesn't work in IPython by default, this might be an easy win short-term (with only a minor decrease in correctness of the type annoations)

xref https://github.com/ipython/ipython/issues/14412, https://github.com/davidhalter/jedi/issues/1990

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: Upstream issue
description: Issue related to pandas dependency

Comments:
Thanks @jorisvandenbossche
BUG: fix inspect usage when pyarrow or jinja2 is not installed
State: closed

This started with trying to fix the xfail when using the string dtype without pyarrow installed, but in practice this was not related to the string dtype, but just to pyarrow not being available (and apparently we only test that case in the build with the future string dtype enabled?).

We need to ensure to return an AttributeError instead of ImportError when accessing attributes on Series/DataFrame, and then `inspect.getmembers` will ignore that error.


- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug

Comments:
Thanks @jorisvandenbossche
BUG (string dtype): fix where() for string dtype with python storage
State: closed

Currently `where(mask, None)` would silently set None in the underlying values, while we only allow setting a string or np.nan/pd.NA.


xref https://github.com/pandas-dev/pandas/issues/54792

Labels:
name: Bug
name: Strings
description: String extension data type and string data

Comments:
Thanks @jorisvandenbossche
TST (string dtype): update all tests in tests/frame/indexing
State: closed

This removes most xfails in `pandas/tests/frame/indexing`. The remaining xfails are actually bugs that need to be fixed (i.e. can't be fixed by just a test update)

The main case that I encountered here is when starting from an empty DataFrame, and then adding columns:

```python
df = pd.DataFrame()
df["a"] = values
```

will result in an object-dtype columns `Index` object (because we start with object dtype for the empty frame, and then preserve that dtype when inserting labels).

And when constructing the expected result through a default `pd.DataFrane(.., columns=["a"])`, you get the future string dtype for the columns `Index`.

xref https://github.com/pandas-dev/pandas/issues/54792

Labels:
name: Strings
description: String extension data type and string data

Comments:
Thanks @jorisvandenbossche 
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.3.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 bec2dbca274a4f983790d069279a4b3aec184f49
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #60193: TST (string dtype): update all tests in tests/frame/indexing'
```

4. Push to a named branch:

```
git push YOURFORK 2.3.x:auto-backport-of-pr-60193-on-2.3.x
```

5. Create a PR against branch 2.3.x, I would have named this PR:

> "Backport PR #60193 on branch 2.3.x (TST (string dtype): update all tests in tests/frame/indexing)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Manual backport -> https://github.com/pandas-dev/pandas/pull/60212
TST (string dtype): avoid hardcoded object dtype for columns in datetime_frame fixture
State: closed

Avoiding the explicit object dtype in construction of the DataFrame fixture allows to remove a few xfails

xref https://github.com/pandas-dev/pandas/issues/54792

Labels:
name: Strings
description: String extension data type and string data

Comments:
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.3.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 34387bddffacb158a60a249b08411a8a1fe44455
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #60192: TST (string dtype): avoid hardcoded object dtype for columns in datetime_frame fixture'
```

4. Push to a named branch:

```
git push YOURFORK 2.3.x:auto-backport-of-pr-60192-on-2.3.x
```

5. Create a PR against branch 2.3.x, I would have named this PR:

> "Backport PR #60192 on branch 2.3.x (TST (string dtype): avoid hardcoded object dtype for columns in datetime_frame fixture)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Manual backport -> https://github.com/pandas-dev/pandas/pull/60198
TST: add extra test case for np.array(obj, copy=False) read-only behaviour
State: closed

Small follow-up on https://github.com/pandas-dev/pandas/pull/60046 (see https://github.com/pandas-dev/pandas/pull/60046/files#r1828348323 for the discussion)

Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
Looks like a good addition, nice way to avoid `copy=None` using both.  (I am not 100% sure if there is a `copy=True` path for this particular case, but it LGTM either way.)
And noticed one more case in `DataFrame.__array__` where we currently were not raising an error for `copy=False` if that was not possible
> (I am not 100% sure if there is a `copy=True` path for this particular case, but it LGTM either way.)

For Series that is covered in the `pandas/tests/base/test_conversion.py::test_to_numpy` tests, for DataFrame will add one more case to the test I was editing anyway.
DOC: added see also for series.dt.round in series.round
State: closed

- [x] closes #60181 
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

<img width="1424" alt="Screenshot 2024-11-04 at 23 12 00" src="https://github.com/user-attachments/assets/7b7d2aa6-5ffe-4be8-96d1-697cfe51945e">



Labels:
name: Docs
name: Datetime
description: Datetime data dtype

Comments:
Thanks @Swati-Sneha
TST: Skip flaky offset test case on WASM
State: closed

e.g. https://github.com/pandas-dev/pandas/actions/runs/11661309317/job/32465371369

Not sure why this case is failing only on WASM, but IMO let's just skip this narrow case to get a clearer green signal

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Unreliable Test
description: Unit tests that occasionally fail

Comments:
Can we reproduce the case with a non-hypothesis written out test case? (or extract that from the failure logs using `reproduce_failure`>)  
Because it might still be worth keeping track of it somewhere in case someone has time to look into it or knows more about it.

But agreed to skip this for now to get green CI.

Opened up https://github.com/pandas-dev/pandas/issues/60205 to track this failure. I wasn't able to locally reproduce this on OSX.

(I suspect that for the Pyodide build we should install `tzdata` and avoid having the build look at the system timezone)
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.3.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 eea95a304795d5c0e72494a645af417e0449bc9f
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #60186: TST: Skip flaky offset test case on WASM'
```

4. Push to a named branch:

```
git push YOURFORK 2.3.x:auto-backport-of-pr-60186-on-2.3.x
```

5. Create a PR against branch 2.3.x, I would have named this PR:

> "Backport PR #60186 on branch 2.3.x (TST: Skip flaky offset test case on WASM)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
String dtype (2.3.x): avoid downcasting object to string in fillna/where/interpolate
State: closed

Noticed while backporting some test updates in https://github.com/pandas-dev/pandas/pull/60180.

For 2.x, we have a deprecation warning about "downcasting" in methods like fillna and interpolate. For example, when starting from object dtype

```python
>>> ser = pd.Series([1, 2, None], dtype=object)
>>> ser.fillna(3)
FutureWarning: Downcasting object dtype arrays on .fillna, .ffill, .bfill is deprecated and will change
in a future version. Call result.infer_objects(copy=False) instead. To opt-in to the future behavior,
set `pd.set_option('future.no_silent_downcasting', True)`
0    1
1    2
2    3
dtype: int64
```

However, because of how this is implemented, when you enable the future string dtype and still have an object dtype column, this will be inferred as string dtype and hence again trigger that warning:

```python
>>> pd.options.future.infer_string = True
>>> ser = pd.Series(["a", "b", None], dtype=object)
>>> ser.fillna("c")
FutureWarning: Downcasting object dtype arrays on .fillna, .ffill, .bfill is deprecated and will change
in a future version. Call result.infer_objects(copy=False) instead. To opt-in to the future behavior,
set `pd.set_option('future.no_silent_downcasting', True)`
0    a
1    b
2    c
dtype: str
```

So this triggers a warning for something that will no longer happen in 3.0, but when testing with `pd.options.future.infer_string = True`, we are essentially already testing behaviour for 3.0, so raising this warning results in a warning the user won't actually see in pandas 3.0.

So I was thinking that we could explicitly not dowcast to string dtype in those methods, since for the new dtype we don't need to keep backwards compatibility with old downcasting behaviour. 

This PR is directly targetting the `2.3.x` branch, given that all this downcasting behaviour is already removed on main.

xref https://github.com/pandas-dev/pandas/issues/54792

Labels:
name: Strings
description: String extension data type and string data
name: Downcasting

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Going to merge this since it blocks several other backport PRs
BUG: Convert output type in Excel for MultiIndex with period levels
State: closed

- [x] closes #60099
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/v3.0.0.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: IO Excel
description: read_excel, to_excel
name: Period
description: Period data type
name: MultiIndex

Comments:
> Looking good - can you also add one test in `excel/test_writers.py` for round-tripping (writes a DataFrame to an excel file, reads it, and asserts its the same as the original DataFrame).

My test case gets called 8 times, 6 times the dtype of result is 'datetime64[us]', and the other two times is 'datetime64[s]'. I don't know why this is happening. I had to do unit conversions for result.

> > Looking good - can you also add one test in `excel/test_writers.py` for round-tripping (writes a DataFrame to an excel file, reads it, and asserts its the same as the original DataFrame).
> 
> My test case gets called 8 times, 6 times the dtype of result is 'datetime64[us]', and the other two times is 'datetime64[s]'. I don't know why this is happening. I had to do unit conversions for result.

I've found that the dtype of `result` depends on the suffix of `tmp_excel`, only in the case of `.ods` the dtype is `datetime64[s]` and in other cases it is `datetime64[us]`. The latest PR has been completed by changing the `expected` to match `result`, please check it out. Thanks a lot!!!
And maybe it can be a new issue to unify the units of `datetime` in different tmp_excel formats.
> Can you add a note in the whatsnew for v3.0.0 under the I/O section.

Sure.
@rhshadrach I'm sorry for the trouble, but could you please let me know if there is anything else to change? Thank you very much!
Thanks @ZKaoChi!
Thanks a lot for all your help!
[backport 2.3.x] TST (string dtype): un-xfail string tests specific to object dtype (#59433)
State: closed

(cherry picked from commit dbeeb1f05bca199b3c1aed979e6ae72074a82243)

Backport of https://github.com/pandas-dev/pandas/pull/59433

Labels:

Comments:
Will wait with further updating this depending on https://github.com/pandas-dev/pandas/pull/60183
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
TST (string dtype): fix invalid comparison error message and update test
State: closed

xref https://github.com/pandas-dev/pandas/issues/54792

Labels:
name: Strings
description: String extension data type and string data

Comments:
Thanks @jorisvandenbossche 
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.3.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 9ec4a9150ef6dbf6da1248b7252141d48203d941
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #60176: TST (string dtype): fix invalid comparison error message and update test'
```

4. Push to a named branch:

```
git push YOURFORK 2.3.x:auto-backport-of-pr-60176-on-2.3.x
```

5. Create a PR against branch 2.3.x, I would have named this PR:

> "Backport PR #60176 on branch 2.3.x (TST (string dtype): fix invalid comparison error message and update test)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Manual backport -> https://github.com/pandas-dev/pandas/pull/60188