BrandonFors/pandas-github-issues huggingface dataset:
DOC: Improve documentation for DataFrame.__setitem__ and .loc assignment from Series
State: closed

- [x] closes #61662
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

The core problem is that when assigning a Series, pandas aligns on index and values in the Series that don't match an index label will result in NaN [DOC: Improve documentation for DataFrame.__setitem__ and .loc assignment from Series · Issue #61662 · pandas-dev/pandas](https://github.com/pandas-dev/pandas/issues/61662), but this behavior is poorly documented.
My proposed solution addresses the issue comprehensively by:

- Adding a complete docstring for DataFrame.__setitem__ with clear explanations and examples
- Enhancing the .loc documentation with specific notes about Series alignment
- Expanding the user guide with a dedicated section on Series assignment and index alignment
- Including comprehensive test cases to ensure the behavior is well-tested

The fix emphasizes that pandas performs index-based alignment rather than positional assignment, which is the source of confusion for many users. The documentation will now clearly explain that when you assign a Series to a DataFrame column, pandas matches values by index labels, not by position, and missing labels result in NaN values.
This solution follows pandas' documentation conventions and provides both reference documentation and practical examples that will help users understand and correctly use this important feature.

Labels:
name: Docs

Comments:
@WillAyd Any thought on this resolution? 
Thanks @niruta25 for the PR

> * Expanding the user guide with a dedicated section on Series assignment and index alignment

I see that "align" is found 16 times when searching "Intro to data structures" section of the docs. This chapter is only preceded by "10 minutes to pandas" so i'm not sure that the linked issue which states "The current documentation is incomplete and vague about how Series alignment works in assignments." is correct that this fundamental paradigm of pandas is not covered in the documentation.

I'm not a member of the documentation team so others may be more positive to these changes, but if I was  to review this PR, I would prefer to see more discussion on the issue itself before proceeding to the PR stage.
I do not have access to merge this PR. Can you please help. 
Thanks @niruta25
TST: update expected dtype for sum of decimals with pyarrow 21+
State: closed

This should fix the failure we started having for the pyarrow nightly build (behaviour change in https://github.com/apache/arrow/pull/44184 to increase the precision of the resulting decimal for sum)

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Arrow
description: pyarrow functionality

Comments:
Apologies for going quickly here, but going to merge this to have green CI on 2.3.x for releasing
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.3.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 ebca3c56c1f9454ef5d2de5bb19ce138e1619504
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #61799: TST: update expected dtype for sum of decimals with pyarrow 21+'
```

4. Push to a named branch:

```
git push YOURFORK 2.3.x:auto-backport-of-pr-61799-on-2.3.x
```

5. Create a PR against branch 2.3.x, I would have named this PR:

> "Backport PR #61799 on branch 2.3.x (TST: update expected dtype for sum of decimals with pyarrow 21+)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Manual backport -> https://github.com/pandas-dev/pandas/pull/61801
DOC: add section about upcoming pandas 3.0 changes (string dtype, CoW) to 2.3 whatsnew notes
State: closed

This is largely copied from the equivalent notes in the 2.2 release notes at https://pandas.pydata.org/pandas-docs/stable/whatsnew/v2.2.0.html#upcoming-changes-in-pandas-3-0, with some updates (and some new content copied from WIP 3.0 release notes in https://github.com/pandas-dev/pandas/pull/61724)

Labels:
name: Docs

Comments:
/preview
LGTM @jorisvandenbossche is the "TODO add link to migration guide" going to be done here or a follow up?
There is a mix of `code-block:: python` and `code-block:: ipython`. Intentional?
> LGTM @jorisvandenbossche is the "TODO add link to migration guide" going to be done here or a follow up?

Planning to merge that migration guide now first and then update here with a link
> There is a mix of `code-block:: python` and `code-block:: ipython`. Intentional?

Fixed (although it was for code blocks where no code prompt was included, and at that point there is not really a difference, I think)
/preview
Website preview of this PR available at: https://pandas.pydata.org/preview/pandas-dev/pandas/61795/
Backport PR #61770 on branch 2.3.x (BUG: Fix unpickling of string dtypes of legacy pandas versions)
State: closed

Backport PR #61770: BUG: Fix unpickling of string dtypes of legacy pandas versions

Labels:
name: Bug
name: Strings
description: String extension data type and string data
name: IO Pickle
description: read_pickle, to_pickle

Comments:
I am adding an ignore for a new numpy deprecation warning. Although it also happens to occur in a pickle test, it is not actually related to the changes in this PR. It just started happening now because the linked numpy PR was merged a few days ago (the same failures can be seen on the 2.3.x branch). 
And it is also only for a test that is already removed in the main branch (testing py2 compat in pickle), so just ignoring here seems fine
TST: assert reading of legacy pickles against current data
State: open

While reviewing https://github.com/pandas-dev/pandas/pull/61770, I noticed that we didn't actually compare the read pickle data to some ground truth expected value, but just to itself (we were essentially doing `assert_equal(result, result)` ..), due to some accidental change in a clean-up many years ago in https://github.com/pandas-dev/pandas/commit/f2246cfa215d01b68aebd2da4afb836d912d248d)

Fixing that here by again creating the expected unpickled data with `create_pickle_data()` during the test run, to compare with the data from the older pickled files.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO Pickle
description: read_pickle, to_pickle
name: Stale

Comments:
can you merge main and see if the pyarrow decimal issue resolves itself?
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
DOC: Add link to WebGL in pandas ecosystem
State: closed

Add link to WebGL in pandas ecosystem.
https://www.khronos.org/webgl/

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks @star1327p
CLN: remove and udpate for outdated _item_cache
State: closed

- [x] closes #61746 
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Thanks @chilin0525
REF: remove unreachable, stronger typing in parsers.pyx
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO CSV
description: read_csv, to_csv

Comments:
Thanks @jbrockmendel
TST: option_context bug on Mac GH#58055
State: closed

- [x] closes #58055 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Thanks @jbrockmendel
CI: Add NumPy 1.26 test job
State: closed

This PR adds a CI job to test Pandas with NumPy 1.26 to ensure compatibility with the latest version.

- Related to issue: [#61588](https://github.com/pandas-dev/pandas/issues/61588)
- Installs NumPy 1.26.0 explicitly and runs the full test suite
- Helps identify future compatibility issues with NumPy releases

### Checklist

- [x] Closes #61588
- [x] Tests added and passed (via CI job)
- [x] All code checks passed (linting, CI)
- [ ] No new type hints were added
- [ ] No doc entry needed (not a new feature or bugfix)

Labels:
name: CI
description: Continuous Integration
name: Dependencies
description: Required and optional dependencies

Comments:
Thanks @Anantanand005 for the PR

closing as superseded by #61806
BUG: Decimal(NaN) incorrectly allowed in ArrowEA constructor with tim…
State: closed

…estamp type

- [x] closes #61775
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Surfaced by #61732

Labels:
name: Arrow
description: pyarrow functionality

Comments:
Huh. the "ArrowInvalid: overflow" here is weird.  will look into it
Thanks @jbrockmendel
BUG[string]: incorrect index downcast in DataFrame.join
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Strings
description: String extension data type and string data

Comments:
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.3.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 cf1a11c1b49d040f7827f30a1a16154c80c552a7
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #61771: BUG[string]: incorrect index downcast in DataFrame.join'
```

4. Push to a named branch:

```
git push YOURFORK 2.3.x:auto-backport-of-pr-61771-on-2.3.x
```

5. Create a PR against branch 2.3.x, I would have named this PR:

> "Backport PR #61771 on branch 2.3.x (BUG[string]: incorrect index downcast in DataFrame.join)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Thanks @jbrockmendel!
Manual backport -> https://github.com/pandas-dev/pandas/pull/61800
BUG: Fix unpickling of string dtypes of legacy pandas versions
State: closed

- [x] closes #61763
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.3.1.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Strings
description: String extension data type and string data
name: IO Pickle
description: read_pickle, to_pickle

Comments:
Thanks very much for the review @jorisvandenbossche, I've added pickles for 2.0-2.2 as extra checks and a whatsnew entry.
BUG: NA.__and__, __or__, __xor__ with np.bool_ objects
State: closed

- [x] closes #58427 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

I expected this to break some other tests, but nope.

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
Thanks @jbrockmendel
BUG: ensure to_numeric down-casts to uint64 for large unsigned integers
State: closed

[to_numeric(..., downcast="unsigned")](cci:1://file:///d:/Github/pandas/pandas/core/tools/numeric.py:48:0-315:21) failed to honour the requested
[uint64](cci:1://file:///d:/Github/pandas/pandas/tests/tools/test_to_numeric.py:580:0-586:44) dtype when values exceeded `np.iinfo(np.int64).max`, returning
`float64` instead and losing integer precision (GH #14422 /
[test_downcast_uint64](cci:1://file:///d:/Github/pandas/pandas/tests/tools/test_to_numeric.py:580:0-586:44)).  
Added a fallback that detects integral, non-negative float results and
safely casts them to `np.uint64`.  All existing logic remains unchanged
for other code paths; the previously xfailed test now passes.


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
ENH: speed up wide DataFrame.line plots by using a single LineCollection
State: open

<!--
Thank you for your contribution to pandas!
Please fill in the checklist below.  Mark completed items with an “x”.
-->

### What does this PR change?

* **Speeds up `DataFrame.plot(kind="line")` when the frame is “wide”.**  
* If the DataFrame has **> 200 columns**, a **numeric index** (e.g. `RangeIndex`
  or integer/float values), is **not** a time-series plot, has **no stacking**
  and **no error bars**, we now draw everything with a single
  `matplotlib.collections.LineCollection` instead of one `Line2D` per column.
* No API changes; behaviour is identical for smaller plots or the excluded
  cases above.

### Performance numbers

| 500 rows × 2000 cols (RangeIndex) | master | this PR | speed-up |
|-----------------------------------|--------|---------|----------|
| `df.plot(legend=False)`           | 0.342 s| 0.069 s | **5×** |

*Benchmarked on pandas **3.0.0.dev0+2183.g94ff63adb2**, matplotlib **3.10.3**, NumPy **2.2.6***  

### Notes

* This PR does _not_ change anything for `DatetimeIndex` plots—those remain on the original per-column path. A follow-up could combine `LineCollection` with the `x_compat=True` workaround (see [#61398](https://github.com/pandas-dev/pandas/issues/61398)) to similarly speed up time-series plots.
* Threshold (`> 200` columns) is a heuristic and can be tuned in review.
* The fast path activates only for numeric indices. Datetime/period/timedelta
  indices still use the original per-column draw, so behaviour there is
  unchanged.

---

- [x] closes **#61532**
- [x] tests added / passed (`pytest pandas/tests/plotting -q`)
- [x] code checks passed (`pre-commit run --all-files`)
- [x] added entry in `doc/source/whatsnew/v3.0.0.rst`

cc @shadnikn @arthurlw – happy to take any feedback 🙂


Labels:
name: Visualization
description: plotting
name: Performance
description: Memory or execution speed performance

Comments:
@jbrockmendel Done in the latest commit, thanks!
chore: remove redundant words in comment
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


remove redundant words in comment

Labels:
name: Docs

Comments:
Thanks @ianlv
TST/CI: temporary upper pin for scipy in downstream tests for compat with statsmodels
State: closed

See https://github.com/statsmodels/statsmodels/issues/9542 / https://github.com/statsmodels/statsmodels/issues/9584

Labels:
name: CI
description: Continuous Integration

Comments:
Thanks @jorisvandenbossche 
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.3.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 7f783db6dcc4ef200643fee845e9565f0a97f37f
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #61750: TST/CI: temporary upper pin for scipy in downstream tests for compat with statsmodels'
```

4. Push to a named branch:

```
git push YOURFORK 2.3.x:auto-backport-of-pr-61750-on-2.3.x
```

5. Create a PR against branch 2.3.x, I would have named this PR:

> "Backport PR #61750 on branch 2.3.x (TST/CI: temporary upper pin for scipy in downstream tests for compat with statsmodels)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Backport -> https://github.com/pandas-dev/pandas/pull/61750