BrandonFors/pandas-github-issues huggingface dataset:
CoW: Delay copy when enlarging df with loc
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

on top of #52051

Labels:
name: Copy / view semantics

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
cc @jorisvandenbossche 
This is the last one, will make a clean up afterward
> This is the last one, will make a clean up afterward

When cleaning up, can maybe also add a note about it in the whatsnew that setitem now tries to delay copy as well.
CoW: Delay copy when inserting Series into DataFrame
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

sits on top of #51698

Labels:
name: Copy / view semantics

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
cc @jorisvandenbossche
API CoW: Return read_only NumPy array from ravel
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

cc @jorisvandenbossche thoughts here?

Labels:
name: Copy / view semantics

Comments:
I'll merge this since it should really make it into 2.0
Sorry for letting this slip through, thanks for merging!

> thoughts here?

My thought is that we should deprecate `ravel()`, but yes as long as we have it, it should behave similarly as the other methods that give a numpy array.
BUG: to_sql with ArrowExtesionArray
State: closed

- [ ] closes #52046 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Includes a fix for an oversight referenced in https://github.com/apache/arrow/issues/34614


Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query
name: Arrow
description: pyarrow functionality

Comments:
Some of the build timeout should be addressed with https://github.com/pandas-dev/pandas/pull/52107
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.0.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 3083ae932e0b122e963608fb25cf3df06d588b03
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #52058: BUG: to_sql with ArrowExtesionArray'
```

4. Push to a named branch:

```
git push YOURFORK 2.0.x:auto-backport-of-pr-52058-on-2.0.x
```

5. Create a PR against branch 2.0.x, I would have named this PR:

> "Backport PR #52058 on branch 2.0.x (BUG: to_sql with ArrowExtesionArray)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
PERF: Improve performance with copy=True and different dtype
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This cuts runtime roughly in half when dtype is different than the dtype from values

```
# main
%timeit DataFrame(arr, dtype="int32", copy=True)
6.01 ms ± 37.9 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

# pr
%timeit DataFrame(arr, dtype="int32", copy=True)
3.54 ms ± 25.4 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

Labels:
name: Performance
description: Memory or execution speed performance
name: DataFrame
description: DataFrame data structure

Comments:
Thanks @phofl
REF: use WrappedCythonOp for GroupBy.std, sem
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Slight slowdown (but there _should_ be a decreased memory footprint) for `std`, pretty nice speedup for `sem`:

```
import pandas as pd
import numpy as np

arr = np.random.randn(10**5, 3)
df = pd.DataFrame(arr)
gpr = np.arange(2).repeat(len(arr) // 2)
gb = df.groupby(gpr)

%timeit gb.std()
790 µs ± 55.9 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)  # <- main
803 µs ± 34.4 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)  # <- PR

%timeit gb.sem()
2.73 ms ± 150 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)    # <- main
796 µs ± 25.4 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)  # <- PR
```

Labels:
name: Refactor
description: Internal refactoring of code
name: Groupby
name: Reduction Operations
description: sum, mean, min, max, etc.

Comments:
Thanks @jbrockmendel
CoW: Delay copy when setting Series or Frame with isetitem
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

sits on top of #51698

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Copy / view semantics

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
cc @jorisvandenbossche
REF: uses_mask in group_any_all
State: closed

any, all, and std go through GroupBy._get_cythonized_result instead of the more-standard WrappedCythonOp.  I'm trying to refactor any/all to use the other path, and as a step toward that am trying to make group_any_all follow the same patterns as the other functions in libgroupby.

The implementation here looks to me like it should behave identically to the existing implementation, but it fails a bunch of tests (mostly in tests/groupby/test_any_all.py).  Hoping @phofl can explain where I'm going wrong.

Labels:
name: Groupby

Comments:
Your suggestion helped, green
thx @jbrockmendel
DataFrame/Series at set value test GH#30649
State: closed


- [ ] closes #30649 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
Hi @liang3zy22 thanks for the PR!

By running the example code in #30649 in the main branch, It seems we still get the errononous behavior, do you agree, i.e. this PR doesn't close that issue? 
Oh, so 30649 is not fixed. My codes just tested what it behaves now.
Yes :-). Want to search for how to fix it? 
Yeah @topper-123 , please give me some info about how to fix this issue. I want to fix it.
This may potentially not be so easy and it can take quite some time to for me also to figure out what need to be fixed here. So you need to either dig into this, or find some other issues you're interested in. Sorry. 

If you're interested in working on some other good starter issue, you can look into some of the issues at https://github.com/pandas-dev/pandas/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22

If you decide you don't want to keep looking into this issue, can you close this PR.
ok, I close my PR. Hope this issue can be fixed soon.
CoW: Optimize Series.reset_index to make lazy copy
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Copy / view semantics

Comments:
Thanks!
BUG: Remove unnecessary validation to non-string columns/index in df.to_parquet
State: closed

- [x] closes #52034 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO Parquet
description: parquet, feather

Comments:
thx @mroeschke 
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.0.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 88f2df5f65e8ff575ee7bdbf2bfa129893573f62
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #52036: BUG: Remove unnecessary validation to non-string columns/index in df.to_parquet'
```

4. Push to a named branch:

```
git push YOURFORK 2.0.x:auto-backport-of-pr-52036-on-2.0.x
```

5. Create a PR against branch 2.0.x, I would have named this PR:

> "Backport PR #52036 on branch 2.0.x (BUG: Remove unnecessary validation to non-string columns/index in df.to_parquet)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
BUG-CoW: DataFrame constructed from Series not respecting CoW
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

right now we back the DataFrame by a read-only array, before the read-only pr this did not respect CoW.

Follow-up will be to do the same for Index.

Labels:
name: Copy / view semantics

Comments:
I'll merge this to get it into 2.0 and to make the follow up
BUG-CoW: from_records not tracking references when called with df
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.



Labels:
name: DataFrame
description: DataFrame data structure
name: Copy / view semantics

Comments:
(has some conflicts now)
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.0.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 db12473eb78b176b90fb8436f2d6e4be4d22f579
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #52029: BUG-CoW: from_records not tracking references when called with df'
```

4. Push to a named branch:

```
git push YOURFORK 2.0.x:auto-backport-of-pr-52029-on-2.0.x
```

5. Create a PR against branch 2.0.x, I would have named this PR:

> "Backport PR #52029 on branch 2.0.x (BUG-CoW: from_records not tracking references when called with df)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
API / CoW: Copy arrays by default in Series constructor
State: closed

- [ ] closes #50776 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

will update the whatsnew from the other copy=True pr

Labels:
name: Series
description: Series data structure
name: Copy / view semantics

Comments:
merging this to be consistent with the DataFrame constructor
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.0.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 d95fc0b17d32b490000277e0f9a10e2de2c63a8d
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #52022: API / CoW: Copy arrays by default in Series constructor'
```

4. Push to a named branch:

```
git push YOURFORK 2.0.x:auto-backport-of-pr-52022-on-2.0.x
```

5. Create a PR against branch 2.0.x, I would have named this PR:

> "Backport PR #52022 on branch 2.0.x (API / CoW: Copy arrays by default in Series constructor)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
CI: xfail strict=True TestClipboard.test_raw_roundtrip in a CI environment
State: closed

Unfortunately clearing out the clipboard in the CI didn't fix the flaky failures. Setting `strict=False` if in our CI

https://github.com/pandas-dev/pandas/actions/runs/4433507399/jobs/7778613334

Labels:
name: IO Data
description: IO issues that don't fit into a more specific label
name: Unreliable Test
description: Unit tests that occasionally fail

Comments:
thx @mroeschke 
Is the clipboard stuff commonly used?  If we're effectively not testing it on the CI, might be worth deprecating support?
No clue if this is actually used. 
I've used it a few times; can be convenient when copy/pasting tables online. But 100% think it should be spun off into it's own IO community project
DOC: remove private parameters from method/constructor signatures
State: closed

- [x] closes #52028
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


The documentation (e.g. for Series constructor) contain the parameters which are used only internally (fastpath in case of Series). Now if a parameter is private and prefixed with an underscore `_` it will not be part of the signature in the output.


Labels:

Comments:
@phofl thanks for the review. Is there any guide on how we do the deprecation?
You should start by opening an issue to discuss it
Created #52028 for the discussion.
Seems like the issue needs more discussion before moving forward with a PR so closing for now, but we can reopen when theres more consensus
DEPR: deprecate axis keyword in GroupBy methods
State: closed

- [x] closes #50405 (Replace xxxx with the GitHub issue number)
- [x] closes #51046
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

@rhshadrach can you confirm this closes #50405

Labels:
name: Groupby
name: Deprecate
description: Functionality to remove in pandas

Comments:
Thanks @jbrockmendel
BUG: Series constructor not respecting CoW when called with BlockManager
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Copy / view semantics

Comments:
BTW we should deprecate accepting managers in the constructor and wean 3rd party libraries off of it
> BTW we should deprecate accepting managers in the constructor and wean 3rd party libraries off of it

If we want to pursue that, I think it would be best to start with proposing and implementing an alternative option to use, since we still need _some_ way to do this for both advanced users (like pyarrow) and subclasses (they need to be able to customize this constructor from a manager, xref https://github.com/pandas-dev/pandas/issues/51772)

(I personally don't have a strong opinion about this. I think it would certainly make sense to have a separate constructor if we were designing it, just not fully sure it is worth changing now)
Addressed all comments, will merge to get in.