BrandonFors/pandas-github-issues huggingface dataset:
REF: Add Manager.column_setitem to set values into a single column (without intermediate series)
State: closed

The `column_setitem` is a method I was adding in https://github.com/pandas-dev/pandas/pull/46958, is something that can be broken off from that PR, and is IMO also a generally useful change.

Currently in some cases to set values into a single column (eg `df.loc[idx, "A"] = value`), we create an intermediate Series for that column, set values into that Series, and then update the DataFrame from that updated Series. I think it would be generally good to avoid creating this intermediate Series, and moving the actual/final "set" operation to the internals (as is done for other set operations).

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
@jbrockmendel apart from the two comments you just added (will update for that), do you have any remaining substantial comments? Otherwise I would want to merge this so I can update https://github.com/pandas-dev/pandas/pull/46958 on top of this.
Given there are no more comments, I am planning to merge this
Backport PR #47059 on branch 1.4.x (REGR: Styler buf and encoding in to_latex and to_html)
State: closed

Backport PR #47059

release note already backported in #47062

Labels:
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap

Comments:
hmm this is failing on backport: https://github.com/pandas-dev/pandas/runs/6523596334?check_suite_focus=true

cc @attack68
TST: add copy/view test for setting columns with an array/series
State: closed

Similarly as https://github.com/pandas-dev/pandas/pull/46979, this is adding some test cases explicitly testing copy/view behaviour for a specific topic: if we set an array or series as a column, is the new column a copy or a view of the original data? 
At this moment, we actually already do make a copy of that data (with one exception being the cached RangeIndex values). So the tests I added here are confirming that current behaviour in tests, and adding a few checks where Copy-on-Write could delay the copy. 



Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Copy / view semantics

Comments:
32-bit test issue: https://github.com/pandas-dev/pandas/runs/6522019534?check_suite_focus=true
Yes, should be fixed now!
nitpicks, otherwise LGTM.  i really liked the comments @jorisvandenbossche !
Thanks @jorisvandenbossche
CI: use github action for pyright
State: closed

pyright has a github action integration https://github.com/microsoft/pyright/blob/main/docs/ci-integration.md

I added the `--skipunannotated` option to prevent pyright from trying to infer missing return annotation (mypy doesn't do that either). That does not make a difference at the moment but it cuts reportGeneralTypeIssues errors in half (if it was enabled).

Labels:
name: CI
description: Continuous Integration
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Closing this. Will try to add mypy to the manual pre-commit stage and then run mypy&pyright through pre-commit.
Thanks @twoertwein. No objection adding it via this PR if the pre-commit version doesn't work out 
> Closing this. Will try to add mypy to the manual pre-commit stage and then run mypy&pyright through pre-commit.

This could slow down development.  When I run `mypy` on my laptop on the pandas source, it can sometimes take 3 minutes to complete.  So adding that amount of time as part of the pre-commit might be burdensome.
This was implemented as a `manual` step in pre-commit, so typing checks won't be run automatically on every local commit. It must be run manually but a manual pre-commit command is now the runner:  https://github.com/pandas-dev/pandas/pull/47075/files#diff-a5a8fc529fd73f3c8aa836fde7c12d7c331b4ca84e48252fd881d2c803c24adfR263
DOC: Fix deprecation warnings in docs for groupby nuisance columns
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Follow up to #47025

Labels:
name: Docs
name: Groupby
name: Deprecate
description: Functionality to remove in pandas
name: Nuisance Columns
description: Identifying/Dropping nuisance columns in reductions, groupby.add, DataFrame.apply

Comments:
thanks @rhshadrach
REGR: Styler `buf` and `encoding` in `to_latex` and `to_html`
State: closed

- [x] closes #47053

is regression should be backported

Labels:
name: Bug
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap

Comments:
thanks @attack68 
@meeseeksdev backport 1.4.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 1.4.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 e50dd367d6df240f34efb1fc9a5fd06d476a31de
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #47059: REGR: Styler `buf` and `encoding` in `to_latex` and `to_html`'
```

4. Push to a named branch:

```
git push YOURFORK 1.4.x:auto-backport-of-pr-47059-on-1.4.x
```

5. Create a PR against branch 1.4.x, I would have named this PR:

> "Backport PR #47059 on branch 1.4.x (REGR: Styler `buf` and `encoding` in `to_latex` and `to_html`)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 1.4.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 e50dd367d6df240f34efb1fc9a5fd06d476a31de
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #47059: REGR: Styler `buf` and `encoding` in `to_latex` and `to_html`'
```

4. Push to a named branch:

```
git push YOURFORK 1.4.x:auto-backport-of-pr-47059-on-1.4.x
```

5. Create a PR against branch 1.4.x, I would have named this PR:

> "Backport PR #47059 on branch 1.4.x (REGR: Styler `buf` and `encoding` in `to_latex` and `to_html`)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
this is leaving behind delete_me.foo files, should use tm.ensure_clean
see #47135 - hopefully fixes problem
BUG: groupby.transform/agg with engine='numba' and a MultiIndex
State: closed

- [x] closes #46867 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Groupby
name: numba
description: numba-accelerated operations

Comments:
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 1.4.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 c4027ad704c452ffb0f5ba962b5b789decbd29bd
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am "Backport PR #47057: BUG: groupby.transform/agg with engine='numba' and a MultiIndex"
```

4. Push to a named branch:

```
git push YOURFORK 1.4.x:auto-backport-of-pr-47057-on-1.4.x
```

5. Create a PR against branch 1.4.x, I would have named this PR:

> "Backport PR #47057 on branch 1.4.x (BUG: groupby.transform/agg with engine='numba' and a MultiIndex)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
BUG: `DataFrame.shift` shows different behavior for `axis=1` when `freq` is specified
State: closed

- [x] closes #47039 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Datetime
description: Datetime data dtype
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
> LGTM, failure unrelated

merge conflict needed resolving. so shouldn't fail this time with main merged.
> Needs approval from @jreback to merge.

green
@meeseeksdev backport 1.4.x
Could not push to auto-backport-of-pr-47051-on-1.4.x due to error, aborting.
ENH: move DataError from core/base.py to error/__init__.py per GH27656
State: closed

- [x] xref #27656. I think it might be best to do this GitHub issue in multiple parts
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/v1.5.0.rst` file if fixing a bug or adding a new feature.


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
thanks @dataxerik
Update mangle_dupe_cols documentation to reflect actual state of implementation
State: closed

- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This pull request updates documentation to reflect #13262 being unresolved.

Labels:
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
TYP: Mypy workaround for NoDefault
State: closed

This uses the mypy-workaround from https://github.com/python/typeshed/pull/7127/files for `NoDefault` (works also for pyright).

`no_default` is public but `NoDefault` is luckily not public, so it should be fine to do the following:

- rename the existing `NoDefault` to `_NoDefault` (implementation and typing)
- define `NoDefault` as `NoDefault = Literal[_NoDefault.no_default]` (implementation and typing; `isinstance` checks need to be re-written to use `is no_default` but all other type annotations can stay the same)
- `no_default` "remains" `no_default = _NoDefault.no_default` (just replacing `NoDefault` with `_NoDefault`)

@simonjayhawkins @jreback @Dr-Irv 


edit:

Some context: The following will work without mypy errors afterwards:
```py
def foo(x: NoDefault | int) -> int:
    if x is no_default:
        return -1
    return x + 1
```

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Something similar could also be done for `NaT` and `NaTType`
>     1. There is a PEP for sentinel values:  https://peps.python.org/pep-0661/ so if that PEP is approved, maybe we should use a pattern here that makes it easy for us to switch to using that feature if it gets approved.

Once we can use this PEP (might take a long time, especially without typing_extensions), we simply need to change the two lines that define `NoDefault` and `no_default` on top of this PR (the PEP also encourages `is` for comparison, not `isinstance`). Technically, we would then need to change `no_default` when switching to this PEP which would technically break the API as it is a public symbol.

>     2. There is a lot of usage of `Literal["left", "right"]` in this PR,  so maybe we should create a type for that in `_typing.py` and use that instead.

Will do

> Once we can use this PEP (might take a long time, especially without typing_extensions), we simply need to change the two lines that define `NoDefault` and `no_default` on top of this PR (the PEP also encourages `is` for comparison, not `isinstance`). Technically, we would then need to change `no_default` when switching to this PEP which would technically break the API as it is a public symbol.

OK, I see the path forward.  However, while `no_default` is a public symbol, we don't document what it does, so I don't think changing it to be a sentinel will break anything.  In any case, that is something that wouldn't need to be done for a few years.


Thanks @twoertwein
ENH: DatetimeArray fields support non-nano
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Enhancement
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
Thanks @jbrockmendel
ENH: Incorproate ArrowDtype into ArrowExtensionArray
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Not fully user facing yet.

Supersedes https://github.com/pandas-dev/pandas/pull/46972

cc @jbrockmendel let me know if this is what you had in mind

Labels:
name: Enhancement
name: Arrow
description: pyarrow functionality

Comments:
yah this is what i  had in mind
Have this in a pretty good state now.

* Not user facing yet
* `BaseConstructorsTests` and `BaseGetitemTests` are running. Will plan to add more in subsequent PRs
> In [30]: arr1.dtype == arr2.dtype

Awesome thanks for the second review.

I was able to address the dtype `__eq__` comparison by adding `pyarrow_dtype` to `_metadata`. I am hoping this will be tested once I add the other base extension tests in subsequent PRs.
Updated & all green
DOC: Fix a typo in documentation for qyear
State: closed

This fixes a typo that I ran across in the [documentation](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Period.qyear.html?highlight=qyear#pandas.Period.qyear). This change corrects the wording to match the given example that is just below.

Labels:
name: Docs
name: Frequency
description: DateOffsets

Comments:
Great, thanks @bjcubsfan
CI: Start Testing on Python 3.11
State: closed

- [ ] xref #46680 (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

It's that time of the year again.

Surprisingly, there are not a lot of failures this year. 

Labels:
name: CI
description: Continuous Integration
name: Stale
name: Python 3.11

Comments:
Thanks for the effort!

[Cython](https://github.com/cython/cython) 0.29.29 and 0.29.30 were released and [NumPy](https://github.com/numpy/numpy) bumped their Cython requirement on the main branch, so I think if you rebase this (and thereby trigger the CI again), it will come a bit further.
I think we already picked up 0.29.29 here(it compiles fine). I can try again with 0.29.30, but sadly these test failures look real.
@lithomas1 Any progress on this effort? Should we open issues for the test failures?
@EwoutH No, they are mainly because an error message changed, and so should be relatively easy to fix. Does anyone downstream need this(can prioritize this if so)? 

I'm a bit short on time right now, so I'll be squeezing in commits here and there, but it might take me a while to fix up all the tests. 

You might also want to hold off reviewing right now, as there are going to be a lot of WIP commits, and I don't want to waste your time reviewing. I can ping you when this is ready.
@github-actions pre-commit
@github-actions pre-commit
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
@lithomas1 Thanks for the effort so far. Are you still working on this?
This PR is currently superseded by #47442. That PR is still blocked by #47467. I expect to finish up #47467 and get it merged within the next week.
Backport PR #47015 on branch 1.4.x (CI: Ensure no-use-pep517 with no-build-isolation with new pip version)
State: closed

Backport PR #47015: CI: Ensure no-use-pep517 with no-build-isolation with new pip version

Labels:
name: Build
description: Library building on various platforms
name: CI
description: Continuous Integration

Comments:
py38_32bit failing since this PR was opened before #47029 was merged. can ignore this and merge once other jobs are green.
Fix: Pandas rolling removes imaginary part of complex
State: closed

- [ ] closes #46619
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
I close this since I stuck this for long time. I not familiar much with cython part. For now, I not able to find examples to handle fused type (complex + float) I get other task which I think can complete. Thank for review.
WIP: ENH Add float[pyarrow] dtype
State: closed

- Basing this off https://github.com/pandas-dev/pandas/pull/46972
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Hello @gsheni! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:













There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-05-15 03:47:56 UTC
@github-actions pre-commit
Thanks for starting this. Just a heads up, I think we won't be expanding support for pyarrow dtypes quite like this (the subclass approach).

If you'd like to help of this front, investigating how `StringArrowArray` and `StringDtype` for pyarrow can be further merged into `ArrowExtensionArray` and `ArrowDtype` respectively would be helpful instead.
DEPR: groupby numeric_only default
State: closed

- [x] closes #46072 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

There are two cases we want to emit a deprecation warning for DataFrameGroupBy:
 - numeric_only is not specified and columns get dropped. In this case emit a warning that the default of numeric_only will change to False in the future.
 - numeric_only is specified to False and columns still get dropped. In this case emit a warning that the op will raise in the future.

Labels:
name: Groupby
name: Deprecate
description: Functionality to remove in pandas
name: Nuisance Columns
description: Identifying/Dropping nuisance columns in reductions, groupby.add, DataFrame.apply

Comments:
Thanks @jreback; filterwarnings has been removed and the helper had been added.
very nice @rhshadrach 
I think this causes the doc build to fail with multiple of these warnings:

> <ipython-input-97-5805a339bf74>:1: FutureWarning: The default value of numeric_only in DataFrameGroupBy.sum is deprecated. In a future version, numeric_only will default to False. Either specify numeric_only or select only columns which should be valid for the function.
>   (bb.groupby(['year', 'team']).sum()

Would probably need to update the documentation to avoid these FutureWarnings.
Thanks @twoertwein - will do a follow up.
DOC: Fix some typos in pandas/.
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
can you merge master; i think this included some patches in the previous PR.
> 

I just merged the current main branch with this RP and found no conflicts.
thanks @Yulv-git