BrandonFors/pandas-github-issues huggingface dataset:
Specify that both ``by`` and ``level`` should not be specified in ``groupby`` - GH40378 
State: closed

- [x] closes #40378 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Ref: https://github.com/pandas-dev/pandas/pull/47778#discussion_r923860446

Labels:
name: Docs
name: Groupby

Comments:
Thanks for the pull request, but I believe this was addressed by https://github.com/pandas-dev/pandas/pull/47778?

Also, please fill out the pull request template to give more context to reviewers. Thanks.
@mroeschke: https://github.com/pandas-dev/pandas/pull/47778#discussion_r923860446

@GivyBoy: When doing a PR that is following up from a conversation, it's helpful for reviews today as well as developers way in the future to reference it in the OP. So I would recommend adding here:

Ref: https://github.com/pandas-dev/pandas/pull/47778#discussion_r923860446
@GivyBoy - I think this looks good; can you merge main and resolve conflicts.
> In the user guide we're losing the sentence
> 
> > When using `.groupby()` on a DatFrame with a MultiIndex, do not specify both `by` and `level`.
> 
> But I think that's okay (API docs seems sufficient to me). @mroeschke - thoughts?

Yeah I think that's fine. Generally I favor things in the API docs over the user guide as I'm guessing the API docs get more visibility.
Thanks @GivyBoy
Fixed metadata propagation in Dataframe.idxmax and Dataframe.idxmin #…
State: closed

…28283

- [x] contributes to #28283  calling .__finalize__ on Dataframe.idxmax, idxmin


Labels:
name: Bug
name: metadata
description: _metadata, .attrs

Comments:
Hello @SomtochiUmeh! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-07-22 18:16:36 UTC
Also, please see https://pandas.pydata.org/docs/development/contributing_codebase.html#pre-commit for how to fix that CI failure
Thanks @SomtochiUmeh
ENH: Add ArrowDype and .array.ArrowExtensionArray to top level
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Moved `ArrowIntervalType` and `ArrowPeriodType` to a separate file `arrow/extension_types.py` to avoid a circular import.

Will add documentation in a separate PR https://github.com/pandas-dev/pandas/pull/47854

Labels:
name: Arrow
description: pyarrow functionality

Comments:
1. Definitely will mark `ArrowDtype` experimental for 1.5 (was planning on the subsequent doc PR)

2. At least for 1.5, I would like users to be able to create other arrow-backed dtypes from the constructor e.g. `Series(..., dtype=int64[pyarrow])` i.e. mimic what is possible now with the `string[pyarrow]` functionality.

3. I'm fine with not deprecating things in 1.5. I would just like to highlight that `"string[pyarrow]"` can be understood by both `StringDtype(storage="pyarrow")` and `ArrowDtype`, and at a high level, the functionality of `ArrowExtensionArray` can be superseded by `StringArrowArray` in the future if the `str` methods are moved over.
`ArrowDtype` I think should be included in the top level because arrow-types-with-parameters cannot always be constructed from a string e.g.

```
In [5]: dtype = pd.core.arrays.arrow.dtype.ArrowDtype(pa.timestamp(unit="s", tz="UTC"))

In [6]: pd.Series([datetime.datetime.now()], dtype=dtype)
Out[6]:
0    2022-07-25 10:58:58+00:00
dtype: timestamp[s, tz=UTC][pyarrow]
```

`ArrowExtensionArray` I only really included since all the arrays are exposed in `pandas.arrays`. One use case for exposing this in `pandas.arrays` is if the user data has an arrow array object they want to bring into pandas e.g.

```
In [7]: pa_array = pa.array([1, 2])

In [8]: pd.Series(pd.core.arrays.arrow.ArrowExtensionArray(pa_array))
Out[8]:
0   1
1   2
dtype: int64[pyarrow]
```

https://github.com/pandas-dev/pandas/pull/47854 would be a start to the documentation of the functionality I was hoping to demonstrate
I'm fine with adding these to the top level (reasoning: just having support for *storage* of these types in a DataFrame is helpful, even if many operations fail).  Agreed with Joris that we shouldn't deprecate `StringDtype(storage="arrow")`, at least not yet.
Okay agreed. Considering deprecation for `ArrowStringArray/StringDtype(storage="pyarrow")` is definitely out of scope here and the near term.
@jorisvandenbossche as discussed in the dev meeting today, documentation noting that this is experimental was made in https://github.com/pandas-dev/pandas/pull/47854/commits/4e3b3b993f250231285f69d951bdf5f131f58f13 in https://github.com/pandas-dev/pandas/pull/47854
Will probably merge this PR this week if no further comments.
Going to merge. Can address any followups if needed in https://github.com/pandas-dev/pandas/pull/47854
CI/TYP: run stubtest
State: closed

Stubtest enforces that pyi files do not get out of sync with the implementation.

stubtest and mypy seems to have a slight discrepancy (had to change annotations so that they both error in the same cases).

stubtest needs pandas dev to be installed. This might not be something people want to do locally: if pandas dev is not installed: `scripts/run_stubtest.py` will 'succeed' but print a warning (the CI has pandas dev installed).

Might want to wait for the CI using mypy 0.971 (#47806 waiting for conda).

@hauntsaninja: The allow list acts more or less like ignore comments, is there an option to error on unused allow list entries?

Closes #47760

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
> I took a second look at the output of `python -m mypy.stubtest pandas --mypy-config-file pyproject.toml --concise` (i.e. don't just run on the pyi files)

What would be the added benefit of doing this? Shouldn't mypy find all those issues already?

> b) There are a lot of overloads where things are typed as keyword only because of `deprecate_nonkeyword_arguments`, but aren't "actually" keyword only. I assume at some point the deprecations will turn into removals.

The idea is that the type annotation is a kind of an added "deprecation" warning when people use a type checker while using deprecated behavior (and probably more importantly it let's us move a bit more quickly with the annotation process). I expect that the implementation will be keyword-only after the 1.5 release (running subtest on all files will be helpful to make all of those changes).
> What would be the added benefit of doing this? Shouldn't mypy find all those issues already?

Yes, sorry. My post was more notes to myself than anything immediately helpful for this PR. I was just thinking through things that would make stubtest more user friendly, so fewer people have to write wrapper scripts around it :-)
I assume this PR doesn't need to wait for mypy 0.971 (seems to already work)
> I was just thinking through things that would make stubtest more user friendly, so fewer people have to write wrapper scripts around it :-)

Two suggestions:

- if stubtest is not meant as a mypy++ (does what mypy does and more), it might be cool to ignore any errors in function bodies as stubtest will not check function internals (might even speed stubtest up).
- if stubtest is meant as mypy++, it might be nice to document that running mypy in addition to stubtest is redundant (assuming stubtest is run on all files not just on files that have a pyi file)
Could go before or after #47806. In either case, the other PR should be rebased before merging.
stubtest is not intended as a "mypy++". Made this clearer in the documentation here: https://github.com/python/mypy/pull/13293

Seeing if we can skip some work / ignore errors that block the actual stubtest logic from running is a good suggestion, I will see what improvements can be made there :-)
Thanks @twoertwein
BUG: fix SparseArray.unique IndexError and _first_fill_value_loc algo
State: closed

- [x] closes #47809
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Only on master, whatsnew omitted. Please review #47779 after this merged. Remind that `pd.unique`/`SparseArray` and `np.unique` is not the same. The former one will keep the original order (which is from pd.core.algorithms.unique) and numpy unique will sort the result. cc @mzeitlin11 

Labels:
name: Bug
name: Sparse
description: Sparse Data Type

Comments:
Since 1.4.3 and main behave differently, I change the issue title according to the 1.4.3 behavior since it's more closer to the users. In whatsnew, since error on main will not be visible to users, I choose to give the bug description from 1.4.3.
Thanks @GYHHAHA!
DOC: add pivot_table note to dropna doc
State: closed

- [ ] closes #47447 
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks for the pull request @Mehgarg 

The pandas documentation is generated automatically from markdown files and docstring (documentation in the code). We never edit the final html files directly.

I think the changes you want to implement should be done to this docstring instead: https://github.com/pandas-dev/pandas/blob/main/pandas/core/frame.py#L6427

Can you revert this html file you're creating and edit the docstring please? Let me know if you need help
Hello @Mehgarg! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-07-29 14:53:38 UTC
Is the failed check expected behavior or do I need to fix something?
Thanks @Mehgarg
TST/CI: xfail test_round_sanity for 32 bit
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

xfailing just to get the CI to green. Might be addressable in the future cc @jbrockmendel 


Labels:
name: CI
description: Continuous Integration
name: Unreliable Test
description: Unit tests that occasionally fail

Comments:
Going to merge this to get the CI back to green. Will open an issue to investigate this issue.
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 1.4.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 8c7b0b268436cb4b3755de8d7cb9447957ec2bf8
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #47803: TST/CI: xfail test_round_sanity for 32 bit'
```

4. Push to a named branch:

```
git push YOURFORK 1.4.x:auto-backport-of-pr-47803-on-1.4.x
```

5. Create a PR against branch 1.4.x, I would have named this PR:

> "Backport PR #47803 on branch 1.4.x (TST/CI: xfail test_round_sanity for 32 bit)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
xref https://github.com/pandas-dev/pandas/issues/47823
TST: Test series construct of dtype timedelta64 #35465
State: closed

- [x] closes #35465 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Timedelta
description: Timedelta data type

Comments:
Hello @Th3nn3ss! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-07-27 12:21:13 UTC
Thanks. I wasn't really sure where to put the tests. I will make changes and update the PR tomorrow. 😄 
> Thanks. I wasn't really sure where to put the tests. I will make changes and update the PR tomorrow. 😄

Maybe `pandas/tests/series/test_constructors.py`
Appears some test build are still failing
Hi @mroeschke  How do I fix the failing build tests?
Try merging in main one more time. We had a flaky test that was affecting all builds last week
Thanks @Th3nn3ss
DEPS: drop np19 support
State: closed


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: Dependencies
description: Required and optional dependencies

Comments:
Hello @fangchenli! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

























There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-08-01 20:03:16 UTC
thanks @fangchenli
BUG: fixed OutOfBoundsDatetime exception when errors=coerce #45319
State: closed

- [ ] closes #45319
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Datetime
description: Datetime data dtype
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
This function that this change is in doesn't care about the error variable, shown by the fact that there is no argument for the errors in the code.  This function will always return NaT, and the raise and coerce cases are dealt with later in the program.
Thanks @srotondo
TST: test for inconsistency due to dtype=string #46512
State: closed

- [x] closes #46512
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] Added an entry in the latest `pandas/core/generic.py` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Strings
description: String extension data type and string data
name: Stale

Comments:
Hello @Shadimrad! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-08-18 23:51:54 UTC
I'll milestone this as 1.5

from https://github.com/pandas-dev/pandas/issues/46512#issue-1181203886

> The behavior for the inplace=False case has changed from 1.4.1 to main since https://github.com/pandas-dev/pandas/pull/45168 allows other na values in the StringArray Constructor.

the testing for `inplace=True` would be further confirming the regression fix, #47763, but with the change for `inplace=False`, would need some work if backporting.
take
Sure. I'll fix it.

On Wed, 17 Aug 2022, 10:01 am Matthew Roeschke, ***@***.***>
wrote:

> ***@***.**** commented on this pull request.
> ------------------------------
>
> In pandas/tests/arrays/string_/test_string.py
> <https://github.com/pandas-dev/pandas/pull/47793#discussion_r948207565>:
>
> > @@ -611,3 +611,11 @@ def test_setitem_scalar_with_mask_validation(dtype):
>          msg = "Scalar must be NA or str"
>      with pytest.raises(ValueError, match=msg):
>          ser[mask] = 1
> +
> +
> +def test_consitency_inplace():
> +    df = pd.DataFrame({"M": [""]}, dtype="string")
> +    df2 = pd.DataFrame({"M": [""]}, dtype="string")
> +    df2.where(df2 != "", np.nan, inplace=True)
>
> Would be good to compare df and df2 to a separately created DataFrame e.g.
>
> expected = pd.DataFrame(...)
> tm.assert_frame_equal(df, expected)
> tm.assert_frame_equal(df2, expected)
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/47793#pullrequestreview-1076116145>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AWI7Q465NG3MD5XUBTTZ7DDVZULFDANCNFSM54BVWMNA>
> .
> You are receiving this because you were assigned.Message ID:
> ***@***.***>
>

I don't think this test is entirely required for 1.5 so removing that milestone. Once ready, we can scope for the 1.6 branch.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
DOC: Fix versionadded for callable in on_bad_lines
State: closed

- [x] closes #47788 (Replace xxxx with the Github issue number)
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).


would backport this


Labels:
name: Docs
name: IO CSV
description: read_csv, to_csv

Comments:
![grafik](https://user-images.githubusercontent.com/61934744/179870294-f5b29d23-7836-4459-9e7e-ae588f3a37c8.png)

Seems correct now
Something went wrong ... Please have a look at my logs.
#47795
BUG: PeriodIndex + TimedeltaArray-with-NaT
State: closed

Implements the same round_ok behavior in astype_overflowsafe that we use in Timestamp/Timedelta _as_reso, i.e. this will lead to more code-sharing in follow-ups.

Labels:
name: Timedelta
description: Timedelta data type
name: Period
description: Period data type

Comments:
I think the `test_round_sanity` 32 bit failures are unrelated, but is it worth xfailing or fixing soon?
> I think the test_round_sanity 32 bit failures are unrelated, but is it worth xfailing or fixing soon?

definitely worth fixing before long
Concatenating unordered ``CategoricalIndex`` overrides indices - GH24845
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Categorical
description: Categorical Data Type

Comments:
Hello @GivyBoy! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



* In the file [`pandas/tests/reshape/concat/test_categorical.py`](https://github.com/pandas-dev/pandas/blob/e2eb81395776baf3b3b6212edc2921b8fc6d6ebc/pandas/tests/reshape/concat/test_categorical.py):

> [Line 262:48](https://github.com/pandas-dev/pandas/blob/e2eb81395776baf3b3b6212edc2921b8fc6d6ebc/pandas/tests/reshape/concat/test_categorical.py#L262): [W292](https://duckduckgo.com/?q=pep8%20W292) no newline at end of file



##### Comment last updated at 2022-07-24 22:37:20 UTC
I wanted to get the conversation started, regarding fixing this issue. I want to know if the solution is acceptable and, if not, what you are looking for. Thanks!
Thanks for the PR @GivyBoy. If you are considering a bug fix, you would better to create an issue then link your pr to it. Also you should pass all the test (or at least all the groupby test) before commit. If disscussion or issue clarification needed, it may be done in the issue you created.
BUG: PeriodIndex fails to handle NA, rather than putting NaT in its place (#46673)
State: closed

- [x] closes #46673 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Period
description: Period data type

Comments:
Thanks @kapiliyer
Specify that both ``by`` and ``level`` should not be specified in ``groupby`` - GH40378
State: closed

- [ ] closes #40378 (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs
name: Groupby

Comments:
Thanks @GivyBoy