BrandonFors/pandas-github-issues huggingface dataset:
REF: refactor describe.py into the methods folder
State: closed

xref: https://github.com/pandas-dev/pandas/pull/50253

It was discussed as a follow up to the above to refactor `describe.py` into `methods`.
- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Refactor
description: Internal refactoring of code

Comments:
Thanks @kostyafarber
Backport PR #50544 on branch 1.5.x (CI fix typing with latest ipython)
State: closed

Backport PR #50544: CI fix typing with latest ipython

Labels:
name: CI
description: Continuous Integration
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
cc @MarcoGorelli Seems to be another typing failure
the IPython and traitlets versions are the same on the 1.5.x branch, don't know why this is failing here but not on main, I can only put it down to it being a different mypy version. OK to `type: ignore`?
Ok by me
This might be failing on main https://github.com/pandas-dev/pandas/actions/runs/3838014340/jobs/6533977845
yes, indeed - let's keep this to the first fix, and then backport https://github.com/pandas-dev/pandas/pull/50562 too?
/
GH49560 without GH50195 (DEPR: Remove int64 uint64 float64 index part 1)
State: closed

This is #49560, but without #50195, as a draft.

Locally, It looks like #49560 doesn't need #50195 to pass the tests, so I'm trying if the same is true in the github CI.

 @jbrockmendel

Labels:

Comments:
This didn't work, so closing.
REF: handle 2D in array_to_datetime
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Refactor
description: Internal refactoring of code
name: datetime.date
description: stdlib datetime.date support

Comments:
Thanks @jbrockmendel
BUG: Follow-up with Timestamp(nanoseconds) keyword only
State: closed

- [x] closes #32174 (Replace xxxx with the GitHub issue number)
- [x] closes #32526 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Timestamp
description: pd.Timestamp and associated methods

Comments:
thanks @mroeschke
CI fix typing with latest ipython
State: closed

closes #50543 

ref

https://github.com/ipython/ipython/blob/01bc9d96e0dd50fad02d7b1e3d2e58113c617b6b/IPython/core/formatters.py#L777-L792

Labels:
name: CI
description: Continuous Integration
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
thx @MarcoGorelli
Updating and re-opening PR #48275
State: closed

- [ ] closes #47953
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Timedelta
description: Timedelta data type
name: Stale
name: Timestamp
description: pd.Timestamp and associated methods

Comments:
Mistyped the issue number in that commit message, should be #47953.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Sorry this went stale, I've been really busy lately, I think I'll be able to find time to clean this up soon.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
Updating and re-opening PR #48129
State: closed

- [ ] closes #43784
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Datetime
description: Datetime data dtype
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Timezones
description: Timezone data dtype
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Sorry this went stale, I've been really busy lately, I think I'll be able to find time to clean this up soon.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
CLN explicit casts to float to avoid implicit casts
State: closed

Similar to #50493 

There are places when an implicit cast happens, but it's not the purpose of the test - the cast might as well be done explicitly then

This would help pave the way towards [PDEP6](https://github.com/pandas-dev/pandas/pull/50424)

Labels:
name: Clean

Comments:
Thanks @MarcoGorelli
API: Series and DataFrame constructors to return shallow copy (i.e. don't share index) from another Series/DataFrame
State: closed

This changes the Series and DataFrame constructors to return a proper shallow copy (i.e. only share data, not attributes like the index) instead of returning a new Series/DataFrame object but with the same manager, for the case of `Series(s)` or `DataFrame(df)` (so no reindexing or casting, and with default of `copy=False`).

See https://github.com/pandas-dev/pandas/issues/49523 for an example.

Strictly speaking this is a breaking change (or notable bug fix, not sure if this was done intentionally in the past).

- [ ] closes #49523
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: API Design
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
@jbrockmendel good to go?
LGTM merge on green
TST: test explicit copy keyword with CoW enabled
State: closed

xref https://github.com/pandas-dev/pandas/issues/50535, part about that we need to test the `copy` keyword as well in those CoW tests.



Labels:
name: Copy / view semantics

Comments:
That depends a bit on what exactly we would be testing. Maybe it is enough to restrict ourselves to just the method call, and have an additional test like this (for this example in addition to `test_rename_columns`):

```
@pytest.mark.parametrize("copy", [True, None, False])
def test_rename_columns_copy(using_copy_on_write, copy):
    df = DataFrame({"a": [1, 2, 3], "b": [4, 5, 6], "c": [0.1, 0.2, 0.3]})
    df2 = df.rename(columns=str.upper, copy=copy)

    if (using_copy_on_write and copy is not True) or copy is False:
        assert np.shares_memory(get_array(df2, "A"), get_array(df, "a"))
    else:
        assert not np.shares_memory(get_array(df2, "A"), get_array(df, "a"))
```

In that case it would probably keep it generally simpler (you still have the complex `if` statement here, but only once). I _think_ it is not necessarily needed to also test the actual modification step (`df2.iloc[0, 0] = 0`) and the resulting behaviour of that, with the different options of `copy`. If we test that a `copy=True` already results in a copy of the data after the method, then we know for sure that modifying the data will not update the original (so no need to test this explicitly?) Similarly for non-COW and `copy=False`, if we test that `df2` shares data with `df`, then we know that updating df2 might update df (and that's not necessarily something to repeatedly test here, as it is strictly speaking not related to the actual method calls like `rename`)

So yes, if we think the above test is sufficient for parametrizing with `copy` (in constrast to the combined test right now in the diff), then it will be simpler to have separate tests.
@phofl thoughts? Shall I do the separate test?
I'll take a close look later today
Yeah, my example test above (https://github.com/pandas-dev/pandas/pull/50536#issuecomment-1371040421) only does the memory check, so indeed we could parametrize that for several methods (to the extent they keep the same column names / structure, to easily refer to the column to check)
@phofl added a parametrized test. I think I added all methods of the list in https://github.com/pandas-dev/pandas/issues/50535 that fit in a single test function. `align`, `swapaxes`/`transpose`, `merge`, `concat` and `to_numpy` will have to be tested separately.
Looks good, can we add swapaxes doing a no-op?
Thx @jorisvandenbossche
BUG: to_datetime incorrect OverflowError
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Datetime
description: Datetime data dtype

Comments:
Thanks @jbrockmendel
REF: move _validate_frequency to more accurate subclass
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Refactor
description: Internal refactoring of code
name: Frequency
description: DateOffsets

Comments:
Thanks @jbrockmendel
BUG: Fix creating DatetimeIndex with BusinessHour Frequency
State: closed

DatetimeIndex creating with BusinessHour Frequency would fail whenever the DatetimeIndex did not include a Timestamp that was also an opening time.

The cause of the issue was a bug where adding/subtracting `0 * BusinessHour` gave wrong results:
```python
import pandas as pd
import datetime as dt
dt.datetime(2020,1,1,10) -0 * pd.offsets.BusinessHour()
```
returns `Timestamp('2020-01-02 09:00:00')` rather than the expected `Timestamp('2020-01-01 10:00:00')`

This PR fixes the problem and adds appropriate assertions to the unit-tests

- [x] closes #49835 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Datetime
description: Datetime data dtype
name: Frequency
description: DateOffsets

Comments:
Great! Thanks @DavidKleindienst
DOC: Add documentation to __add__
State: closed

- [x] closes #44784 
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


I'm not sure what the convention on documenting (or not documenting) dunder methods is. If there's a better place to add documentation to resolve #44784 , happy to do that!

Labels:
name: Docs
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
Thanks for your review @datapythonista ! 

Pushed a commit to update the example using animal names. Thanks for pointing that out!

To your initial question: I am not aware of any location where this docstring would get rendered the docs, because -- exactly to your point -- there doesn't seem to be a section for operators. Happy to either (a) add such a section (or subsection, etc.) to the docs or (b) relocate this documentation elsewhere, if either of those are desireable.
Excellent. Thank you @datapythonista ! Pushed a few more commits to address your comments and added `__add__` to the docs. I'm going to start work on a reusable doc string for the other magic methods, pending comment from @MarcoGorelli .
good one - thanks @jessica-writes-code !
DOC: Add groupby-specific docstrings for take
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs
name: Groupby

Comments:
Thanks @rhshadrach
PERF: ArrowExtensionArray comparison methods
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.0.0.rst` file if fixing a bug or adding a new feature.

Perf improvement in `ArrowExtensionArray` comparison methods when array contains nulls. Improvement is from avoiding conversion to object array. 

```
import pandas as pd
import numpy as np

data = np.random.randn(10**6)
data[0] = np.nan

arr = pd.array(data, dtype="float64[pyarrow]")

%timeit arr > 0

# 80.5 ms ± 619 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)    <- main
# 4.05 ms ± 11.4 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  <- PR
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Arrow
description: pyarrow functionality

Comments:
Thanks @lukemanley
ENH: Added a new format in parsing.pyx
State: closed

- [x] closes #50514
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
> tests are failing, can you fix this up please?

I looked but couldn't figure out what was wrong,  could you help
> > tests are failing, can you fix this up please?
> 
> I looked but couldn't figure out what was wrong, could you help

I'd suggest you start by stepping through this part of the code (in particular the tokens) to understand what it does - once you've really understood it, you should be ready to submit a fix
> > > tests are failing, can you fix this up please?
> > 
> > I looked but couldn't figure out what was wrong, could you help
> 
> I'd suggest you start by stepping through this part of the code (in particular the tokens) to understand what it does - once you've really understood it, you should be ready to submit a fix

Yeah I'll do that
closing for now then, there's no point in guessing fixes - thanks anyway for your PR!
> closing for now then, there's no point in guessing fixes - thanks anyway for your PR!

ok
Update doc for pd.MultiIndex.droplevel to reflect inplace behaviour
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs
name: MultiIndex

Comments:
Sure, thanks @LakshyAAAgrawal
Revert "STYLE use pandas-dev-flaker (#40906)
State: closed

So, #40906 consolidated a bunch of custom checks into a flake8 extension

As it stands, flake8 is the slowest pre-commit hook we have (even without the extension - see [timings](https://github.com/pandas-dev/pandas/pull/50518#issuecomment-1368265690))

In https://github.com/pandas-dev/pandas/pull/50160, Joris tried adding ruff, which would more than an order of magnitude faster (!), and has more checks. Several projects are moving towards it

So, I suggest we revert #40906 , so that we can rip out flake8 and can replace it with `ruff` without losing the custom checks

This is mostly a pure revert of #40906, with some adjustments for things which have since changed

I removed the `import pandas.core.common as com` one, as that one was `noqa`'d in a few places anyway (if the check isn't a flake8 extension, then it's not so easy to turn off the check, we it's done for a whole file) and doesn't seem too important anyway

---

Overall, I think that this + the ruff PR would be a big improvement to developer workflow. It would mean more hooks (for now, maybe then can be consolidated anyway), but they would run a lot faster. And I'd expect the speed of the checks to matter much more to developer workflow than the speed of them

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
thanks - merging to clear the queue then, the only failures are the sqlalchemy ones, which hopefully #50580 will address
wip
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
ok, even without pandas-dev-flaker, flake8 is still the slowest hook

gonna try to push forwards with removing it in favour of ruff then

```python
$ python hook-timings.py
pandas-errors-documented          0.04
title-capitalization              0.05
validate-min-versions-in-sync     0.06
pip-to-conda                      0.07
end-of-file-fixer                 0.07
pg8000-not-installed-CI           0.09
seed-check-asv                    0.10
invalid-ea-testing                0.10
future-annotations                0.10
sync-flake8-versions              0.11
no-bool-in-core-generic           0.11
cython-casting                    0.13
no-return-exception               0.16
incorrect-backticks               0.18
rst-backticks                     0.22
np-testing-array-equal            0.22
trailing-whitespace               0.27
double-quote-cython-strings       0.45
rst-directive-colons              0.52
flake8-pyi                        0.66
use-pd_array-in-core              0.68
validate-errors-locations         0.70
rst-inline-touching-normal        0.71
unwanted-patterns                 0.88
use-io-common-urlopen             1.10
cpplint                           1.43
codespell                         1.44
flake8-rst                        2.84
check-test-naming                 3.16
debug-statements                  4.27
absolufy-imports                  4.29
isort                             5.41
cython-lint                       5.55
vulture                           7.86
sphinx-lint                      11.85
pyupgrade                        17.31
black                            43.71
flake8                           57.68
dtype: float64
```