BrandonFors/pandas-github-issues huggingface dataset:
Update cython version for asv conf
State: closed

- [x] closes #44067



Labels:
name: Benchmark
description: Performance (ASV) benchmarks

Comments:
we bumped the min to support 3.10
Modified enviornment and requirements too
> Do any of the benchmarks change dramatically with this change?

I changed this on the 1.3.x tracker (https://simonjayhawkins.github.io/fantastic-dollop/) a couple of weeks back. Haven't noticed any significant changes.
FWIW ive changed my local asv.conf.json to use py39
DOC: Clarify DST rounding behavior in Timestamp/DatetimeIndex
State: closed

xref https://github.com/pandas-dev/pandas/issues/44287
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them


Labels:
name: Docs
name: Datetime
description: Datetime data dtype

Comments:
just wondering out loud if we could/should actually do this in a tz-aware way for `.round`, `.ceil`, `.floor`; e.g. are these fundamentally different ops that say subtraction that we could *always* do this w/o going ambiguous

cc @mroeschke 
@jreback found discussion https://github.com/pandas-dev/pandas/pull/22560#issuecomment-419334705 and  https://github.com/pandas-dev/pandas/pull/22560#issuecomment-419684301 that rounding in a tz-aware might not actually be possible and falling back to rounding based on actual, UTC time may give unexpected results.

my question is can we leave in thr tz and do a round operation w/o introducing ambiguity?
Hmm since the rounding operations happen on epoch timestamps, rounding could happen on the UTC epoch timestamps and relocalized without ambiguity i.e. `.tz_convert("UTC").round(...).tz_convert(tz)`. But this flies counter to https://github.com/pandas-dev/pandas/pull/22560#issuecomment-419684301 and the notion of "round relative to the wall time"
@mroeschke rereading i am wondering if it's always possible in freq < day (eg hour / min) to do the round ops in local time (and not convert-round-localize) which can introduce ambiguity
Yeah for small rounding frequencies this can be achieved. Looking at the UTC offsets I think the limit is sub 15 minutes (I see a +05:45 UTC offset for example)
thanks @mroeschke
Fixed regression in Series.duplicated for categorical dtype with bool categories
State: closed

- [x] closes #44351
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Regression
description: Functionality that used to work in a prior pandas version
name: Categorical
description: Categorical Data Type

Comments:
Added the test and modified the release note
@meeseeksdev backport 1.3.x
thanks @phofl 
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 1.3.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -m1 9f54f707fa9f79ec225f715e4b802f8f91c45b52
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #44356: Fixed regression in Series.duplicated for categorical dtype with bool categories'
```

4. Push to a named branch :

```
git push YOURFORK 1.3.x:auto-backport-of-pr-44356-on-1.3.x
```

5. Create a PR against branch 1.3.x, I would have named this PR:

> "Backport PR #44356 on branch 1.3.x (Fixed regression in Series.duplicated for categorical dtype with bool categories)"

And apply the correct labels and milestones.

Congratulation you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove `Still Needs Manual Backport` label once the PR gets merged.

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 1.3.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -m1 9f54f707fa9f79ec225f715e4b802f8f91c45b52
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #44356: Fixed regression in Series.duplicated for categorical dtype with bool categories'
```

4. Push to a named branch :

```
git push YOURFORK 1.3.x:auto-backport-of-pr-44356-on-1.3.x
```

5. Create a PR against branch 1.3.x, I would have named this PR:

> "Backport PR #44356 on branch 1.3.x (Fixed regression in Series.duplicated for categorical dtype with bool categories)"

And apply the correct labels and milestones.

Congratulation you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove `Still Needs Manual Backport` label once the PR gets merged.

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
CLN: split giant dt accessor tests
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
what you don't like long tests :-<>
> what you don't like long tests :-<>

it was a PITA to debug in my branch working on DayDST
changed shape argument for ndarray from int to tuple in ./core/strings/object_array.py
State: closed

xref #37715


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
pandas/core/strings/object_array.py:65: error: Argument "dtype" to "ndarray" has incompatible type "Union[ExtensionDtype, str, dtype[object_], dtype[Any], Type[object]]"; expected "Union[dtype[Any], None, type, _SupportsDType[dtype[Any]], str, Union[Tuple[Any, int], Tuple[Any, Union[SupportsIndex, Sequence[SupportsIndex]]], List[Any], _DTypeDict, Tuple[Any, Any]]]"  [arg-type]

@jbrockmendel How are you today? Do you think these changes could work?
lgtm cc @simonjayhawkins 
thanks @nickleus27
BUG: frame.loc[2:, 'z'] not setting inplace when multi-block
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
is this user facing at all?
> is this user facing at all?

yes, will update with whatsnew
whatsnew added + green
cc @phofl if you can have a look
CI: Fix import of coo_matrix after scipy dev upgrade
State: closed

New scipy_dev renamed ``coo`` to ``_coo`` to make it private

Labels:
name: CI
description: Continuous Integration
name: Dependencies
description: Required and optional dependencies

Comments:
Fixes the scipy affected tests. The numpy affected tests are still failing
Best guess is this np.quantile PR https://github.com/numpy/numpy/pull/18203 is causing the current build failures.
I think it is https://github.com/numpy/numpy/commit/cd7a02a4db7e760b881f3feeb832ffd84fa8645a
@phofl duplicate of #44362?
The scipy part yes
TYP: timestamps.pyi
State: closed

Should type annotations from a pyi files also be added to the pyx file (if possible)?

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
maybe worth it to use something more descriptive than `_S` for the type var
> maybe worth it to use something more descriptive than _S for the type var

IIRC thats what they used in the microsoft stubs that i based this on
ENH: option to change the number in `_dir_additions_for_owner`
State: closed

- [x] closes #37996
- [x] tests added / passed
- [x] Allows to optionally change the number of columns from  a dataframe that are added to `dir`, which is frequently also the number of columns suggested by tab completion, by using the option `pandas.set_option('display.max_dir_additions', 100)` to a higher number than 100 or None for an unlimited number of columns. Changing the value of `display.max_dir_additions` will only take effect for existing dataframes after a column is added or deleted, as the result of `dir` is cached.

Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
> why are we not just using max_columns for this?

If you have a dataframe with 10 000 columns you will not want to have them all pretty printed, but you may still want to have all columns suggested by tab completion.  (edit: so you need to be able to set them to a different value)
really? you are going to scroll thru them?

i don't really like adding yet another option 

if people really want to scroll then just increase the default 
No you do not scroll thru them.  Instead you type the first two (or so) letters and press the tab button. The editor will  then suggest names for columns that start with those letters. This is called tab completion. Except that, currently, it will not suggest names of columns that are beyond the first 100 columns since these are not known to `dir`. 

For instance, if you grab a jupyter notebook and take some random data

```
import string
import pandas as pd
import numpy as np
columns = [u + v for u in string.ascii_lowercase for v in string.ascii_lowercase]
data = np.random.normal(0, 1, (100, len(columns)))
df = pd.DataFrame(data, columns=columns)
```

then `df.e<tab>` will not return any suggestions for columns. (It will suggest things like df.eq though, but because this is a method.) With this pull request, if you set `pd.set_option('display.max_dir_items', None)` then it will suggest `ea` etc. until `ez`. However if you were to set at the same time `pd.set_option('display.max_columns', None)` then `df` will print all 26 * 26 columns. 
i c ok then

need tests
and update the options documentation
and a ehatsnew note
Failed tests seem related to `pandas/tests/window/test_rolling` and unrelated to the current PR.

Is it expected that some of these tests fail? 
cc @pandas-dev/pandas-core and @arw2019 if any comments here.
thanks @TheDerivator
TST: use custom parametrization for consistency in base extension array tests
State: closed

Follow-up on https://github.com/pandas-dev/pandas/pull/44242, as that broke geopandas CI. This adds back the custom parametrization, which also makes it consistent with the other occurrence of this test (for a different op) which is still using this:

https://github.com/pandas-dev/pandas/blob/057c6f81b464c6bbb667d326e203ad0dbd17cbde/pandas/tests/extension/base/ops.py#L116-L127

Since this is the only occurrence of `frame_or_series` in the base extension tests, I think it's not warranted to make it a fixture. If we would rather want this, we should add it to `tests/extension/conftest.py` instead (can always be done in a follow-up).

Labels:

Comments:
> we should add it to tests/extension/conftest.py

i dont understand.  why would we duplicate the fixture already in the top-level conftest?
> why would we duplicate the fixture already in the top-level conftest?

Because it makes it clear what is expected to be implemented by the downstream user. In practice there are currently no duplicated fixtures though.
Since this will fixup downstream CI + make this consistent within the same file, going to merge this. If we want to change the use of the fixture, we can do that in a later PR.
DOC: df.to_html documentation incorrectly contains min_rows optional param
State: closed

- [x] closes #44304
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [NA] whatsnew entry

Moved min_rows parameter out of the common docstring that is used by both DataFrame.to_html and DataFrame.to_string and reintroduced the parameter in DataFrame.to_string only

Labels:
name: Docs

Comments:
thanks @loicdiridollou 
reordering parameters that are not keyword only is technically a breaking change even though it is unlikely that users would pass all parameters by position.

Maybe needs a release note.
BUG: is_dtype_equal(dtype, "string[pyarrow]") raises if pyarrow not installed
State: closed

- [x] closes #44276
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry

Added instance to check string preventing import error


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
do we actually have a test which exercises this? e.g. not on a build where pyarrow is installed?
> do we actually have a test which exercises this? e.g. not on a build where pyarrow is installed?

I do not have pyarrow running on my system and got the initial import error. Actually I am able to run this on a system where pyarrow is not installed. It does not matter as pyarrow is not imported, so installation will not matter.
Whatsnew note added
pls merge master as well
Done, put the test into a new file as specified
@jorisvandenbossche @jreback any updates for me on this?
@jorisvandenbossche @jreback any updates for me on this?
@shubham11941140 you need to merge master
Done.
Thanks @shubham11941140
DOC: remove note about groupby.apply side-effect
State: closed

The described behavior has been fixed in 0.25

- [x] closes #44318
- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry, _comment:_ wouldn't say this is applicable here


Labels:
name: Docs
name: Groupby

Comments:
thanks @k-dominik 
thx for the quick responses @jreback!
Backport PR #44315 on branch 1.3.x (Pin aiobotocore to lower than 2.0 )
State: closed

(cherry picked from commit 28c7f7628dd6b862e1eed29ff71e2e8363ca1eee)

- [x] xref #44315


Labels:
name: CI
description: Continuous Integration
name: Dependencies
description: Required and optional dependencies

Comments:
@simonjayhawkins Failing pipelines are related. Looks like our env files changed on master, so that our pins are not enough. Should we adress this here?
The failing envs are picking up aiobotocore-2.0.0, so we can just pin those envs too?
> Looks like our env files changed on master

python 3.7 is dropped on master so there has been many changes to the ci on master compared to 1.3.x
Pinned the other two envs
Thanks @phofl
ENH: Add decimal parameter to read_excel
State: closed

- [ ] closes #14403
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Enhancement
name: IO Excel
description: read_excel, to_excel

Comments:
thanks @lithomas1 very nice!
BUG: failure to cast all-int floating dtypes when setting into int dtypes
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry

Overlap with (but not rebased on top of) #44275

Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
user facing at all?
> user facing at all?

yes, just updated with whatsnew
small comment, otherwise lgtm
Thx @jbrockmendel
Pin aiobotocore to lower than 2.0
State: closed

- [x] xref #44311


Labels:
name: CI
description: Continuous Integration
name: Dependencies
description: Required and optional dependencies

Comments:
LGTM pending green

do we need a comment about conditions under which the pin can be reverted?
Would have tracked this through the issue, but could add a comment to 
nice merge on green
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.3.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 28c7f7628dd6b862e1eed29ff71e2e8363ca1eee
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #44315: Pin aiobotocore to lower than 2.0'
```

4. Push to a named branch :

```
git push YOURFORK 1.3.x:auto-backport-of-pr-44315-on-1.3.x
```

5. Create a PR against branch 1.3.x, I would have named this PR:

> "Backport PR #44315 on branch 1.3.x (Pin aiobotocore to lower than 2.0)"

And apply the correct labels and milestones.

Congratulation you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove `Still Needs Manual Backport` label once the PR gets merged.

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
thanks @phofl 
@meeseeksdev backport 1.3.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.3.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 28c7f7628dd6b862e1eed29ff71e2e8363ca1eee
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #44315: Pin aiobotocore to lower than 2.0'
```

4. Push to a named branch :

```
git push YOURFORK 1.3.x:auto-backport-of-pr-44315-on-1.3.x
```

5. Create a PR against branch 1.3.x, I would have named this PR:

> "Backport PR #44315 on branch 1.3.x (Pin aiobotocore to lower than 2.0)"

And apply the correct labels and milestones.

Congratulation you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove `Still Needs Manual Backport` label once the PR gets merged.

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
BUG: PeriodIndex in pytables
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

I've ONLY checked that the edit here fixes the commented-out test, not sure if thats sufficient to close #7796 

Labels:
name: Bug
name: IO HDF5
description: read_hdf, HDFStore
name: Period
description: Period data type

Comments:
ok maybe worth a note (but could also wait to close that issue).
CLN/TYP: address TODOs, ignores
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Clean
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Thanks @jbrockmendel
Detect CPORT header in SAS files
State: closed

Refers https://github.com/pandas-dev/pandas/issues/15825 . Doesn't fix the issue (might be unfixable), but gives a more helpful warning.


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: IO SAS
description: SAS: read_sas

Comments:
lgtm. @bashtage if any comments.
I just deleted most of the CPT binary to cut down on wasted space - unit tests still pass.
> I just deleted most of the CPT binary to cut down on wasted space - unit tests still pass.

If you want to keep the shortened CPT file (and why not), you should be sure to rebase and squash to a single commit so that any history is removed from the commits.
> > I just deleted most of the CPT binary to cut down on wasted space - unit tests still pass.
> 
> If you want to keep the shortened CPT file (and why not), you should be sure to rebase and squash to a single commit so that any history is removed from the commits.

Oh yeah, my bad! Done.
thanks @lordgrenville
BUG: partially-inferring pydatetime objects
State: closed

- [x] xref #40111
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Datetime
description: Datetime data dtype

Comments:
cc @mroeschke if any comments
DEPR: unused 'errors' keyword in where, mask
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Deprecate
description: Functionality to remove in pandas

Comments:
thanks @jbrockmendel and @simonjayhawkins
TYP: _ensure_data and infer_dtype_from_array
State: closed


Labels:
name: Typing
description: type annotations, mypy/pyright type checking
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
status here?
@simonjayhawkins status of this?
Thanks for the pull request, but it appears to have gone stale. Feel free to reopen when you have time to merge main and continue.
TYP: remove a `type: ignore`
State: closed


Labels:
name: Typing
description: type annotations, mypy/pyright type checking
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
status here?
Thanks for the PR, but it appears to have gone stale. Feel free to reopen when you have time to revisit.
BUG: Period.__eq__ numpy scalar (#44182 (comment))
State: closed

- [x] fixes https://github.com/pandas-dev/pandas/pull/44182#issuecomment-956851322
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry



Labels:
name: Bug
name: Period
description: Period data type

Comments:
I added three tests but only one of them fails without the code change.

Fixed by the code update:
```python
Period("2000-01", "M") == np.array(0)
```

Additional tests not affected by the code update:
```python
idx = period_range("2007-01", periods=20, freq="M")
idx == np.array(idx[10])

Period("2000-01", "M") == np.int_(0)
```


This used to return `True` (but now returns `False`), so I think we are not yet entirely there:

```python
p = pd.Period("2000-01", "M")
p == np.array(p)
```
thanks @mathause
BENCH: Add more numba rolling benchmarks
State: closed

- [x] xref #https://github.com/pandas-dev/pandas/pull/44176#issuecomment-953441787
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them


Labels:
name: Benchmark
description: Performance (ASV) benchmarks

Comments:
thanks @mroeschke