BrandonFors/pandas-github-issues huggingface dataset:
BUG: Groupby median on timedelta column with NaT returns odd value (#…
State: closed

…57926)

Handle NaT correctly in group_median_float64

- [x] closes #57926
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Groupby

Comments:
Thanks @JJLLWW 
@WillAyd - why the backport? I didn't think this was a regression.
Ah OK - I thought we were doing bugfixes on that branch now. Didn't realize it was limited to regressions. Can just keep this on 3.x then
This might be just me, but I like to keep patch versions as small as possible since bugfixes can induce new bugs and would like patch upgrade to be as safe as possible for users. But in the past, I know others have been more inclined to backport more than I prefer - cc @phofl for any thoughts.
Agree with @rhshadrach 

we backported fixes for new features that were introduced in the version before but generally not much else
> we backported fixes for new features that were introduced in the version before but generally not much else

Ah, I think that's where I remember a disagreement - thanks. And yea, I've come around to doing bug fixes for new features.
Clean up more Cython warning
State: closed

Overlooked this in the recent PR. I think this is the last of Cython warnings.

`(warning: /home/pandas/pandas/_libs/parsers.pyx:1605:26: noexcept clause is ignored for function returning Python object)`

Labels:
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
Thanks @WillAyd
PERF: DataFrame(dict) returns RangeIndex columns when possible
State: closed

Discovered in https://github.com/pandas-dev/pandas/pull/57441

Also removed a seemingly unused `_sort_msg`

Labels:
name: Performance
description: Memory or execution speed performance
name: Index
description: Related to the Index class or subclasses
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
Going to merge, but happy to follow up if needed
Remove versioneer / setuptools
State: closed

trying to follow the lead set by https://github.com/numpy/numpy/pull/24196

@lithomas1

Labels:
name: Build
description: Library building on various platforms
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Closing since there hasn't been much activity here, but feel free to reopen if/when you have time to revisit
@lithomas1 we are dropping setuptools support for 3.0 right?
I don't think the setuptools stuff is maintained anymore. (I just forgot to remove it)

I think my main objection was to the versioneer stuff.
Not actively working on this - let's clear the queue
Backport PR #57029 on branch 2.2.x (DOC: Add `DataFrame.to_numpy` method)
State: closed

Backport PR #57029: DOC: Add `DataFrame.to_numpy` method

Labels:
name: Docs

Comments:
@datapythonista 

Just to confirm - changes to docs should be backported if we want them to appear in 2.2.x, since the docs would be built from the 2.2.x branch?
Correct. It's only the website that is updated for every commit, and it's not useful to backport things. If you want the docs (sphinx) to contain this in 2.2.x besides in 3.0 you should backport it.
DOC: #38067 add missing holiday observance rules
State: closed

- [x] closes #38067

Updated docs with missing functions.

`weekend_to_monday` and `next_monday` do indeed do the exact same thing, and I agree that `next_monday` should be deprecated. Any thoughts?


Labels:
name: Docs

Comments:
Thanks @dontgoto 

> weekend_to_monday and next_monday do indeed do the exact same thing, and I agree that next_monday should be deprecated. Any thoughts?

Yeah that makes sense to deprecate one of them if they are redundant

> Yeah that makes sense to deprecate one of them if they are redundant

Can I put in a deprecation notice for 4.0.0? Or something like that?

My two cents here - I think "weekend_to_monday" is more accurate and descriptive, BUT if that's the one we keep it would make the most sense to also change the current "previous_friday" to "weekend_to_friday".  I think it's clearest to keep the before/after and next/previous pairs as matched as possible ... so whichever gets deprecated, the pair should be either:

* weekend_to_monday and weekend_to_friday
* next_monday and previous_friday

I prefer the former but wouldn't care too much if folks prefer the latter.
> * weekend_to_monday and weekend_to_friday
> 
> * next_monday and previous_friday

I find the former function names make more sense , because in the end they only shift when on a weekend, the latter implies that weekdays are also shifted, but they're not.

On the other hand, the latter ones have been in the user guide since forever, the former ones not.

@dontgoto I agree on the "makes more sense" ... even though the others have been documented I think that can be got around by deprecating them for a while before they go away
BUG: #29049 make holiday support offsets of offsets
State: closed

- [x] closes #29049 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v3.0.0.rst` file if fixing a bug or adding a new feature.

The summing of offsets is generally not possible, so the bug is fixed by simply flattening the asymmetric list of lists of offsets that one might easily end up with when defining holidays in relation to each other.

In theory it is possible to sum consecutive `DataOffset(n, weekday=None)` in the offset list, but I did not want to introduce even more complexity for such little gain.


Allowing asymmetrical lists of lists and their element dtypes as parameters is not very pretty, but the only solution to this problem that does not involve a major refactoring of the offset classes to allow representation of lists of offsets in a single composite offset e.g. ["TUE(1)", 3]. Summing of composite offsets is not possible because additions between weekday and regular day shift offsets are not associative, not distributive, and not commutative.

Another option would be to just enhance the error message given to the end user when trying to chain offsets in the manner shown in the issue and let them do the aggregation to a simple list of offsets themselves.

Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
Thanks @dontgoto 
Thank you for the input @mroeschke
BUG: Fix bad datetime64[ns] array to str dtype conversion in Series ctor which causes odd read_csv behaviour
State: closed

- [x] closes #57512
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Dates were being converted to nanosecond timestamps by numpy's `ndarray.astype` when converting an `M8[ns]` array to `object` during `ensure_string_array`. By wrapping the array with a `DatetimeArray` EA, we make sure that the dates get nicely converted in `ensure_string_array`.

Also added a check to `ensure_string_array` since it was converting na values disregarding the `convert_na_value` argument which caused missing dates to come out as `NaN` instead of `NaT`.

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
Fix tagging within Dockerfile
State: closed

When the user within a docker container does not match the user on the host machine (this is the case by default on Linux, not sure of all OSes) you cannot use git to inspect the worktree:

```sh
root@6de0debf3870:/home/pandas# git log
fatal: detected dubious ownership in repository at '/home/pandas'
To add an exception for this directory, call:

	git config --global --add safe.directory /home/pandas
```

This prevents builds within docker from being tagged with the appropriate git revision

Labels:

Comments:
Thanks @WillAyd
DateOffset add to ndarray
State: closed

This is WIP to implement changes needed to allow DateOffset subclasses e.g. MonthEnd to be directly added to numpy.ndarray, e.g. `numpy.array(['2022-01-01'], dtype='datetime64') + pandas.offsets.MonthEnd(1)`

I need some guidance on:

- cython mechanism to check that passed ndarray is datetime64 dtype i.e.cython equivalent of pandas.api.types.is_datetime64_any_dtype
- how to approach adding unit tests? two ideas I have
  - somehow duplicate existing offset tests but run them on ndarray instead of Series?  looks on the surface to be more difficult to implement
  - apply a set of offsets to a predefined Series and ndarray equivalent, and then check results match

Many thanks!

- [x] closes #47014
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Labels:
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Frequency
description: DateOffsets
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
DEPR: Execute all csv deprecations
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Stale

Comments:
Looks like this PR's gotten too large, esp. with date_parser removal.

I'll split it up.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Since it looks like some of these have been chipped away already, going to close. Feel free to open PRs to enforce the deprecations if you have time
Remove Cython warnings
State: closed

  warning: /Users/runner/work/pandas/pandas/pandas/_libs/tslibs/util.pxd:188:38: noexcept clause is ignored for function returning Python object
  warning: /Users/runner/work/pandas/pandas/pandas/_libs/tslibs/util.pxd:193:40: noexcept clause is ignored for function returning Python object

Labels:
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
Thanks @WillAyd
DOC: Update docs with the use of meson instead of setup.py 
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Not sure if this is still needed: https://github.com/pandas-dev/pandas/blob/main/gitpod/gitpod.Dockerfile#L38C5-L38C45

Labels:
name: Docs

Comments:
I would go ahead and update the gitpod as well - no reason to still be using setuptools anywhere in our code base
I think that the file [`gitpod.Dockerfile`](https://github.com/pandas-dev/pandas/blob/main/gitpod/gitpod.Dockerfile#L38C5-L38C45) is not being used. Since it's not being called here: https://github.com/pandas-dev/pandas/blob/main/.gitpod.yml#L9C3-L9C19  (but of course, I could be wrong). 

Plus it seems that there is some pending work for gitpod: https://github.com/pandas-dev/pandas/pull/54046

Maybe updating gitpod could be done on a separate PR? 

 
Thanks @DeaMariaLeon
DOC: Getting started tutorials css adjustments
State: closed

- [x] closes #57912 
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

I implemented what I suggested in the issue.


Labels:
name: Docs

Comments:
/preview
Website preview of this PR available at: https://pandas.pydata.org/preview/pandas-dev/pandas/57916/
Thanks @SilvestriStefano. Agreed looks a lot better
BUG: pretty print all Mappings, not just dicts
State: closed

This was discovered in https://github.com/ibis-project/ibis/issues/8687

Due to a strict `isinstance(x, dict)` check, custom mapping types would instead get treated as mere iterables, so only the keys would get printed as a tuple:

```python
import pandas as pd
from ibis.common.collections import frozendict
pd.Series([frozendict({"a": 5, "b": 6})])
# 0    (a, b)
# dtype: object
```

- [x] NA: closes #xxxx
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] NA Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
that was a pretty nice experience setting up a dev environment locally, pass along my compliments to the rest of the pandas maintainers @mroeschke !
Thanks @NickCrews
CI: Make pep8 validation work with multiple files simultaneously
State: closed

The validation is bottlenecked due to the repeated sys calls, subprocess spawning, and pep8 calls that only validate a single file each. Running the validation on a list of files is approximately as fast as running it on a single file. This PR makes the first step, enabling the pep8 validation of a list of files, but for now still running it only on a single file. It will be followed by a few more PRs. The end result will be a complete refactoring of `validate_docstrings.py`, that will then take about 10 seconds to validate all docstrings.

- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.

@datapythonista 

Labels:
name: Docs
name: CI
description: Continuous Integration
name: Stale

Comments:
I'm travelling and I could only revew with the phone, but looks good.

Quick question, would it make sense to use ruff instead of flake8 for this?
Thanks for the quick review!

> Quick question, would it make sense to use ruff instead of flake8 for this?

I think it would be slightly faster, but someone would have to configure ruff to behave the same way as we intend flake8 to behave here. On my machine, the whole `code_check.sh docstrings` validation finishes in about 10 seconds, 7s of those are separate calls to `numpydoc.validate` (I put in a PR there that brings this down to 5.5s), and the remaining 3s are split between our `validate.py`and pep8. 

So I think even with ruff we would at best get to 5.5s instead of 8.5s if we keep `numpydoc.validate`. Might still be worth an issue to discuss the (future) role of ruff here and in other places.


Yep, trying to improve 10s doesn't make sense. I know we are using ruff in the CI nowz bit I didn't check the exact state. The question was more about consistency, I didn't check if we already replaced flake8 with ruff.
I see ruff being used as part of the pre-commit hooks. And there it let a docstring error that I introduced in a non-replaced docstring through. It's also replaced flake8 for the docstring formatting, but back when I introduced my error, only the ``code_check.sh`` showed up as failed.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the PR but appears there not much appetite for this change so closing
REF/PERF: Use concat(..., ignore_index=True) when index doesn't matter
State: closed

For internal `concat` usages where the result index is set later, setting `ignore_index=True` to avoid More Work calculating the result concat index

Labels:
name: Refactor
description: Internal refactoring of code
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Thanks @mroeschke
CI: Improve API of --ignore_errors in validate_docstrings.py
State: closed

Follow up of #57879, to use `--ignore_errors PR02,SA01` instead of `--ignore_errors * PR02,SA01`.

I also make the script ignore the specified errors when running for a single document (i.e. `validate_docstrings.py pandas.Series -i SA01` was reporting the SA01 error before this PR, and it won't with the changes here).

CC: @mroeschke 


Labels:
name: Docs
name: CI
description: Continuous Integration

Comments:
> It's probably overly safe, but one could verify that only one CLI parameter is specifying globally ignored errors, otherwise one set would overwrite the other

It shouldn't overwrite it, I'm not sure if I explicitly tested it, but the way I implemented was to allow to do `validate_docstrings.py -i PR01 -i SA01` and ignore each of them:

```python
ignore_errors[None].update(set(error_codes.split(",")))
```

What I don't check is if there are duplicates, but that's not likely going to be useful since we just have one error being fully ignored, and I don't think we'll be adding any or many more errors to the script.

What I'm not sure is what happens if an object (e.g. `pandas.Series.head`) contains a typo, maybe it's ignored silently.
Thanks @datapythonista 
Thanks for all the work on this @datapythonista! `code_checks.sh` is looking awesome with thse new improvements. Just wanted to check and make sure this was the last change you had pending. If this is the final update, I can go ahead and open new issues to address these. 
Yes, no more changes to the ignore functions in `code_checks.sh`. You should also have the triage permissions now.
Opened [DOC: Enforce Numpy Docstring Validation (Parent Issue) #58063](https://github.com/pandas-dev/pandas/issues/58063) as a parent issue for fixing docstrings based on the refactoring in code_checks.sh 

Feel free to swing by and help out! 🙂
DOC: update link in benchmarks.md
State: closed

- [x] closes #57850
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

The [H20.ai benchmark link](https://h2oai.github.io/db-benchmark/) in the [benchmarks webpage](https://pandas.pydata.org/community/benchmarks.html) contains the outdated benchmark results (e.g. the last benchmark was performed July 2021 and it used the pandas==1.2.5). DuckDB published a new benchmark results (e.g. the last benchmark was December 2023 with pandas==2.1.1) in the [link](https://duckdblabs.github.io/db-benchmark/) with the same approach described in [their blog](https://duckdb.org/2023/11/03/db-benchmark-update). So it's a good idea to replace the benchmark link. Thank you @rootsmusic for the issue.

Labels:
name: Docs

Comments:
/preview
Website preview of this PR available at: https://pandas.pydata.org/preview/pandas-dev/pandas/57903/
Thank you @yukikitayama, very nice! 
Thanks for your review @datapythonista 👍
DOC: Remove doc of deprecated week and weekofyear
State: closed

- [x] closes #57887 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Labels:
name: Docs
name: Datetime
description: Datetime data dtype

Comments:
/preview
Website preview of this PR available at: https://pandas.pydata.org/preview/pandas-dev/pandas/57901/
Thanks @Aloqeely