BrandonFors/pandas-github-issues huggingface dataset:
don't repeat yourself in pyproject.toml
State: closed

- [x] simplifies pyproject.toml writing, and so maintenance. zero change

Labels:

Comments:
this method is applied in dask project since two months, so I presume not bad.
hummm, maybe there was a reason ? https://github.com/pandas-dev/pandas/blob/main/scripts/generate_pip_deps_from_conda.py
> hummm, maybe there was a reason ? https://github.com/pandas-dev/pandas/blob/main/scripts/generate_pip_deps_from_conda.py

thanks for your PR - you could update that script too if you wanted to do this
> > hummm, maybe there was a reason ? https://github.com/pandas-dev/pandas/blob/main/scripts/generate_pip_deps_from_conda.py
> 
> thanks for your PR - you could update that script too if you wanted to do this

Well, I thought it was a low hanging fruit, now I see it's a bit too complex for me. I'm happy if anyone takes the idea, it was just to help
Could you reference where dask does this?
> Could you reference where dask does this?

https://github.com/dask/dask/blob/main/pyproject.toml#L66-L70
Thanks but since we have a script enforcing these are consistent in the file (as opposed to the pip extras being consistent), I think keeping the verboseness is fine so closing.
ENH/PERF: pyarrow timestamp & duration conversion consistency/performance
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.1.0.rst` file if fixing a bug or adding a new feature.

A few related changes:

1. ArrowExtensionArray.\_\_getitem\_\_(int) will now return a Timestamp/Timedelta for non-nano timestamp/duration types to be consistent with nanosecond types. Previously non-nano types returned python native datetime/timedelta.
2. ArrowExtensionArray.\_\_iter\_\_ will now yield Timestamp/Timedelta objects for non-nano types to be consistent with nanosecond types.
3. ArrowExtensionArray.to_numpy now allows for zero-copy for timestamp/duration types

Submitting as a single PR since there are a number of tests that require consistency across these methods and trying to split the nano/non-nano behavior from the performance improvements is tricky. 

These were somewhat motivated by:

```
import pandas as pd
import pyarrow as pa

N = 1_000_000
arr = pd.array(range(N), dtype=pd.ArrowDtype(pa.timestamp("s")))

%timeit arr.astype("M8[s]")
# 5.29 s ± 162 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)          -> main
# 137 µs ± 4.88 µs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)   -> PR

%timeit pd.DatetimeIndex(arr)
# 6.31 s ± 560 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)          -> main
# 67.6 µs ± 3.03 µs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)  -> pr
```


Labels:
name: Performance
description: Memory or execution speed performance
name: Arrow
description: pyarrow functionality
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
Thanks @lukemanley
DEPR: make Series.agg aggregate when possible
State: closed

The doc string for `Series.agg` says: `"A passed user-defined-function will be passed a Series for evaluation."` which isn't true currently. In addition, `Series.agg` currently will not always aggregate, when given a aggregation function. This PR fixes those issues.

It can be noted that both the behavior and that doc string are correct for `DataFrame.agg`, so this PR aligns the behavior of `Series.agg` and `DataFrame.agg`.

I'm not sure if this is too drastic as a bug fix, or we want it as part of a deprecation process, but just putting this up there, will see where it goes.

EDIT: In the new version the old behavior is deprecated rather than just removed. 

Labels:
name: Deprecate
description: Functionality to remove in pandas
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
I stumbled upon this in https://github.com/pandas-dev/pandas/pull/53208. I think changing this behavior is good but should be deprecated IMO to ensure `f` actually produces a reduced result
I've made a new version with deprecation.
In the new deprecation version I BTW found out that while `DataFrame.agg` isn't affected by this PR when given a single callable or a dict of callables, it was actually affected when given a list of callables...

@mroeschke - good here?
Thanks @topper-123
DOC: Move whatsnew for Arrow CSV fixes
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Backporting #52087 is blocked on another PR being backported.

Marking as draft for now.
DOCS: improving "to_excel" function description. Explained more about engines
State: closed

The basic description looked a bit confusing and did not explain why one engine should be used over the other. Also it didn't mention that both need to be installed and which was the default option.  I had 'openpyxl' installed in a previous project, and every time the file was 2x the normal size compared to using 'xlsxwriter' or opening and closing in Excel directly (manually). 

Labels:

Comments:
Feel free to reopen when you have time to circle back here
Remove old versionadded from DataFrame.map
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Method is new, so no need for those

Labels:
name: Docs

Comments:
Thanks, @phofl.
Update contributing_docstring.rst
State: closed

Change "Convention for examples" to use Series name as 'ser' instead of 's'.

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
thanks for your pr - closing as there's already one open for this
Change Naming Scheme of Series
State: closed

closes #53271 

**Checklist:**
- [x] I modified all versions of series with the naming scheme s to ser.
- [x] pre-commit checks pass. 


Labels:
name: Docs

Comments:
It looks like all the pre-commit checks are passing on my end when I run:

pre-commit run --files <files you have modified>

pre-commit run --all-files pass as well
thanks but there's already an open PR, please find another issue to contribute to
wait, sorry, my bad - looks like you'd expressed interest before the other contributor - reopening this one and closing the other one then
/preview
Website preview of this PR available at: https://pandas.pydata.org/preview/53312/
Well done @austinburnett ! 
LGTM but we need someone else with the super powers to merge (and double check). 
Like @MarcoGorelli maybe?
> Well done @austinburnett ! LGTM but we need someone else with the super powers to merge (and double check). Like @MarcoGorelli maybe?

Thank you! LGTM! I look forward to contributing more to Pandas!
Hahahah thanks @MarcoGorelli. I look forward to contributing more to Pandas!
BUG: read_sql reading duplicate tz aware columns
State: closed

- [ ] closes #44421 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query
name: Timezones
description: Timezone data dtype

Comments:
Going to merge, but happy to follow up if needed
@mroeschke thanks! Any chance this bugfix could be backported to pandas 2.0.x too? 
Typically the backport branch is only maintained until the following minor release or just after. Now that the latest version is 2.2.0, it is extremely unlikely that any further patch releases will happen in the 2.0.x branch.
CLN: Make minor docstring fix for `DataFrame.drop`
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks @deepyaman
CI: Pin Numba<0.57 in ARM build
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: CI
description: Continuous Integration
name: ARM
description: aarch64 architecture

Comments:
@mroeschke
I see numba 0.57.0 dropped on conda-forge just under a day ago. Maybe pinning might help?

The segfault comes from ``test_numba_vs_cython``.
thx @mroeschke 
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.0.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 9d38f40767da08ff46b807fd9565666a7aaf079c
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #53309: CI: Pin Numba<0.57 in ARM build'
```

4. Push to a named branch:

```
git push YOURFORK 2.0.x:auto-backport-of-pr-53309-on-2.0.x
```

5. Create a PR against branch 2.0.x, I would have named this PR:

> "Backport PR #53309 on branch 2.0.x (CI: Pin Numba<0.57 in ARM build)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
DEPS: Test numba on PY 3.11
State: closed

Appears numba 0.57 which supports Python 3.11 IIRC is now available so it should be safe to enable testing: https://anaconda.org/conda-forge/numba

Labels:
name: Dependencies
description: Required and optional dependencies
name: numba
description: numba-accelerated operations

Comments:
thx @mroeschke
DOC: Fixing EX01 - Added examples
State: closed

- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Towards https://github.com/pandas-dev/pandas/issues/37875

I don't know why the May 18 commits (of a branch that is already deleted) are shown here, but the  changes are the ones I wanted to include on this PR

Labels:
name: Docs

Comments:
Thanks @DeaMariaLeon 
Thank you @mroeschke
ENH: Enable using a boolean loc in a non-boolean index
State: closed

- [x] closes #52102 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
DOC: Change "Convention for examples" to use Series name as ser instead of s
State: closed

- [] closes #53271 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
> thanks @Jakaho ! there's a couple missing: `def head` and `def plot`, but the rest looks good!

Hey @MarcoGorelli I believe I finished up the remaining changes at https://github.com/pandas-dev/pandas/pull/53312.
> > thanks @Jakaho ! there's a couple missing: `def head` and `def plot`, but the rest looks good!
> 
> Hey @MarcoGorelli I believe I finished up the remaining changes at #53312.

indeed, let's go with that one - @Jakaho for the future, if someone has already commented that they're working on an issue, please wait at least a week before taking over. Thanks anyway for your PR!
BUG: to_datetime re-parsing Arrow-backed objects
State: closed

- [ ] closes #52545
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Ideally, we could push this down into DTA/DTI, but that doesn't seem to support Arrow arrays ATM.

Labels:
name: Datetime
description: Datetime data dtype
name: Arrow
description: pyarrow functionality

Comments:
> Also, does something similar happen to `to_timedelta`?

I think so, I'll try to get at it at a followup.

I think in the future, it might also make sense for DF inputs with Arrow backed ``year``, ``month``, etc. columns to convert to Arrow Backed dt columns.

e.g.
```
>>> import pandas as pd
+ /Users/thomasli/opt/miniconda3/envs/pandas-dev/bin/ninja
[1/1] Generating write_version_file with a custom command
>>> df = pd.DataFrame({'year': [2015, 2016],
...                    'month': [2, 3],
...                    'day': [4, 5]})
>>> df = df.astype("int64[pyarrow]")
>>> pd.to_datetime(df)
0   2015-02-04
1   2016-03-05
dtype: datetime64[ns]
```
should return something with dtype: ``timestamp[ns][pyarrow]``


Main thing I'm worried about with this PR is doing the wrong thing for ``utc=True`` (e.g. localizing instead of converting and vice versa).

(cc @phofl in case any thoughts and to double check my logic).
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks @lithomas1 
Does the to_datetime docstring need to be updated to reflect non-DTI return?
DOC: simplify pandas footer
State: closed

## Description

This pull request addresses issue #51536 by proposing a solution to simplify the pandas theme footer.

The sphinx theme currently requires an extra template to specify the footer content, which limits customization options. 
For the moment there seems to be no way to get rid of the template and set everything in the config file,
However, after research and explorationI have found a potential workaround that allows us to define the footer content directly in the `conf.py` configuration file using the `html_context` setting.


## Proposed Changes

- Updated `conf.py` to define the footer content using the `html_context` setting.
- Modified the `pandas_footer.html` template to render the footer content defined in `conf.py`.

Your feedback and suggestions on this approach are highly appreciated.

Labels:
name: Docs
name: Web
description: pandas website

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
ENH: Add sort keyword to unstack
State: closed

- [ ] closes #15105 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Enhancement
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
might address #9514?
> might address #9514?

AFAICT this is fixed on main outside of these changes
Any other comments @jbrockmendel? 
nothing here, go for it
REF: Move sas pyx files to _libs
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This simplifies the meson build a lot. I've comfirmed that this compiles correctly and passes the sas tests with both meson and setuptools.

Labels:
name: Refactor
description: Internal refactoring of code
name: Build
description: Library building on various platforms
name: IO SAS
description: SAS: read_sas

Comments:
LGTM.  I don't know enough about meson to provide much of a comment there, but I suppose the proof that it is right is in the CI run. 
Thanks @lithomas1
BUG: dtype of DataFrame.idxmax/idxmin incorrect for empty frames
State: closed

- [x] closes #53265
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
Thanks @topper-123