BrandonFors/pandas-github-issues huggingface dataset:
DEPR: Series.argsort NA behavior
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Series.argsort has NA behavior different from EA.argsort/nargsort (xref #43840) that seems really useless to me.  This deprecates that behavior so in the future we can just pass through to the EA.

We _could_ expose the EA's na_position keyword, but let's wait on that until someone asks for it.  AFAIK this isn't really used.

Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
```
--- a/doc/source/whatsnew/v2.1.0.rst
+++ b/doc/source/whatsnew/v2.1.0.rst
@@ -295,7 +295,6 @@ Other API changes
 
 Deprecations
 ~~~~~~~~~~~~
-- Deprecated the behavior of :meth:`Series.argsort` in the presence of NA values; in a future version these will be sorted at the end instead of giving -1 (:issue:`54219`)
 - Deprecated 'broadcast_axis' keyword in :meth:`Series.align` and :meth:`DataFrame.align`, upcast before calling ``align`` with ``left = DataFrame({col: left for col in right.columns}, index=right.index)`` (:issue:`51856`)
 - Deprecated 'downcast' keyword in :meth:`Index.fillna` (:issue:`53956`)
 - Deprecated 'fill_method' and 'limit' keywords in :meth:`DataFrame.pct_change`, :meth:`Series.pct_change`, :meth:`DataFrameGroupBy.pct_change`, and :meth:`SeriesGroupBy.pct_change`, explicitly call ``ffill`` or ``bfill`` before calling ``pct_change`` instead (:issue:`53491`)
@@ -319,6 +318,7 @@ Deprecations
 - Deprecated the ``axis`` keyword in :meth:`DataFrame.ewm`, :meth:`Series.ewm`, :meth:`DataFrame.rolling`, :meth:`Series.rolling`, :meth:`DataFrame.expanding`, :meth:`Series.expanding` (:issue:`51778`)
 - Deprecated the ``axis`` keyword in :meth:`DataFrame.resample`, :meth:`Series.resample` (:issue:`51778`)
 - Deprecated the behavior of :func:`concat` with both ``len(keys) != len(objs)``, in a future version this will raise instead of truncating to the shorter of the two sequences (:issue:`43485`)
+- Deprecated the behavior of :meth:`Series.argsort` in the presence of NA values; in a future version these will be sorted at the end instead of giving -1 (:issue:`54219`)
```
updated+green
Thanks @jbrockmendel
CI: Try out pre-commit.ci again
State: closed

Will need to enable the pre-commit Github app 

Labels:
name: CI
description: Continuous Integration
name: Code Style
description: Code style, linting, code_checks

Comments:
does it fit into the time limit now? I remember that being an issue, but now we've moved from flake8 to ruff and are building black using the wheels
I still need to turn on the pre-commit Github app to test with this PR, but I'm hopeful that we shouldn't timeout anymore
@MarcoGorelli looks like pre-commit CI completes in less than 3 minutes and the autofix PR feature is working pretty nicely
oh wow, it's actually within the limit now! neat
> @MarcoGorelli looks like pre-commit CI completes in less than 3 minutes and the autofix PR feature is working pretty nicely

Pls turn off autofix. It was really annoying the last time it was briefly on, at least for me.

Is there a way to make it trigger on a comment instead (like the old comment bot)?
> Pls turn off autofix. It was really annoying the last time it was briefly on, at least for me.

Sure, just curious what you found annoying about autofix?

There is a separate command for running the fix if autofix is turned off
> > Pls turn off autofix. It was really annoying the last time it was briefly on, at least for me.
> 
> Sure, just curious what you found annoying about autofix?
> 
> There is a separate command for running the fix if autofix is turned off

Mainly it just autopushing fixes, so then my local branch and the branch on github would be out of sync and conflict.
(so then I'd have to do a push and then pull).

Is there an opt out mechanism for users, or is autofix by repository?
> Is there an opt out mechanism for users, or is autofix by repository?

I think it's autofix by repo.

The latest commit has autofix turned off. A comment command to autofix is perfectly fine
Since this new configuration is essentially preserves the same checking and autoupdating (the config not PRs) state as before just with pre-commit CI going to merge. Can follow up if needed
DOC: Removed InvalidComparison and LossySetItemError from docs
State: closed

- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Towards https://github.com/pandas-dev/pandas/issues/37875

https://github.com/pandas-dev/pandas/pull/48088
https://github.com/pandas-dev/pandas/pull/48687

@jorisvandenbossche @mroeschke : Could you comment if this is OK please? thanks.
Suggested by @MarcoGorelli

Labels:
name: Docs

Comments:
I'm still slightly concerned that overtime that these exceptions could leak in public APIs. Maybe it's good to document that they should be internal errors?
Do you mean put them back to the documentation and write a note? 
> Do you mean put them back to the documentation and write a note?

Yes, I would be partial to keep them in the docs and note in the docstrings that they are internal errors
maybe we should just add these to

https://github.com/pandas-dev/pandas/blob/f5a5c8d7f0d1501e5d8ff31b3b5f24c916137d9c/scripts/validate_docstrings.py#L44-L52

and note in the docstring that they're private and get re-raised, and not add examples to them
@MarcoGorelli stylers? shouldn't we do it for errors? 
yeah I mean to add these two errors to that set
~It looks green @MarcoGorelli ...~
edit: I think I have to merge again
> edit: I think I have to merge again

hey - was there a merge conflict? if not, no need to merge
FIX: query() and isin() fail on period type column
State: closed

- [X] closes #54199
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [X] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
hi @mroeschke , I've made some changes to address the failing testcases and pushed to this branch `dannyi96:period_type_issue`.
Do i create a new PR & discard this ?
DOC: Fixing EX01 - Added examples
State: closed

- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Towards https://github.com/pandas-dev/pandas/issues/37875

Example by itself because I was having problems with it

Labels:
name: Docs

Comments:
Thanks @DeaMariaLeon
TST: add a test with freq=1MS to increase coverage of shift
State: closed

xref #52064

Added a test with `freq=1MS` to increase coverage of `shift`. The reason: existing tests don't cover a case if the index of a DataFrame is a `DatetimeIndex`.

Labels:
name: Datetime
description: Datetime data dtype
name: Needs Tests
description: Unit test(s) needed to prevent regressions

Comments:
@MarcoGorelli, , could you please take a look at this pr?
DOC: Fixing EX01 - Added examples
State: closed

- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Towards https://github.com/pandas-dev/pandas/issues/37875



Labels:
name: Docs

Comments:
`Error: /home/runner/work/pandas/pandas/pandas/core/frame.py:2739:EX01:pandas.DataFrame.to_feather:No examples section found`
Could you merge in main once more. I merged your earlier PR first
TST: Use numpy random generator for testing data 
State: closed

This PR diff is large, but essentially it enables a ruff rule `NPY002` to use the numpy random `Generator` instead of the legacy random methods.

The change is replacing `np.random` with `np.random.default_rng(2)` before calling the data generation method. I think starting to use a fixed seed is good to make failures more reproducible.

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Looks like tests are passing here so going to merge. Happy to follow up if needed
REF: more greppable names for ujson functions
State: closed

Discussed with @WillAyd 

Labels:
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
```
pandas/io/json/_table_schema.py:16: error: Module "pandas._libs.json" has no attribute "ujson_loads"  [attr-defined]
pandas/io/json/_json.py:24: error: Module "pandas._libs.json" has no attribute "ujson_dumps"  [attr-defined]
pandas/io/json/_json.py:24: error: Module "pandas._libs.json" has no attribute "ujson_loads"  [attr-defined]
pandas/io/excel/_xlsxwriter.py:264: error: Module has no attribute "ujson_dumps"  [attr-defined]
pandas/io/excel/_odswriter.py:250: error: Module has no attribute "ujson_dumps"  [attr-defined]
```
Thanks @jbrockmendel
CLN: Adjust namespace handling in to_xml
State: closed

- [X] Relates to [note](https://github.com/pandas-dev/pandas/pull/54043#discussion_r1258888096) in #54043
- [X] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/v2.1.0.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO XML
description: read_xml, to_xml

Comments:
Thanks @ParfaitG
CI: Bump pandas-gbq to one that doesn't pin pyarrow
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Dependencies
description: Required and optional dependencies
name: Mothballed
description: Temporarily-closed PR the author plans to return to
name: Stale
name: Arrow
description: pyarrow functionality

Comments:
Might be worth bumping the min version to https://github.com/googleapis/python-bigquery-pandas/releases/tag/v0.18.1 so users also don't get bit by pyarrow being pinned
Ugh 0.18.1 isn't on conda-forge.

Is it fine to just go directly to 0.19.x?
> Ugh 0.18.1 isn't on conda-forge.
> 
> Is it fine to just go directly to 0.19.x?

Yeah that should be fine
A few more places need bumping otherwise looks good
Not entirely making sense of the error, but looks like there is a conflict between pyarrow dependencies
(aws-sdk-cpp) and google-bigquery dependencies(grpcio) fighting over openssl 1 and openssl 3.

I think what needs to happen is that we need to bump pyarrow from 7 to something newer. 

PDEP-10 stops this from happening for 2.1, but maybe we can bump as soon as 2.2?
There seems to be no support plan in that PDEP, stopping us from bumping pyarrow as we please.

cc @jorisvandenbossche (this relates to the issue with pyarrow effectively being pinned on CI)

This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Going to close to clear the queue for now
I think I may have tackled this in https://github.com/pandas-dev/pandas/pull/55524?
Nice, thanks Matt.
Fix UserWarning in DataFrame.plot.scatter when invoked with c="b"
State: closed

- [x] closes #53908
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Visualization
description: plotting

Comments:
Thanks @GianlucaFicarelli
Fix deprecation warning on Python 3.12
State: closed

`Constant.n` is now deprecated for removal in 3.14; it appears that `Constant.value` has existed since at least 3.9.

- [n/a] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [n/a] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: expressions
description: pd.eval, query

Comments:
I already have a PR for this #53743.

Thanks for the effort though.
Thanks for the PR, but since this is covered in https://github.com/pandas-dev/pandas/pull/53743 will close in favor of that PR
Oops, thought I looked for existing ones; not sure why I didn't find it earlier.
BUG: HDFStore select for large integer
State: closed

- [x] closes #53323
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Used decimal conversion instead of float to prevent precision loss

Labels:
name: IO HDF5
description: read_hdf, HDFStore

Comments:
Thanks @SanjithChockan
BUG: read_sql_table unable to read views
State: closed

- [x] closes #52969 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Looks like `read_sql_table` isn't implemented for sqllite, but this PR enable `read_sql` to work with views


Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
Hey @mroeschke !

Didn't realise we don't have read_sql_table implemented for sqllite! Looks great to me, some minor thoughts above :)
BUG: PeriodDtype and IntervalDtype holding references to itself when object is deleted
State: closed

Fixes this issue

```
In [1]: import pandas as pd

In [2]: import weakref

In [3]: pdt = pd.PeriodDtype(freq="D")

In [4]: ref = weakref.ref(pdt)

In [5]: del pdt

In [6]: import gc; gc.collect()
Out[6]: 1119

In [7]: ref()
Out[7]: period[D]

In [8]: idt = pd.IntervalDtype("interval[int64, right]")

In [9]: ref2 = weakref.ref(idt)

In [10]: del idt

In [11]: import gc; gc.collect()
Out[11]: 615

In [12]: ref2()
Out[12]: interval[int64, right]
```

1. For `IntervalDtype` the cache didn't really seem to improve performance now it's not used
2. For `PeriodDtype` just caching the period freq code instead of the created object

```
# PR

In [2]: %timeit pd.IntervalDtype("interval[int64, right]")
11.1 µs ± 13 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)

In [3]: %timeit pd.PeriodDtype(freq="D")
22.2 µs ± 135 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

# main

In [2]: idt = pd.IntervalDtype("interval[int64, right]")

In [3]: %timeit pd.IntervalDtype("interval[int64, right]")
14 µs ± 28.6 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)

In [4]: pdt = pd.PeriodDtype(freq="D")

In [5]: %timeit pd.PeriodDtype(freq="D")
22.2 µs ± 182 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)
```

Labels:
name: Period
description: Period data type
name: Interval
description: Interval data type

Comments:
mypy complaint, otherwise looks good
TST: add test with freq=2M to increase coverage asfreq
State: closed

xref #52064

Added a test in which a Series has a PeriodIndex with `freq=2M` to increase coverage `asfreq`. 
The reason: existing tests don't cover a case if the index of a Series is a PeriodIndex with `freq=2M`, then asfreq converts time series to a new one with the frequency `freq=M`.

Labels:
name: Period
description: Period data type
name: Needs Tests
description: Unit test(s) needed to prevent regressions

Comments:
@MarcoGorelli, could you please take a look at this pr?
ENH: `ewm()` default to using DatetimeIndex as `times` when `halflife` is a timedelta
State: closed

Allow `ewm()` to use a DatetimeIndex as the `times` parameter if it isn't provided and `halflife` is a timedelta convertible.

Before:
```
>>> now = pd.Timestamp.now()
>>> idx = [now + pd.Timedelta(seconds=i) for i in range(1, 4)]
>>> df = pd.DataFrame({'a': [1,2,3]}, index=idx)
>>> df.ewm(halflife=pd.Timedelta(seconds=1)).mean()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/gsiano/.local/lib/python3.10/site-packages/pandas/core/generic.py", line 11743, in ewm
    return ExponentialMovingWindow(
  File "/home/gsiano/.local/lib/python3.10/site-packages/pandas/core/window/ewm.py", line 386, in __init__
    raise ValueError(
ValueError: halflife can only be a timedelta convertible argument if times is not None.
>>> df.ewm(halflife=pd.Timedelta(seconds=1), times=df.index).mean()
                                   a
2023-07-18 10:45:18.007264  1.000000
2023-07-18 10:45:19.007264  1.666667
2023-07-18 10:45:20.007264  2.428571
>>> 
```

After:
```
>>> now = pd.Timestamp.now()
>>> idx = [now + pd.Timedelta(seconds=i) for i in range(1, 4)]
>>> df = pd.DataFrame({'a': [1,2,3]}, index=idx)
>>> df.ewm(halflife=pd.Timedelta(seconds=1)).mean()
                                   a
2023-07-18 10:46:46.589891  1.000000
2023-07-18 10:46:47.589891  1.666667
2023-07-18 10:46:48.589891  2.428571
```

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
> This would be an API behavior change so this isn't possible as is. Was this discussed in an issue?

No, it wasn't. Should I create an issue? Or is something like this not worth an API change?
You could create an issue, but IMO this behavior is less explicit so IMO I am not sure this is a good change
Thanks for opening up https://github.com/pandas-dev/pandas/issues/54187, going to close this PR for now until there has been more opt-in from other core developers in the issue
DOC: changed description of value_counts function 
State: closed

Changed the description of value_counts function

- [x] closes #48635
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Thanks @ArchPh03nix
Fix for #54167
State: closed

- [x] closes #54167
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).


Labels:

Comments:
I can check only for less than 0 instead of less than or equal to 0. Am I right?
Hi,
I am new to the library. May I know where should I add the tests and the what's new. 
Could you also tell me how can I test this?
You can review https://pandas.pydata.org/docs/development/contributing.html
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
BUG: iter(ser.str) did not raise TypeError
State: closed

- [x] closes #54173 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Strings
description: String extension data type and string data

Comments:
Looks like tests are passing here so merging. Happy to follow up if needed
Backport PR #54169 on branch 2.0.x (DEBUG: CY3 test_add_out_of_pydatetime_range)
State: closed

Backport PR #54169: DEBUG: CY3 test_add_out_of_pydatetime_range

Labels:
name: Unreliable Test
description: Unit tests that occasionally fail
name: Dependencies
description: Required and optional dependencies

Comments:
Will reopen if we decide to release a 2.0.4