BrandonFors/pandas-github-issues huggingface dataset:
REGR: Series.astype raising TypeError when converting a pyarrow integer type with nulls to numpy float
State: closed

- [x] closes #57093
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.2.1.rst` file if fixing a bug or adding a new feature.


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Regression
description: Functionality that used to work in a prior pandas version
name: Arrow
description: pyarrow functionality

Comments:
The regression here was fixed by @phofl in #57233. I've updated this PR to just add a test for the regression for pyarrow integers reported in #57093
REGR: DataFrame.sort_index not producing stable sort
State: closed

- [x] closes #57151
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.2.1.rst` file if fixing a bug or adding a new feature.


Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Index
description: Related to the Index class or subclasses

Comments:
Thanks @lukemanley
Fix docstring
State: closed

- [ ] xref #57111 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks @thiphan94
WEB: Adding Effective pandas 2 book to home page
State: closed

@mattharrison I think you discussed this with @phofl, I need the affiliated link before it's published.

Also, I changed the color palette of the book cover to reduce the size of the file. I don't think it's easy to notice, but if you want to have a look, to make sure you're happy with it. Thanks!


Labels:
name: Web
description: pandas website

Comments:
/preview
Website preview of this PR available at: https://pandas.pydata.org/preview/pandas-dev/pandas/57164/
Hi @datapythonista . Can I get an email address to attribute referrals from the website?
Here's the link https://store.metasnake.com/effective-pandas-book/lhte7
Thanks @mattharrison, this should be ready now 
/preview
Website preview of this PR available at: https://pandas.pydata.org/preview/pandas-dev/pandas/57164/
Thx @datapythonista
CI: Add macOS M1 CI
State: closed

- [ ] closes #47443 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Build
description: Library building on various platforms
name: CI
description: Continuous Integration
name: OS X
description: Related to Mac OS & hardware issues (M1)

Comments:
self-merging as this is green. 

As always, happy to address comments in a followup.
Timestamp 'freq' is removed in GH 14146, not moved to kw-only arg. Removal is …
State: closed

…already captured under 'Removal of prior version deprecations/changes'

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Sure thanks @kevonscott
BUG: Fix to_dict with datelike types and orient=list
State: closed

- [ ] closes #57125 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug

Comments:
Very nice. In a similar vein I opened issue https://github.com/pandas-dev/pandas/issues/57063 - I wonder what other places we have in the code base where we have historically exposed nanoseconds-since-epoch values out to end users. Would be nice to get rid of all of them I think
CI: Remove redundant check
State: closed

I removed 2 checks that's already covered in `ruff`'s [`PGH003`](https://docs.astral.sh/ruff/rules/blanket-type-ignore/) and [`PGH004`](https://docs.astral.sh/ruff/rules/blanket-noqa/)

The second commit is just an example that the check works. Will revert before merging if approved.

```bash
ruff....................................................................................................Failed
- hook id: ruff
- exit code: 1

pandas/compat/numpy/function.py:209:18: PGH003 Use specific rule codes when ignoring type issues
pandas/tests/computation/test_compat.py:30:18: PGH004 Use specific rule codes when using `noqa`
Found 2 errors.

```

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: CI
description: Continuous Integration
name: Code Style
description: Code style, linting, code_checks

Comments:
Thanks for the work on this @tqa236. I only see one of the errors failing in the CI. Is this working as expected?
hello @datapythonista , the CI works as expected. The 2 errors are caught by `ruff` and made the pre-commit job failed, as shown [here](https://results.pre-commit.ci/run/github/858127/1706650692.CSnx5xuWQBKrRHyaVvxxWw). In addition, I think recently even `mypy` will also catch the bare type ignore error, and raise that error again in the `mypy` check. 

```
mypy.....................................................................Failed
- hook id: mypy
- duration: 77.99s
- exit code: 1

pandas/compat/numpy/function.py:209: error: "type: ignore" comment without error code (consider "type: ignore[return-value]" instead)  [ignore-without-code]
Found 1 error in 1 file (checked 1449 source files)
```
thanks for the review, @datapythonista, I reverted the last commit. The PR is ready.
Thanks again @tqa236 

If interested, it would be great to see what ruff (pylint) checks are covered by what is run by pylint. It would be great if we could just switch over to ruff with most of the rules being covered
@mroeschke I can look into it. Do you mind if I bump `ruff` to the latest version? A lot of new `pylint` rules are added recently, but I was trying to avoid creating conflict with your work in https://github.com/pandas-dev/pandas/pull/56863
> Do you mind if I bump ruff to the latest version?

Go for it. I am happy to keep that branch updated with whatever changes you make
CI: Fix _get_dst_hours for numpy 2.0 change
State: closed

xref https://github.com/numpy/numpy/pull/25570

I don't think we can make this annotation backwards compatible, so removing the annotation all together.

Labels:
name: Timezones
description: Timezone data dtype
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
Going to merge to get the CI back to green
Can you open up a followup issue? Thanks.
Sure https://github.com/pandas-dev/pandas/issues/57154
BUG: Index(Series) makes array read only for object dtype
State: closed

- [ ] closes #57130 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Copy / view semantics

Comments:
Thanks @phofl
DEPR: removed deprecated argument `obj` from GroupBy `get_group`
State: closed

- xref #53571
- removed deprecated argument `obj` from GroupBy `get_group`
- added an entry in the doc/source/whatsnew/v3.0.0.rst



Labels:
name: Groupby
name: Deprecate
description: Functionality to remove in pandas

Comments:
@MarcoGorelli, could you please take a look at this PR? CI failures aren't related to my changes.
Thanks @natmokval!
Fix typo in missing_data.rst
State: closed

<img width="736" alt="Screen Shot 2024-01-29 at 10 58 11 AM" src="https://github.com/pandas-dev/pandas/assets/48232/1c2b9c87-9acc-43a6-8fe2-19a3a131c9c1">


Labels:
name: Docs

Comments:
Thanks @davetapley
Try to enable some aspects of LSAN
State: closed


Labels:
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Just removed ASAN from CI so this has a ways to go before its worth trying again
CI: avoid DeprecationWarning in validate_min_versions_in_sync script
State: closed

this warning is showing up in CI because of the extra whitespace here

https://github.com/pandas-dev/pandas/blob/09cb36710aab9d9a724602aa7ccfde33fbc51f63/scripts/tests/data/deps_expected_random.yaml#L41

which then goes to the `except` arm in 

https://github.com/pandas-dev/pandas/blob/ffc583966451fd26cdd4c350018362f6fadd02a3/pandas/util/version/__init__.py#L122-L125

which throws the warning

Labels:
name: CI
description: Continuous Integration

Comments:
Thanks @MarcoGorelli
CI: autouse add_doctest_imports
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Just trying this out to see impact to the CI

Ref: https://github.com/pytest-dev/pytest/issues/11873
Effectively reverts #45667

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: CI
description: Continuous Integration

Comments:
@mroeschke - this gets doctests back to green, not noticing an appreciable slowdown.
Thanks @rhshadrach
REGR: Fix to_numpy for masked array with non-numeric dtype
State: closed

- [ ] closes #56991 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
sorry to ask this here, but its somewhat related. Should the following return `pd.NA` or `np.nan` for the na_value? It is returning `pd.NA` but that isn't allowed in a float64 array:

```
In [1]: import pandas as pd

In [2]: from pandas.core.arrays._utils import to_numpy_dtype_inference

In [3]: from pandas._libs import lib

In [4]: arr = pd.array([1, pd.NA], dtype="int64[pyarrow]")

In [5]: to_numpy_dtype_inference(arr, dtype="float64", na_value=lib.no_default, hasna=True)
Out[5]: (dtype('float64'), <NA>)
```
> Should the following return pd.NA or np.nan for the na_value? It is returning pd.NA but that isn't allowed in a float64 array:

Personally I would expected `np.nan` here
#57233
@mroeschke good to merge?
Thanks @phofl
DEPR: Enforce deprecation of ArrayManager
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
Thanks @Dr-Irv, @jbrockmendel, and @mroeschke. I think this is ready for another look. Two outstanding issues:

 - [The arrays method is used in 8 places in pandas.core](https://github.com/pandas-dev/pandas/pull/57118#discussion_r1468915971)
 - Removal of `internals.base`. This will involve moving a significant amount of code around, so I'd like to handle in a followup.
Thanks @rhshadrach
BUG: fix `Series.value_counts` with `sort=False` returns result sorted on values for Series with string dtype
State: closed


- [x] closes #55224 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
thx @yuanx749
fix: bug solution
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Thanks for the pull request, but I would suggest you follow this guide and develop locally first to ensure the fix works correctly before a review https://pandas.pydata.org/pandas-docs/stable/development/contributing_codebase.html#test-driven-development
DOC: fix PR02 errors in docstrings - pandas.Series.interpolate, pandas.read_hdf, pandas.HDFStore.append
State: closed

All PR02 Errors resolved in the following cases:

1. scripts/validate_docstrings.py --format=actions --errors=PR02 pandas.Series.interpolate     
2. scripts/validate_docstrings.py --format=actions --errors=PR02 pandas.read_hdf      
3. scripts/validate_docstrings.py --format=actions --errors=PR02 pandas.HDFStore.append 

OUTPUT:

1. scripts/validate_docstrings.py --format=actions --errors=PR02 pandas.Series.interpolate     
```
################################################################################
################################## Validation ##################################
################################################################################

Docstring for "pandas.Series.interpolate" correct. :)
```

2. scripts/validate_docstrings.py --format=actions --errors=PR02 pandas.read_hdf      
```
################################################################################
################################## Validation ##################################
################################################################################

Docstring for "pandas.read_hdf" correct. :)
```

3. scripts/validate_docstrings.py --format=actions --errors=PR02 pandas.HDFStore.append 
```
################################################################################
################################## Validation ##################################
################################################################################

8 Errors found for `pandas.HDFStore.append`:
	PR01	Parameters {'complib', 'complevel', 'axes', 'columns', 'errors'} not documented
	PR07	Parameter "key" has no description
	PR07	Parameter "min_itemsize" has no description
	PR07	Parameter "nan_rep" has no description
	PR07	Parameter "chunksize" has no description
	PR07	Parameter "expectedrows" has no description
	PR07	Parameter "encoding" has no description
	SA01	See Also section not found
```


- [x] xref https://github.com/pandas-dev/pandas/issues/57111
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs
name: Clean

Comments:
Thanks @jordan-d-murphy
CI: Use new documentation previewer
State: closed

- [X] closes #50832

We've got a documentation previewer service for few months now, but the whole infrastructure powering it was just a PoC not very suitable for the long term (it didn't have error control, was running in a screen session, the workflow wasn't optimal, a github token was hardcoded, it required to remove old previews manually...).

Now we've got a service that IMHO is ready for production. Everything is built independently from pandas in this repo:
https://github.com/pandas-dev/github-doc-previewer/. In the pandas side the only thing we need is to call a GitHub action (check the diff in this PR).

For most people there won't be a significant difference between the previous PoC and the new system, as the UX is the same. Besides stability the main differences are:
- A preview will be generated only for the PRs where it's requested (with the `/preview` comment). Previously we published the previews of every PR.
- The new service is pandas-agnostic. If there is interest, any other repo can use it, the only thing needed is that the GitHub organization of the project is added to a setting in the server. The whole `pandas-dev` organization is of course added. Not sure if we have any other repo where this can be useful, but only adding the GitHub action is enough to have previews enabled.
- The previous point has a small implication. The urls of the previews now contain the org and repo. So, instead of pandas.pydata.org/preview/1234/ the new url is now pandas.pydata.org/preview/pandas-dev/pandas/1234/

Any feedback or problem please let me know.

Labels:
name: Docs
name: CI
description: Continuous Integration

Comments:
I'll be merging this soon if there are no objections, so we can use the previewer again (with the new service).
Thanks all your hard work here @datapythonista 
I had to increase the timeout on nginx, since the requests stays life for a bit more than one minute until the preview is ready and the comment can be posted, but now everything seems to be working fine with the new version.
DEPR: Enforce deprecation of removing axis from all groupby ops
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Groupby
name: Deprecate
description: Functionality to remove in pandas

Comments:
Thanks @rhshadrach
DOC: Update meeseeksmachine reference
State: closed


Labels:
name: Docs

Comments:
Wait, ... they changed it? 😱 

I remember using ``@meeseeksdev`` still worked, at least a couple weeks back.
Idk I tried using it in https://github.com/pandas-dev/pandas/pull/57078 and wasn't getting a response from the old command. Also probably best to change this since meeseeksdev points to a user profile now
@MeeseeksDev hello
Helloooo @datapythonista, I'm Mr. Meeseeks! Look at me!
I have the feeling that using `@meeseeksdev` is like when we use `/preview`, not really tagging that user, but just the keyword for the bot. I still see in all the docs that the right command is `@meeseeksdev`, and from the above test seems like it's still working. Happy to get this merged if I'm missing something, but in general I don't think this is needed.
Ah OK maybe I was impatient with the response time of the bot so I'll close this PR