BrandonFors/pandas-github-issues huggingface dataset:
REF (string): de-duplicate str_isfoo methods
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Refactor
description: Internal refactoring of code
name: Strings
description: String extension data type and string data
name: backported

Comments:
Thanks @jbrockmendel
TST (string): more targeted xfails in test_string.py
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Strings
description: String extension data type and string data
name: backported

Comments:
Thanks @jbrockmendel
TST (string): copy/view tests
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Strings
description: String extension data type and string data
name: backported

Comments:
Thanks @jbrockmendel
Fix docstring timestamps (Issue #59458)
State: closed

related to #59688 
- [x] Addresses some tasks of #59458 
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks @mimistiv
DOC: fix broken link for governance in the team web page
State: closed

- [ ] closes #59693 
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `web/pandas/about/team.md` file to correct the broken link.


Labels:

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
Changed to GroupBy
State: closed

- [ ] closes #59695  (Replace xxxx with the GitHub issue number)
 I have tried changing the groupby to GroupBy. Please check the hyperlink if it works.


Labels:
name: Docs

Comments:
On it 
Hey @mroeschke , I made the necessary changes. Could you please check and merge the PR . I really would like to contribute to such amazing projects like pandas.
Thanks @doshi-kevin
ENH: add basic DataFrame.from_arrow class method for importing through Arrow PyCapsule interface
State: open

See https://github.com/pandas-dev/pandas/issues/59631

For now, this adds the most basic method, just converting an Arrow tabular object, without exposing any keyword arguments (or without exposing it in `pd.DataFrame()` directly)


- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Stale
name: Arrow
description: pyarrow functionality

Comments:
cc @kylebarron 
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
I think it'd be great to have this for pandas 3. Do you have the time to finish this @jorisvandenbossche ?

I think we're missing a release note in this PR, and maybe we could also mention this new feature in this section? https://pandas.pydata.org/docs/user_guide/pyarrow.html#data-structure-integration
Seems reasonable.
[pre-commit.ci] pre-commit autoupdate
State: closed

<!--pre-commit.ci start-->
updates:
- [github.com/astral-sh/ruff-pre-commit: v0.5.0 → v0.6.3](https://github.com/astral-sh/ruff-pre-commit/compare/v0.5.0...v0.6.3)
- [github.com/asottile/pyupgrade: v3.16.0 → v3.17.0](https://github.com/asottile/pyupgrade/compare/v3.16.0...v3.17.0)
<!--pre-commit.ci end-->

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Looks like we may need to do a mypy upgrade first before pursuing this first so closing
TST (string dtype): resolve all infer_string TODO/xfails in pandas/tests/arrays
State: closed

Together with the test changes in https://github.com/pandas-dev/pandas/pull/59678 and https://github.com/pandas-dev/pandas/pull/59682, this PR then resolves the remaining `infer_string` TODO or xfails in `pandas/tests/arrays`

xref https://github.com/pandas-dev/pandas/issues/54792/

Labels:
name: Strings
description: String extension data type and string data
name: backported

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.3.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 4f328f08df90906198b3a3f955ab321018964f0a
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #59686: TST (string dtype): resolve all infer_string TODO/xfails in pandas/tests/arrays'
```

4. Push to a named branch:

```
git push YOURFORK 2.3.x:auto-backport-of-pr-59686-on-2.3.x
```

5. Create a PR against branch 2.3.x, I would have named this PR:

> "Backport PR #59686 on branch 2.3.x (TST (string dtype): resolve all infer_string TODO/xfails in pandas/tests/arrays)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Manual backport: https://github.com/pandas-dev/pandas/pull/60020
String dtype: map builtin str alias to StringDtype
State: closed

This ensures that the builtin type `str` is also mapped to the future default string dtype (if `infer_string` is enabled), i.e. to make `.astype(str)` behave the same as `.astype('str")`.

In general we rely on numpy for mapping builtin types to dtypes. For example `int` is mapped to `np.dtype("int64")` by passing that value to `np.dtype(..)` in `pandas_dtype()`. 
But now that we have a default dtype that is mapped to one of those builtin types, we can no longer rely on numpy doing that. Implementation wise, for `pandas_dtype(str)` I currently added a special case checking explicitly for `str` and then importing the dtype lazily and returning it. 

An alternative could be to handle this through the ExtensionDtype `Registry.find()`. However, that currently relies on using `ExtensionDtype.construct_from_string()` (i.e. we would have to expand the typing there to allow type objects as well next to strings, which is of course also possible). And maybe it is better to have control centrally about builtin aliases, instead of allowing external dtypes to map to those as well.

xref https://github.com/pandas-dev/pandas/issues/54792/

Labels:
name: Strings
description: String extension data type and string data
name: backported

Comments:
Thanks @jorisvandenbossche
fix: use fastpath for PyCapsule export when starting from pyarrow-backed Series, respect requested_schema
State: closed

Follow-up to https://github.com/pandas-dev/pandas/pull/59587/files (so, I haven't added a new whatsnew)

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Arrow
description: pyarrow functionality

Comments:
@jorisvandenbossche @WillAyd fancy taking a look?
Thanks for the follow up @MarcoGorelli
String dtype: implement _get_common_dtype
State: closed

This PR implements `StringDtype._get_common_dtype` to control the behaviour in concat etc.

This fixes two things:
- When concatting with a numpy string-like dtype (S/U), we can preserve our StringDtype instead of converting to object dtype (as happens now). In general we don't store such numpy dtypes in pandas objects, but in certain corner cases we are passing numpy dtypes directly to `find_common_type`.
- When concatting different instances of StringDtype. In practice this shouldn't happen often, but can certainly occur. In this case, I decided to give precedence to the pyarrow / pd.NA: 1) if pyarrow is installed, that's the default and therefore logical pick in this case as well, and 2) pd.NA is opt in, so if you have a mix, assume the user explicitly opted in and propagate that preference.

xref https://github.com/pandas-dev/pandas/issues/54792





Labels:
name: Strings
description: String extension data type and string data
name: backported

Comments:
Thanks @jorisvandenbossche
infer
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
what does this infer?
Fix groupby agg preserve subclass type
State: closed

- [x] closes #59667 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.3.0.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Closing as described in https://github.com/pandas-dev/pandas/issues/59667#issuecomment-2324674201
TST (string dtype): fix and clean up arrow roundtrip tests
State: closed

xref https://github.com/pandas-dev/pandas/issues/54792

The `arrow_compat` tests were already passing (after my previous PR fixing arrow import/export, I assume). 
The `arrow_roundtrip` tests needed a fix in the expected dtypes. 

Labels:
name: Strings
description: String extension data type and string data
name: backported

Comments:
Thanks @jorisvandenbossche
fix typos
State: closed

_cf_: https://github.com/pandas-dev/pandas/issues/59640

fixes implemented via shell script


Labels:
name: Docs

Comments:
Thanks @musvaage 

FYI: pandas, and many other open source projects, use https://github.com/codespell-project/codespell to help catch these spelling mistakes. If interested you might want to update their corpus so other projects can benefit. 
I have added most of these typos in https://github.com/codespell-project/codespell/pull/3531. Thank you.

Without context,  `experimential` could be changed into `experimental` or `experiential`, so I have added both suggestions.
@DimitriPapadopoulos

pandas/io/pytables.py

reverted  this change which shouldn't have been implemented

```diff
-         """return a tuple of my permutated axes, non_indexable at the front"""
+         """return a tuple of my permuted axes, non_indexable at the front"""
```
I added `permutate` to the _rare_ dictionary, I will remove it altogether.
DOC: Remove unnecessary parentheses in `:func:` & `:meth:` role
State: closed

- [x] closes #59662
- [x] ~[Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature~ N/A
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] ~Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.~ N/A
- [x] ~Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.~ N/A

---

The func & meth role's parentheses are unnecessary as Sphinx will add them to the rendered output.

```rst
.. bad
:func:`read_csv()`

.. good
:func:`read_csv`
```

We're introducing a new rule to the sphinx-lint ([issue](https://github.com/sphinx-contrib/sphinx-lint/issues/114) & [PR](https://github.com/sphinx-contrib/sphinx-lint/pull/115)) and, since sphinx-lint utilizes pandas doc in the CI, we'd like to also fix the issue in pandas.


Labels:
name: Docs

Comments:
Thanks @mattwang44 great to see a new sphinx-lint rule for this in the future
DOCS: fix docstring validation error for pandas.Series
State: closed

Partially addresses https://github.com/pandas-dev/pandas/issues/59592

Fixes
```
-i "pandas.Series.dt.normalize PR01" \
-i "pandas.Series.dt.qyear GL08" \
```

Labels:

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
ENH: expose `to_pandas_kwargs` in `read_parquet` with pyarrow backend
State: closed

- [x] closes #49236, #56842
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Enhancement
name: IO Parquet
description: parquet, feather

Comments:
@jorisvandenbossche would you be able to review? Tagging you since you mentioned this in #56842.
> Would be better if also add `to_pandas_kwargs` to `read_parquet` explicitly.

I wasn't sure if I should do this since it is an engine specific option. Happy to add it though.
Looking forward to upgrading to this fix. Due to this issue, I am currently using pickles, which is not ideal.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
(I see that in https://github.com/pandas-dev/pandas/pull/57044, we had some discussion if we actually want this .. Will move that part of the discussion to the issue)
@jorisvandenbossche I think I've addressed your comments and resolved merge conflicts.
Thanks @kleinhenz!
Fixed Typos #59640
State: closed

- [✅] closes #59640 


Labels:
name: Clean

Comments:
@asishm could you please review 
Thanks @callumfrederiksen 
pandas/tests/io/test_stata.py

the ISO 3166 code and the Internet TLD for the
Republic of Senegal respectively are SN and .sn

```diff
- Value labels for column ethnicsn are not unique. These cannot be converted to
+ Value labels for column ethics are not unique. These cannot be converted to
```
increase verbosity
State: closed

increase verbosity of error message by including the type causing the error in the error message

- [ ] closes #59659
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions. -> not applicable
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature. -> minor change


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
Is this PR associated with an issue?
@mroeschke I've opened one now #59659
can I re-open a new PR regarding this? seems like this PR has become stale
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
TST: Add SparseArray _arith_method test to fail on incorrect comparison operator
State: closed

- [x] Part of #58517 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] ~Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.~ Not applicable
- [ ] ~Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.~ Not applicable

Before, if the line [core/arrays/sparse/array.py:1768](https://github.com/pandas-dev/pandas/blob/v2.2.1/pandas/core/arrays/sparse/array.py#L1768) is changed from  `if len(self) != len(other):` to `if len(self) > len(other):`, all tests still passed. Now they will fail if the comparison operator is changed to anything besides `!=`.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Sparse
description: Sparse Data Type

Comments:
@mroeschke Could you take a look at the change? Thanks!
Thanks @jsngn
PERF: CategoricalDtype.update_dtype
State: closed

If a `CategoricalDtype` is passed to `CategoricalDtype.update_dtype`, this API will attempt to unnecessarily re-validate the categories if it was not None.

`CategoricalDtype.update_dtype` is called in constructors like `Categorical.__init__` and `Categorical._simple_new` where there is an attempt to update the passed dtype with `ordered=False` if it was `None`. A fully validated `CategoricalDtype` should just return itself if passed to `update_dtype`

```python
In [1]: import pandas as pd

In [2]: cdtype = pd.CategoricalDtype(categories=list(range(100_000)), ordered=True)

In [3]: base_dtype = pd.CategoricalDtype(ordered=False)

In [4]: %timeit base_dtype.update_dtype(cdtype)
2.5 μs ± 11.5 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)

In [4]: %timeit base_dtype.update_dtype(cdtype)
865 ns ± 2.26 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Categorical
description: Categorical Data Type

Comments:
Thanks for the fix @mroeschke !
Looks like tests are passing here so merging. Happy to follow up if needed