BrandonFors/pandas-github-issues huggingface dataset:
PERF: faster _coerce_to_data_and_mask() for astype("Float64")
State: closed

- [ ] closes #60066
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Use `np.isnan` for floats instead of `libmissing.is_numeric_na`. Also add a fast path for booleans

Prev:
![image](https://github.com/user-attachments/assets/2e76a59f-c5f1-4670-b9cb-3df71b7b34dc)
New:
![image](https://github.com/user-attachments/assets/27ba0f21-a578-4afe-9f5f-2f01a389a17d)



Labels:
name: Performance
description: Memory or execution speed performance
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
Failures seem unrelated: `FAILED pandas/tests/io/test_parquet.py::TestParquetPyArrow::test_timezone_aware_index[timezone_aware_date_list1] - AssertionError: DataFrame.index are different`
The results from asv don't seem as impressive as the results above, guessing the scale of these tests is much smaller.
![image](https://github.com/user-attachments/assets/b5c83eec-b7b4-4ba8-b48f-5c6240a158c2)

@rhshadrach friendly ping
Thanks @auderson
BUG: isin check incorrect with uint64 dtype
State: closed

- [ ] closes #59609 
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Stale

Comments:
These test failures come from changing the ```np_find_common_type``` method, which the ```isin``` method uses to cast the values to a common type. The problem is that when taking a int64 and uint64, the common type is float64, causing precision errors. I instead had ```np_find_common_type``` return an object type, which fixed the issue for the bug but obviously caused a lot of test issues.

Should I only edit inside the ```isin``` method, or should I change the ```np_find_common_type```? Any advice would be helpful.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
TST (string dtype): fix xfails in test_algos.py
State: closed

Remaining xfails in `test_algos.py`

xref https://github.com/pandas-dev/pandas/issues/54792

Labels:
name: Strings
description: String extension data type and string data

Comments:
Does #59433 cover this?
Ah, sorry, should have checked the open PRs first. Will put that on my to do list to review (for next week)
Covered by https://github.com/pandas-dev/pandas/pull/59433, so closing this PR.
BUG: preserve (object) dtype in factorize
State: closed

Noticed this issue in https://github.com/pandas-dev/pandas/pull/60116 (which adds an xfail for it):

```python
>>> pd.options.future.infer_string = True
>>> idx = pd.Index(["string", "with", "object", "dtype"], dtype=object)
>>> idx
Index(['string', 'with', 'object', 'dtype'], dtype='object')
>>> idx.factorize()
(array([0, 1, 2, 3]),
 Index(['string', 'with', 'object', 'dtype'], dtype='str'))    # <--- now str dtype
```

I think in general a function like factorize should preserve the dtype, even though a better dtype could be inferred here.

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
Thanks @jorisvandenbossche 
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.3.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 13926e5e298acf328b0c1347f008ef3f9c4eb078
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #60118: BUG: preserve (object) dtype in factorize'
```

4. Push to a named branch:

```
git push YOURFORK 2.3.x:auto-backport-of-pr-60118-on-2.3.x
```

5. Create a PR against branch 2.3.x, I would have named this PR:

> "Backport PR #60118 on branch 2.3.x (BUG: preserve (object) dtype in factorize)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Manual backport -> https://github.com/pandas-dev/pandas/pull/60174
TST (string dtype): duplicate pandas/tests/indexes/object tests specifically for string dtypes
State: closed

We have some `Index` tests specifically for object dtype. Some of those will already get tested through the builds that enable the string dtype by default, but so this PR essentially duplicated those tests and:

- updated the object dtype ones to explicitly use `dtype=object`, and deleted the ones that are specific to strings
- for the duplicated ones that are relevant for strings, parametrized all tests for the various string dtypes (`any_string_dtype` fixture)

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Strings
description: String extension data type and string data

Comments:
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.3.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 d8905e4bee2aa0e096ed7831fea7d395d7657120
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #60117: TST (string dtype): duplicate pandas/tests/indexes/object tests specifically for string dtypes'
```

4. Push to a named branch:

```
git push YOURFORK 2.3.x:auto-backport-of-pr-60117-on-2.3.x
```

5. Create a PR against branch 2.3.x, I would have named this PR:

> "Backport PR #60117 on branch 2.3.x (TST (string dtype): duplicate pandas/tests/indexes/object tests specifically for string dtypes)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Manual backport in https://github.com/pandas-dev/pandas/pull/60131
TST (string dtype): add explicit object vs str dtype to index fixture
State: closed

Also for when we switch the default, that we keep testing object dtype

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Strings
description: String extension data type and string data

Comments:
The two failing builds are failing on main as well, so going to merge this (blocking a follow-up PR)
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.3.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 7bd594c81acb5f6428e9ef54ba5a9da1f2860a89
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #60116: TST (string dtype): add explicit object vs str dtype to index fixture'
```

4. Push to a named branch:

```
git push YOURFORK 2.3.x:auto-backport-of-pr-60116-on-2.3.x
```

5. Create a PR against branch 2.3.x, I would have named this PR:

> "Backport PR #60116 on branch 2.3.x (TST (string dtype): add explicit object vs str dtype to index fixture)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Manual backport at https://github.com/pandas-dev/pandas/pull/60136
Fixes: [#60084] Added a new file test_timestamp_hash.py
State: closed

This pull request addresses the problem described in..
- [ ] closes #60084  (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://github.com/pandas-dev/pandas/issues/60084#issue-2606527029) if fixing a bug or adding a new feature
 The new test file created ensures that 2 timestamp Objects, created from the same String but in the context of different time zones are equal and have the same hash value after converting to Co ordinated universal time(UTC).


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
This PR seems to have gone stale. I'll take over this issue thanks!
Closing as the parent issue was closed
Backport PR #60109 on branch 2.3.x (CI/TST: Update pyreadstat tests and pin xarray on CI)
State: closed

Backport PR #60109: CI/TST: Update pyreadstat tests and pin xarray on CI

Labels:
name: CI
description: Continuous Integration
name: Dependencies
description: Required and optional dependencies

Comments:
@WillAyd @jorisvandenbossche - good to backport this, yea?
Yes, thanks!
CI/TST: Update pyreadstat tests and pin xarray on CI
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

xarray issue: https://github.com/pydata/xarray/issues/9661


Labels:
name: CI
description: Continuous Integration
name: Dependencies
description: Required and optional dependencies

Comments:
Merging to get the CI green.
Fix BUG: read_sql tries to convert blob/varbinary to string with pyarrow backend
State: closed

- [x] closes #59242 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This PR allows for bytes based data to be returned instead of throwing an exception like before

Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query
name: Stale
name: Arrow
description: pyarrow functionality

Comments:
The most recent change on main seems to also have 17 failing tasks so I _think_ these failures are unrelated to the change in my PR but please let me know if that's wrong!
trying to figure out why this test is failing https://github.com/pandas-dev/pandas/actions/runs/12484030959/job/34840808563?pr=60105 despite passing locally and in the other unit test checks - seems to be due to a library version mismatch between the different unit test CI checks and the `Future infer strings` check
@kastkeepitjumpinlikekangaroos when running that locally you would need to set an environment variable:

```sh
PANDAS_FUTURE_INFER_STRING=1 python -m pytest pandas/tests/io/test_sql.py
```

Can you reproduce the error if you do that?
> @kastkeepitjumpinlikekangaroos when running that locally you would need to set an environment variable:
> 
> ```shell
> PANDAS_FUTURE_INFER_STRING=1 python -m pytest pandas/tests/io/test_sql.py
> ```
> 
> Can you reproduce the error if you do that?

@WillAyd ah yes thank you!! was able to reproduce and fix the failing test locally. Waiting on CI now but should be passing there too. Please let me know if there's another way to be checking for the `infer_string` config option other than the way I'm currently doing it (`pd.options.future.infer_string`).
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
DOC: to_latex braces in headers must be escaped (#60063)
State: closed

- [x] closes #60063
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Any braces used in the headers must be escaped with `{{` or `}}`, and it is not trivial to figure out what goes wrong. I have added that the braces must be escaped, as agreed in the linked issue.

Labels:
name: Docs
name: IO LaTeX
description: to_latex

Comments:
Thanks @GregersSR
BUG: `read_csv` with chained fsspec TAR file and `compression="infer"`
State: closed

- [x] closes #60028
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: IO CSV
description: read_csv, to_csv

Comments:
tests needs `ffspec` to be installed. What do you think @rhshadrach?
Use the decorator `td.skip_if_no("fsspec")`.
CI failures seems unrelated

cc: @rhshadrach 
Improved the test function to validate the dataframe inputs as well
```
(pandas-dev) kev@mac pandas % pytest pandas/tests/io/test_common.py::test_read_csv_chained_url_no_error
+ /opt/homebrew/Caskroom/miniforge/base/envs/pandas-dev/bin/ninja
[1/1] Generating write_version_file with a custom command
============================================================================================================ test session starts =============================================================================================================
platform darwin -- Python 3.10.15, pytest-8.3.3, pluggy-1.5.0
PyQt5 5.15.9 -- Qt runtime 5.15.8 -- Qt compiled 5.15.8
rootdir: /Users/kev/self/pandas
configfile: pyproject.toml
plugins: localserver-0.0.0, qt-4.4.0, cov-5.0.0, hypothesis-6.115.5, cython-0.3.1, anyio-4.6.2.post1, xdist-3.6.1
collected 1 item

pandas/tests/io/test_common.py .

------------------------------------------------------------------------------------------ generated xml file: /Users/kev/self/pandas/test-data.xml ------------------------------------------------------------------------------------------
============================================================================================================ slowest 30 durations ============================================================================================================
0.01s call     pandas/tests/io/test_common.py::test_read_csv_chained_url_no_error

(2 durations < 0.005s hidden.  Use -vv to show these durations.)
```
CI looks unrelated

> CI looks unrelated

The test that's failing is the one being added here.

https://github.com/pandas-dev/pandas/actions/runs/11700807499/job/32585578297?pr=60100#step:8:83
> The test that's failing is the one being added here.

Any ideas to fix it? Since the CI was `minimum versions` I assumed that it was just outdated version of Tarfile or pandas
Looks like a bug with the version of fsspec being tested (`TarContainedFile` in fsspec didn't define flush). You'll want to specify a minimum version in `skip_if_no` (2023.1.0)
Thanks @KevsterAmp
[TST] Parse arrow_dtype as datetime index
State: closed

- [X] closes #57930(Replace with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Stale

Comments:
As feedback : 
- On my pandas version (not main as I don't have a C compiler), the returned dataframes have the following dtypes : 
   - index : datetime64[ns]
   - int64[pyarrow]
I would have expected the index to also be a pyarrow dtype, but unsure if I need to enforce that.
I checked some of the fails : 
- The new test fails even if was stated to be working on main https://github.com/pandas-dev/pandas/issues/57930#issuecomment-2248564090. Strange.
- There is an error on ISORT error, but Ruff corrects it differently 🙄 



This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Keep it open please 🫣
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
@mroeschke : I have done all you requested.
BLD: relax meson/meson-python requirements
State: closed

This makes bugfixes from later meson/meson-python fixes available to build pandas. For eg: python 3.13t support in meson, needs an up-to-date version of meson.

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

cc @rgommers 

Labels:
name: Build
description: Library building on various platforms
name: Python 3.13

Comments:
Thanks for the PR.

I'm OK with bumping, but I'd prefer leaving the pin in.
(This has "protected" us from bugs/breaking changes in the latest meson in the past.)

CI looks to be just red in general across pandas unfortunately. 
(If someone doesn't fix the failing builds over the next couple of days, I'll go ahead and take a look myself.)
> I'm OK with bumping, but I'd prefer leaving the pin in.
> (This has "protected" us from bugs/breaking changes in the latest meson in the past.)

Hard pins in general are never the right thing to do in `main` (unless the dependency is too unstable, which meson/meson-python are not), so if you really want to keep an `==` then I suggest:
1. Put a comment above it that this is a pre-emptive pin only, and distro maintainers can feel free to drop the pin
2. Amend your release procedure to bump the pinned version to the latest released one before each release.

If you don't do that, you will end up creating problems for distros, as well as missing out on lots of bug fixes which tend to be needed for users on niche platforms and niche compilers in particular.
Also +1 to remove the pin. I think it was really useful when we swapped over to Meson, but its probably outlived its usefulness. I've used Meson on a few other projects with only setting a minimum bound, and haven't run into issues. 
CI is fixed now, can you merge main.
Thanks @isuruf