BrandonFors/pandas-github-issues huggingface dataset:
Adjust Docker File Key Value Format
State: closed

- [x] closes #61036
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.

Labels:

Comments:
Thanks @arthurlw
CI/TST: Fix xfail in test_columns_dtypes_not_invalid for pyarrow nightly
State: closed

e.g. https://github.com/pandas-dev/pandas/actions/runs/13639721895/job/38126633833

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: CI
description: Continuous Integration

Comments:
Merging to get the CI to green
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.3.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 72839535bc0222686a97437457adb48edc531acb
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #61042: CI/TST: Fix xfail in test_columns_dtypes_not_invalid for pyarrow nightly'
```

4. Push to a named branch:

```
git push YOURFORK 2.3.x:auto-backport-of-pr-61042-on-2.3.x
```

5. Create a PR against branch 2.3.x, I would have named this PR:

> "Backport PR #61042 on branch 2.3.x (CI/TST: Fix xfail in test_columns_dtypes_not_invalid for pyarrow nightly)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
BUG: Recognize chained fsspec URLs
State: closed

- [ ] closes #48978
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/v3.0.0.rst` file if fixing a bug or adding a new feature.

The new `(::[A-Za-z0-9+\-+.]+)*` part of the regex will allow for optional `::<alphanumeric+-.>`-like strings  before the `://` section of the URL, which will enable it to recognize chained fsspec URLs.

Labels:
name: IO Network
description: Local or Cloud (AWS, GCS, etc.) IO Issues

Comments:
Just confirming that there isn't a solution alluded to in https://github.com/pandas-dev/pandas/issues/48978#issuecomment-1813692546 where `fsspec` provides an API already to validate fsspec URLs?
> Just confirming that there isn't a solution alluded to in [#48978 (comment)](https://github.com/pandas-dev/pandas/issues/48978#issuecomment-1813692546) where `fsspec` provides an API already to validate fsspec URLs?

I checked the latest fsspec API reference and couldn't find any public functions for validating URLs.
https://filesystem-spec.readthedocs.io/en/latest/api.html

Thanks @snitish 
Thanks @snitish!

I thought the bug was going to get closed the same week as when I documented the root cause and the bullet point solution.  Took a few years in the end!
Wow, I filed this bug in October 2022.  I am super pleased to have it fixed.  It would be churlish to complain about the amount of time it's taken--instead, I am just impressed by how inexorable progress seems to be!
Thanks @JMBurley for the outline you provided - it helped me fix this bug in no time.
@ligon glad I could be of help  :)
Could this fix also be applied to 2.3.x ?
BUG: fix DataFrame.__setitem__ throwing a ValueError when setting a column with a 2D object array
State: closed

- [x] closes #61026
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
DOC: Correct typos in Working with text data
State: closed

Correct typos in Working with text data.
https://pandas.pydata.org/docs/dev/user_guide/text.html

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks @star1327p
Bump to `pypa/cibuildwheel@v2.23.0`; build WASM wheels against Pyodide 0.27
State: closed

- [x] follow-up of https://github.com/pandas-dev/pandas/pull/60756#issuecomment-2607902892
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This bumps to https://github.com/pypa/cibuildwheel/releases/tag/v2.23.0 that was released a while ago. The notable highlights are that the project now _officially_ supports and tests the `ubuntu-22.04-arm` images, and bumps to Pyodide 0.27, matching the version used in the nightly wheels.

Other ancillary items from the release notes: PyPy 3.11 is now supported, but PyPy has been marked with a TODO and is not addressed here. 

Labels:
name: CI
description: Continuous Integration

Comments:
Thanks @agriyakhetarpal
Issue 60550 fix
State: closed

# Fix-#60550

#Removed this condition as a fix for issue #60550 :
![image](https://github.com/user-attachments/assets/70db04c3-7065-479e-bbc4-fe4c8800305c)

#Added this condition
 if percentiles == []:
        percentiles.append(0.5) # By default, if percentiles is empty then appending 50th percentile. 

![image](https://github.com/user-attachments/assets/5f2f122a-a536-4498-8d23-5527fdab824b)


Labels:

Comments:
@Abhibhav2003 The number of files changed in the PR looks incorrect. Please fix it.
BUG: Fix bug in to_datetime that occasionally throws FloatingPointErr…
State: closed

…or when called on a float array with missing values

- [ ] closes #58419
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/v3.0.0.rst` file if fixing a bug or adding a new feature.

The cause of the error is a numpy array created via np.empty(). Sometimes it contains garbage values which subsequently get passed to `np.round` causing errors.

Labels:
name: Datetime
description: Datetime data dtype
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
Thanks @snitish 
Pandas v.2.2.3, the bug is still there. I am processing a pd.Series filled with float timestamps. Overflows while `to_datetime` conversion with following `dt.round` to `ms`:
` time = pd.to_datetime(time, unit='ms', errors='coerce').dt.round(freq='ms')`
The error is related to `pd.to_datetime`, you can see below.

More details:
File "../pandas/core/tools/datetimes.py", line 1067, in to_datetime
values = convert_listlike(arg._values, format)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "../pandas/core/tools/datetimes.py", line 407, in _convert_listlike_datetimes
return _to_datetime_with_unit(arg, unit, name, utc, errors)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "../pandas/core/tools/datetimes.py", line 512, in _to_datetime_with_unit
arr = cast_from_unit_vectorized(arg, unit=unit)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "conversion.pyx", line 149, in pandas._libs.tslibs.conversion.cast_from_unit_vectorized
File "../numpy/_core/fromnumeric.py", line 3710, in round
return _wrapfunc(a, 'round', decimals=decimals, out=out)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "../numpy/_core/fromnumeric.py", line 57, in _wrapfunc
return bound(*args, **kwds)
^^^^^^^^^^^^^^^^^^^^
FloatingPointError: overflow encountered in multiply
@vgolskiy yes, the fix will be released with 3.0 (unless this gets backported to the 2.3.x branch)
What workaround is recommended until pandas 3 is released?
@RyuuOujiXS you can call `to_datetime()` after dropping `NaN`s, then add back the `NaN`s in the right places.

@snitish , so it was the NaN value that was causing overflow? Also, after removing NaNs, it is better to return NaTs as the data will have a `datetime` type by that moment.
@RyuuOujiXS I described the workaround in the last comment here https://github.com/pandas-dev/pandas/issues/58419#issuecomment-2773359135
TST: Add test for groupby with datetime and NaT values
State: closed

- [x] closes #35202
- [x] [Tests added and passed]
- [x] All [code checks passed]
- [ ] Added [type annotations]
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Thanks @asharmalik19
BUG: Fix MultiIndex alignment issue in Dataframe-Series binary operat…
State: closed

…ion in case of different levels

- [ ] closes #61009
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/v3.0.0.rst` file if fixing a bug or adding a new feature.

In case of arithmetic operations between a `DataFrame` with MultiIndex columns and a `Series`, if the two multiindexes don't have the same levels, the output values are all `nan` (see example in the original issue).
A similar issue for `DataFrame`-`DataFrame` arithmetic operations was fixed in #60538. This PR addresses the bug in the `DataFrame`-`Series` case with `axis=1` (i.e. aligning DataFrame columns with Series MultiIndex).

The fix itself is simple - we just use
```
right = other._reindex_indexer(join_index, ridx)
```
instead of 
```
right = other.reindex(join_index, level=level)
```
because `Series.reindex` needs the new MultiIndex to have identical levels to the current index. Instead, we simply use the `ridx` object that we already have from a previous `Index.join` call, which does handle partially overlapping levels well.

Labels:
name: Bug
name: MultiIndex

Comments:
Thanks @snitish
Update describe.py
State: closed

Updated the code to exclude 0.5 percentile

Excluded the 0.5 percentile from the code where a value is passed.

Added a test case to validate if 0.5 percentile is excluded or not


Labels:

Comments:
As mentioned before, you have an open pull request in https://github.com/pandas-dev/pandas/pull/60986 and we prefer you update that pull request with updates.

If you need guidance on how to do that see https://pandas.pydata.org/docs/development/contributing.html#updating-your-pull-request. Closing this PR in favor of your old one
PERF: use `blk.dtype` in `where()` & `_setitem_frame()`
State: closed

- [ ] closes #61010 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

this pr is a one-line change to the code here:
https://github.com/pandas-dev/pandas/blob/d1ec1a4c9b58a9ebff482af2b918094e39d87893/pandas/core/generic.py#L9735-L9737


### Performance comparison:

#### test script:
```python
import numpy as np
import pandas as pd
import timeit

for width in [10, 1000, 1000_00, 1000_0000]:
    df = pd.DataFrame(np.random.randn(1, width))
    mask = df > 0.5
    tm = timeit.timeit("df.where(mask)", number=10, globals=globals())
    print(width, tm)
```

#### for _dt in cond.dtypes:

10 0.002963045029900968
1000 0.006705133942887187
100000 0.40306550299283117
10000000 46.55275956704281
![image](https://github.com/user-attachments/assets/2f028c89-b868-45f2-9f01-dc0bcd9ba446)

#### for _dt in cond.dtypes.unique():

10 0.0028260269900783896
1000 0.002695770002901554
100000 0.042065858957357705
10000000 6.068146598991007
![image](https://github.com/user-attachments/assets/ade4f6a5-cc1a-47e4-8633-73c90522e64b)

#### for _dt in [blk.dtype for blk in cond._mgr.blocks]:

10 0.0009857049444690347
1000 0.0011893719201907516
100000 0.003112988080829382
10000000 0.13763279700651765
![image](https://github.com/user-attachments/assets/086371f8-a0f4-4d09-8130-d10ce0697565)


Labels:
name: Performance
description: Memory or execution speed performance

Comments:
@mroeschke 
Hi, do I need to add this in whatsnew? 2.3.0 or 3.0.0?
This pattern is also found in `_setitem_frame`:

https://github.com/pandas-dev/pandas/blob/5da9eb726c915412928ee44a0631a74bda614556/pandas/core/frame.py#L4276-L4289

#### Before
![image](https://github.com/user-attachments/assets/b21f596b-b292-4cf8-9106-e4da99eb4bb1)
#### After
![image](https://github.com/user-attachments/assets/4fcaf7bf-8872-4279-a3a7-f06a3f565131)


>  Hi, do I need to add this in whatsnew? 2.3.0 or 3.0.0?

`v3.0.0.rst` please
Thanks @auderson
DOC: Update DataFrame.drop() docstring 
State: closed

Updated to reflect existing behavior of allowing any iterable

- [x] closes #59890 
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks @carsfox
removed "if we" typo in is_dtype() doc
State: closed

- [x] closes #61011
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.


Labels:
name: Docs

Comments:
Thanks @arthurlw
docs: include option 'delete_rows' into `DataFrame.to_sql`
State: closed

- [x] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Hello folks, I missed to add the documentation in this place so it is a patch to the original [PR](https://github.com/pandas-dev/pandas/pull/60376).

https://pandas.pydata.org/pandas-docs/dev/reference/api/pandas.DataFrame.to_sql.html#
There's no mention to `delete_rows` ☝️ .

Please let me know if there's any other place :)

cc @WillAyd , @mroeschke 

Labels:
name: Docs
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
Thanks @gmcrocetti
TST: parametrize test_common
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Clean

Comments:
Thanks @fangchenli
DOC: Fix missing a closing bracket in contributing codebase
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks @chilin0525
TST: Change sqlite test query string values to single quotes
State: closed

xref https://github.com/pandas-dev/pandas/actions/runs/13505086719/job/37732738398

Might be a new failure interaction with `sqlite3` and sqlite 3.49 being released recently https://www.sqlite.org/changes.html, but if this query is backward compatible we might as well use this form.

closes https://github.com/pandas-dev/pandas/issues/60989

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
Since this is passing on sqlite 3.48 (tested on ubuntu-arm) and 3.49 (tested on ubuntu-(x86)), going to merge to get to green
BUG: fix read_json ignoring the dtype with the pyarrow engine
State: closed

- Added code to check if dtypes have been passed in, if they are, read the JSON with the schema built from those dtypes
- Split read JSON function up for the respective engines

- [X] closes #59516  (Replace xxxx with the GitHub issue number)
- [X] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [X] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [X] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO JSON
description: read_json, to_json, json_normalize
name: Arrow
description: pyarrow functionality

Comments:
Could any reviewers point me in the right direction with the failing test? Can't seem to figure out what is causing it
I believe that issue appears in pyarrow < 16. In your test you can check that pyarrow version and filter the warning if that is detected
> I believe that issue appears in pyarrow < 16. In your test you can check that pyarrow version and filter the warning if that is detected

Looks like that has worked, thanks!
Thanks @will-larkin
BUG: fix PeriodIndex.difference producing incorrect results
State: closed

- [x] closes #58971 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v3.0.0.rst` file if fixing a bug or adding a new feature.


Labels:
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
Update test_describe.py
State: closed

Added test case

Modified the .describe() method to avoid adding the 50th percentile by default when a custom percentiles list is provided.
Added a test case to verify the behavior when a single percentile is passed.

Labels:

Comments:
@mdjaved24 Hi, I noticed you already open a PR (https://github.com/pandas-dev/pandas/pull/60986) related to same issue, you should commit and push to same PR instead of creating two separate PRs.
Agreed. Please continue working on this feature in https://github.com/pandas-dev/pandas/pull/60986 so closing this one out