BrandonFors/pandas-github-issues huggingface dataset:
BUG: Fix pd.read_html handling of rowspan in table header
State: closed

- [x] closes #60210(Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.

From the original thread:
```
s = '<table><tr><th rowspan="2">A</th><th>B</th></tr><tr><td>1</td></tr><tr><td>C</td><td>2</td></tr></table>'
buf = io.StringIO(s)
print(pd.read_html(buf)[0])
#    A                  B
#    A Unnamed: 1_level_1
# 0  1                NaN

# Expected:
#    A  B
# 0  A  1
# 1  C  2
```
The bug is due to rowspan > 1 in the header row which leads to overflow into the body rows. Current logic does not handle this case. I fix it by overflowing the partial rows from the header into the body (and similarly from body to footer if any).

Labels:
name: Bug
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap

Comments:
Thanks @snitish
BUG: Fix keyerror bug when indexing multiindex columns with NaT values
State: closed

- [x] closes #60388 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
Thanks @snitish
CI: avoid bad combination of pyopenssl and cryptography versions
State: closed

We are running into https://github.com/conda/conda/issues/13619#issuecomment-2075825121 in some of our envs, seeing what it gives to force a newer pyopenssl version.

Labels:
name: CI
description: Continuous Integration

Comments:
I think we can close this PR since this error isn't showing up on main anymore
BUG: ValueError when printing a DataFrame with DataFrame in its attrs
State: closed

- [x] closes #60455(Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This PR replaces `concat` with slice in `_truncate_horizontally` as in `_truncate_vertically` to avoid the ValueError.
https://github.com/pandas-dev/pandas/blob/a4fc97e92ed938260728e3f6c2b92df5ffb57b7f/pandas/io/formats/format.py#L697-L699


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: metadata
description: _metadata, .attrs

Comments:
Hmm, I don't understand why there is a mypy error after this change:
 pandas/io/formats/string.py:82: error: Cannot determine type of "tr_col_num"  [has-type]

I don't get this error locally.
> Looks good. Could you add a whatsnew entry in `v3.0.0.rst`?

Not sure where the right place is, I added in the `Other` section.
Thanks @yuanx749 
Many thanks @yuanx749
BUG: Groupby.sum, DataFrame.sum and Series.sum for object type should be NA instead of 0 for all-nan values
State: closed

- [ ] partly addresses #60229
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added note to whatsnew/v3.0.0.rst

Labels:
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
@snitish - This looks good to me if you'd like to reopen. I'm handling the string case in #60936
@rhshadrach sure I'd be happy to. There was some debate on the original thread on whether the default sum value for `object` dtype should be `NA` or `""` (empty string). Any thoughts on this?
> There was some debate on the original thread on whether the default sum value for `object` dtype should be `NA` or `""` (empty string). Any thoughts on this?

For object dtype, I'm only seeing support for NA in that thread. Am I missing any contrary opinions? The only hesitation I see is [@Dr-Irv on how this impacts users](https://github.com/pandas-dev/pandas/issues/60229#issuecomment-2529761812), but I do not see an easy deprecation path and I am comfortable with listing this in the breaking changes for 3.0

When I say NA here, I mean "the NA value for the given dtype". For object dtype, the NA value is `np.nan`.
Thanks for approving, @WillAyd. This PR still doesn't address the issue for `DataFrame.sum()` and `Series.sum()`. I'm working on that.
Closing this so someone else can take over. I ran into too many issues trying to support this behavior for object dtype, and not sure if it's worth it.
DOC: Examples added for float_format in to_csv documentation
State: closed

Closes #60363 


Labels:
name: Docs

Comments:
@rhshadrach Please check if the examples are appropriate.
@mroeschke The reviewed changes are corrected, have a check and let me know if further changes are required.
@mroeschke The code is changed accordingly to the reviewed changes, don't know why the tests are failing, can you please help me with it?
Thanks @Uvi-12 
@mroeschke Thank you for the reviews, that helped a lot.
String dtype: coerce missing values in indexers for string dtype Index
State: closed

Closes #59879

Re-taking part of the changes in https://github.com/pandas-dev/pandas/pull/60329 that I had removed from that PR to let that just focus on fixing looking up its "native" missing value for string dtype. This PR then relaxes that restriction to allow looking up other missing values as well.



Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Strings
description: String extension data type and string data

Comments:
lgtm. I don't think the macOS failures are related?
Thanks @jorisvandenbossche
[BUG FIX] Issue #59459 - Incorrect pandas version when using Gitpod environment. Implemented patch.
State: closed

- [x] Closes #59459
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

We’ve implemented a temporary fix that should restore Gitpod environment usability. We’ve tweaked the Gitpod environment code so that it deletes [the buggy file](https://github.com/pandas-dev/pandas/issues/59459#issuecomment-2309555876) (build/cp310/_version_meson.py) at the end of the Gitpod build process. Now, contributors should be able to run pandas immediately upon opening it in Gitpod. We’ve yet to solve the underlying issue, but this patch resolves [Issue #59459](https://github.com/pandas-dev/pandas/issues/59459).



Labels:
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
TST: Test .loc #25548 for matched and unmatched indices of Series
State: closed

- [x] Part of #25548 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] ~~Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.~~
- [x] ~~Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.~~

A test to ensure that .loc assignment of matched and unmatched indices of series yields the right output.

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Thanks @DhruvBShetty
DOC: Added missing links to optional dependencies in getting_started/install.html
State: closed

Closes #60364  


Labels:
name: Docs

Comments:
@rhshadrach I need help with this- the PR is having an issue with ci/circleci: test-linux arm, despite everything is alright. And please check the added link, if it's alright I will proceed on adding the other missing links.
![Screenshot 2024-11-29 at 11 16 16 AM](https://github.com/user-attachments/assets/39fe85c0-2965-4e3e-b76d-cf4cf48d5991)

> the PR is having an issue with ci/circleci: test-linux arm

You can ignore this build.
@rhshadrach All the links for the optional dependencies have been added. If everything looks good, please proceed with merging the PR to make the links go live. Let me know if there’s anything else that needs to be addressed.


Thanks @Uvi-12
Fix indices docstrings, DataFrameGroupBy, SeriesGroupBy, Resampler
State: closed

- [x] Part of issue #60365
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).


Labels:
name: Docs
name: Groupby
name: Resample
description: resample method

Comments:
Thanks @gvmmybear!
ENH: Add sort_columns parameter to combine_first
State: closed

- [x] fixed #60427  
- [x] [Tests added and passed] if fixing a bug or adding a new feature
- [x] All [code checks passed]
- [x] Added [type annotations]to new arguments/methods/functions.
- [x] Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.


This PR enhances the combine_first method in pandas.DataFrame by adding a new parameter, sort_columns, which allows users to control whether the result's column order should be sorted lexicographically or preserve the original order of the calling DataFrame (self).

Currently, combine_first always returns a DataFrame with columns sorted in lexicographical order, which may not be desirable for users who want to maintain the column order of the original DataFrame

```

df1 = pd.DataFrame({"B": [1, 2], "A": [3, 4]})
df2 = pd.DataFrame({"A": [5]}, index=[1])

result = df1.combine_first(df2)
# Current behavior:
#     A  B
# 0  3  1
# 1  4  2

```

With the new sort_columns parameter:

Default Behavior (sort_columns=True): Columns remain sorted as before.
New Behavior (sort_columns=False): Columns retain the order from the original DataFrame (self).


```

df1 = pd.DataFrame({"B": [1, 2], "A": [3, 4]})
df2 = pd.DataFrame({"A": [5]}, index=[1])

result = df1.combine_first(df2, sort_columns=False)
# New behavior:
#     B  A
# 0  1  3
# 1  2  4

```

Tests: Added new test cases in pandas/tests/frame/methods/test_combine_first.py to validate:
Default behavior with sort_columns=True.
Column order preservation with sort_columns=False.

Documentation:
Updated the docstring for combine_first with examples showcasing the new parameter.
Added a changelog entry in doc/source/whatsnew/v3.0.0.rst.


This enhancement maintains backward compatibility, as the default behavior (sort_columns=True) remains unchanged. The new parameter provides additional flexibility for users who need control over column order.






Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
ENH: Support kurtosis (kurt) in DataFrameGroupBy and SeriesGroupBy
State: closed

- [x] closes #40139 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

DataFrameGroupBy and SeriesGroupBy currently support mean, std and skew (the first 3 moments) but not kurtosis (the 4th moment). This change addresses that. I implemented kurtosis in cython in similar fashion to skewness. I've verified that the output of this function matches that of DataFrame.kurt().

Labels:
name: Groupby
name: Reduction Operations
description: sum, mean, min, max, etc.

Comments:
Thanks for the review @rhshadrach. 

- Addressed your comments
- Added test case for skipna=False (by default it's true)
- Added test case for float64[pyarrow] (we already have one for float64)
- Added test case for constant data. Note that the result here is 0.0, consistent with `DataFrame.kurt()` and `Series.kurt()`
Apologies @snitish - this fell off my radar. Should be able to get to it today or tomorrow.
Thank you @rhshadrach!
Thanks @snitish
upgrade to the latest version of PyData Sphinx Theme
State: closed

upgrade PyData Sphinx Theme from 0.14 to the latest 0.16

it is needed to get the latest font-awesome pack to be able to display the new twitter-x icon.

https://github.com/pandas-dev/pandas/pull/60426

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
I can confirm that updating to the latest version of sphinx-theme fixes the issue with "X" icon.
![image](https://github.com/user-attachments/assets/fad17ef1-bf24-46e1-b6b4-0b1baf39547c)
the website is available as an artifact here: https://github.com/pandas-dev/pandas/actions/runs/12044327291/artifacts/2242724668

Thanks. Could you run pre-commit over your changes?
> Thanks. Could you run pre-commit over your changes?

@mroeschke all checks are passing except ci/circleci: test-linux-arm build which is failing due to an unrelated issue. All PRs have this build failing.
Thanks let's give this a try
everything seems to be working now!
![image](https://github.com/user-attachments/assets/4953c56f-1832-4d30-9628-fd40d2f78a5c)
replace twitter->X
State: closed

sphinx theme prefers using "icon_links" instead of custom keywords

https://github.com/pydata/pydata-sphinx-theme/blob/main/docs/conf.py#L165

Labels:
name: Docs

Comments:
Thanks @partev 
the icon doesn't seem to be displaying correctly:
![image](https://github.com/user-attachments/assets/978b6e5c-ba7e-4401-a556-a15a4e6c585e)
even though it is referring to the correct icon name: https://fontawesome.com/icons/square-x-twitter

@mroeschke the problem is that twitter-x icon was added in fontawesome 6.4.2 https://fontawesome.com/changelog, while pandas website uses 6.1.2

how can we upgrade fontawesome?
DOC: Clarifying pandas.melt method documentation by replacing "massage" by "reshape"
State: closed

Meanwhile, "massage" is correct in a colloquial sense to mean transforming or reshaping data. This is far from accessible for a non-English speaker (as I am).

Using the term `reshape` or `transform` is more meaningful while being accurate.

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks @ohe
BUG: Fix formatting of complex numbers with exponents
State: closed

- [ ] closes #60393 
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/v3.0.0.rst` file if fixing a bug or adding a new feature.

```
>>> input_array=[(1.8816e-09+0j), (3.39676e-09+0j)]
>>> pd.Series(input_array)
0    1.881600e-09+0.000000e+                    00j
1    3.396760e-09+0.000000e+                    00j
dtype: complex128
```

The issue was due to the regex in the _trim_zeros_complex function not being able to handle +/- in the exponent. It assumes the complex number string to be of the form (-)xxx(+/)yyyj. I fixed the regex so it doesn't split on +/- if preceded by an exponent.
Post the fix:

```
>>> input_array=[(1.8816e-09+0j), (3.39676e-09+0j)]
>>> pd.Series(input_array)
0    1.881600e-09+0.000000e+00j
1    3.396760e-09+0.000000e+00j
dtype: complex128
```

Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
@mroeschke Thanks for your feedback on #60405. Raised a new PR with amendments.
Thanks @snitish
TST: Avoid hashing np.timedelta64 without unit
State: closed

Avoids

```bash
____________________ TestValueCounts.test_value_counts_nat _____________________
[gw3] linux -- Python 3.13.0 /opt/hostedtoolcache/Python/3.13.0/x64/bin/python

self = <pandas.tests.test_algos.TestValueCounts object at 0x7f7c3f35dd20>

    def test_value_counts_nat(self):
        td = Series([np.timedelta64(10000), NaT], dtype="timedelta64[ns]")
        dt = to_datetime(["NaT", "2014-01-01"])
    
        for ser in [td, dt]:
            vc = algos.value_counts_internal(ser)
            vc_with_na = algos.value_counts_internal(ser, dropna=False)
            assert len(vc) == 1
            assert len(vc_with_na) == 2
    
        exp_dt = Series({Timestamp("2014-01-01 00:00:00"): 1}, name="count")
        result_dt = algos.value_counts_internal(dt)
        tm.assert_series_equal(result_dt, exp_dt)
    
>       exp_td = Series({np.timedelta64(10000): 1}, name="count")
E       ValueError: Can't hash generic timedelta64

pandas/tests/test_algos.py:1257: ValueError
```

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.3.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 0b6cece3acda1ae6e4f582d8276851b02aeac1ea
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #60416: TST: Avoid hashing np.timedelta64 without unit'
```

4. Push to a named branch:

```
git push YOURFORK 2.3.x:auto-backport-of-pr-60416-on-2.3.x
```

5. Create a PR against branch 2.3.x, I would have named this PR:

> "Backport PR #60416 on branch 2.3.x (TST: Avoid hashing np.timedelta64 without unit)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
DOC: Add type hint for squeeze method
State: closed

- [x] closes #60384 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

This PR adds the missing type hint to the `squeeze` method in order to be consistent with the other methods in `core/generic.py`.


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
CI failures seem unrelated
Thanks @jasonmokk
BUG: fix NameError raised when specifying dtype with string having "[pyarrow]" while PyArrow is not installed
State: closed

I thinks this is more like a bug than enhancement, as quoted from the OP:

> This conforms to the description in [Installation Guide](https://pandas.pydata.org/docs/getting_started/install.html#optional-dependencies): If the optional dependency is not installed, pandas will raise an ImportError when the method requiring that dependency is called.

- [x] closes #57928 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Arrow
description: pyarrow functionality

Comments:
Thanks @yuanx749