BrandonFors/pandas-github-issues huggingface dataset:
DataFrame.eval error fixed
State: closed

- [#59062 ] closes #59062  (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
Difference between calamine and openpyxl readers fixed
State: closed

- [ #59186 ] closes #59186  (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO Excel
description: read_excel, to_excel
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Closing as stale. @Riyazul555 - if you'd like to continue, address the comments above and we'd be happy to reopen!
DOC: Fix for excelwriter engine for ods files
State: closed

From https://github.com/pandas-dev/pandas/pull/58831#discussion_r1642841812

Changing odswriter references to odfpy

```py
from pandas.io.excel._util import get_writer
get_writer('odswriter') # fails with ValueError: No Excel writer 'odswriter'
get_writer('odf') # works returning pandas.io.excel._odswriter.ODSWriter
```

That said, the library doesn't seem to be maintained. Maybe we should consider deprecating.

Labels:
name: Docs
name: IO Excel
description: read_excel, to_excel

Comments:
Thanks @asishm 

> That said, the library doesn't seem to be maintained. Maybe we should consider deprecating.

I would be +1 for that
BUG: Fix .dt.microsecond accessor for pyarrow-backed Series
State: closed

- [x] closes #59154(Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Datetime
description: Datetime data dtype
name: Arrow
description: pyarrow functionality

Comments:
Thanks @Aloqeely
update algo.take to solve #59177
State: closed

- [x] closes #59177
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
need to work on 

FAILED pandas/tests/dtypes/test_generic.py::TestABCClasses::test_abc_hierarchy[ABCNumpyExtensionArray-inst6-ABCExtensionArray-subs2]
There are a few others `if isinstance(xxx, (ABCIndex, ABCSeries, ExtensionArray))` like tests in pandas code (algorithms.py, common.py, categorial.py, blocks.py) and I wonder if I should find them all and append `ABCNumpyExtensionArray` ?
Thanks @mutricyl
DEPR: group by one element list gets scalar keys
State: closed

- [x] closes #58858

changed the behavior: group `keys` to be `1-tuples` instead of `scalar` when grouping with a `length-1 list`

Labels:
name: Groupby
name: Deprecate
description: Functionality to remove in pandas
name: API - Consistency
description: Internal Consistency of API/Behavior

Comments:
I deprecated the behavior for `groups`: group `keys` are now 1-tuples instead of scalars when grouping with a length-1 list.
I fixed the tests except for this one `test_groupby_raises_category_on_category` in `pandas/tests/groupby/test_raises.py`. I got errors like:
```
FAILED pandas/tests/groupby/test_raises.py::test_groupby_raises_category_on_category[by3-False-idxmin-False-agg] - ValueError: Can't get idxmin of an empty group due to unobserved categories. Specify observed=True in groupby instead.
```
@rhshadrach, could you please advise me on how to fix this test?
> @rhshadrach, could you please advise me on how to fix this test?

Sorry for the delay! I believe you've resolved this, let me know if not.
> Sorry for the delay! I believe you've resolved this, let me know if not.

Thanks @rhshadrach for reviewing this PR. I moved the FutureWarning from Grouping to GroupBy.groups as you suggested and added a note to v3.0.0. Could you please take a look at my changes?
Thanks @natmokval!
TST: Added script to enforce usage of match argument for tm.assert_produces_warning 
State: closed

- [x] closes #58290 
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.

Added a script called `enforce_match_arg_in_assert_produces_warning.py` which checks that all usages of tm.assert_produces_warning make use of the `match` argument. This is done to ensure that users are always displayed the correct warning message. Uses of `tm.assert_produces_warning(None)` have been excluded as no warning is produced in this case. 

Furthermore, to ensure that the existing code doesn't fail the `pre-commit` stage, all files that currently do not use the match argument have been excluded. 

Once the PR is merged in, we can open up another issue to shorten the list of "excluded" files. 

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: CI
description: Continuous Integration
name: Stale

Comments:
@Aloqeely Just fixed the issue with `Sequence` import. It works now with `TYPE_CHECKING`. Is there anything else left to be addressed? If not, is it good to merge?
LGTM. cc @mroeschke 
@mroeschke - are we going ahead with this PR? Any reason we’d like to hold it off for now? cc @Aloqeely 
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the contribution, and sorry this was not reviewed before. Maybe silly question, but since `assert_produces_warning` is our own function, why would we want to have a script to check if it's being called with the `match` parameter, when we can simply make the parameter required?

I'm ok with adding `match` (not sure if there are cases where it doesn't make sense). But this seems an overkill.

Maybe worth start by fixing some of the cases in somehow small PRs, and if that looks good, then just raise if `match` is not called (assuming there is agreement).

What do you think?
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
@chaarvii Thanks for the PR. 

Closing to clear queue. we can reopen if you're still interested in working on this.
REF: Move methods in core/reshape/util.py to where they are used
State: closed

2 unit tests were removed since `cartesian_product` now strictly operates on `list[np.ndarray]` (i.e. `MultiIndex.codes`) and the unit tests were testing `Index` inputs

Labels:
name: Refactor
description: Internal refactoring of code
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Going to merge. Happy to follow up if needed
TST: Address UserWarning in matplotlib test
State: closed

e.g. https://github.com/pandas-dev/pandas/actions/runs/9759909211/job/26937595679?pr=59136

closes #58851

Labels:
name: Visualization
description: plotting
name: Warnings
description: Warnings that appear or should be added to pandas

Comments:
I think this issue is related:  https://github.com/pandas-dev/pandas/issues/58851
Thanks @Dr-Irv! Yeah It appears something changed on the matplotlib side to make "automatic" legend labeling now raise a UserWarning 
Going to merge to get CI to green
header, update in basic.rst M #59130
State: closed

closes #59130
![Screenshot 2024-07-01 220007](https://github.com/pandas-dev/pandas/assets/170221536/3f95dbd5-2ac2-4cf8-9a51-80ba62d536f6)
By changing the header to "With speedup (ms)" and "Without speedup (ms)", the documentation will directly highlight the performance improvements. This makes it easier for users to understand the value of these libraries without needing to know about older versions.

.. csv-table::
    :header: "Operation", "With speedup (ms)", "Without speedup (ms)", "Ratio"
    :widths: 25, 25, 25, 25

    ``df1 > df2``, 13.32, 125.35,  0.1063
    ``df1 * df2``, 21.71,  36.63,  0.5928
    ``df1 + df2``, 22.04,  36.50,  0.6039


Labels:

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
Avoid empty lines with spaces to be transformed to empty string
State: closed

- [x] closes #59147  (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Thanks for the PR! I'm not sure if this fixes the problem in the linked issue. Can you write a test that asserts the result of `read_html` on `'<table><tr><td> </td></tr></table>'` is not an empty list?
@Aloqeely addressed your comments and removed the new named argument
Are there any implications of passing `skip_blank_lines=False` as the default now? I'm sure that would break some existing code.
To be quite frank I'm not very familiar with the `read_html` code, ping @mroeschke 
> Are there any implications of passing `skip_blank_lines=False` as the default now? I'm sure that would break some existing code. To be quite frank I'm not very familiar with the `read_html` code, ping @mroeschke

The difference now would be that, every line with only spaces would be included as a new row in the DataFrame
I am not too familiar with the `read_html` code either, but given the issue I lean toward @Aloqeely opinion that this PR might cause more of a behavior change as opposed to a "bug fix". And reading the original issue it seems like the docs were clear enough given this edge case.

I would lean toward more updating this docs as opposed to changing the code.
I understand, accordingly I will close this pr, so do you suggest @Aloqeely and @mroeschke to update the docs to say that data elements with only spaces will not be interpreted as rows and all spaces in the html elements would be stripped as a sidenote?
ENH: Add a Series method which checks whether a Series is constant
State: closed

- [x] closes #58806 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Thanks for the PR, as but the issue is still marked `Needs Discussion` and therefore this feature needs more discussion before proceeding with a PR. Closing but we can reopen when there is more agreement on moving forward with this feature in the issue
Add "N / A" to `STR_NA_VALUES`
State: closed

This is a value that I just came across in some data that is in the spirit of the other NA value. I don't think this will be a false positive any more so than some of the other values in this list.

- [x] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature. A test here would just be duplicating the implementation.
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature. I don't think this deserves an entry?

Labels:

Comments:
PS, I think it is usually bad taste for a test to exactly copy the contents of the implementation like this. Can I just make it so we reference STR_NA_VALUES directly in the test, and remove `_NA_VALUES` from that test?
Thanks for the PR but its not really a feasible goal for us to track down every NA value a user could come across, and dealing with whitespace like this is not something I see of being value added to pandas. I think you may just need to handle this in your own process
That is definitely true, I already have a selection of additional values I check for. This is just the most egregious one that I thought would.be applicable.for everyone.

Even if we don't get to perfection, I think this PR still improves the situation and is a good contribution. Are you worried that slowly this list will balloon to 100 different examples? Trying to figure out if there is some mitigation o can do to address your concern. Thank you, I appreciate your time and effort!
Thanks for the PR, but agreed that it's probably not worth expanding these default at this point so closing
BUG: Fix difference functionality for Period Indexes
State: closed

- [x] closes #58971
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Labels:

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
DataFrame.eval error fixed
State: closed

- [x] closes #59062  (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
I have moved the test in the respective file
Do check it out @chaoyihu and let me know about terms[i].value .....what you want me o do further?
Thanks
I am closing this PR
Supersede by #59189
remove ops div class to solve #21374
State: closed

- [x] closes #21374
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] Existing tests updated/extended
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v3.0.0.rst` file if fixing a bug or adding a new feature.

`core.computation.div.ops.Div` class that was introduced to manage unexpected upcast from float32 to float64 looks to be responsible of the imaginary part of complexes being dropped. It seams that numpy and numexpr have evolved since #12388 and this fix is no more needed.
This PR removes `core.computation.div.ops.Div` class and extends tests to cover for both floats and complex.


Labels:
name: Complex
description: Complex Numbers
name: expressions
description: pd.eval, query

Comments:
Thanks @mutricyl
ENH: Allow adjust=False when times is provided
State: closed

- [ x ] closes [#54328](https://github.com/pandas-dev/pandas/issues/54328) (partially)
- [ x ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ x ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ x  ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ x ] Added an entry in the latest `doc/source/whatsnew/v3.0.0.rst` 

This change enables EWMA to be calculated recursively ("infinite history", i.e. `adjust=False`) for irregular-interval time series. Previously `new_wt` was held constant in the ewm run loop, but now `new_wt` is updated each step through the series under the narrow condition that `adjust=False` and `com=1`.

There may be a more elegant expression that will generalize to `com` values different than one, but using `1 - old_wt` works nicely if `halflife` is the only provided decay parameter since we are guaranteed `com=1`. The additional parameter restrictions to force this condition seem acceptable since there is already [some restriction](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.ewm.html) on decay parameters.

> Exactly one of `com`, `span`, `halflife`, or `alpha` must be provided if `times` is not provided. If `times` is provided, `halflife` and one of `com`, `span` or `alpha` may be provided.

FWIW I confirmed that results match polars and @azmyrajab's nice [test function](https://github.com/pandas-dev/pandas/issues/54328#issuecomment-2072214710):

```
ema_no_adjust = ema_test(vals, dt_seconds, half_life=half_life_seconds)[-1]
ema_no_adjust_pl = ema_polars(df["val"].rename_axis(index="ts").rename("val"), half_life,
                              by="ts").select("val")[-1].item()

print(ema_no_adjust)
print(ema_no_adjust_pl)
print(df.ewm(halflife=half_life, times=id, adjust=False).mean().iloc[-1]['val'])

0.2062994740159002
0.2062994740159002
0.2062994740159002
```



Labels:
name: Window
description: rolling, ewma, expanding

Comments:
pre-commit is finally green but a lot of the actions are failing with the same message: 

`Error: Failed to CreateArtifact: Received non-retryable error: Failed request: (409) Conflict: an artifact with this name already exists on the workflow run`

Seeing lots of PRs get these errors. Maybe a recent GHA setup change is breaking some tests? 
CI all green :)
Thanks @tserrao
REF: Make concat not stateful.
State: closed

All the logic used to be held in a `_Concatenator` class. The statefullness wasn't really needed since `concat` is a one-shot method so broke it out into its functions

Labels:
name: Refactor
description: Internal refactoring of code
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
```bash
| Change   | Before [039edee0] <main>   | After [0becd77e] <ref/concat>   |   Ratio | Benchmark (Parameter)                                                                |
|----------|----------------------------|---------------------------------|---------|--------------------------------------------------------------------------------------|
| -        | 46.8±4μs                   | 41.8±0.1μs                      |    0.89 | join_merge.ConcatIndexDtype.time_concat_series('string[pyarrow]', 'has_na', 0, True) |

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.
```
Since test are passing, going to merge. Happy to follow up if needed
Hey, just a note that this has [broken](https://github.com/geopandas/geopandas/actions/runs/9817406953/job/27108741741) concat override in geopandas, where we're now hitting `'object' object has no attribute 'objs'`. Just wanted to clarify if that is expected or just an unwanted consequence of refactoring (i.e. whether I should fix it on geopandas side).
It was an intended consequence of the refactor given that `concat` isn't stateful anymore. I think I can come up with a fix that will make this non-breaking for geopandas

To confirm it appears that geopandas overrides `__finalize__`? Just curious the reason for that
> To confirm it appears that geopandas overrides __finalize__? Just curious the reason for that

To ensure that we can correctly propagate custom metadata. That used to be an information about projection and an active geometry column, right now it is only active geometry column. Without that, the resulting GeoDataFrame would lose the track of active geometry and require manual re-assignment of active geometry.