BrandonFors/pandas-github-issues huggingface dataset:
BUG: groupby.groups with NA categories fails
State: closed

- [x] closes #61356 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

There is a slight code duplication here, but we don't need to rely on Cateorical's codes because we can just directly use groupby's. We also can't use `groupby` to implement `Index.groupby` because the former only works in the case where the `values` are exhaustive.

Labels:
name: Bug
name: Groupby
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Categorical
description: Categorical Data Type

Comments:
Thanks @rhshadrach
DOC: Added constructor parameters to DateOffset docstring for API consistency #52431
State: closed

 - Added the constructor signature for DateOffset.
 - No functional changes were made, only documentation improvements.
 - Part of the issue #52431 is addressed by this.

Labels:

Comments:
I'm a bit confused here, as the description of the PR doesn't match the changes. I'll be closing this, as the change here doesn't seem very useful. The issue you refer to is about documenting the parameters of the constructors of the mentioned classes, not adding an example of constructing the class. Also, the examples should go into their section and format.
BUG: Raise ValueError for non-string columns in read_json orient='table' (GH19129)
State: closed

- Closes #19129
- Adds validation to ensure all column names are strings when using orient='table' in read_json
- Raises a clear ValueError if invalid column names are found
- Adds a new unit test to pandas/tests/io/json/test_json_table_schema.py to cover the invalid input case
- Ran pytest and pre-commit hooks successfully

Looking forward to feedback. Thanks!

Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: IO JSON
description: read_json, to_json, json_normalize
name: Stale

Comments:
pre-commit.ci autofix
Hi, it seems there is already a PR #60945 addressing this issue.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
As mentioned, it appears we already have https://github.com/pandas-dev/pandas/pull/60945 addressing this issue so closing in favor of that PR since it was opened first
Improve documentation for MonthEnd and YearBegin offsets
State: closed

This pull request improves the documentation for two commonly used offset constructors in Pandas: MonthEnd and YearBegin.

Changes include:

Clarified the purpose and behavior of each offset class

Added runnable examples (doctest-compliant) to demonstrate usage

Improved parameter descriptions where necessary

These changes aim to make the documentation more accessible and clear for both new and experienced users of Pandas’ time series functionality.


Labels:

Comments:
Looks like something went wrong with this PR, it contains lots of unrelated changes. I think better to close this, as I think it'll be much simpler to open a new PR with the documentation changes than fixing this PR. Please, feel free to reopen for the intended work. I recommend using a new branch (not `main`) to work on pull requests, that should help avoid the problems here.
DOC: Removed self-reference to `DataFrame.resample` in the "See also" section.
State: closed

- [ ] ~closes #xxxx (Replace xxxx with the GitHub issue number)~
- [ ] ~[Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature~
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] ~Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.~
- [ ] ~Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.~


Labels:

Comments:
Closing this. I realized that `DataFrame.resample` and `Series.resample` share the same docstring, so listing both in "See also" makes sense.
Test Cython divmod fix for Windows
State: closed

https://github.com/cython/cython/pull/6801 should fix the issues we were seeing in https://github.com/pandas-dev/pandas/pull/61261, but this commit is not apart of the Cython nightly wheels yet.

Labels:
name: Build
description: Library building on various platforms

Comments:
Looks like that Cython PR fixes the Windows build. Closing until we the Cython nighties are updated
DOC: Updated `groupby.expanding` arguments
State: closed

- [ ] ~closes #xxxx (Replace xxxx with the GitHub issue number)~
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] ~Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.~
- [ ] ~Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.~


Labels:
name: Window
description: rolling, ewma, expanding

Comments:
Thanks @arthurlw
Add warning to `.groupby` when null keys would be dropped due to default `dropna`
State: closed

- [X] closes #61339
- [X] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [X] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

TODO:

- [X] Check performance for `codes` check approaches (`codes.min()` was about 3x faster)
- [ ] Run full test suite to ensure nothing broke
- [ ] Add tests/implementation for `.pivot_table`/`.stack`/etc. (possibly in a follow-up PR?)

Labels:
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
TST: Testing for mixed int/str Index
State: closed

- [x] closes #54072 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Index
description: Related to the Index class or subclasses

Comments:
@xaris96 Can you fix the broken tests and remove all your debug files from this PR please?
@datapythonista okay. no problem. i would like to ask about the pr. Do we need to fix the failed tests that already existed to pass the pr? we are new to this and need some guidance :)
I think the main idea of the issue is to add the mixed index to the tests, see what it fails, fix the bugs of things that don't work with an index with mix types, and if there are tests that need to be updated, also fix that. Since I think several things will be broken, maybe you can open a first PR with just adding the index with mixed types, then write the failures in a comment to the issue, and then address them in individual PRs. But up to you, whatever you consider it makes things easier. @jbrockmendel anything you'd like to add?
@datapythonista  thanks for the advise. it is really helpful!
@datapythonista also one more question. is there any problem or do we create a mess if we making changes and update this pr? or this is the common tactic?
> @datapythonista also one more question. is there any problem or do we create a mess if we making changes and update this pr? or this is the common tactic?

A PR is mainly a UI for your fork's branch. The usual way of working is to just keep updating your fork's branch, until reviewers are happy and it's merged into the project's main branch.
> anything you'd like to add?

seems like a reasonable approach.
@mroeschke okay 
@datapythonista Hi, I’m encountering an issue with test_rolling_var_numerical_issues, even though I haven't changed anything related to it. It was passing before, but now it's failing. Could you help me understand what might be causing this?
@mroeschke Hi.. i also commented above that I’m encountering an issue with test_rolling_var_numerical_issues, even though I haven't changed anything related to it. It was passing before, but now it's failing. Could you help me understand what might be causing this?
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
Mixed int string 
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Feels like this was superseded by #61349. Please let me know if I missed something and there is a reason to reopen this PR.
@datapythonista yeah you are right. sorry for the mess its our first time!
Update groupby().first() documentation to clarify behavior with missing data (#27578)
State: closed

This PR enhances the docstring for `GroupBy.first()` to clarify:
- It returns the first *non-null* value per column
- It differs from `.nth(0)` and `.head(1)` in how it treats missing values
- Includes comparative examples for better understanding

Fixes part of issue #27578

Ready for review.

Labels:
name: Docs
name: Groupby

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
Fix #61072: inconsistent fullmatch results with regex alternation
State: closed

in PyArrow strings
Fixes an issue where regex patterns with alternation (|) produce different results between str dtype and string[pyarrow] dtype. When using patterns like "(as)|(as)", PyArrow implementation would incorrectly match "asdf" while Python's implementation correctly rejects it. The fix adds special handling to ensure alternation patterns are properly parenthesized when using PyArrow-backed strings

- [ ] closes #61072  

Labels:
name: Bug
name: Strings
description: String extension data type and string data

Comments:
> I think this ends up using pyarrow's `match_substring_regex` function here; therefore, this would be a limitation of the pyarrow implementaiton and would need fixing there instead of pandas

Thanks for the feedback! I agree the issue stems from PyArrow’s match_substring_regex not enforcing full-string matching. To ensure consistent Series.str.fullmatch behavior, I propose a pandas workaround: wrap patterns in ^...$ for PyArrow arrays (e.g., pat = f"^{pat}$") while keeping the | grouping logic. I’ll:

Implement the workaround.
Add tests for foo|bar, empty strings, and re.IGNORECASE.
Document the PyArrow limitation in the docstring.
File a PyArrow bug report and link it here.
I’ll update the PR shortly. Please let me know if this approach works or if you suggest an alternative
Thanks but I prefer not to add temporary workarounds in pandas until it's properly implemented in PyArrow. If interested in still working on this, I would suggest working on this on the Arrow repository so closing as an upstream issue
DOC Update link to "The Grammar of Graphics" book
State: closed

Update link to "The Grammar of Graphics" book.
https://doi.org/10.1007/0-387-28695-0

Original link does not work:
https://www.cs.uic.edu/~wilkinson/TheGrammarOfGraphics/GOG.html

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks @star1327p
BUG: Fixed issue with bar plots not stacking correctly when 'stacked' and 'subplots' are used together
State: closed

- [x] closes #61018
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Added check for when stacked and subplots are used in conjunction for bar plots. And logic dictating offsets for individual subplots to account for plots with non-concurrent columns being graphed. 

Currently, does not take into account column order in subplot entry (eg: (A, B) vs (B, A)) when stacking

Labels:
name: Visualization
description: plotting

Comments:
Thanks @eicchen02
ENH/TST: unset_index method #60869
State: closed

- [x] closes #60869 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Thanks for the PR, but the linked issue has not been triaged yet and therefore a PR would be too early. Closing this PR, but I would suggest tackling issue that do not have the `needs triage` or `needs discussion` labels.
DOC: Updated `groupby.ewm` arguments
State: closed

- [ ] ~closes #xxxx (Replace xxxx with the GitHub issue number)~
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] ~Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.~
- [ ] ~Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.~




Labels:
name: Groupby
name: Window
description: rolling, ewma, expanding

Comments:
Thanks @arthurlw
CI: Have dedicated Python 3.13 job instead of using Python dev
State: closed

We've been testing Python 3.13 using the `Python dev` job. Since Python 3.13 has been available since last October, we should be able to test this version with a dedicated job with all of our optional dependencies. 

Additionally, this new job caught an unclosed `sqlite3` engine, so modified some `test_sql.py` tests (and removed unnecessary parametrizations)

Labels:
name: CI
description: Continuous Integration
name: Python 3.13

Comments:
Going to merge since green. Happy to follow up if needed
DEPS: Clean unused dependencies
State: closed

* When installing pytables, it appears `blosc2` is generally a dependency now https://github.com/PyTables/PyTables/blob/9afcc380d93192460e7badd1baf568592cadad26/pyproject.toml#L78
* I don't think we use `google-auth` or `gitdb` anywhere now

Labels:
name: Build
description: Library building on various platforms
name: Dependencies
description: Required and optional dependencies

Comments:
Looks good, I guess xarray was pinned to not allow the latest versions because of the test failures in the CI.

@shoyer @FrancescAlted if you have any comment here...
Since this PR is green, going to merge. Happy to follow up if needed
add test case for mixed string and int
State: closed

- [ ] closes #54072 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
Fix: AttributeError when using .iloc with pyarrow-backed Series in Pandas #61311
State: closed

fix : #61311
### Solution:
The problem arises when the code tries to access `.max()` and `.min()` on `ArrowExtensionArray`. To fix this, we can replace `.max()` and `.min()` with `np.max()` and `np.min()`, which can handle these arrays correctly.

Here is the modification to be made:

```python
# check that the key does not exceed the maximum size of the index
if np.max(arr) >= len_axis or np.min(arr) < -len_axis:
    raise IndexError("positional indexers are out-of-bounds")
```

---

Labels:

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
PERF: Restore old performances with .isin() on columns typed as np.ui…
State: closed

…nt64

- [ ] closes #60098 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Only if dtypes are equal (e.g uint64 vs uint64, uint32 vs uint32...)

%timeit data["uints"].isin([np.uint64(1), np.uint64(2)]) # 17ms (!)
The last line, with older numpy==1.26.4 (last version <2.0), is even worse: ~200ms.

Labels:
name: Performance
description: Memory or execution speed performance
name: Regression
description: Functionality that used to work in a prior pandas version
name: isin
description: isin method

Comments:
pre-commit.ci autofix
Implicit conversion to float64 happens only whith uint64/int64.
I reverted the PR #46693 to provide an example based on initial issue #46485:

```
import pandas as pd
import numpy as np
test_df = pd.DataFrame([{'a': 1378774140726870442}], dtype=np.uint64)

print(1378774140726870442 == 1378774140726870528) 
#False

print(test_df['a'].isin([1378774140726870528])[0])
#True

print(test_df['a'].isin([1])[0])
#False
```

The second test must be False and was handled by the PR #46693
because there is implicit conversion to float64.
But if we change it to:

```
print(test_df['a'].isin([np.uint64(1378774140726870528)])[0])
#False
```

The result is correct because in this case there is no implicit conversion so it's not necessary to use object.
Regarding the performance, it's resolves partially the issue #60098:

Before:

```
import pandas as pd, numpy as np
data = pd.DataFrame({
    "uints": np.random.randint(10000, size=300000, dtype=np.uint64),
    "ints": np.random.randint(10000, size=300000, dtype=np.int64),
})

%timeit data["uints"].isin([np.uint64(1), np.uint64(2)]) # 239ms
```

After:
```
import pandas as pd, numpy as np
data = pd.DataFrame({
    "uints": np.random.randint(10000, size=300000, dtype=np.uint64),
    "ints": np.random.randint(10000, size=300000, dtype=np.int64),
})

%timeit data["uints"].isin([np.uint64(1), np.uint64(2)]) # 4ms
```
Thanks @pbrochart
DOC: Add missing punctuation to merging.rst
State: closed

Add missing punctuation to ``merging.rst``.

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
/preview
Website preview of this PR available at: https://pandas.pydata.org/preview/pandas-dev/pandas/61315/
Thanks @star1327p