BrandonFors/pandas-github-issues huggingface dataset:
Fix/devcontainer qt deps
State: closed

- [x] closes #61037 
- [x] [Tests not added and not passed]Test: Manually ran pytest inside the devcontainer to confirm PyQt5 and pytest-qt are functional.
- Adding platform: linux/amd64 to the docker-compose.yml dev service to work around image compatibility issues on Apple Silicon.
- Switching from direct Dockerfile builds to docker-compose.yml via updates in .devcontainer/devcontainer.json:
Removed "dockerFile" setting
Added "service", "workspaceFolder", and "dockerComposeFile"
- Installing qt5-qmake and qtbase5-dev via apt-get to support pytest-qt, which is required for the test suite. However this is commented in order to avoid redundant tools installed on not arm/Arch64 plattform just like the original file. The user should uncomment it.

These changes resolve build failures seen on Apple Silicon when using the VS Code Remote - Containers extension.

Why this matters:

Apple Silicon machines often encounter architecture compatibility issues when building development containers, especially when Python packages need to compile C/C++ or Qt-based code. These changes ensure a smooth devcontainer build experience on both ARM64 and x86_64 environments.

## Introduction

If you use mac Silicon, you should uncomment   "# platform: linux/amd64" of docker-compose.yml file and     #-y qt5-qmake qtbase5-dev\ of Dockerfile. If you use another plattform, just build dev container or docker container as usual, nothing was changed.


Labels:
name: Build
description: Library building on various platforms

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
CLN: Fix code formatting to address pre-commit and build failures
State: closed

Run ruff --fix and ruff format to fix style issues flagged by pre-commit.
This resolves common RUF003 errors (e.g., ambiguous hyphens) seen across multiple PRs.

Labels:

Comments:
I think as per the documentation I should rename my commit to CLN: Code cleanup.
Not sure where you saw pre-commit flagging issues, but the CI is green for main, I think all is good (based on our validations).

Maybe you run things locally with different versions of the validation tools?
> Not sure where you saw pre-commit flagging issues, but the CI is green for main, I think all is good (based on our validations).
> 
> Maybe you run things locally with different versions of the validation tools?

Yeah makes sense — but even on a clean main, pre-commit fails for me unless I run Ruff. Even adding just a comment triggers it, so I don’t think it’s a version mismatch. Might be that pre-commit is catching things CI isn’t?

Also seeing the docstring validation failing — happy to fix it if needed, or we can just drop the PR if it’s not worth changing right now. Let me know what you prefer!


We do check pre-commit in the CI, with main and a clean install of the pre-commit tools if I'm not wrong. And it's green for main, so it seems like you've got something different than our CI locally.

I'll run pre-commit to this PR to see how it looks like.
pre-commit.ci autofix
Ahh got it — makes sense now. I’ll revert those changes to the test docstrings. Thanks for pointing that out!
The problem was that after making the code change, I was manually running ruff format and ruff check --fix, which applied to the entire directory. After running pre-commit, I realized that everything outside the pandas/ directory is skipped, along with pandas/tests/, when using pre-commit run --all-files.
DEP: update python-calamine to 0.3.2
State: closed

- [x] closes https://github.com/pandas-dev/pandas/issues/61186
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This PR updates the calamine engine dependency to version `0.3.2`, which includes the upstream fix for leading-zero truncation in VLOOKUP-derived Excel cells.


Labels:
name: Dependencies
description: Required and optional dependencies

Comments:
It'd be good to either link an existing issue, or to write a description, so we can understand why we should merge this.
@datapythonista Thanks for the reminder! I’ve now added the issue link for clarity. Since this is my first PR involving a package version bump, I wanted to ask:
1. Does the pandas community support this upgrade to calamine >= 0.3.2? If it’s not aligned with project goals, I’m happy to close the PR.
2. If the upgrade is approved, would you like me to add a unit test covering the original issue scenario?

Thanks! 
Thanks for the context. This looks reasonable. In general we support dependencies that are two years old. Given there is a bug, I guess it can make sense to require a version that is newer than that. @mroeschke @rhshadrach do you have an opinion on whether it's more important to be compatible with older versions one of the libraries to read Excel, or to prevent a bug it has?
> Given there is a bug, I guess it can make sense to require a version that is newer than [2 years].
> ...
> do you have an opinion on whether it's more important to be compatible with older versions one of the libraries to read Excel, or to prevent a bug it has?

Every optional dependency has bugfixes on just about every release, I think a bugfix would only justify bumping the minimum version if it is of an serious nature. Security issues as well. I do not think this is the case here.
Let's close this then. I assume many users create a fresh environment and they'll get a new version anyway. For the people who has environments with the old dependency, hopefully they can find information that leads them to upgrading the buggy version.
WEB: Replace os.path with pathlib.Path in pandas_web.py
State: closed

Replaces `os.path` with `pathlib.Path` in `pandas_web.py`, as suggested by @datapythonista in [61578](https://github.com/pandas-dev/pandas/pull/61578). No functional changes, verified site generation remains correct

- [x] Ran pre-commit check

Labels:
name: Web
description: pandas website

Comments:
Thanks for the review! updated to make sure all paths are created as `Path` early and used consistently. Removed redundant casts too.
Addressed all the latest comments, switched to `Path.rglob`, cleaned up `base_url`, and ensured consistent `Path` usage. Let me know if anything else looks off, I also retested on local after the changes
/preview
Website preview of this PR available at: https://pandas.pydata.org/preview/pandas-dev/pandas/61604/
Thanks for the clean up here @iabhi4 , very nice PR
REF: Replace os.path with pathlib.Path in pandas_web.py
State: closed

Replaces `os.path `with `pathlib.Path` in `pandas_web.py`, as suggested by @datapythonista in [61578](https://github.com/pandas-dev/pandas/pull/61578). No functional changes, verified site generation remains correct
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).


Labels:

Comments:
I'm resolving the conflicts, it's my previous PR which just got merged
WEB: remove "String data type" from "Roadmap points pending a PDEP" section.
State: closed

> pandas is in the process of moving roadmap points to PDEPs (implemented in August 2022). During the transition, some roadmap points will exist as PDEPs, while others will exist as sections below.

This one is now covered by PDEP-14 which has been accepted and therefore no longer pending a PDEP.

Labels:
name: Web
description: pandas website

Comments:
I think that point was obsolete for a long time before PDEP-14. Thanks for taking care of this @simonjayhawkins
DOC/ENH: Holiday exclusion argument
State: closed

- [x] closes #54382 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Frequency
description: DateOffsets

Comments:
I'm trying to figure out what the Pyodide test failing means, but struggling a bit. @mroeschke any insight in where I should look to resolve the failed check? 

Also trying to just rerun the checks, but couldn't figure out how to do that without making an empty commit 
> I'm trying to figure out what the Pyodide test failing means

I think that's unrelated to your PR, so no need to worry about it
Thanks for all your help on the PR so far! I really appreciate it. Are there any other changes I should make?
Thanks @sharkipelago
ENH: improve optional import error message
State: closed

- [x] closes #61521 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
I'm not sure how to fix the CI errors: 
```
Error: Failed to CreateArtifact: Received non-retryable error: Failed request: (409) Conflict: an artifact with this name already exists on the workflow run
```
One change is missing: https://github.com/pandas-dev/pandas/blob/675c81c9dc82ff9be797c94f3fc8dddb5dc44894/pandas/tests/io/test_parquet.py#L337
I expect the CI will pass once that’s fixed.
thanks @yuanx749, confirmed on the previous CI errors that the fastparquet was the issue
@rhshadrach - can you kindly review this? thanks!
Thanks @KevsterAmp
Add table of contents support to ecosystem page
State: closed

This PR adds a table of contents to the Ecosystem page using the Markdown `TocExtension`.

- Inserted `[TOC]` placeholder in `ecosystem.md`
- Enabled `toc` extension in `web.py`
- Configured `toc_depth` for h2 and h3 headers
- Closes #61587


Labels:
name: Web
description: pandas website

Comments:
@akafle01 do you have time to continue the work on this?
> @akafle01 do you have time to continue the work on this?

Yes I will work on it today
/preview
Website preview of this PR available at: https://pandas.pydata.org/preview/pandas-dev/pandas/61595/
Thanks @akafle01 for the contribution. I was checking this, as the table of contents looked different than in the other pages, and realized besides adding the ToC we needed to do some refactoring. I better add the ToC to Ecosystem in #61655 where I've implemented the refactor, so we can test both things together.

I'll close this PR, sorry about it, as the work you've been doing is great, just more practical to add this in the other PR.
WEB: Restore website width and improve table of contents style
State: closed

Follow up of #58791

This restore the website width (happy to make the content of PDEPs narrower, but it's a bit trickier so I'll leave it to another PR).

I also styled the table of contents more similar to the code blocks, which personally I think it looks a bit nicer.

Before:
![Screenshot at 2025-06-06 23-23-22](https://github.com/user-attachments/assets/7a025b7c-c5b6-4e6a-b921-627f4e3439cf)

After:
![Screenshot at 2025-06-06 23-22-37](https://github.com/user-attachments/assets/178642b6-8376-4992-bef0-224fddb2c97d)

CC @rhshadrach 

Labels:
name: Web
description: pandas website

Comments:
Thanks @datapythonista
WEB: Moving maintainers to inactive status
State: closed

A sad PR, but after checking with some maintainers, they confirmed that they became inactive and wish to be changed status. I think 3 more people will also be moved unfortunately, but still waiting for confirmation from them.


Labels:
name: Admin
description: Administrative tasks related to the pandas project
name: Web
description: pandas website

Comments:
/preview
Website preview of this PR available at: https://pandas.pydata.org/preview/pandas-dev/pandas/61591/
Avoid re-enabling the GIL at runtime
State: closed

The GIL gets dynamically reenabled because the shared utility Cython build does not include the `freethreading_compatible` directive.

- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [X] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [X] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Build
description: Library building on various platforms
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
Thanks @lysnikolaou
BUG: Require sample weights to sum to less than 1 when replace = True
State: closed

- [x] closes #61516
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
 1. I've added this as an exception, but I've also considered doing this as a warning, since the issue is

    - easy to encounter - one case occured in user_guide
    - hard to explain within a short error message - requires specific equation and some level of understanding of probability
    - has minimal effect on majority of use cases - as long as the variance isn't huge, the bias this error introduces is minimal, and for very specific cases e.g. the village sampling from the op warning should suffice

 2. I also deviated a bit from original issue in that i'm allowing `max(w) * n / sum (w) == 1`. As far as I understand, it doesn't introduce any bias, and it allows some specific uses, e.g. `df.sample(n=1,weights=[1,0])`

 3. If this is implemented as an exception, I believe that there is no way for the numpy exception `"Fewer non-zero entries in p than size"` to occur, as:

    - `w` - list of weights
    - `n` - number of non-zero weights in `w`
    - `size` - size of the sample to be drawn
    - `len` - length of `w`

    I'm assuming that

    - the weights are normalized, so `sum(w) = 1`, as it doesn't change anything and makes the math easier
    - the weights are equal, since if any of them is larger than the others, `max(w)` and `max(w) * size` will increase
    - therefore `w = [1/n, 1/n, ..., 1/n]` or any combination like `w = [1/n, 0, 1/n, 0, ..., 1/n]` where `n` is the number of non-zero weights
    - for the `"Fewer non-zero entries in p than size"` to occur, `size > n`
    - for the `"Invalid weights: If replace=False, "` to occur, `size * max(w) / sum(w) >= 1`
    - `size * max(w) / sum(w) >= 1`
    - `size * max(w) >= 1`, as `sum(w) = 1`
    - `size * 1/n >= 1`, as `max(w) = 1/n`
    - `size >= n`

Thanks @microslaw!
DOC: Fix typos
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
pre-commit.ci autofix
DOC: Validate versions.json before building docs #61573
State: closed

Adds a JSON validity check for `versions.json` directly inside `pandas_web` during context generation. This ensures malformed JSON (e.g., trailing commas) is caught early, preventing issues like the broken version dropdown in #61572

- [x] Closes #61573

Labels:
name: CI
description: Continuous Integration
name: Web
description: pandas website

Comments:
> Can you move this check to main, and just open the file and load it as json, with a comment about what's for.

Thanks for the review @datapythonista, just to confirm: when you say "move this check to main", do you mean the `main()` function itself, or the` __main__` block. My assumption is the `__main__` block, so we catch the error before any file system operations, just want to double check!

Added the trailing comma to `versions.json` to test the failure. Leaving the `sys.exit(1)` in place for now. Once we see the CI result, happy to adjust that if we want to just warn instead of fail

Better inside the `main` function. The idea is that the `if __name__ == "__main__":` block is only executed when the script is called. If for whatever reason I want to import the script from another file to generate the website, that block won't execute, but I still want to validate the file.

We don't want to warn. What I'm saying is that when you do `json.load()` this will already raise an exception, show an error message and crash the script. So, catching the exception to show an error and crash the script is probably not needed, since it'll already happen, and the code is significantly simpler, just two lines.

If the default error message is not great, then I'm happy with your approach. But instead of `sys.exit` I think it's better to raise a new exception from the previous one (Python can chain exceptions).

Btw, when this is done, if you are interested, it'd be nice to open a new PR to use `pathlib` instead of `os.path` in the whole script. For this PR is better to use `os.path` for consistency.
Thanks for such a detailed clarification. Have pushed the updated changes.

> Btw, when this is done, if you are interested, it'd be nice to open a new PR to use `pathlib` instead of `os.path` in the whole script

happy to follow up with a separate PR for the `pathlib` changes once this one is in


Thanks for the update. I think catching the exception as you did was indeed better, the error message here is not helpful enough. But personally, I would just raise a RuntimeError with the good error message from the original one.

Feel free to remove the comma, after those changes I think we can get this merged. Thanks!
Raising `RuntimeError` now as suggested. Thank you so much for your help in this
TST: Remove match= in test_setitem_invalid to avoid PytestWarning
State: closed

test only change - This PR removes the use of `match=""` in `test_setitem_invalid` within `base/setitem.py`. 

Using an empty string as a match pattern triggers a `PytestWarning` in newer versions of pytest, so the `match` argument has been removed since the message was not being validated

- [x] closes #61557 
- [x] Ran pre-commit check

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
> Personally I'd just remove the match parameter (unless we have a check to make sure it exists).
> 
> But either way this looks good, thanks for taking care of it @iabhi4.

I tend to agree with @datapythonista, unless there is a specific check that we are looking for and in this case it does not appear to be, I would leave out the `match` parameter.

```
with pytest.raises((ValueError, TypeError)):
```
@mroeschke I think the changes you requested have been implemented, this should be ready to merge
Thanks @iabhi4
DataFrames Class update
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Thanks @SaraTammame for the contribution, but this seems to be some personal tests, so I guess we can close this.

If there is a contribution you want to make, please try to link an issue or provide a description, so we have more context when reviewing.
BUG: Fix JSON typo that breaks javascript in the docs
State: closed

Closes #61571 

As opposed to Python, JSON doesn't accept commas after the last element of a dict, and it's strict about it. We added this as a typo (I think I did the same during a release, I'll create an issue to validate this JSON in the CI) when updating the JSON that provides the versions for the documentation drop down. This seems to be breaking all the javascript in our docs.

@mroeschke I would merge this before waiting for the CI, but up to you.

Labels:
name: Bug
name: Docs

Comments:
Is this failure related? https://github.com/pandas-dev/pandas/actions/runs/15474985199/job/43567905639?pr=61572

```bash
Writing evaluated template result to /home/runner/work/pandas/pandas/doc/build/html/_static/nbsphinx-code-cells.css

Extension error (pydata_sphinx_theme):
Handler <function update_config at 0x7f4fee50fbe0> for event 'builder-inited' threw an exception (exception: Expecting property name enclosed in double quotes: line 17 column 5 (char 392))
```
I'm pretty sure it's not, but no idea where this exception is coming from, I'm having a look now and see if I can understand what's going on. I'll rerun the job
Ok, I think it's kind of related, but I think this is failing because it's fetching the `versions.json` published online, which has the typo. I checked this is also broken in `main` (locally), so I'm going to merge, and hopefully we don't see this CI error anymore.
The CI after merging also failed, since building the website fetches the file in production (kind of nonsense, but that's how the sphinx theme works), so we were in a loop where we can't fix the website because production is broken, and we can't update production because the website is broken. I updated the file in the server directly, and now everything should be fine. The dropdown works for me already, but the file with the typo may be cached for others I guess.