BrandonFors/pandas-github-issues huggingface dataset:
Bump meson
State: closed

Meson 1.4.0 adds first class support for NumPy > 2.0 detection https://mesonbuild.com/Release-notes-for-1-4-0.html#new-numpy-custom-dependency

Not an urgent upgrade but figured worth starting

Labels:
name: Build
description: Library building on various platforms
name: Stale

Comments:
I believe the pypy failure is a Cython issue - see https://github.com/cython/cython/issues/6099

Generally I was surprised that the amount of warnings changed with this PR. But I fixed a few / ignored others (mostly for MSVC). Would be nice to get a lot of the MSVC warnings fixed up but I think that will take a dedicated initiative.

This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Closing since there hasn't been much activity here, but feel free to reopen if/when you have time to revist
BUG: Handle Series construction with Dask, dict-like, Series
State: closed

Should hopefully fix `test_construct_dask_float_array_int_dtype_match_ndarray` which is failing because dask `Series` now raises on `__bool__`?

Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: Series
description: Series data structure
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
thx @mroeschke
CI: Remove ASAN job
State: closed


Labels:
name: CI
description: Continuous Integration

Comments:
I think for now can just remove this until we can root cause it
If you want to come back to this at a later time to debug and not completely remove some of the configurations, you could do:


```diff
--- a/.github/workflows/unit-tests.yml
+++ b/.github/workflows/unit-tests.yml
@@ -160,7 +160,7 @@ jobs:
 
     - name: Test (not single_cpu)
       uses: ./.github/actions/run-tests
-      if: ${{ matrix.name != 'Pypy' }}
+      if: ${{ matrix.name != 'Pypy' || matrix.name != 'ASAN / UBSAN'}}
       with:
         preload: ${{ matrix.preload }}
         asan_options: ${{ matrix.asan_options }}
```
Thanks @WillAyd
CI: make docstring checks "instantaneous"
State: closed


- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.

This PR reduces the `check_code.sh docstrings` runtime to under 10 seconds. 

The key changes are in [`validate_pep8_for_examples`](https://github.com/dontgoto/pandas/blob/14e3915f21dc35fcdcc5ce198b35245fb51a68dd/scripts/validate_docstrings.py#L151), which was refactored to call the pep8 CLI once with a list of files to analyse. The functions in the call stack above were changed to handle lists instead of single objects as well. I also added a new test since the `prefix` parameter was not properly tested before.

Let me know if you'd rather see this split into multiple PRs.

I'd also wait until #57879 is merged as there will be quite a few merge conflicts.

Unrelated code checks seem to be failing here and in other PRs.

Labels:

Comments:
@MarcoGorelli @datapythonista 
Sorry @dontgoto, this is now conflicting. Results look very impressive, thanks for the work on this. If you can split it in different PRs that would be great, it'll make reviewing easier.
ENH: add warning when export to tar, zip in append mode
State: closed

- [ ] closes #56778 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO Data
description: IO issues that don't fit into a more specific label
name: Warnings
description: Warnings that appear or should be added to pandas

Comments:
tagging @twoertwein since you commented on the issue
Can you add a test case for this, e.g., calling to_csv with mode="a"?
pre-commit.ci autofix
@twoertwein test added
Thanks @quangngd
DOC: add note to how pandas deduplicate header when read from file
State: closed

- [ ] closes #57792 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

<img width="904" alt="image" src="https://github.com/pandas-dev/pandas/assets/30631476/3d5ee26f-8b6c-4493-bcfd-5491cd68417d">



Labels:
name: Docs
name: IO CSV
description: read_csv, to_csv

Comments:
/preview
Website preview of this PR available at: https://pandas.pydata.org/preview/pandas-dev/pandas/57874/
Thanks @quangngd
DEP: Deprecate CoW and no_silent_downcasting options
State: closed

- [ ] closes #59502 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Deprecate
description: Functionality to remove in pandas
name: Copy / view semantics
name: Stale

Comments:
Looks good, but appears to be valid test failures
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
I think we would want this, but it appears this has gone stale so going to close to clear the queue
DOC: Fix rendering of whatsnew entries
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.
cc @rhshadrach those render in a weird box if you have a space in the beiginning

Labels:
name: Docs

Comments:
/preview
Website preview of this PR available at: https://pandas.pydata.org/preview/pandas-dev/pandas/57871/
We tend to keep those and clean them up when releasing, it’s a good placeholder until we cut the release 
Thanks @phofl
DEPR: Deprecate remaining copy usages
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Deprecate
description: Functionality to remove in pandas
name: Copy / view semantics

Comments:
Thanks @phofl
CLN: enforce deprecation of `NDFrame.interpolate` with `ffill/bfill/pad/backfill` methods
State: closed

xref #53607

enforced deprecation of `NDFrame.interpolate` with `"ffill” / "bfill” / “pad” / “backfill"` methods

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Clean

Comments:
Thanks for sticking with this @natmokval! 
Thanks @mroeschke for helping me with this PR. Enforcing this deprecation was more complicated than I expected.
Fix stubname equal suffix in wide to long function
State: closed

- [x] closes #46939(Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature. (Will add after review)

As `pandas` allows an index in a multiindex and a column to have the same name, I think it's reasonable to allow stubname equal suffix.

However, I think the root cause is in the `melt` function. It's stated in the docstring that

```
    value_name : scalar, default 'value'
        Name to use for the 'value' column, can't be an existing column label.
```

But if `value_name == var_name`, illogical result is returned. Example:

```
df = pd.DataFrame(
    {
        "A": {0: "a", 1: "b", 2: "c"},
        "B": {0: 1, 1: 3, 2: 5},
        "C": {0: 2, 1: 4, 2: 6},
    }
)
pd.melt(
    df,
    id_vars=["A"],
    value_vars=["B"],
    var_name="myVarname",
    value_name="myVarname",
)
```

Result:

![image](https://github.com/pandas-dev/pandas/assets/25203655/e4c01d61-8c44-4e89-a858-c811cc722ed4)

I think it's more reasonable to also raise an error if `value_name == var_name` in `melt`

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Hi @mroeschke, can you take a look at this PR, please? It should be ready for review.
hi @datapythonista, thank you for the review. I also force pushed to the branch because I noticed that you also didn't make any comment on the actual content of the PR. Now this PR should be ready for review.
Thanks for the update here @tqa236. The fix you implemented seems a hack, not the proper fix. You should identify what's causing the exception, which seem kind of unrelated, and fix that. Please let me know if you want to have a look more in detail to the actual problem, otherwise we can close this PR.
thank you for taking a look @datapythonista, unfortunately it's been a year and the context of this issue is no longer fresh in my mind. Let's close this PR.
DOC: Add examples for creating an index accessor
State: closed

Picked up issue #49202. This adds examples for creating index, dataframe, and series accessors.
- [X] closes #49202 
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).


Labels:
name: Docs

Comments:
pre-commit.ci autofix
pre-commit.ci autofix
pre-commit.ci autofix
Can you have a look at the CI errors please?
The checks are still failing, could you please fix them?
Hi @datapythonista I believe we have fixed all the CI errors. Could you please re-review?
/preview
Website preview of this PR available at: https://pandas.pydata.org/preview/pandas-dev/pandas/57867/
Thanks @sjalkote
Fix 28021: Add 5% padding in timeseries line plot
State: closed

- [x] closes #28021 
- [x] All code checks passed
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` (**will do this once this PR is reviewed**)
- [x] Tests passed (`pytest pandas/tests/plotting/test_datetimelike.py`)

Labels:
name: Stale

Comments:
Was hunting through commits and found https://github.com/pandas-dev/pandas/commit/3795272ad8987f5af22c5887fc95bb35c1b6bb69#diff-b742bb6314b1f7b55dd7a0987ae6102f529a8d4faa6a5524ba9122d47138ad23R393 which does the exact thing that I have done. But I don't know if this is the right way to do it because it was removed. I am new to this and would like some references to do this better.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. Additionally, I don't think adding arbitrary tolerances is a sufficient solution here. Closing but let's continue to discuss solutions in the original issue
DOC: Fix F821 error in docstring
State: closed

- [ ] xref #22900 (I think this PR can also close the linked issue as now `pandas` and `numpy` are considered defaults import before linting docstring)

The command below returns no error now.

```
flake8 --doctest --ignore="" --select=F821 pandas/ | grep -v "F821 undefined name 'pd'" | grep -v "F821 undefined name 'np'"
```

Labels:
name: Docs

Comments:
@tqa236 should we add a check in the CI for this?
Hello @datapythonista, I fully agree that fixing linting errors without enforcing them is quite inefficient. But I wonder if the CI check is a long term solution as it's based on `flake8 --doctest` while we're moving more and more to `ruff`.

There's an open issue on `ruff` to track this feature: https://github.com/astral-sh/ruff/issues/3542

In my opinion, we can wait for the feature to be implemented in `ruff` and enable it here rather than implementing our own check with `flake8 --doctest` for 2 reasons
- The code in all of the official docstrings are already run and checked if there's any syntax errors already.
- What's left for this CI check is only the internal docstring (in `conftest`,... ), and the number of errors is quite low (it in fact reduces compared to your result last year from the issue). 

Let me know what you think.
Sounds good, up to you, I was just wondering if you considered it. Thanks for all the details and for the continued work on this.
Thanks @tqa236
DOC: RT03 fix for various DataFrameGroupBy and SeriesGroupBy methods
State: closed

- [x] xref https://github.com/pandas-dev/pandas/issues/57416
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
/preview
Website preview of this PR available at: https://pandas.pydata.org/preview/pandas-dev/pandas/57862/
@quangngd we changed how we specify the pending errors in `code_checks.sh` and you'll have to resolve the conflicts. Sorry for the inconvenience.
@datapythonista resolved
Thanks @quangngd
PERF: Remove slower short circut checks in RangeIndex__getitem__
State: closed

- [ ] closes #57712 (Replace xxxx with the GitHub issue number)

Although the `all()` and `not any()` checks were supposed to be for short-circuiting, I think these cases are not common enough to always check in this if branch. The `not any()` check will get reduced by `np.flatnonzero`. The `all()` check cannot short circuit unlike the check in `_shallow_copy`.

A probably more "common case":

```python
In [1]: from pandas import *; import numpy as np
+ /opt/miniconda3/envs/pandas-dev/bin/ninja
[1/1] Generating write_version_file with a custom command

In [2]: N = 10**6

In [3]: s = Series(date_range("2000-01-01", freq="s", periods=N))

In [4]: s[np.random.randint(1, N, 100)] = NaT

In [6]: %timeit s.dropna()
7.49 ms ± 333 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  # main

In [5]: %timeit s.dropna()
7.22 ms ± 239 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  # PR
```

The worst case where `if key.all()` would have gotten hit (will be improved by https://github.com/pandas-dev/pandas/pull/57812)

```python
In [1]: import pandas as pd, numpy as np, pyarrow as pa
+ /opt/miniconda3/envs/pandas-dev/bin/ninja
[1/1] Generating write_version_file with a custom command

In [2]: ri = pd.RangeIndex(range(100_000))

In [3]: mask = [True] * 100_000

In [4]: %timeit ri[mask]
2.82 ms ± 31.4 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  # PR

In [4]: %timeit ri[mask]
2.41 ms ± 23.8 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  # main
```


Labels:
name: Performance
description: Memory or execution speed performance
name: Index
description: Related to the Index class or subclasses

Comments:
thx @mroeschke
PERF: Skip _shallow_copy for RangeIndex._join_empty where other is RangeIndex
State: closed

- [ ] closes #57852 (Replace xxxx with the GitHub issue number)


```python
In [1]: from pandas import *; import numpy as np
+ /opt/miniconda3/envs/pandas-dev/bin/ninja
[1/1] Generating write_version_file with a custom command

In [2]: df = DataFrame({"A": np.arange(100_000)})

In [3]: df_empty = DataFrame(columns=["B", "C"], dtype="int64")

In [4]: %timeit df_empty.join(df, how="inner")
369 µs ± 4.17 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)  # main

In [4]: %timeit df_empty.join(df, how="inner")
126 µs ± 1.74 µs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)  # PR
```


Labels:
name: Performance
description: Memory or execution speed performance
name: Index
description: Related to the Index class or subclasses

Comments:
thx @mroeschke
DOC: Remove duplicated Series.dt.normalize from docs
State: closed

`Series.dt.normalize` is listed both as an attribute and as a method. Removing the attribute instance, since it's a method.


Labels:
name: Docs

Comments:
> Does this need to be backported to 2.2.x?

No big difference, but if we do the docs of the 2.2 version won't have the duplicate. So I added it to the milestone.
Thanks @datapythonista
CLN: Remove unused functions
State: closed

It's pretty hard to be 100% sure that these non-private functions are unused, but I'm pretty sure that they are not commonly used, if at all, after some searches in Google, GitHub, and pandas doc.

Labels:
name: Clean

Comments:
Keep them coming! @tqa236 
@mroeschke thanks! I'm combing through `vulture` to see what can be removed. May I know where the documentation about the private directories is, please? It'll surely help me decide if a report is a true or false positive
> May I know where the documentation about the private directories is, please?

https://pandas.pydata.org/docs/reference/index.html#api-reference
Thank you!
DOC: Remove Dask and Modin sections in scale.rst in favor of linking to ecosystem docs.
State: closed

- [x] closes #57831 
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.2.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 d4ddc805a03586f9ce0cc1cc541709419ae47c4a
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #57843: DOC: Remove Dask and Modin sections in scale.rst in favor of linking to ecosystem docs.'
```

4. Push to a named branch:

```
git push YOURFORK 2.2.x:auto-backport-of-pr-57843-on-2.2.x
```

5. Create a PR against branch 2.2.x, I would have named this PR:

> "Backport PR #57843 on branch 2.2.x (DOC: Remove Dask and Modin sections in scale.rst in favor of linking to ecosystem docs.)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Thanks @yukikitayama 

(Backporting since the dask dependency changes were backported too)
Thank you for reviewing @mroeschke !
improve accuracy of to_pytimedelta
State: closed

```
>>> a = pd.Timedelta(1152921504609987375)
>>> a.to_pytimedelta()
datetime.timedelta(days=13343, seconds=86304, microseconds=609988)
>>>
```

609988 for microseconds but expected is 609987

```
>>> a = pd.Timedelta(1152921504609987374)
>>> a.to_pytimedelta()
datetime.timedelta(days=13343, seconds=86304, microseconds=609987)
>>>
```

In this example, a difference of 1 nanosecond, shouldn't result in a difference of 1 microsecond when converted from ns to us unit.

Labels:
name: Timedelta
description: Timedelta data type

Comments:
Ok, but the reason this behavior is occurring I don't believe is related to python's round function. It's because python's integer division goes through a floating point which introduces accuracy issues for large integers (and in this case, result is rounded anyway). The issue is with the division, not the rounding done by timedelta.
Sorry, you are correct. I didn't realize the problem with the floating point precision.

I wonder what's the impact on performance of this change, did you execute a benchmark to compare the execution times of both implementations?

@MarcoGorelli @jbrockmendel thoughts on this change?
Added whatsnew, are there existing benchmarks for comparing performance?
Thanks @rohanjain101