BrandonFors/pandas-github-issues huggingface dataset:
REF: check monotonicity inside _can_use_libjoin
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Refactor
description: Internal refactoring of code
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
@lukemanley any interest in taking over on this one?  im at risk of letting it fall through the cracks
> @lukemanley any interest in taking over on this one?  im at risk of letting it fall through the cracks

Sure, I'll take this one.
It looks like this is now down to one test failure: `test_setitem_mix_of_nan_and_interval`

The underlying issue currently exists on `main` and is similar to #54847.

Now that `Index.is_monotonic_increasing` gets called as part of `Index.is_unique` we're now hitting this behavior in the test:

```
import pandas as pd
from decimal import Decimal

idx = pd.Index([Decimal("NaN"), Decimal("NaN")])

# raises InvalidOperation: [<class 'decimal.InvalidOperation'>]
idx.is_monotonic_increasing
```

We could either check for nulls in `algos.is_monotonic` or put a try/except around `Index.is_unique` to catch this case. Kind of annoying either way it seems. Any other options?

> or put a try/except around Index.is_unique to catch this case

Where would this live?  some places would be uglier than others.

> Where would this live? some places would be uglier than others.

Added to the existing try/except in `IndexEngine._do_monotonic_check`

This is now passing all tests.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
gentle ping @jbrockmendel if you have any thoughts on this
LGTM.  Is the perf issue you noted addressed?  One more merge main and ping on green pls
@jbrockmendel - merged main and now green. I think this is ready
Thanks @jbrockmendel 
thanks @lukemanley
ENH: Implement convert_dtypes on block level
State: closed

The current implementation is pretty bad in the sense that we will always end up splitting blocks even if we don't want to touch the block at all. Pushing this to the Block level let's us solve that problem.

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
cc @mroeschke 
Thanks @phofl
Implement masked algorithm for mode
State: closed

- [ ] closes #45776 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
planning on merging this soonish if no objections
Raises a ValueError for not handled dtypes in pyarrow Conversion.
State: closed

Raises a ValueError in PandasColumn._dtype_from_pandasdtype for the following unhandled dtypes: 'b', 'B', 'S', 'a', 'V'.

- [X] closes #55332 (Replace xxxx with the GitHub issue number)
- [X] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [X] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [X] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Interchange
description: Dataframe Interchange Protocol

Comments:
thanks for your PR - not sure this is the solution to be honest https://github.com/pandas-dev/pandas/issues/55332#issuecomment-1751720378
BLD: Fix gitpod meson version
State: closed

- [X] closes #55214 
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

The issue is that meson is out of date in gitpod. 

`pip install --upgrade meson`

The above command fixes the issue, so I added it to the build instructions in gitpod.yml. It could be added to the dockerfile instead, but I decided to be cautious. 

I added the update command to both the `init` and `command` sections because gitpod sometimes skips the `init` section and goes straight to the `command` section without installing anything.

With the fix:
```
gitpod@theuerc-pandas-jy3jlfp1dg6:/workspace/pandas$ python
Python 3.10.8 (main, Dec  6 2022, 14:13:21) [GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
+ /usr/local/bin/ninja
[1/1] Generating write_version_file with a custom command
>>> pandas.to_datetime("2022-01-01")
Timestamp('2022-01-01 00:00:00')
>>> 
```

I'm running the tests in gitpod overnight and I'll check the log file in the morning.

#### Update: 

These are the results:

661 failed, 211445 passed, 7990 skipped, 2426 xfailed, 15 xpassed, 405 warnings, 58 errors in 4524.22s (1:15:24)

The failed tests are the ones that interact with either S3 or a nonexistent SQL server, so I think everything works.


Labels:
name: Admin
description: Administrative tasks related to the pandas project
name: Stale

Comments:
I'll update it every so often until someone with write access comes along again.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the PR, but this is currently getting overhauled in https://github.com/pandas-dev/pandas/pull/56375 which should also solve your original issue so closing in favor of that PR
COMPAT: Fix warning with numba >= 0.58.0
State: closed

- [ ] closes #55247 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: numba
description: numba-accelerated operations

Comments:
pre-commit.ci autofix
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
pre-commit.ci autofix
@mroeschke 

Can you take another look at this?
This would be nice to have for 2.2.
Looks good just a comment
Thanks @lithomas1
Bug fix where df.rolling doesn't work with certain datetime64 index t…
State: closed

- [x] closes #55299
- [x] Tests added and passed.
- [x] All code checks passed.
- [x] Added an entry in bug fixes in the latest `doc/source/whatsnew/v2.2.0.rst` file.

This PR fixes an issue where in `df.rolling`, `rolling._win_freq_i8` is always given in nanoseconds but `rolling._index_array` is given in the unit corresponding to the dtype (e.g. s, ms, us, ns). This causes a problem when the window is calculated if the dtype is not `datetime64[ns]`. This code returns the index array in nanoseconds so it matches with `rolling._win_freq_i8`.

Labels:

Comments:
Thanks for the PR and the late triage in the old issue. We have a PR already in progress to fix this issue https://github.com/pandas-dev/pandas/pull/55173 so going to close in favor of that PR.
CLN: Clean up some iteration logic in tslib
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Clean

Comments:
I don't think the multi-iterator is necessary, we only iterate on values not results.

cc @jbrockmendel 
thanks @lithomas1
BUG: `date_range` `inclusive` parameter behavior doesn't match interval notation
State: closed

- [X] closes #55293 
- [X] closes #46331

- [X] Added an entry in the latest `doc/source/whatsnew/v2.1.3.rst` file if fixing a bug or adding a new feature.

This pr fixes the issues with `inclusive` in `date_range`

Got rid of the problematic expression which seems to be `i8values[] == start/end_i8`, also the `and` in line 498 should have been an `or`. However after those fixes the if-statement was over complicated, so I simplified it. 

Labels:
name: Bug
name: Datetime
description: Datetime data dtype

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
This pr is ready for review
> In this case the test seems a bit cumbersome, was it asserting cases with the behavior we consider a bug in this PR?

Yes, as far as I could tell the tests had a few mistakes in them, also they were written in a really strange and unnecessarily complex way. 
For example this line:
```python
expected = date_range(bday_start, bday_end, freq="D")
```  
should get the expected range, but the `inclusive` argument is not passed to it

Thanks for the feedback and the work on this @PiotrekB416 

Marco will be able to provide better feedback when he has the time to review, but what makes sense to me is:

- Leave the existing tests as they are in this PR
- Add a clearer brand new test that hits the specific bug you are fixing. If it makes sense we can test more on it
- Only if there is any specific part of the existing tests that fail after your changes, remove that part, in a ver controlled way of what's the behavior we were testing to be true, that we now want to be changed
- In a follow up, we can do a proper refactoring of the tests of this functionality

While what you are doing here seems reasonable, it feels strange that the complex logic you are making much simpler here wasn't implemented for a reason, and that we may be missing some use case where your changes fix a bug, but introduce a new one. Changing too much here makes difficult to see if that will be the case. I feel like the approach I'm proposing should minimize the chances of it happening. What do you think?
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
doc/source/whatsnew/v0.23.1.rst modified
State: closed

- [x] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Thanks but this was fine as is. Closing
DOC move resample bug whatsnew notes to 2.2
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thx!
Thanks @MarcoGorelli (do the changes to 2.1.2.rst need to be backported then?)
i hadn't backported it to begin with so I don't think so
ENH: case_when function
State: closed

- [x] closes #39154 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Continues the work started by @ElHoussineT(#50343)

- uses `pd.Series.mask` under the hood
- function implementation, as well as a Series method

Labels:

Comments:
Hi @mroeschke @rhshadrach @phofl  can I pls get some feedback on this PR. thanks
Please merge with current `main`.  Your PR is deleting code recently merged.
@Dr-Irv thanks for the feedback. I have merged with current main - i still notice extra files that shouldnt pop up for review, since it has nothing to do with this PR. open to suggestions on how to fix this. thanks
> @Dr-Irv thanks for the feedback. I have merged with current main - i still notice extra files that shouldnt pop up for review, since it has nothing to do with this PR. open to suggestions on how to fix this. thanks

I don't think you did it correctly.  I am just looking at `_typing.py`, and it shows you deleted stuff.

Do
```bash
git fetch upstream
git merge upstream/main
git push origin case_when
```

thanks @Dr-Irv i will close this PR, I created a new one - hopefully the issues wont resurface. #55390
> thanks @Dr-Irv i will close this PR, I created a new one - hopefully the issues wont resurface. #55390

I looked briefly, and see that the issue is gone. But others will need to approve that PR.
Valid capitalization errors #32550
State: closed

- [x] Fix capitalization at [Fix capitalization among headings in documentation files #32550](https://github.com/pandas-dev/pandas/issues/32550)
- [x] https://github.com/pandas-dev/pandas/blob/main/doc/source/development/contributing_docstring.rst
- [x] https://github.com/pandas-dev/pandas/blob/main/doc/source/development/contributing_documentation.rst

Labels:
name: Docs

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
ExtensionArray.interpolate() method and tests
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Adds a default interpolate method for numeric EAs.

Partially fixes #40252 by removing fill_value from interpolate signiture. This doesn't fully close as I haven't implemented interpolate for integers - should interpolating an IntegerArray return a IntegerArray or FloatArray? 

Is this the right way to go about testing interpolate? I couldn't think of a simpler way to go about it.

This works for Series(EA).interpolate(), but falls over for DataFrame(EA).interpolate() as axis=1 is passed. 

Labels:
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale and the checks are all not passing yet. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
BUG: df.resample('MS', closed='right') incorrectly places bins
State: closed

- [x] closes #55271
- [x] closes #55281
- [x] closes #55282
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

~todo: extra tests for 'B' with closed ='right' (linked issue below), whatsnew~

This fixes a regression from 2.1.0 - but, it also happens to fix two older bugs as a side-effect. So, I've put this in the 2.1.2 milestone

Labels:
name: Resample
description: resample method

Comments:
Thanks @MarcoGorelli 
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.1.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 98f5a787940e02966f2747edbcbd992f0031c277
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am "Backport PR #55283: BUG: df.resample('MS', closed='right') incorrectly places bins"
```

4. Push to a named branch:

```
git push YOURFORK 2.1.x:auto-backport-of-pr-55283-on-2.1.x
```

5. Create a PR against branch 2.1.x, I would have named this PR:

> "Backport PR #55283 on branch 2.1.x (BUG: df.resample('MS', closed='right') incorrectly places bins)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
thanks

ah I put the wrong milestone, this should've just gone into 2.2 as it's just a regression since main, rather than since 2.1

will fixup the whatsnew entry later
@MarcoGorelli ping on the whatsnews, they cause merge conflicts on all back ports
here you go, sorry for the delay https://github.com/pandas-dev/pandas/pull/55309
Add remove_from_default_na options to read_csv, read_excel...
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

https://github.com/pandas-dev/pandas/issues/52493
As already known, pd.read_csv(io.StringIO("a\nNone")).a[0] is 'None' on pandas 1 but NaN on pandas 2.

Pandas uses NaN to represent missing values, but "None" often means 'No options satisfy the condition' in real data.
Even though this is an unintentional breaking change, this bug is considered to be too late to be fixed now.

I was trying to update libraries we use and bumped into this problem.
So, I suggest to add a parameter 'remove_from_default_na' 

`read_excel(path, remove_from_default_na="None")
`

which is also a straightforward solution to the issue below.
https://github.com/pandas-dev/pandas/issues/19156

Without this parameter, the workaround would be something like

> from pandas._libs.parsers import STR_NA_VALUES
> read_excel(path, na_values=STR_NA_VALUES-set(["None"]), keep_default_na=False)

Thank you for the great library.

Labels:

Comments:
Thanks for the pull request, but it looks like this new feature breaks a lot of checks, and we require a discussion and buy-in from core devs before moving forward with a pull request. I would suggest providing more context in one of your linked issues before moving forward here so closing for now.