BrandonFors/pandas-github-issues huggingface dataset:
DOC: update docstrings for ffill and bfill (no longer aliases for fillna with method)
State: closed

Follow-up on https://github.com/pandas-dev/pandas/pull/54728 to do the same for `bfill`, and also updated both `ffill` and `bfill` to include the parameter section (now that only lived in fillna)

Labels:
name: Docs

Comments:
Nice thanks @jorisvandenbossche
BUG: repr aligning left for string dtype columns
State: closed

- [ ] closes #54797 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

cc @jorisvandenbossche this is a bit hacky, but changing format_array breaks index-related stuff. Don't feel comfortable to do this now, can be a follow up (we are casting to object anyway, so this does not impact performance)

Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
> this is a bit hacky, but changing format_array breaks index-related stuff.

Looking at the code in `Index._format_with_header`, it seems we just never use `format_array` in case of object dtype (i.e. current strings). And in the code path for non-object dtype calling `format_array`, we specify `justify="left"`. That's the reason for this behaviour, and so something we would have to change (or let the value depend on the dtype; I don't fully understand the reasoning for this `justify="left"` for other dtypes)
Yeah that's correct, we can set justify=all, but this screws with Index formatting if the index has a NaN because we are doing something weird. But that's too big for this here, that's why I went with the short term solution
> I don't fully understand the reasoning for this `justify="left"` for other dtypes

You can get this "bug" as well for other data types because of this left alignment. For example with numeric index:

```
In [14]: df = pd.DataFrame({99999999999999999: [1, 2, 3], 1: [4, 5, 6]})

In [15]: df
   99999999999999999  1                
0                  1                  4
1                  2                  5
2                  3                  6
```

(will move this to the issue)
Infer string storage based on infer_string option
State: closed

- [x] closes #54793
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Strings
description: String extension data type and string data
name: Arrow
description: pyarrow functionality

Comments:
I'll add some docs to this. 
Updated
WARN: Remove non-actionable warning in value_counts
State: closed

- [x] closes #54775 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

cc @rhshadrach maybe?

Labels:
name: Groupby
name: Regression
description: Functionality that used to work in a prior pandas version
name: Categorical
description: Categorical Data Type

Comments:
Thanks @phofl - changes look great. Looks like we didn't have any test coverage here - I've opened #54836
BUG: creating dataframe from masked numpy array turns missing string …
State: closed

…values into NaN

- [x] closes #54706 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Stale

Comments:
take

Could you address the failing tests?
@jbrockmendel Hi I am facing some issue while testing the code with the following

 ` # GH#18463
            warnings.warn(
                f"Mismatched null-like values {a} and {b} found. In a future "
                "version, pandas equality-testing functions "
                "(e.g. assert_frame_equal) will consider these not-matching "
                "and raise.",
                FutureWarning,
                stacklevel=find_stack_level()`
                
                
causing the test to fail while running 
                
 `pytest pandas/tests/series/test_constructors.py`
 
 
 it has been 10 days but there has been no response on slack regarding the query. Kindly help 
> [...] causing the test to fail while running
>
> pytest pandas/tests/series/test_constructors.py

That generally means you are doing something like `tm.assert_series_equal(result, expected)` and result/expected contain different NA-like values, e.g. one has `np.nan` while the other has `None`.  To avoid the warning, they need to match.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
TYP: simple return types
State: closed

"simple" meaning, non-unions and non-overload return types (at least in most cases).

xref #47521

edit: sorry for changes in many files - I grep'ed for missing return types inferred by pyright and then fixed the resulting type errors.

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Thank you @rhshadrach for your thorough review! I think I addressed your comments as best as I can.
Thanks @twoertwein 
hm, the docs timed out on main as well - did the annotation cause the docs to fail?
I think that's unrelated (my running theory is that something processing the whatsnew files is not parallel safe and hanging)
CI: Unpin Cython
State: closed

- [ ] closes #54756 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: CI
description: Continuous Integration
name: Dependencies
description: Required and optional dependencies

Comments:
Something is going wrong on the Cython end I think ...

Pushing this to 2.1.1.
I'm gonna merge to get Cython unpinned for now. Happy to address followups in another PR.
@lithomas1 - We've been pinning Cython in the ASVs; should this be unpinned too?

https://github.com/pandas-dev/pandas/blob/753d0ed38adae2acbd928c06cce9a320253fff2e/asv_bench/asv.conf.json#L45
Ah - maybe I'm confused. Looks like we're only unpinning Cython in the CI?
Yeah, ASVs have to stay pinned since there was a perf regression with Cython 3 (so we'd have spurious regressions if we unpinned).
CI: Remove strict option
State: closed

It is there by default in asv > 0.6.0

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: CI
description: Continuous Integration
name: Benchmark
description: Performance (ASV) benchmarks

Comments:
Thanks @lithomas1
REGR: Index.union loses python string dtype
State: closed

- [x] closes #54745 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

cc @mroeschke

Labels:
name: Index
description: Related to the Index class or subclasses

Comments:
Thanks @phofl
BUG: pd.SparseDtype('int', 0) == pd.SparseDtype('int', pd.NA) returns pd.NA
State: closed

BUG: pd.SparseDtype('int', 0) == pd.SparseDtype('int', pd.NA) returns pd.NA

- [X] closes #54770(Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Sparse
description: Sparse Data Type

Comments:
Can you investigate the tests failures?
> Can you investigate the tests failures?

I am not able to setup the debug to on vs code and even after trying multiple machine I am on the same page.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
Fix roundtripping with pyarrow schema
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

that's a regression on main as well, it's currently returning NA

cc @mroeschke

Labels:
name: Arrow
description: pyarrow functionality

Comments:
Nice thanks @phofl
PERF: skip libjoin fastpath for MultiIndex
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Addresses #54646 for `MultiIndex` but leaves `RangeIndex` as is. The `RangeIndex` case
seems to have a few side effects - I'll add comments separately to #54646 for `RangeIndex`. 


Also, simplifies the logic in `Index.join` which improves perf for a few `MultiIndex` and non-`MultiIndex` cases.


`multiindex_object` benchmarks:

```
       before           after         ratio
-      18.7±0.2ms       16.6±0.4ms     0.89  multiindex_object.SetOperations.time_operation('monotonic', 'ea_int', 'symmetric_difference', False)
-      19.0±0.7ms       16.5±0.2ms     0.87  multiindex_object.SetOperations.time_operation('monotonic', 'int', 'intersection', False)
-       60.6±10ms         49.8±1ms     0.82  multiindex_object.SetOperations.time_operation('monotonic', 'string', 'intersection', None)
-        40.1±1ms       32.2±0.4ms     0.80  multiindex_object.SetOperations.time_operation('monotonic', 'ea_int', 'intersection', None)
```

`join_merge` benchmarks:

```
       before           after         ratio
-      9.98±0.2ms       8.99±0.1ms     0.90  join_merge.ConcatIndexDtype.time_concat_series('Int64', 'has_na', 1, True)
-      12.2±0.3ms      10.5±0.05ms     0.86  join_merge.ConcatIndexDtype.time_concat_series('int64[pyarrow]', 'has_na', 1, True)
-        13.9±2ms       11.8±0.2ms     0.85  join_merge.ConcatIndexDtype.time_concat_series('datetime64[ns]', 'has_na', 1, False)
-        26.3±3ms      22.3±0.05ms     0.85  join_merge.ConcatIndexDtype.time_concat_series('string[python]', 'monotonic', 1, False)
-        6.50±1ms      5.19±0.05ms     0.80  join_merge.ConcatIndexDtype.time_concat_series('datetime64[ns]', 'monotonic', 1, False)
-         133±2ms         51.1±2ms     0.38  join_merge.Align.time_series_align_int64_index
-       105±0.9ms       23.7±0.5ms     0.22  join_merge.Align.time_series_align_left_monotonic
```



Labels:
name: Performance
description: Memory or execution speed performance
name: MultiIndex

Comments:
Thanks @lukemanley
CI: Pin more cython
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Build
description: Library building on various platforms
name: CI
description: Continuous Integration

Comments:
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.1.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 ce6aa67314546f15a90668ad95ea7af88d80036a
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #54759: CI: Pin more cython'
```

4. Push to a named branch:

```
git push YOURFORK 2.1.x:auto-backport-of-pr-54759-on-2.1.x
```

5. Create a PR against branch 2.1.x, I would have named this PR:

> "Backport PR #54759 on branch 2.1.x (CI: Pin more cython)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
@meeseeksdev backport 2.1.x
CI: Pin Cython on CI
State: closed

- [ ] ref #54756 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: CI
description: Continuous Integration

Comments:
Thanks @phofl 
Looks like python dev needs to be pinned as well, PR incoming.
I think we also need to pin cython in `.github/workflows/unit-tests.yml`
BUG: merge raising for ea int and numpy float
State: closed

- [ ] closes #46178 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Ideally I'd like to get this in, but not a blocker

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
```
mypy.....................................................................Failed
- hook id: mypy
- duration: 105.59s
- exit code: 1

pandas/core/reshape/merge.py:1393: error: Item "dtype[Any]" of "dtype[Any] | ExtensionDtype" has no attribute "construct_array_type"  [union-attr]
pandas/core/reshape/merge.py:1396: error: Item "dtype[Any]" of "dtype[Any] | ExtensionDtype" has no attribute "construct_array_type"  [union-attr]
Found 2 errors in 1 file (checked 1413 source files)
```
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.1.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 9939c323e636d580e8ca1690aef40b6822ff02f7
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #54755: BUG: merge raising for ea int and numpy float'
```

4. Push to a named branch:

```
git push YOURFORK 2.1.x:auto-backport-of-pr-54755-on-2.1.x
```

5. Create a PR against branch 2.1.x, I would have named this PR:

> "Backport PR #54755 on branch 2.1.x (BUG: merge raising for ea int and numpy float)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Thanks @phofl
REGR: groupby.count returning string dtype instead of numeric for string input
State: closed

- [ ] closes #54751 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Groupby
name: Regression
description: Functionality that used to work in a prior pandas version
name: Blocker
description: Blocking issue or pull request for an upcoming release

Comments:
Thanks @phofl
PERF: Increase threashold for using binary search in IndexEngine
State: closed

# Checklist

- [x] closes #54550 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

# Explanation

So after a bit of research for #54550, I found that the source of the behavior is coming from PR #22826 at [this line](https://github.com/pandas-dev/pandas/blob/0f437949513225922d851e9581723d82120684a6/pandas/_libs/index.pyx#L361) and particularly this comment [here](https://github.com/pandas-dev/pandas/pull/22826#discussion_r220715797). 

In the comment, @rtlee9 makes an excellent explanation for the reason behind the threshold. The default behavior is a linear search (`O(n)`) in the pandas index and #22826 added a binary search optimization for monotonically increasing index which is in `O(m*log(n))` where `m` is the number of target indexes.

# Proposed Solution

I'm proposing we increase the limit, which was set conservatively to 5. We could at least increase it to n / (2 * log2[n]) which is derivable from m*log2[n]<n and dividing per 2 for keeping a certain margin. I'll open a PR for that.

# New results of benchmarking + homemade script

Here is a screenshot of `asv continuous -f 1.1 -E virtualenv upstream/main HEAD -b ^indexing.*Index*` output (done on Macbook Air m1, inside the docker container provided by pandas):

![Capture d’écran, le 2023-08-23 à 09 05 41](https://github.com/pandas-dev/pandas/assets/5565274/a5eda5c8-2d9a-4ebd-a337-c82f0f5094a0)

Here is also a new output for the script use to demonstrate the problem in #54700: 

```
With all at once: num_indexes=1 => 0.00048s
With all at once: num_indexes=2 => 0.00045s
With all at once: num_indexes=3 => 0.00038s
With all at once: num_indexes=4 => 0.00035s
With all at once: num_indexes=5 => 0.00035s
With all at once: num_indexes=6 => 0.00045s
With all at once: num_indexes=7 => 0.00042s
With all at once: num_indexes=8 => 0.00066s
With all at once: num_indexes=9 => 0.00049s
With all at once: num_indexes=10 => 0.00048s
```

Thanks in advance for the review!

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Performance
description: Memory or execution speed performance

Comments:
Thanks @l3robot 
> Thanks @l3robot 

Glad to contribute!