BrandonFors/pandas-github-issues huggingface dataset:
ENH: Add `method='table'` for EWM.mean
State: closed

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Enhancement
name: Window
description: rolling, ewma, expanding

Comments:
@meeseeksdev backport 1.3.x
thanks @mroeschke 
Something went wrong ... Please have  a look at my logs.
This is failing ci. reverting to get ci back to green

@mroeschke can you open a new PR for this.
CLN: simplify Styler clear mechanics and tests
State: closed

- [x] follows #42187 
- [x] checks an item off #40675

The current mechanics are unstructured and the tests is not future proof (if new attributes are added tests will not pick these up and fail). This PR generalises the construction and future proofs the tests. No user facing change.


Labels:
name: Refactor
description: Internal refactoring of code
name: Clean
name: Styler
description: conditional formatting using DataFrame.style

Comments:
sure
PERF: CategoricalIndex.get_indexer
State: closed

- [x] closes #42249
- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

```
import pandas as pd
import string, itertools
data_unique = pd.CategoricalIndex(
            ["".join(perm) for perm in itertools.permutations(string.printable, 3)]
)
cat_list = ["a", "c"]

%timeit data_unique.get_indexer(cat_list)
106 µs ± 6.14 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)  # <- PR
39 ms ± 616 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)  # <- 1.2.4
290 ms ± 3.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  # <- master
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Index
description: Related to the Index class or subclasses

Comments:
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.3.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 73ed010db5292c31b3bb562010acacaf4247ce70
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #42270: PERF: CategoricalIndex.get_indexer'
```

4. Push to a named branch :

```
git push YOURFORK 1.3.x:auto-backport-of-pr-42270-on-1.3.x
```

5. Create a PR against branch 1.3.x, I would have named this PR:

> "Backport PR #42270 on branch 1.3.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
@meeseeksdev backport 1.3.x
cc @jbrockmendel guess this won't backport easily
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.3.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 73ed010db5292c31b3bb562010acacaf4247ce70
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #42270: PERF: CategoricalIndex.get_indexer'
```

4. Push to a named branch :

```
git push YOURFORK 1.3.x:auto-backport-of-pr-42270-on-1.3.x
```

5. Create a PR against branch 1.3.x, I would have named this PR:

> "Backport PR #42270 on branch 1.3.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
DEPR: Index.get_loc with method
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

We only use it one place internally; this will allow us to simplify get_loc, which will marginally improve perf.

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Deprecate
description: Functionality to remove in pandas

Comments:
We can probably work around it, but just as a heads up, this is also used externally in Xarray in a pretty extensively used code path (#5721).
Is it correct that the `tolerance` argument will also need to be removed as part of this? I think the `tolerence` is only relevant to inexact methods. The `tolerance` argument is not currently marked as deprecated, but maybe its implicit given the relationship?
> Is it correct that the tolerance argument will also need to be removed as part of this? I

yes.
trying to enforce this I found that for DatetimeIndex cases the alternative `get_indexer([key], ...)` doesn't work because get_loc does string parsing that get_indexer doesn't.

I'd be OK with either a) requiring users to do the parsing themselves before calling get_indexer or b) reverting this deprecation.
PERF: IntervalIndex.intersection
State: closed

- [x] closes #42240
- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

```
import pandas as pd
import numpy as np

N = 10 ** 5
left = pd.IntervalIndex.from_breaks(np.arange(N))
right = pd.IntervalIndex.from_breaks(np.arange(N - 3, 2 * N - 3))

%timeit left.intersection(right)
146 ms ± 1.82 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  # <- master
5.6 ms ± 277 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)  # <- PR
```

We're also going to get a nice bump in join_monotonic/_inner_indexer, but not big enough to warrant not-special-casing IntervalIndex's intersection.

Labels:
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.3.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 569a615a89fada930cef8de588958bd5b477232c
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #42268: PERF: IntervalIndex.intersection'
```

4. Push to a named branch :

```
git push YOURFORK 1.3.x:auto-backport-of-pr-42268-on-1.3.x
```

5. Create a PR against branch 1.3.x, I would have named this PR:

> "Backport PR #42268 on branch 1.3.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
@meeseeksdev backport 1.3.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.3.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 569a615a89fada930cef8de588958bd5b477232c
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #42268: PERF: IntervalIndex.intersection'
```

4. Push to a named branch :

```
git push YOURFORK 1.3.x:auto-backport-of-pr-42268-on-1.3.x
```

5. Create a PR against branch 1.3.x, I would have named this PR:

> "Backport PR #42268 on branch 1.3.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
@jbrockmendel this failed to backport if you can do manual

cc @simonjayhawkins
TST IntegerArray bounds error closing GH21796
State: closed

- [x] closes #21796
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them



Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
Thanks for the pr @TAJD! Think the test failure is because the test assumes int64, but on 32-bit builds or windows, the default integer size will be int32. So I'd suggest just not trying to match as completely or could use numpy's platform int
Thanks @mzeitlin11 - hopefully by allowing 32 or 64 in the regex on the error message tests the required functionality.
this is fine, though the message actually is *slightly* misleading, you are actually trying to cast to a nullable type, but the underlying casting is bubbled up here.

so merging this, but would be neat to look into having a slightly more informative message here.
CLN: Group tests for apply/agg/transform with a string argument
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

Reorg of apply tests, grouping all those that take a string argument together. Plan to next create `test_list` and `test_dict`, then all other test files in apply will have (at least partially) UDFs, which make up a majority of the tests.

Had to rename a few tests because of collisions, but otherwise purely a move.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby
name: Clean

Comments:
@jreback - greenish
CLN: simplify take2d_axis0 generated code
State: closed

For example, on master we generate:

```
@cython.wraparound(False)
@cython.boundscheck(False)
def take_2d_axis0_int8_int32(const int8_t[:, :] values,
                                    ndarray[intp_t] indexer,
                                    int32_t[:, :] out,
                                    fill_value=np.nan):
    cdef:
        Py_ssize_t i, j, k, n, idx
        int32_t fv

    n = len(indexer)
    k = values.shape[1]

    fv = fill_value

    IF False:
        cdef:
            const int32_t *v
            int32_t *o

        # GH#3130
        if (values.strides[1] == out.strides[1] and
            values.strides[1] == sizeof(int32_t) and
            sizeof(int32_t) * n >= 256):

            for i in range(n):
                idx = indexer[i]
                if idx == -1:
                    for j in range(k):
                        out[i, j] = fv
                else:
                    v = &values[idx, 0]
                    o = &out[i, 0]
                    memmove(o, v, <size_t>(sizeof(int32_t) * k))
            return

    for i in range(n):
        idx = indexer[i]
        if idx == -1:
            for j in range(k):
                out[i, j] = fv
        else:
            for j in range(k):
                out[i, j] = values[idx, j]
```

On this pr:

```
@cython.wraparound(False)
@cython.boundscheck(False)
def take_2d_axis0_int8_int32(const int8_t[:, :] values,
                                    ndarray[intp_t] indexer,
                                    int32_t[:, :] out,
                                    fill_value=np.nan):
    cdef:
        Py_ssize_t i, j, k, n, idx
        int32_t fv

    n = len(indexer)
    k = values.shape[1]

    fv = fill_value


    for i in range(n):
        idx = indexer[i]
        if idx == -1:
            for j in range(k):
                out[i, j] = fv
        else:
            for j in range(k):
                out[i, j] = values[idx, j]
```

Not too impactful since just generated code, but makes looking at all the take helpers clearer (and the file shorter).


Labels:
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff
name: Clean

Comments:
thanks @mzeitlin11
Set threadsafe behavior for numba parallelism
State: closed

- xref #https://github.com/numba/numba/issues/6839#issuecomment-867977583
- [x] tests added / passed

Applicable for 1.3.0 since this behavior can appear for the new `method="table"` feature for rolling/expanding

cc @jbrockmendel 


Labels:
name: numba
description: numba-accelerated operations

Comments:
With this i don't get segfaults, but am getting test failures
> With this i don't get segfaults, but am getting test failures

Which ones in particular?
> Which ones in particular?

All of the test_table_method_rolling_methods tests
Got it. Might not be the best thing to hardcode a particular threading backend for numba either. I'll just document this in another PR.
BUG: Fix to_datetime() cache behaviour to not omit duplicated output values
State: closed

- [x] closes #42259 
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Datetime
description: Datetime data dtype

Comments:
Added a note

thanks @zyc09 
@meeseeksdev backport 1.3.x
CLN: allow label memviews to be None
State: closed

Saves some unnecessary memory allocations, plus moving towards making `rank_sorted_1d` easier to call. Also have done something similar for allowing `mask` to be None, but split that off this pr to keep it simpler.


Labels:
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff
name: Clean

Comments:
Small comments, generally looks very nice
thanks @mzeitlin11
CI: fewer workers on windows
State: closed

- [x] closes #42236

Seeing if this helps for #42236



Labels:
name: CI
description: Continuous Integration
name: Windows
description: Windows OS

Comments:
set this on all the windows builds
I've milestoned 1.3 or we can instead backport later if this improves things.

since it's only CI related, no harm in backporting though.
thanks @mzeitlin11 
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
ENH: Create directories when missing in to_* methods
State: closed

This is the most fundamental approach to creating new folders when a user specifies a path that is missing in file creation methods like `to_csv`, `to_excel`, `to_feather` etc. We can do this in [io/common.py:`get_handle()`](https://github.com/pandas-dev/pandas/blob/50a59bac1f2bcbf9ac87c7a4160d4355be512294/pandas/io/common.py#L521) conditionally when the file buffer provided is in a writable mode.

- [x] closes #24306
   - Can use the existing #42250 to extend documentation for each of the `to_*` methods (@Joeperdefloep)
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: IO Data
description: IO issues that don't fit into a more specific label
name: Stale

Comments:
So do we want default behaviour to be that directories are always created or an extra argument like `mkdirs=True` that needs to be passed? I tried to do it that way, but passing the argument through gave a lot of changes, and I thought the conclusion from #24306 was to just add it to the documentation. However, I think the change in behaviour is most welcome 🎉 

I can add that to the documentation. Then that would be in all `to_something` that can be a file?
Exposing an argument to control this might be helpful and shouldn't be too much work (and would allow us to reinstate the two tests I removed). I'll defer to the maintainers as to whether this should be on or off by default, but I agree that changing these to create missing directories by default is much more intuitive for novice users.
Then I'll also wait with the documentation until this is resolved :)
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
Thanks for the PR, but as mentioned, the team isn't keen on supporting an API that creates directories and instead would prefer a documentation approach like in https://github.com/pandas-dev/pandas/pull/42250/files. Closing.
REGR: groupby with as_index=False on an empty frame
State: closed

- [x] closes #41998
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

No whatsnew because this was introduced in 1.3


Labels:
name: Groupby
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
This looks right to me, but merits another pair of eyeballs
this looks good, thanks @rhshadrach 
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
ENH: allow attrs to be propagated via pd.concat
State: closed

- [x] closes #41828
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

@TomAugspurger @lithomas1 I have made a fix to this issue and I wonder if do you mind have a review, please? I will add the whatsnew when this PR is about to be merged.

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: metadata
description: _metadata, .attrs

Comments:
Hello @xiki-tempula! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-12-01 08:51:44 UTC
@ivalaginja Thanks for the review. I have fixed the tests.
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
@github-actions @jreback @TomAugspurger Would you mind have a review of this PR, please? Thank you.
I won't have time this week.
can you merge master and will look
@jreback Thanks very much for the review. I have updated the changes.
can you merge master and will look
@jreback Thanks for the help. I have merged the master.
@jreback Thanks for the review. I have used parameterized on the tests and added the test for the empty dataframe and series to the test.
I'm not quite sure of how to use the assert_equal to test integers so I have left it there.
@jreback Thanks, I have the what's new updated. I'm not quite sure of "ping on green" means, if you are talking about the tests, they are 1 cancelled and 31 successful checks for the last commit f49f027.
There is one test failed due to timeout.
thanks @xiki-tempula
DOC: Clarify relation between DateOffset and relativedelta
State: closed

dateutil.relativedelta can be constructed either from two date/datetime objects passed as positional arguments, or explicitly constructed in terms of year, month, day etc passed as keyword arguments. DateOffset only supports the keyword argument form of relativedelta.

- [x] related to #41847
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Docs
name: Frequency
description: DateOffsets

Comments:
Thanks @daviewales
DOC: create folders in to_csv #24306
State: closed

- [x] closes  #24306
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them



Labels:
name: IO CSV
description: read_csv, to_csv

Comments:
one question would be if we also want this in `to_excel`
Thanks for the pr @Joeperdefloep! Based on the linked issue, it seems like the idea was to add a doc note, and there was no consensus about adding this feature. I'd be surprised by automatic directory creation being the default just based on python behavior, eg 
```
>>> with open("no_dir/text.txt", "x") as handle:
...     handle.write("hello")
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
FileNotFoundError: [Errno 2] No such file or directory: 'no_dir/text.txt'
```
Hi @mzeitlin11 thanks for your comment. I know that it would not eb default Python behaviour. My proposed PR is also only an upate in the docs on how to create a new folder so that users can wrap it in a function. #42255 does indeed implement this behaviour and there is discussion there on whether this should be implemented and/or exposed.
> Hi @mzeitlin11 thanks for your comment. I know that it would not eb default Python behaviour. My proposed PR is also only an upate in the docs on how to create a new folder so that users can wrap it in a function. #42255 does indeed implement this behaviour and there is discussion there on whether this should be implemented and/or exposed.

My bad @Joeperdefloep, commented on the wrong pr :)
yeah, actually also slightly mine, should've said *take* on the issue
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
Why are you adding an empty file called `file.xlsx`?
thanks @Joeperdefloep lgtm.

@alimcmaster1 @lithomas1 if any comments.
thanks @Joeperdefloep
Updated the return type section of to_datetime 
State: closed

Updated the return type section of to_datetime to include list-like mixed timezone inputs

- [x] closes #42229
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry - Updated the return type section of to_datetime to include list-like mixed timezone inputs


Labels:
name: Docs
name: Datetime
description: Datetime data dtype

Comments:
@mroeschke - updated as per you suggestion 
Removed the additional line , Thanks for patience and feedback @mroeschke 
Thanks @prabha-git 
Hi @prabha-git and @mroeschke ! It seems that the resulting comment is half/half (half speaking about timezone, half speaking about time offset...). Therefore I created https://github.com/pandas-dev/pandas/pull/42494 to complete this PR.
PERF/ENH: allow Generator in sampling methods
State: closed

- [x] closes #38100 
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

The perf gap is pretty crazy:

```

In [10]: rng = np.random.default_rng()

In [11]: ser = pd.Series(np.zeros(10 ** 6))

In [12]: %timeit ser.sample(n=10 ** 4)
13.8 ms ± 231 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

In [13]: %timeit ser.sample(n=10 ** 4, random_state=rng)
569 µs ± 8.11 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

```

Leaving #28440 open because I think the end goal should still be to make using these the default. But assuming that change isn't too easy since will be a breaking change?

Labels:
name: Enhancement
name: Performance
description: Memory or execution speed performance

Comments:
thanks @mzeitlin11 

might need to slightly revist after we drop 1.17 (shortly)