BrandonFors/pandas-github-issues huggingface dataset:
PERF: Use list comprehension to join strings (#41753)
State: closed

PERF: Use list comprehension to join strings (#41753)

Labels:
name: Performance
description: Memory or execution speed performance

Comments:
Thanks @vrserpa - if you fetch and merge upstream/master that should fix the pre-commit failure

Are you interested in writing a linting check for this in [pandas-dev-flaker](https://github.com/pandas-dev/pandas-dev-flaker)?
Hi @MarcoGorelli, I will write a linting check for [pandas-dev-flaker](https://github.com/pandas-dev/pandas-dev-flaker).
awesome, thanks, let me know if you want/need any help with that - I expect you'd be able to model it off of https://github.com/asottile/pyupgrade/blob/5fb168667ae73f157dd579344708e1cdfb0c0341/pyupgrade/_plugins/generator_expressions_pep289.py
Thanks @MarcoGorelli! I'm going to have a look at this.
Hi @vrserpa - can you fetch and merge upstream/master?
Ok!
can you fixup the pre-commit failures please? https://pandas.pydata.org/docs/development/contributing_codebase.html#pre-commit
Sorry, @MarcoGorelli. I did the check with pre-commit now.
@MarcoGorelli will this be automatically checked in the CI/precommit? 
once https://github.com/pandas-dev/pandas-dev-flaker/pull/26 is in, yup!
ENH: add ignore index to DataFrame / Series.sample
State: closed

- [x] closes #38581
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

Picking up #38594. Left in 1.3 for now since all comments were addressed except small typing issues and seems like a nice enhancement, but let me know if it should be moved.

Labels:
name: Enhancement

Comments:
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.3.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 902be0267dfe3c2ec9263a92919ed0c2cca7ce18
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #42168: ENH: add ignore index to DataFrame / Series.sample'
```

4. Push to a named branch :

```
git push YOURFORK 1.3.x:auto-backport-of-pr-42168-on-1.3.x
```

5. Create a PR against branch 1.3.x, I would have named this PR:

> "Backport PR #42168 on branch 1.3.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
thanks @mzeitlin11 and @erfannariman for the original!
cc @meeseeksdev backport 1.3.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.3.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 902be0267dfe3c2ec9263a92919ed0c2cca7ce18
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #42168: ENH: add ignore index to DataFrame / Series.sample'
```

4. Push to a named branch :

```
git push YOURFORK 1.3.x:auto-backport-of-pr-42168-on-1.3.x
```

5. Create a PR against branch 1.3.x, I would have named this PR:

> "Backport PR #42168 on branch 1.3.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
@mzeitlin11 looks like a conflict on 1.3 if you do a manual PR (cc @simonjayhawkins)
> @mzeitlin11 looks like a conflict on 1.3 if you do a manual PR (cc @simonjayhawkins)

Yep I can take care of it later today
REGR: preserve Int32 dtype on setitem
State: closed

- [x] closes #42099
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
PERF: Check identity first before comparing the objects
State: closed

 - [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

This is a regeression in #41952. Checking the identity first avoids long comparison of e.g. tuples - checking the id first is also how `PyObject_RichCompareBool(a, b, Py_EQ)` works.

Here are results from asv:

```
       before           after         ratio
     [1ff6970c]       [b8285515]
-      11.0±0.1ms      7.09±0.06ms     0.64  algos.isin.IsInWithLongTupples.time_isin

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.

```


Labels:
name: Performance
description: Memory or execution speed performance

Comments:
great, ping on green.
thanks @realead 
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
DOC: share swaplevel docstring between DataFrame and Series (#42120)
State: closed

- [ ] closes #42120 
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] tests added / passed
- [ ] whatsnew entry

This commit
- adds examples to the Series.swaplevel docstring
- reuses the same template in Series.swaplevel and DataFrame.swaplevel docstrings


Labels:
name: Docs

Comments:
looks fine. @MarcoGorelli can you have a look
STYLE: Remove in-line comments in flake8 configuration
State: closed

- [x] closes #42141

Hi @MarcoGorelli, how does this look?
Let me know if this needs any changes.
Thanks

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Nice, thanks - could you do this for `flake8-rst` section too please?
> Nice, thanks - could you do this for `flake8-rst` section too please?

Sure!
thanks @RhnSharma
DOC: Substitute example in DataFrame.to_markdown docstring (#42126)
State: closed

- [ ] closes #42126 
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
Nice, thanks @neinkeinkaffee ! Did you try building these pages locally to check that they show up fine? If so, could you post a screenshot?

see [here](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#building-the-documentation) for how to build the docs locally, you'll probably want to use the `--single` option with `pandas.DataFrame.to_markdown`
Sure! This is how they look like:

<img width="697" alt="Screenshot 2021-06-20 at 15 27 52" src="https://user-images.githubusercontent.com/1748562/122677903-496fb500-d1dc-11eb-89f6-679b1841d0c5.png">
<img width="704" alt="Screenshot 2021-06-20 at 15 28 12" src="https://user-images.githubusercontent.com/1748562/122677905-4aa0e200-d1dc-11eb-8346-4d80145a5f16.png">



Thanks - From the first screenshot you posted (DataFrame.to_markdown), it looks like the Series example is still the one being substituted into the docstring :thinking: 
OMG, you're right, weird, I could swear it was alright when I checked earlier! Will check what happened there
Here it is one more time:

<img width="672" alt="Screenshot 2021-06-20 at 17 20 14" src="https://user-images.githubusercontent.com/1748562/122681366-d0785980-d1eb-11eb-9cb4-84bf21236b9d.png">

we could also add `to_markdown` to `user_guide/io/rst`. It recently had some output methods added, but I didn't even know this one existed. 
Good idea - @neinkeinkaffee if you wanted to do that (in a separate PR) it'd be welcome, else we can open it as an issue for others to take

CI failures here are probably unrelated, if you fetch and merge upstream/master once CI is fixed they'll probably be fine
@MarcoGorelli Sure, I'll add it to userguide/io.rst and submit a separate PR for that.

Regarding the failing CI check of this one, do you mean I should just merge my branch with master again and that might fix it?
Yeah, that's right - latest PRs seem to be mostly green, so if you fetch and merge upstream/master now that might be enough
Updated with the most recent commits from master, hopefully that'll make the checks go green. I'll work on updating the io.rst on a separate branch.
Actually, having a second look at io.rst I see that the documentation for each of the read and write methods is quite extensive there, containing lots of tips and tricks. Not sure if I can come up with something like that for `to_markdown` on the spot, it looks like a bigger project to be honest :)
No worries, I can open it up as an issue and someone else will take it! Thanks for your help on this one!
@simonjayhawkins I've seen some recent docs PRs being tagged as 1.3, is that an OK milestone for this one too?
![](https://i.fluffy.cc/MKjDMmwXxZlqRTwnn070gVqZ8Rwh6d3p.gif)
Can't combine DatetimeIndex MultiIndex with numeric MultiIndex
State: closed

- [ ] closes #23478
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry



Labels:
name: Datetime
description: Datetime data dtype
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: MultiIndex
name: Index
description: Related to the Index class or subclasses
name: Stale

Comments:
@AlexGregory96 are there other concat tests in this file? I think https://github.com/pandas-dev/pandas/blob/b0082d2503a9c5b4732acfe09716f49670e0fe8f/pandas/tests/reshape/concat/test_index.py#L192 would be a more natural location
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
this needs a rebase
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
pls merge master and address comments
thanks @Alex-Gregory-1
DEPR: `Styler.render` in favour of `Styler.to_html`
State: closed

This PR deprecates the method `Styler.render()` in favour of `Styler.to_html()`, which has been recently introduced and produces the same result under the default case of no arguments.

Additionally this PR has added a test to a new file `pandas/tests/io/formats/style/test_deprecated.py`, and it has moved some of the previous Styler deprecation tests into this file.


Labels:
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap
name: Deprecate
description: Functionality to remove in pandas
name: Styler
description: conditional formatting using DataFrame.style

Comments:
can you rebase again, thanks
thanks @attack68
REGR: dont warn on unpickle Timestamp
State: closed

- [x] closes #41949
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Datetime
description: Datetime data dtype
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
BUG: use hash-function which takes nans correctly into account for ExtensionDtype
State: closed

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them


`pandas/tests/arrays/sparse/test_dtype.py test_hash_equal` fails on Python3.10 (see #41940). This is due to the fact, that since Python3.10 different nan-objects have different hashes.

In #41988 a hash function with the old pre-Python3.10 behavior was introduced. This function is used in ExtensionDtype to keep the correct behavior also with Python3.10.


Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: hashing
description: hash_pandas_object
name: Python 3.10

Comments:
thanks @realead 
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
REGR: Series.mode with boolean and pd.NA
State: closed

- [x] closes #42107
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
@jbrockmendel feel free to tag with 1.3 if regressions.
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
DOC: added examples to align function's documentation
State: closed

- [ ] closes #42125
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Docs
name: Stale

Comments:
Hello @meiruv! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

* In the file [`pandas/core/generic.py`](https://github.com/pandas-dev/pandas/blob/7116772be261cf7eb117e64b3dd3abd87e25fe46/pandas/core/generic.py):

> [Line 8547:1](https://github.com/pandas-dev/pandas/blob/7116772be261cf7eb117e64b3dd3abd87e25fe46/pandas/core/generic.py#L8547): [W293](https://duckduckgo.com/?q=pep8%20W293) blank line contains whitespace



##### Comment last updated at 2021-07-13 11:54:34 UTC
Hey @meiruv - are you still working on this? No hurry, just checking if it's still active / if you need help with anything
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
Hi @meiruv - still working on this? Looks like it's pretty close - no hurry if you are, would just be good to know
closing as per https://github.com/pandas-dev/pandas/issues/42125#issuecomment-901025821
Move Check for deprecated messages without sphinx directive to pre-commit
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks
name: Stale

Comments:
Hi Noora - are you still working on this? Need help with anything? (no hurry of course)
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
HI Marco

Just spotted this! I’ll pick up the sprints in September. Go ahead and work on anything I began in the meantime.

-N

> On 11 Jul 2021, at 10:22, Marco Edward Gorelli ***@***.***> wrote:
> 
> 
> Hi Noora - are you still working on this? Need help with anything? (no hurry of course)
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub <https://github.com/pandas-dev/pandas/pull/42128#issuecomment-877768354>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGSVKEDFKRFJYTI3D2DI3PDTXFPDXANCNFSM467HB5EA>.
> 


thanks Noora, no worries - closing for now then
BUG: Unignore .c files under _libs/src. (#42116)
State: closed

- [x] closes #42116

Not 100% what tests I could write for this. This works locally. git status shows no untracked files while `rg on_bad_lines` now shows results from `pandas/_libs/src/parser/tokenizer.c`

Labels:
name: Stale

Comments:
this is to not include the source in the distribution? where is this breaking?
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
@dalejung can you address https://github.com/pandas-dev/pandas/pull/42117#issuecomment-873667846 and merge upstream/master (should fix ci)
Thanks for the PR, but it appears to have gone stale. Let us know if you're still interested in working on this and we can reopen. Closing.
REGR: DatetimeIndex.intersection #42104
State: closed

- [x] closes #42104
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Datetime
description: Datetime data dtype
name: Regression
description: Functionality that used to work in a prior pandas version
name: Frequency
description: DateOffsets
name: Index
description: Related to the Index class or subclasses
name: setops
description: union, intersection, difference, symmetric_difference

Comments:
I've milestoned 1.3, but looks like a trvial backport if we want to include in 1.2.5. 

@jbrockmendel can you add a release note
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
Added stride to iloc documentation
State: closed

Stride examples were added to iloc documentation.

Labels:
name: Docs

Comments:
Thanks for the contribution @bizzyvinci. While I understand that strides on a dataframe may be useful to you, I think it's a rare use-case, and I would prefer not to add extra noise in an already long and complex documentation page. And I don't think it adds much value either, since in the current page we already show that using slices with `iloc` is possible, so I think we don't need to show all possible options of slices, the Python documentation already shows that.

So, -1 on this PR. But I'll wait for a second opinion, and see if other maintainers find this is an improvement.
Agreed especially since slices with `iloc` are already demonstrated. Thanks for the contribution @bizzyvinci but going to close this pull request. We would be happy to have other pull requests closing issues with the label `good first issue`
Ok. Thanks for the feedback.
DOC-Modified documentation for the issue GH42106
State: closed

- [ ] closes #42106 




Labels:
name: Docs
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
> Looks good to me

Hi MarcoGorelli, I just wanted to know when can this PR be merged with the master/main.
Hi MarcoGorelli
Web/doc CI check was failing and as a result I had to update this PR with the master branch. Can you please approve the workflows.
Hey @raghuabhishek 

yup, done

>  I just wanted to know when can this PR be merged with the master/main.

this looks good to me, but I'll leave it open for a bit in case others have comments
> 
> 
> Hey @raghuabhishek
> 
> yup, done
> 
> > I just wanted to know when can this PR be merged with the master/main.
> 
> this looks good to me, but I'll leave it open for a bit in case others have comments

Thanks @MarcoGorelli 
thanks @raghuabhishek 
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
PERF: lib.generate_slices
State: closed

```
import numpy as np
import pandas._libs.lib as lib

np.random.seed(0)
inds = np.random.randint(0, 100, 1000000)
ngroups = inds.max() + 1

%timeit lib.generate_slices(inds, ngroups)
```
Master:
`1.86 ms ± 44 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)`

This pr:
`859 µs ± 127 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)`

Adding the `cython` decorators makes a huge difference here


Labels:
name: Performance
description: Memory or execution speed performance

Comments:
@meeseeksdev backport 1.3.x
thanks @mzeitlin11 
Something went wrong ... Please have  a look at my logs.
REGR: Subclassing `Styler` and addressing `from_custom_template`
State: closed

- [x] closes #42053

More info there - note that this makes the broken functionality re-available but is not backwards compatible due to inherited jinja2 templates.

The documentation  is updated for the correct application.

![Screen Shot 2021-06-17 at 23 48 54](https://user-images.githubusercontent.com/24256554/122476543-b3af0c80-cfc6-11eb-943c-7f3a7fc902e8.png)


Labels:
name: Styler
description: conditional formatting using DataFrame.style

Comments:
can you merge master and have the CI run again.
thanks @attack68 
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
BUG: RecursionError with CategoricalIndex.get_indexer
State: closed

- [x] closes #42088
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Categorical
description: Categorical Data Type

Comments:
cool. merge & backport on green.
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
This fix seems to have caused a large slowdown on the `get_indexer` benchmarks: https://pandas.pydata.org/speed/pandas/#indexing.CategoricalIndexIndexing.time_get_indexer_list?python=3.8&Cython=0.29.21&p-index='monotonic_incr'&commits=cf5852bf-fce7f9eb

The regression overview (https://pandas.pydata.org/speed/pandas/#regressions?sort=1&dir=desc) lists it as a 1000x slowdown, but that's only because https://github.com/pandas-dev/pandas/pull/42042 first improved the performance a lot (which might be a bit suspicious?). Compared to the timing before that, it's only 4-5x slowdown. With the below code, I see locally a ~9x slowdown on master compared to 1.2.5.


```python
import string, itertools
data_unique = pd.CategoricalIndex(
            ["".join(perm) for perm in itertools.permutations(string.printable, 3)]
)
cat_list = ["a", "c"]

%timeit data_unique.get_indexer(cat_list)
52.8 ms ± 5.56 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)  # <-- pandas 1.2.5
417 ms ± 22.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  # <-- master
```

I _think_ it has to do with the fact that before we called the Engine.get_indexer on the codes, while now in the base class version we do that with the `.categories`, which means in this case that both `self` and `target` are cast to object dtype and thus use the Engine.get_indexer for object dtype.


I opened https://github.com/pandas-dev/pandas/issues/42249 to keep track of this.