BrandonFors/pandas-github-issues huggingface dataset:
DOC: Remove multiple blank lines in ipython directives
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

Related to #26788

This PR gets rid of all warnings of the form "UserWarning: Code input with no code at..." when building the docs. When there are multiple blank lines, e.g.

```
foo = 5


def bar():
    pass
```

one of the blank lines is reported as having no code. That said, I'm not sure if this is the right approach, as the double blank likes are PEP8 compliant.  Perhaps this should be reported upstream instead and the multiple blank lines left in the docs.

cc @jorisvandenbossche @TomAugspurger 


Labels:
name: Docs
name: CI
description: Continuous Integration

Comments:
I would argue that since *.rst documents are not code some flexibility is allowed. Especially since the examples are rather code snippets, often without context, their structure might also be unknown-context based, e.g. function separation in a class is 1-line and in a module 2-lines. Since PEPs purpose is primarily to aid readability, and readability doesn't seem to suffer here I'd favour the changes. Possibly the readability even improves and it gets rid of the warnings! 👍 

Originally, there was a problem with the IPython directive and using single blank lines (https://github.com/ipython/ipython/issues/11362). I think that's the reason we removed any blank line around functions, and with some auto formatting that changed to 2 blank lines. 

In any case, as long as it now works properly (no error in the resulting HTML), I am perfectly fine with a single blank line (I agree that for examples this is totally fine)
> I would argue that since *.rst documents are not code some flexibility is allowed. Especially since the examples are rather code snippets, often without context, their structure might also be unknown-context based, e.g. function separation in a class is 1-line and in a module 2-lines. Since PEPs purpose is primarily to aid readability, and readability doesn't seem to suffer here I'd favour the changes. Possibly the readability even improves and it gets rid of the warnings! 👍

sure in an ideal world. but we need to be as mechanically checked as possible. e.g. different devs can have different opinons on this (and they maybe dated, esp mine, haha). 

so we ideally have a single canonical checked way of doing things.

in this case I dont' think we actually check this but maybe would be nice to.
thanks @rhshadrach
DOC: Remove deprecated usage of Index.__xor__
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

ref #37374 cc @jbrockmendel 


Labels:
name: Deprecate
description: Functionality to remove in pandas
name: Index
description: Related to the Index class or subclasses

Comments:
thanks @rhshadrach
DOC: specify regex=True in str.replace
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

ref #36695

When passing a callable repl, str.replace will raise if regex is False. So it seems to me that in the future, when the default is changed, we should be ignoring the regex argument completely (and update the documentation to that effect). This is slightly magical, but I think better than having a default that raises.

If this is the correct way forward, then we don't need to warn when repl is callable.

cc @dsaxton @jorisvandenbossche


Labels:
name: Docs
name: Strings
description: String extension data type and string data
name: Deprecate
description: Functionality to remove in pandas

Comments:
@rhshadrach If I'm understanding correctly, I think we would not want to skip this warning when a callable is passed, because that behavior will go from not raising to raising in the future. That way users who are implicitly relying on this won't be surprised when suddenly their code starts failing. It almost feels like the warning message could even be _more_ verbose in this case.

Not sure if we'd want to update this here too, but I just noticed the docstring for this method is grammatically a bit odd:
https://github.com/pandas-dev/pandas/blob/67c9385787c4d854b75a6f2c04fdf6886fdb1a06/pandas/core/strings/accessor.py#L1234
Thanks @dsaxton - I think that would not be the case if we followed my suggested way forward in the OP (and comment in the code, but perhaps that comment is too terse). Thoughts on this approach, where we would be _ignoring_ the regex keyword when repl is callable?

> Not sure if we'd want to update this here too, but I just noticed the docstring for this method is grammatically a bit odd

I think this would be okay to address here, will do so.

> I think that would not be the case if we followed my suggested way forward in the OP (and comment in the code, but perhaps that comment is too terse). Thoughts on this approach, where we would be _ignoring_ the regex keyword when repl is callable?

Ah, I misunderstood. Personally I like the idea of being consistent in how `regex` is handled and breaking loudly if a callable is passed along with `False` for `regex` (either implicitly or not). I think it'd make the method easier to reason about, and forces end users to be explicit about what they want to do.
Makes sense @dsaxton - updated this PR to only fix documentation.
yep this lgtm. thanks @rhshadrach
DOC: Remove deprecated use of level for agg functions
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

ref #40869, cc @phofl

Labels:
name: Docs
name: Groupby
name: Deprecate
description: Functionality to remove in pandas

Comments:
I assumed these would give warnings when building the docs, and there weren't any. Just did some searching through the userguide and didn't find any. However, there are documented cases of `Series.sub` and `Series.div` using level. I think these are not deprecated (but maybe should be as well?)
Sorry I forgot this. I changed them to using groupby when deprecating the keyword

Edit: Yes they are not deprecated
> I assumed these would give warnings when building the docs, and there weren't any. Just did some searching through the userguide and didn't find any. However, there are documented cases of `Series.sub` and `Series.div` using level. I think these are not deprecated (but maybe should be as well?)

`.sub` and so on are not deprecated; I think we need the level keyword (as it provides broadcasting behavior)
thanks @rhshadrach
[ArrowStringArray] TST: parametrize str.split tests
State: closed

the test (and benchmark) changes broken off from #41085 as a precursor to #41085 and #41372 (which currently makes changes to the str.split path, although I may break that PR up also)

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Strings
description: String extension data type and string data

Comments:
these are the current timings for master (will hopefully get improvements for object fallback by breaking off changes in  #41372 and also by using pyarrow kernels #41085)

```
 25.00%] ··· strings.Split.time_rsplit                                                                                                                   ok
[ 25.00%] ··· ============== ========== ==========
              --                     expand       
              -------------- ---------------------
                  dtype         True      False   
              ============== ========== ==========
                   str        82.2±0ms   33.5±0ms 
                  string      66.9±0ms   29.9±0ms 
               arrow_string   88.0±0ms   33.7±0ms 
              ============== ========== ==========

[ 50.00%] ··· strings.Split.time_split                                                                                                                    ok
[ 50.00%] ··· ============== ========== ==========
              --                     expand       
              -------------- ---------------------
                  dtype         True      False   
              ============== ========== ==========
                   str        107±0ms    56.8±0ms 
                  string      92.4±0ms   54.3±0ms 
               arrow_string   111±0ms    58.0±0ms 
              ============== ========== ==========
```
Thanks Simon!
REF: add internal mechanics to separate index/columns sparsification in Styler
State: closed

This is not a user facing change.

It adds only the necessary code to permit separate control of index/column MultIndex sparsification. Essentially it removes the `get_option("display.multi_sparse")` from a low level method to a higher level method where it can be overwritten by user-args later.

partly addresses #41142 

Labels:
name: Refactor
description: Internal refactoring of code
name: Styler
description: conditional formatting using DataFrame.style

Comments:
@jreback ping this is green and your requested changes made.
thanks @attack68
REF: re-use machinery for DataFrameGroupBy.nunique
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Groupby

Comments:
wow cleaned that up
TYP: SelectionMixin
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

@simonjayhawkins thoughts on the FrameOrSeries vs FrameOrSeriesUnion that ive ignored here?

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
@jbrockmendel i've pushed a commit removing the ignores. feel free to revert and do that as a follow-up.
Neat thanks
yeah thanks @jbrockmendel and @simonjayhawkins this is like magic.
DEPR: kind kwarg in _maybe_cast_slice_bound
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

When I tried just removing it that broke some dask tests.

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Deprecate
description: Functionality to remove in pandas

Comments:
sure
REF: document casting behavior in groupby
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Groupby
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
and perf question
updated + green
REF: do less in Grouping.__init__
State: closed

making it incrementally easier to reason about what `self.grouper` is

Labels:
name: Groupby

Comments:
pls rebase (though i see a similar PR adding some of this machinery), so either ok (suspect they overlap a bit)
rebased + green
@jreback gentle ping; this has become a blocker for #41373
thanks. not super happy with adding state, but as you are working towards more robust ways this is fine.
BUG: loc casting to float for scalar with MultiIndex df
State: closed

- [x] closes #41369
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

In theory this would solve the issue, but I am not sure if this is desirable. We cast the row to a series, which forces the dtype conversion. If we loop in reverse we retrieve a column as a series which would avoid the conversion. Would you mind having a look @jbrockmendel ?

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: MultiIndex

Comments:
can you rebase
Done
@jreback greenish, failure unrelated
@phofl can you merge master to resolve conflicts
yeah this looks good (just resolved conflicts)
merged master
Thanks @phofl
ENH: retain masked EA dtypes in groupby with as_index=False
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Enhancement
name: Groupby

Comments:
can you rebase and will look again
can you rebase again
just pushed.  this should be correct, but im also working on another branch that could serve as a state-reducing preliminary
Question: it's certainly nice to already preserve the dtype with `as_index=False`, but is it worth adding special code for it now if supporting EAs in the Index will also solve it? (assuming it does)
> Question: it's certainly nice to already preserve the dtype with as_index=False, but is it worth adding special code for it now if supporting EAs in the Index will also solve it? (assuming it does)

Good question.  Better to think of this as a disentangling/refactor of BaseGrouper/Grouping with a bonus upside of fixing the broken cases (in fact,  #41529 does most of this _except_ for fixing the as_index cases).  If/when we get the general EA Index , we'll be able to come back and trim this down some more.
gentle ping; id like to reclaim momentum on the groupby stuff
might call this a bug fix
> might call this a bug fix

i think id rather call it an ENH, dont want users to think that other functions for which we havent yet done this are bugs
CLN: Remove raise if missing only controlling the error message
State: closed

- [x] xref #41170

The removed tests don't make sense anymore, since all is printed now but without a fixed order

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Clean

Comments:
thanks @phofl 
Did this cause a performance regression?

https://pandas.pydata.org/speed/pandas/index.html#indexing.NumericSeriesIndexing.time_loc_slice?python=3.8&Cython=0.29.21&p-index_dtype=%3Cclass+%27pandas.core.indexes.numeric.UInt64Index%27%3E&p-index_structure=%27nonunique_monotonic_inc%27&commits=67c9385787c4d854b75a6f2c04fdf6886fdb1a06
This tick really looks similar to the one a few weeks earlier. But I will have a look.

Practically this should only impact cases where a KeyError is raised
@TomAugspurger if you wouldn't mind creating an issue about this regression 
Yeah, looks like a false alarm. The latest benchmarks are back to normal.
oh ok great!
Pin fastparquet to leq 0.5.0
State: closed

- [x] xref #41366

Pinning for now to get ci passing

Labeled the issue with 1.3 for now so that we don't forget it

Labels:
name: CI
description: Continuous Integration
name: Dependencies
description: Required and optional dependencies

Comments:
great merge on green.
Shoould we backport? CI on 1.2.x is currently failing
@meeseeksdev backport 1.2.x
r
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:

Comments:
@taytzehao it would be preferable if you could experiment with changes and pull requests in your own github account, and submit pull requests to pandas only when they are ready and viable for developers to review. Thanks.
CLN: groupby
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Groupby

Comments:
web/docs failing? weird. can you rebase
rebased + green
REF: test_invalid_dtype in numeric index tests
State: closed

The invalid dtype tests for all the numeric index types are currently done in `tests.numeric.test_numeric.py::TestFloat64Index::test_invalid_dtype`. This PR moves them to the appropriate test class (`TestInt64Index` etc.).

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Index
description: Related to the Index class or subclasses

Comments:
thanks @topper-123
ENH: Implemented optional initializer for pd.concat (#40820)
State: closed

- [x] closes #40820 
- [x] tests added / passed (passed on azure checks)
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

A proposed solution for issue #40820, to accept an optional argument ``initializer`` which allows for concatenating empty sequences and return the object passed to initializer -- which can be useful for cases where users are running pd.concat on sequence variables which might potentially be empty -- and would like the result of the concatenation to be initialized to an empty Series/DataFrame

The new argument is made optional, and the ValueError is still thrown for cases where the ``initializer`` is not specified -- this should maintain expected behaviour / backward compatibility 


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
As stated in https://github.com/pandas-dev/pandas/issues/40820#issuecomment-834604151, I don't think `concat` should be responsible for determining the return type for an empty input.
> @yushao2 sympathize with the issue. we don't anything like this and don't think have this magically just work is a good idea. not to mention adding complexity.

no worries, I will rescind the PR :) cheers
whatsnew about new engine param for to_sql (follow up to #40556)
State: closed

- [ ] xref #36893
- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

Follow up to #40556, per this comment https://github.com/pandas-dev/pandas/pull/40556#issuecomment-831413175

Labels:
name: Docs
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
@jreback here's the follow up PR with whatsnew
thanks @yehoshuadimarsky
BUG : Min/max markers on box plot are not visible with 'dark_background' (#40769)
State: closed

- [x] closes #40769
- [x] tests added / passed
- [x] Ensure all linting tests pass, see here for how to run them

Color in boxplot.py should be a hex value. Letter based color will be at matplotlib level. Using same color as that of boxes for the caps as well. The color will be fetched based on the selected theme. We are over riding color with "k" , rather need to use the one set at matplotlib level.
Before Fix :

Caps for the box plot is not visible :
![![image](https://user-images.githubusercontent.com/9417467/117279259-2634b480-ae7f-11eb-933c-350cff0133de.png)](https://user-images.githubusercontent.com/9417467/117279207-1a48f280-ae7f-11eb-9f0f-fe477d80dc77.png)


After Fix :

Caps for the box plow will be visible in dark background mode :
![image](https://user-images.githubusercontent.com/9417467/117279282-2df45900-ae7f-11eb-8107-9fe3ae2fe15f.png)


Whats New : Box Plot's Caps will have same color as boxes unless color is explicitly specified by user arguments. Hence theme changes will not adversly effect caps color.

Labels:
name: Bug
name: Visualization
description: plotting

Comments:
sorry, @yeshsurya a bit unlucky, I _think_ this failure isn't related to your change, probably you will need another rebase 
thanks very much, very nice!
REF: move masked dispatch inside _ea_wrap_cython_operation
State: closed

After the groupby cleanups, masked dispatch can be moved inside `_ea_wrap_cython_operation` without adding args. Seemed like the preferred place for it in the original pr.

Labels:
name: Refactor
description: Internal refactoring of code
name: Groupby
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
thanks @mzeitlin11
TST/CLN: remove frame replace class
State: closed

Class doesn't seem to have any purpose here


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Clean
name: replace
description: replace method

Comments:
is there any meaningful organization of the tests within the class?
> is there any meaningful organization of the tests within the class?

Not currently. There are a bunch of tests for regex-related functionality, so that at least could be split out into its own file if you think that makes sense.
> Not currently. There are a bunch of tests for regex-related functionality, so that at least could be split out into its own file if you think that makes sense.

We put the file-per-method structure in place last year.  Unless there's a compelling reason, id prefer not to break that pattern, but instead use classes to organize tests within those files if necessary.
That makes sense, will close this then and put up something later to try to split this up
BTW that test organization process is ongoing.  if this is something you're interested in im happy to point you at parts of the tests that havent gotten much attention
Yeah always happy to know where time/attention is most valuable
> where time/attention is most valuable

FWIW i think of test organization less about being high priority than as being something useful i can do when burned out on more difficult tasks.

A few areas of test organization that i either didnt get to or stalled out on
- tests/arithmetic/ there are many comments to the effect of "TODO: parametrize or de-duplicate".  There's tons of these and a lot of duplication.
- groupby/, apply/, resample/, reshape/ i haven't looked at the test organization at all, no idea if these have a a reasonable layout (roughly defined as "if i want to know if we have tests for X, is it obvious where to look?")
- many tests in frame/methods/test_foo and series/methods/test_foo can likely be shared
Thanks that's good to know will keep in mind!
BUG: replace with regex raising for StringDType
State: closed

- [x] closes #35977, closes #41333
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

2 separate issues at play here - regex was ignored in #41333 (specific to the `replace_list` path) but then if `replace_regex` was hit with anything but an object type that would just raise.


Labels:
name: Bug
name: Strings
description: String extension data type and string data
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: replace
description: replace method

Comments:
> Thanks @mzeitlin11 generally lgtm for the fix in hand, although we should probably be dispatching to the `_str_replace` method on the EA.

Agreed, but I think that decision could be handled independently of fixing this. There's some additional complexity because I think `_str_replace` assumes replacement occurs with a string, which is not guaranteed by the general replace function (though it would be a strange thing to do). 
> Agreed, but I think that decision could be handled independently of fixing this.

sure.
thanks @mzeitlin11 very nice!