BrandonFors/pandas-github-issues huggingface dataset:
TYPING: type hints for core.indexing
State: closed

cc @WillAyd 

Any is added in a separate commit in case we can't come to an agreement ;)

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
i wonder if there is a decorator that we could use in tests to turn on reveal_type and show a summary
Can you merge master? Just pushed in change to remove ABCs from _typing so should run CI again to double check
sure
I am with @WillAyd here about the type ignore's. I am afraid that we are never going to revisit these; we did this for a while with flake8 and it was a disaster; so I would either: actually define the type or type somewhat less until this can be fixed.
> I am with @WillAyd here about the type ignore's. I am afraid that we are never going to revisit these; we did this for a while with flake8 and it was a disaster; so I would either: actually define the type or type somewhat less until this can be fixed.

fair enough. but this defeats the purpose of why I was adding type hints... to make the types of the function parameters and return types known to other functions.
Backport PR #27510 on branch 0.25.x (BUG: Retain tz transformation in groupby.transform)
State: closed

Backport PR #27510: BUG: Retain tz transformation in groupby.transform

Labels:
name: Groupby
name: Timezones
description: Timezone data dtype

Comments:
Hmm not sure why the tests didn't run.
Did this need a backport? I think this was just an existing bug. 
I guess https://github.com/pandas-dev/pandas/wiki/Maintainer-Overview#backporting isn't explicit about whether non-regression bug fixes should back ported. For smaller ones like this, I'm personally fine with it, but we should agree as a group.
I had thought backporting was mainly for regression bugs, but I'm fine as well to backport this one. 
we should disable codecov just generally :-<
DOC:Remove DataFrame.append from the 10min intro
State: closed

Remove the `append` section from 10 min intro doc as complexity of that is very different than `list.append`

- [x] closes #27518



Labels:
name: Docs

Comments:
As a note, the DataFrame.append API docs have a good discussion of this https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.append.html#pandas.DataFrame.append.

We need to determine whether the "append" use-case (incrementally building a DataFrame) is important enough to warrant inclusion in the 10min intro.
I think as we already have explained `concat` at https://pandas.pydata.org/pandas-docs/stable/getting_started/10min.html#concat, it should suffice for a beginner and it is anyway better than iteratively using `append`.  
I'd prefer a warning and link to append docs about performance rather than outright removal
CI should be fixed if you fetch and merge master then push.

> I'd prefer a warning and link to append docs about performance rather than outright removal

Note that we do still discuss DataFrame.append in the user guide: https://pandas.pydata.org/pandas-docs/stable/user_guide/merging.html#concatenating-using-append. I just don't think it's valuable enough to include in your first 10 minutes to learning pandas.

That said, we may want to head of the style of iteratively building a DataFrame, like you would with a list, by leaving a comment at the bottom of https://pandas.pydata.org/pandas-docs/stable/getting_started/10min.html#object-creation saying something like

> Adding a column to a DataFrame is relatively fast. Adding a row, however, requires a full copy of the data and a construction of a new Index, which may be expensive. Rather than iteratively building a DataFrame by appending records, we recommend passing the pre-built list of records to the DataFrame constructor. See ... for more.

How does that sound @WillAyd?
Sounds good!
@jreback I have made the required changes.
@TomAugspurger Done!
Thanks @sameshl!
more type hints for io/formats/format.py
State: closed

another pass to follow-on from #27418

Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Thanks! Haven't looked in detail yet but FYI I think we'll want to get #27424 in as a precursor. There's some failures showing up in there in the io.formats file so may uncover some others here as well since that's a central module getting touched

Haven't checked failures in that PR in detail yet so if you have time / interest in taking a look would be appreciated
thanks @simonjayhawkins if you can address the questions (in this PR) in a followup would be great
> thanks @simonjayhawkins if you can address the questions (in this PR) in a followup would be great

no problem. i'm hoping some issues will resolve themselves as more type hints are added to imported functions and the default return type of  `Any` is replaced with more specific types.

so i'm tending towards favoring more liberal use `# type: ignore` in preference to `cast`.

However, I think we may start to encounter more and more issues with the untyped _lib.pyx imports. mypy has good understanding of isinstance, but not of the `is_*` functions.

have spent some time looking at stubbing and overloading, no joy yet. so we may need to add casts after every `is_*` call.
BUG: display.precision of negative complex numbers
State: closed

- [x] closes #27484
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: Complex
description: Complex Numbers

Comments:
LGTM
lgtm. going to see if I can fix the CI on 0.25.1 first.
can you rebase this @mroeschke 
Rebased and green.
Thanks @mroeschke
PERF: speed up MultiIndex.is_monotonic by 50x
State: closed

The current logic for `MultiIndex.is_monotonic` relies on `np.lexsort()` on `MultiIndex._values`. While the result is cached, this is slow as it triggers the creation of `._values`, needs to perform an `O(n log(n))` sort, as well as populate the hashmap of a transient `Index`.

This PR significantly speeds up this check by directly operating on `.codes` when `.levels` are individually sorted. This means we can leverage `libalgos.is_lexsorted()` which is `O(n)` (but has the downside of needing `int64` when `MultiIndex` compacts levels).
```
       before           after         ratio
     [5bd57f90]       [3320dded]
-      31.9±0.9ms         627±50μs     0.02  index_cached_properties.IndexCache.time_is_monotonic_decreasing('MultiIndex')
-      29.6±0.7ms         528±80μs     0.02  index_cached_properties.IndexCache.time_is_monotonic('MultiIndex')
-        29.9±1ms         524±90μs     0.02  index_cached_properties.IndexCache.time_is_monotonic_increasing('MultiIndex')
```

- [ ] closes #27099
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Performance
description: Memory or execution speed performance
name: MultiIndex

Comments:
thanks @qwhelan
PERF: speed up IntervalIndex._intersection_non_unique by ~50x
State: closed

I've been backfilling `asv` data and noticed the following regression in `IntervalIndexMethod.time_intersection_both_duplicate` (see [here](https://qwhelan.github.io/pandas/#index_object.IntervalIndexMethod.time_intersection_both_duplicate?machine=T470-W10&os=Linux%204.4.0-17134-Microsoft&ram=20822880&p-param1=100000&commits=83eb2428-cdc07fde)):
![Screenshot from 2019-07-20 02-30-17](https://user-images.githubusercontent.com/440095/61577005-7407df00-aa96-11e9-9b13-7b323377babd.png)

This regression was missed as the benchmark was added in #26711, which was after introduction in #26225.

This PR both simplifies the `IntervalIndex._intersection_non_unique` logic (now equivalent to `MultiIndex._intersection_non_unique`) and provides a `~50x` speedup:
```
       before           after         ratio
     [9bab81e0]       [2848036e]
     <interval_non_unique_intersection~1>       <interval_non_unique_intersection>
-      12.6±0.1ms         725±30μs     0.06  index_object.IntervalIndexMethod.time_intersection_both_duplicate(1000)
-         4.96±0s         96.7±6ms     0.02  index_object.IntervalIndexMethod.time_intersection_both_duplicate(100000)
```
The new numbers are about `10x` faster than the old baseline.

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Performance
description: Memory or execution speed performance

Comments:
@jschendel 
thanks @qwhelan
Use Matplotlib built-in converters for python date(time)s
State: closed

In Matplotlib we have and maintain unit converters for the python build in datetime objects and `np.datetime64`, so there is no need to provide another converter. This would fix https://github.com/pandas-dev/pandas/issues/27479

Labels:

Comments:
Thanks @dstansby, will take a look.

It sounds like you're more knowledgeable than most of us about matplotlib.  Can you offer any insight into #26676?
Hmm this clearly isn't as easy as I thought it would be, since the pandas converters offer extra functionality.
BUG: quantile segfaults on invalid quantile values
State: closed

- [x] closes #27470
- [ ] waiting for #27584 to be resolved (move `check_percentiles` out of `NDFrame`)
- [ ] waiting for #27526 to be resolved (#20405 cythonized `Groupby.quantile` with functionality regression)
- [ ] whatsnew entry


Labels:
name: Groupby
name: Segfault
description: Non-Recoverable Error

Comments:
fixes this here makes sense.  is the underlying problem in numpy or in one of our cython functions?
>  is the underlying problem in numpy or in one of our cython functions?

I think this line is the problem
https://github.com/pydata/pandas/blob/d48306e77c0a708e0ee33a2aa1da7e267df52ef6/pandas/_libs/groupby.pyx#L758-L759

it calculates an index by using the quantile value as a fraction of array length.

> if we were to move that to PandasObject to share across NDFrame and GroupBy would that be even 
> simpler?

Yeah, `_check_percentile` is what this does. I don't think it fits in `PandasObject`, maybe in lib.pyx instead? it lives on NDFrame but is actually a standalone static method.
Let's put the check in the cython function, so as to preclude any chance of the segfault showing up, e.g. if we call it from somewhere else
Logical, but thorny. Is the overall approach in pandas to put input validation in cython in the python  caller? could also usea python wrapper which is the only cython caller.

@WillAyd  suggests reusing `NDFrame._check_percentiles` instead. It's often inconsistent standards about where validation should happen that creates these sort of bugs, so maybe a bigger issue here.

Whatever there's agreement on is fine by me.
> but thorny

what is thorny about it?  I haven't looked at this too specifically, just in principle would rather segfault-prevention happen as close to the fault as possible
blocked by #27526
waiting for #27584
This was a simple fix, but it now depends on too many other small things. I'm sure someone else will get it eventually.
Groupby transform cleanups
State: closed

- [x] whatsnew

Related #27389 (several issues there)
closes #27486

Labels:
name: Groupby
name: API Design

Comments:
Hello @pilkibun! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

















There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-07-25 18:18:14 UTC
Can you write release notes for each of the user-facing changes? If you merge master we should have a 0.26.0.rst now.
this also needs a whatsnew note. I am not so concerned about the invalid case (e.g. .transform('foo')), but something that looked legitimate and maybe even did *something*, though it may have been wrong, is now raising
@jreback green
thanks @pilkibun 
:+1:
CLN: remove unused row_bool_subset
State: closed

@jreback can you confirm my hunch that the `result.ndim == 2` block in groupby.ops was for Panel?

Tracing back the blame, row_bool_subset was introduced [here](https://github.com/pandas-dev/pandas/commit/b15ae85e1f647821755d35bedd6143ad1e9c3678) in 2012 to close #152 and it started being called only for result.ndim == 2 [here](https://github.com/pandas-dev/pandas/commit/9bfdc3c037e926143e708463d76fcdae3fb709f2)

Labels:
name: Groupby
name: Clean

Comments:
Haven't looked deeply but seems like this is more for `ohlc` than Panel right?
I speculate that it’s for Panel because if it is an agg op that should be a reduction, so we shouldn’t get a 2D result unless we started with 3D.  But “speculate” is definitely the important word in that sentence
> so we shouldn’t get a 2D result unless we started with 3D

Yea that's not quite true with ohlc though. To illustrate:

```python
In [20]: ser = pd.Series([1], index=[pd.Timestamp('today')])

In [21]: ser.groupby([1]).ohlc()
Out[21]:
   open  high  low  close
1     1     1    1      1
```

Not sure what would specifically hit this branch but I think ohlc test coverage is weak so just need to be careful
Dangit Will, you're gonna make me go and do actual critical thinking instead of mechanically reading the coverage results...
Hmm that example doesn't break on this branch, so must not go through this code path.  Still, the point that test coverage is spotty is well-taken.
thanks @jbrockmendel 

I don't really remember where this was used, does seem like its related to Panel though.
Remove .. versionadded:: 0.18
State: closed

Removed ``.. versionadded:: 0.18.0`` and related text blocks.

Labels:
name: Docs

Comments:
is the policy something like keeping these for N releases?  if so, is/should these be checked for in code_checks?
There isn't a policy at the moment, the previous rounds I've just removed on my own initiative.

I'm not convinced that a policy check is practical, there could be many minor versions or fewer larger versions, and a check wouldn't catch that.
> There isn't a policy at the moment, the previous rounds I've just removed on my own initiative.

I think it makes to create a policy, but we can do that afterwards.  That being said, `0.18.0` is pretty safe to remove.  I would consider `0.19.x` as well (either here or in a new PR).
thanks @topper-123 

yeah certainly anything < 0.20.3 is fair game at this point.
CLN: avoid runtime imports
State: closed

Edits to lib.is_scalar are to put slower python-space checks later so they may get short-circuited.

Everything else should be self-explanatory.

Labels:
name: Performance
description: Memory or execution speed performance

Comments:
> any perf differences?

PR
```
In [3]: %timeit is_scalar(None)                                                                                                                                                                                                                   
55.1 ns ± 0.745 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)
```

master
```
In [3]: %timeit is_scalar(None)                                                                                                                                                                                                                   
588 ns ± 25.7 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
```

Presumably anything that is caught by the `isinstance(Fraction, Number)` would take a hit, but the difference when I pass an int or float is within the margin of error, and it isn't clear what else would be caught by that isinstance check.  Also no difference when passing a list, which is what we'd expect.
i will look at things tomorrow
ping
thanks
Fix #27451 (doc + validation)
State: closed

- [X ] closes #27451 
- [ X] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
Hello @ddebernardy! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-07-19 07:06:38 UTC
Can you add a test to ensure that this raises?
@jschendel @WillAyd 

> Can you add a test to ensure that this raises?

There is one already.

https://github.com/pandas-dev/pandas/blob/master/pandas/tests/generic/test_generic.py#L361

I imagine the negative `float` is currently getting caught as a negative `n` somewhere downstream in the code.
Looks like doctest is failing:

https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=14665&view=logs&jobId=435ca956-9126-505f-566f-fff31072e2ba

Can you merge master and take a look?
@ddebernardy can you rebase
Thanks for the PR! Looks like this has gone stale though so closing. ping if you'd like to pick it back up
PERF: restore performance for unsorted CategoricalDtype comparison
State: closed

Fixes a performance regression introduced in #26403 very shortly before `0.25.0rc0` was cut, which can be seen [here](https://qwhelan.github.io/pandas/#indexing.CategoricalIndexIndexing.time_getitem_slice?machine=T470&num_cpu=4&os=Linux%205.0.0-20-generic&ram=20305904)
![Screenshot from 2019-07-17 23-29-08](https://user-images.githubusercontent.com/440095/61434418-c0aeb700-a8ea-11e9-8f02-8b7bb6dee96f.png)

When comparing `CategoricalDtype`s with `ordered=False` (the default), a hash is currently done that is relatively slow even for a small number of categories. If we check if the categories happen to be the same and in the same order, we see a significant speedup in the equal case. The `.equals()` function pre-exits, so overhead in the non-equal case should be minimal.

`asv` results:
```
       before           after         ratio
     [a4c19e7a]       [6810ff2f]
     <unsorted_cats~1>       <unsorted_cats>
-       179±0.4μs       25.6±0.4μs     0.14  indexing.CategoricalIndexIndexing.time_getitem_slice('monotonic_decr')
-         179±2μs         25.2±1μs     0.14  indexing.CategoricalIndexIndexing.time_getitem_slice('monotonic_incr')
-       180±0.4μs       25.1±0.3μs     0.14  indexing.CategoricalIndexIndexing.time_getitem_slice('non_monotonic')
```

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Performance
description: Memory or execution speed performance
name: Categorical
description: Categorical Data Type

Comments:
thanks @qwhelan
BUG: avoid incorrectly inheriting result
State: closed

This has caused problems in a number of local branches, but I don't have a good test case on master.  Suggestions welcome.  Still though, the failure mode is pretty clear, so this fixes it.

Labels:
name: Groupby
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
Hmm not immediately apparent here but what problems does this fix?
> Hmm not immediately apparent here but what problems does this fix?

You're right that ATM this is a solution in search of a problem.  This becomes necessary in the branch that is the follow-up to #27428 (which is itself a follow-up to #27411).  I can close and re-open when it becomes actually necessary, but am hoping I can find a useful example in the interim, since the failure mode is clear.
> if you have a test for this would be best, but if you can revise sligthly can push thru

Yah, I'd rather leave this open for now until it becomes necessary (following #27428) or I can come up with a test.
> > if you have a test for this would be best, but if you can revise sligthly can push thru
> 
> Yah, I'd rather leave this open for now until it becomes necessary (following #27428) or I can come up with a test.

not averse but if we can't break the current code then maybe wait on this.
Closing in favor of #27567, which includes this as a subset
CLN: simplify maybe_convert_objects, soft_convert_objects
State: closed

Block.replace has code for dispatching to either maybe_convert_objects or soft_convert_objects, but the former is only reached from the tests.  So rip that out and use soft_convert_objects.

Both of the functions have branches that are unreachable, so rip those out, along with adding types and validation.

Labels:
name: Internals
description: Related to non-user accessible pandas implementation
name: Clean

Comments:
ping
thanks
TST: Add test for operations on DataFrame with Interval CategoricalIndex
State: closed

- [x] closes #27415
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Categorical
description: Categorical Data Type
name: Interval
description: Interval data type
name: DataFrame
description: DataFrame data structure

Comments:
Yah these definitely don't belong in tests.indexing.  Is there anything DataFrame-specific about them, or could similar tests be written for Series etc?  If the latter, then these would go in tests.arithmetic.  If they are DataFrame-specific, then tests.frame.test_arithmetic
thanks @yanglinlee
BUG: fix+test quantile with empty DataFrame, closes #23925
State: closed

- [x] closes #23925
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Bug
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff
name: DataFrame
description: DataFrame data structure

Comments:
thanks @jbrockmendel
PLT: Delegating to plotting backend only plots of Series and DataFrame methods
State: closed

- [X] xref #26747
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Visualization
description: plotting

Comments:
Sounds good.

On Wed, Jul 17, 2019 at 10:45 AM Marc Garcia <notifications@github.com>
wrote:

> *@datapythonista* commented on this pull request.
> ------------------------------
>
> In pandas/plotting/_core.py
> <https://github.com/pandas-dev/pandas/pull/27432#discussion_r304482395>:
>
> > @@ -375,7 +375,7 @@ def boxplot(
>          >>> type(boxplot)
>          <class 'numpy.ndarray'>
>      """
> -    plot_backend = _get_plot_backend()
> +    plot_backend = _get_plot_backend("matplotlib")
>
> I think this function is pandas.plotting.boxplot(df) and not df.boxplot()
> which iirc is boxplot_frame, and in this PR still delegates the call to
> the backend.
>
> Personally I'd deprecate pandas.plotting.boxplot more than anything else.
> Afaik it's the same as df.boxplot() or df.plot(kind='box'), each with a
> different implementation. But if we don't deprecate, my preference would be
> to try to make things as simple as possible, and avoid special cases.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/27432?email_source=notifications&email_token=AAKAOIWWMPLROLNDO4SSWNTP745BHA5CNFSM4IERMUL2YY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOB6XUVOA#discussion_r304482395>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKAOITWMXEKVFIO4XVD5KDP745BHANCNFSM4IERMULQ>
> .
>
CLN/REF: stop allowing iNaT in DatetimeBlock
State: closed

Sits on top of #27411 

xref #16674.  The broader goal is de-special-casing code in internals.


Labels:
name: Internals
description: Related to non-user accessible pandas implementation
name: Clean

Comments:
ok will look after resolve / merge #27411 
lgtm. can you rebase just to be sure. ping on green.
ping
thanks