ZeroDivisionError when groupby rank with method="dense" and pct=True
State: closed

- [x] closes #23666 
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Groupby
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
the first thing to always write is a test
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/23864?src=pr&el=h1) Report
> Merging [#23864](https://codecov.io/gh/pandas-dev/pandas/pull/23864?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/20ae4543c1d8838f52229830bfae0cc8626801bb?src=pr&el=desc) will **increase** coverage by `0.06%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/23864/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/23864?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #23864      +/-   ##
==========================================
+ Coverage   92.28%   92.35%   +0.06%     
==========================================
  Files         161      161              
  Lines       51500    51557      +57     
==========================================
+ Hits        47528    47613      +85     
+ Misses       3972     3944      -28
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.75% <ø> (+0.06%)` | :arrow_up: |
| #single | `42.46% <ø> (+0.14%)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/23864?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/formats/console.py](https://codecov.io/gh/pandas-dev/pandas/pull/23864/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvY29uc29sZS5weQ==) | `74.24% <0%> (-1.52%)` | :arrow_down: |
| [pandas/core/arrays/timedeltas.py](https://codecov.io/gh/pandas-dev/pandas/pull/23864/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL3RpbWVkZWx0YXMucHk=) | `96% <0%> (-0.45%)` | :arrow_down: |
| [pandas/plotting/\_misc.py](https://codecov.io/gh/pandas-dev/pandas/pull/23864/diff?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19taXNjLnB5) | `38.68% <0%> (-0.31%)` | :arrow_down: |
| [pandas/core/indexes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/23864/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9iYXNlLnB5) | `96.32% <0%> (-0.17%)` | :arrow_down: |
| [pandas/core/arrays/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/23864/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2RhdGV0aW1lcy5weQ==) | `98.37% <0%> (-0.14%)` | :arrow_down: |
| [pandas/tseries/offsets.py](https://codecov.io/gh/pandas-dev/pandas/pull/23864/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvb2Zmc2V0cy5weQ==) | `96.84% <0%> (-0.14%)` | :arrow_down: |
| [pandas/core/ops.py](https://codecov.io/gh/pandas-dev/pandas/pull/23864/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvb3BzLnB5) | `94.14% <0%> (-0.14%)` | :arrow_down: |
| [pandas/core/config.py](https://codecov.io/gh/pandas-dev/pandas/pull/23864/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29uZmlnLnB5) | `87.04% <0%> (-0.13%)` | :arrow_down: |
| [pandas/io/sas/sas\_xport.py](https://codecov.io/gh/pandas-dev/pandas/pull/23864/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfeHBvcnQucHk=) | `90.14% <0%> (-0.1%)` | :arrow_down: |
| [pandas/core/groupby/ops.py](https://codecov.io/gh/pandas-dev/pandas/pull/23864/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9vcHMucHk=) | `96.72% <0%> (-0.07%)` | :arrow_down: |
| ... and [74 more](https://codecov.io/gh/pandas-dev/pandas/pull/23864/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/23864?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/23864?src=pr&el=footer). Last update [20ae454...74f9e0b](https://codecov.io/gh/pandas-dev/pandas/pull/23864?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Hello @Koustav-Samaddar! Thanks for updating the PR.

 - There are no PEP8 issues in the file [`pandas/tests/groupby/test_rank.py`](https://github.com/Koustav-Samaddar/pandas/blob/88f5cb46af13af7aeb1c9143196491d41e467b83/pandas/tests/groupby/test_rank.py) !



##### Comment last updated on December 01, 2018 at 20:52 Hours UTC
Also make sure you check the Travis failures. Think you have a styling issue somewhere
> Also make sure you check the Travis failures. Think you have a styling issue somewhere

I skimmed through the Travis logs and wasn't able to find (/didn't understand) what the issue was. I didn't modify any of the source files that I saw were mentioned in the logs. There was an error/issue raised by scripts/tests/test_validate_docstrings.py but I couldn't understand the logs from it.

I rechecked with pep8 and flake8 and couldn't find anything sticking out.
The failure is in `ci/code_checks.sh` if you run locally should give you visibility into issue.

@datapythonista FYI failures in that script don't provide a useful error message in CI currently. Not sure if that's on your radar
@WillAyd the error message `./pandas/tests/groupby/test_rank.py:304:44: W292 no newline at end of file` is coming from `flake8`.

If you mean that they are difficult to find, that's addressed in #22854, which should make them much easier and quicker to find. We have a permissions problem with Azure, that I expect to be fixed early next week, so with some luck we have it merged very soon.
I ran `LINT=true ci/code_checks.sh` on my machine and it didn't fail anything. However, on code submission, Travis is still failing.
The travis failure was unrelated to the linting, it was caused by a url that could not be fetched. I rerun it let's see if it succeeds now.
> grp_size[i, 0] shouldn't be 0 but is in the case where there is a group with only one member with a non NaN value and another index also has the same value but in a different group && method="density". In this case the correct grp_size[i, 0] should be 1 and it being 0 is what causes BUG 23666.

I don't buy this. why would a grp_size *ever* be 0?
> > grp_size[i, 0] shouldn't be 0 but is in the case where there is a group with only one member with a non NaN value and another index also has the same value but in a different group && method="density". In this case the correct grp_size[i, 0] should be 1 and it being 0 is what causes BUG 23666.
> 
> I don't buy this. why would a grp_size _ever_ be 0?

If I have a group where all the values are `NaN`s, then `grp_size[i, 0]` should be zero. However, the `if out[i, 0] == NaN` block makes it so that all the members (not sure if right term) of such a group get caught there, and never make it past to the check of `if grp_size[i, 0] == 0`.

TL;DR there are valid cases where `grp_size[i, 0] == 0` but they will never enter the condition that checks `if grp_size[i, 0] == 0`
@datapythonista  When I run ci/code_checks.sh on my local machine it doesn't raise any errors. However, in Travis I'm getting `The command "ci/code_checks.sh" exited with 1.`. I still am unable to ascertain from the Travis logs what's going wrong/failing.

Any help is greatly appreciated!
> there are valid cases where grp_size[i, 0] == 0 

and what is an example of this
don’t reference the existing code
rather show an input an output that has no group size yet has a result
> > there are valid cases where grp_size[i, 0] == 0
> 
> and what is an example of this
> don’t reference the existing code
> rather show an input an output that has no group size yet has a result

Here's an example `df`:
```
   A    B
0  1  3.0
1  1  4.0
2  2  NaN
3  2  NaN
```

If we were to do `df.groupby("A")` this would result in two groups:
`{1: Int64Index([0, 1], dtype='int64'), 2: Int64Index([2, 3], dtype='int64')}`

Here, the second group (`A == 2`) is an example of a valid group with `grp_size[i, 0] == 0`.

Hope this helps!

and what would the result frame look like 
Sorry again.
On running `df.groupby('A').rank(method="dense", pct=True)` the correct result should be:
```
     B
0  0.5
1  1.0
2  NaN
3  NaN
```

I don't know how to link a snippet of code from a commit, but in test_rank.py I changed the testing function to include the permutations of grp_sizes and NaNs that you requested in a previous comment.
@Koustav-Samaddar ok that result looks ok to me. still have some comments.
lgtm. @WillAyd over to you.
Thanks @Koustav-Samaddar !