ENH/API: accept list-like percentiles in describe (WIP)
State: closed

Closes https://github.com/pydata/pandas/issues/4196

This is for frames. I'm going to refactor this into generic since to cover series / frames.

A couple questions:
- API wise, I added a new kwarg `percentiles`. For backwards compat, we keep the `percentile_width` kwarg. I changed the default `percentile_width` from 50 to `None` (but the default output is the same) Cases:
  1. You specify `percentile_width` and `percentiles` -> `ValueError`
  2. You specify neither `percentile_width` nor `percentiles` -> `percentile_width` set to 50 and same as before
  3. You specify one of those, everything goes as expected.
- I'm accepting either decimals (e.g. [0.25, .5, .75]) or percentages (e.g. [25, 50, 75]). Those two are equivalent in output. Should I move this logic to the `.quantile` to be more consistent?
- I'm choosing to not sort the provided percentiles. It's easy for the user to sort, but hard for them to unsort if for some reason they want it in a specific order. I'd rather not add another kwarg.


Labels:
name: API Design
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
I'm close on this... a quick question though. There's also a describe for object types (strs or datetime). Notice the different index order for the last one (this is all current behavior):

``` python


# strs only
In [31]: df2 = pd.DataFrame({"C2": ['a', 'a', 'b', 'c']})

In [32]: df2.describe()
Out[32]: 
       C2
count   4
unique  3
top     a
freq    2

[4 rows x 1 columns]

# datetime only
In [28]: df = DataFrame({"C1": pd.date_range('2010-01-01', periods=4, freq='D')})

In [29]: df
Out[29]: 
          C1
0 2010-01-01
1 2010-01-02
2 2010-01-03
3 2010-01-04

[4 rows x 1 columns]

In [30]: df.describe()
Out[30]: 
                         C1
count                     4
unique                    4
first   2010-01-01 00:00:00
last    2010-01-04 00:00:00
top     2010-01-01 00:00:00
freq                      1

[6 rows x 1 columns]

# mix of timestamp and strs
In [33]: df = pd.concat([df, df2], axis=1)

In [35]: df.describe()
Out[35]: 
                         C1   C2
count                     4    4
first   2010-01-01 00:00:00  NaN
freq                      1    2
last    2010-01-04 00:00:00  NaN
top     2010-01-01 00:00:00    a
unique                    4    3

[6 rows x 2 columns]
```

So the index gets sorted. Is it worth breaking backwards compat to keep the index in a sensible order? I'm not sure.

yeh these should be in a sensible order I think
u can put it in API changes

Moved to generic (I'm not sure it was worth it; the code got pretty messy with a bunch of if / else.), updated docs. Should be good when travis says so.

ok, in theory `ndim>=3` should work via `.apply` FYI

you can put tests in test_generic.py (you can do specific tests or have it create them generically)

@jreback was there anything else you saw here? I think it''s ready.

didn't realize their is an argument percentile_width

I think u should just rename it to percentiles and make it what u have for percentiles (and if it's a scalar then the meaning is unchanged)

I think too confusing with that argument (which is prob not used much at all) - yours is much more useful 

Should we do any warning / deprecation? I should be able to handle that very easily.

On May 11, 2014, at 3:46 PM, "jreback" <notifications@github.com<mailto:notifications@github.com>> wrote:

didn't realize their is an argument percentile_width

I think u should just rename it to percentiles and make it what u have for percentiles (and if it's a scalar then the meaning is unchanged)

I think too confusing with that argument (which is prob not used much at all) - yours is much more useful

—
Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/7088#issuecomment-42782700.

sure why don't I deprecate perentile_width and replace with percentile 

otherwise functionality is the same 

Added a note about this deprecation to #https://github.com/pydata/pandas/issues/6581.

Anything else?

@jorisvandenbossche Does my deprecation note [here](https://github.com/pydata/pandas/pull/7088/files#diff-03b380f521c43cf003207b0711bac67fR3509) look ok? That's how the numpy guide said to do it for _objects_. I assumed it was similar for keyword arguments.

Some comments:
- is it necessary that both the percentage-style (50) and number-style (0.5) are allowed? This seems to make it a bit complex to me (what if you use the first style, but want to give 0.5%, then you have to use the other style and it becomes confusing I think)
- the deprecation of percentile_width is not in the release notes
- About the deprecation note in the docstring. I think this is OK, but another option is to keep the parameter in the parameter section, but start its explanation with like 'Deprecated and will be removed. Use instead ... ' (the numpy docstring standard does not really mention this how to do, I think in pandas we mostly mention it in the parameter section itself instead of a seperate note (although we maybe mostly forget it ...), eg http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_html.html?highlight=deprecated).

@jorisvandenbossche thanks for the comments.

I was hoping that accepting both percentages and raw decimals would be less confusing, since I can never remember which we expect.

I actually had a longer reply written and then I realized why it was confusing. I'll switch it back to just expecting decimals between `[0, 1]`, which is at least consistent with `quantile`.

@TomAugspurger I think `quantile` was originally designed this way to match R, however, now it dismatches `np.percentile` which uses [0,100]. In any case, too late to change now in pandas I think, and indeed most important to be consistent within pandas with `quantile`. 

BTW, nice and informative FutureWarning! +1

@TomAugspurger why are you using `Counter` rather than `pd.core.algorithm.value_counts()` again?

`Counter` DOES not sort, while `value_counts` does.

On windows `test_generic/test_describe_object` is failing on the datetimes because of the arbitrary order from `Counter` (as the counts are all 1 for some reason it picks the 2nd one).

```
======================================================================
FAIL: test_describe_objects (pandas.tests.test_generic.TestDataFrame)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\Users\Jeff Reback\Documents\GitHub\pandas\build\lib.win-amd64-2.7\pandas\tests\test_generic.py", line 997, in test_describe_objects
    assert_frame_equal(result, expected)
  File "c:\Users\Jeff Reback\Documents\GitHub\pandas\build\lib.win-amd64-2.7\pandas\util\testing.py", line 573, in assert_frame_equal
    check_exact=check_exact)
  File "c:\Users\Jeff Reback\Documents\GitHub\pandas\build\lib.win-amd64-2.7\pandas\util\testing.py", line 520, in assert_series_equal
    assert_almost_equal(left.values, right.values, check_less_precise)
  File "testing.pyx", line 58, in pandas._testing.assert_almost_equal (pandas\src\testing.c:2465)
  File "testing.pyx", line 93, in pandas._testing.assert_almost_equal (pandas\src\testing.c:1793)
  File "testing.pyx", line 139, in pandas._testing.assert_almost_equal (pandas\src\testing.c:2338)
AssertionError: Timestamp('2010-01-02 00:00:00') != Timestamp('2010-01-01 00:00:00')

----------------------------------------------------------------------
Ran 6971 tests in 194.439s

FAILED (SKIP=130, failures=1)
```

```
2.7\pandas\core\generic.py(3576)describe_categorical_1d()
-> if data.dtype == object:
(Pdb) n
> c:\users\jeff reback\documents\github\pandas\build\lib.win-amd64-2.7\pandas\core\generic.py(3585)describe_categorical_1d()
-> elif issubclass(data.dtype.type, np.datetime64):
(Pdb)
> c:\users\jeff reback\documents\github\pandas\build\lib.win-amd64-2.7\pandas\core\generic.py(3586)describe_categorical_1d()
-> names = ['count', 'unique']
(Pdb) p data
0   2010-01-01
1   2010-01-02
2   2010-01-03
3   2010-01-04
Name: C1, dtype: datetime64[ns]
(Pdb) n
> c:\users\jeff reback\documents\github\pandas\build\lib.win-amd64-2.7\pandas\core\generic.py(3587)describe_categorical_1d()
-> asint = data.dropna().values.view('i8')
(Pdb) n
> c:\users\jeff reback\documents\github\pandas\build\lib.win-amd64-2.7\pandas\core\generic.py(3588)describe_categorical_1d()
-> objcounts = compat.Counter(asint)
(Pdb) p asint
array([1262304000000000000, 1262390400000000000, 1262476800000000000,
       1262563200000000000], dtype=int64)
(Pdb) l
3583                        result += [top, freq]
3584
3585                elif issubclass(data.dtype.type, np.datetime64):
3586                    names = ['count', 'unique']
3587                    asint = data.dropna().values.view('i8')
3588 ->                 objcounts = compat.Counter(asint)
3589                    result = [data.count(), len(objcounts)]
3590                    if result[1] > 0:
3591                        top, freq = objcounts.most_common(1)[0]
3592                        names += ['first', 'last', 'top', 'freq']
3593                        result += [lib.Timestamp(asint.min()),
(Pdb) p compat.Counter(asint)
Counter({1262390400000000000: 1, 1262563200000000000: 1, 1262304000000000000: 1, 1262476800000000000: 1})
(Pdb) p asint
array([1262304000000000000, 1262390400000000000, 1262476800000000000,
       1262563200000000000], dtype=int64)
(Pdb) p pd.algorithms.value_counts
*** AttributeError: AttributeError("'module' object has no attribute 'algorithms'",)
(Pdb) p pd.core.algorithms.value_counts
<function value_counts at 0x00000000076FDBA8>
(Pdb) p pd.core.algorithms.value_counts(asint)
1262304000000000000    1
1262563200000000000    1
1262476800000000000    1
1262390400000000000    1
dtype: int64
(Pdb) p pd.core.algorithms.value_counts(asint,sort=True)
1262304000000000000    1
1262563200000000000    1
1262476800000000000    1
1262390400000000000    1
dtype: int64
(Pdb)
```

Ahh I missed that one. I'll switch it over to use value counts and fix the test so that it isn't ambiguous.

awesome just put up a pr and I can test