WIP: Add value_counts() to DataFrame
State: closed

- abstract bin generation from cut to use elsewhere. Panel goes to ndarray on
  apply so that's a future TODO.

I'm relatively certain that this works, but I don't have explicit tests
for it. I'm hoping others who are interested (cough cough @rockg) will
be willing to create some of them. Key to test are that binning,
normalization and sorting all work correctly.

Will close #5377 when completed.


Labels:
name: Enhancement
name: API Design
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
@jtratner looks fine...(obv need some tests as you have indicated)

yeah, I'm pretty sure there's a bug with the binning that I need to address
and flesh out some of the edge cases (esp dup columns :-/). Should this
wait until 0.14 b/c it's a new feature?

let's put this off to 0.14....

Yep, that's what I was thinking too.

Added some documentation comments.

Thanks for the comments! I'll incorporate those when I return to finish
this up.

When would someone use this ? I can't think of when I would use this :s

Should fill in the NaNs with 0, since count is 0 if no values appear.

In general I think anything that you can do for a series you should be able to do for a column/columns of a DataFrame.  I think there are many uses for this especially when binning data--essentially it's a histogram function that you can apply to bunch of columns at once rather than going through each series.

> In general I think anything that you can do for a series you should be able to do for a column/columns of a DataFrame

A column of a DataFrame _is_ a Series (so the first half I agree). However, I strongly disagree to extending all Series functionality to DataFrame for the sake of it, rather things should be extended only where they makes sense... Personally I can't see this particular one making sense, but am open to seeing an example where it does.

> I think there are many uses for this especially when binning data...

So, could you give a specific example?

Exactly my point that it's a series of Series so it's a natural extension to be able to do a Series function to a series of Series.

One example is if there are is a DataFrame of different price timeseries and the user is interested in counts in bins of some increments (e.g., $10 or $5 or some other measure).  Or one could think of binning returns on a DataFrame or binning measurement data.  Another one is if there are states of different variables in each column one would be interested in the counts of these states for the various columns.  There are other ways to do the counts, but I think the binning is the real key to the usefulness of value_count.

It is just waiting tests thanks to implementation work already done by @jtratner.

I've ran into a couple situations in the past couple weeks where I could have used value counts on a DataFrame. Specifically needing to get value counts for flows from period to period:

``` python
In [12]: df.head()
Out[12]: 
                                1   2   3   4   5   6   7   8
HRHHID      HRHHID2 PULINENO                                 
12807008622 87001   1         NaN  nn  nn  nn  nn  nn  nn  nn
                    2         NaN  nn  nn  nn  nn  nn  nn  nn
                    3         NaN  ee  ee  ee  ee  ee  ee  ee
17141220290 87001   1         NaN  un  un  ne  ee  ee  ee  ee
                    2         NaN  ee  ee  ee  ee  ee  ee  ee
```

Granted, using `df.apply(lambda x: x.value_counts())` isn't onerous, but It wasn't the first thing I turned to (`df.value_counts` was).

This is a pretty trivial addition overall - I just haven't had time to get
to it (been super busy these past two weeks). Sorry guys.

@jtratner up 2 u if you want this for 0.13.1

@jtratner ?

@jtratner still working on this?

@jtratner  status?

@jtratner i  can take this if you want

I don't know if this bug is still at the "needing use cases" stage, but I have a dataframe where the columns are categorical variables. I want to prune values for these variables which appear below a certain frequency in the dataset. So doing value_counts() on a dataframe is the first step for that.

closing as stale