VIS/ENH Hexbin plot
State: closed

This is just 10 minutes of copy-paste cargo-culting to gauge interest, I haven't tested anything yet.

``` python
In [1]: df = pd.DataFrame(np.random.randn(1000, 2))

In [2]: df.plot(kind='hexbin', x=0, y=1)
Out[2]: <matplotlib.axes.AxesSubplot at 0x10eb76e10>
```

![hexbin](https://f.cloud.github.com/assets/1312546/1506888/ef198d6a-4953-11e3-8ecc-891d22def090.png)

It's not terribly difficult to do this on your own, so my feeling wouldn't be hurt at all if people are -1 on this :)

EDIT: oops I branched from my `to_frame` branch.  I'll clean up the commits.


Labels:
name: Enhancement
name: Visualization
description: plotting

Comments:
looks interesting

Colorbar on by default?

![hexbin](https://f.cloud.github.com/assets/1312546/1507077/a4b336e0-4962-11e3-95d9-c9ddf1c5dd90.png)

Also, apparently the default matplotlib colormap, `jet`, is widely despised.  I'm looking around for a better default.

Thoughts on a default color pallet?  [Seaborn](https://github.com/mwaskom/seaborn/blob/master/seaborn/rcmod.py#L17) seems to use `cubehelix` (screenshot)

![hexbin](https://f.cloud.github.com/assets/1312546/1609065/44088960-552b-11e3-95fb-824372ad86e6.png)

[This](http://nbviewer.ipython.org/urls/github.com/olgabot/prettyplotlib/raw/master/examples/pcolormesh.ipynb) may be helpful. @olgabot seems to use different cmaps for different data values. That may be a bit too much for pandas.

EDIT: oh I also think this should spend some time incubating on master.  Once the RC branch is forked, I'll add some release notes for .14;

Oh, also, I hate the [matplotlib argument names](http://matplotlib.org/api/pyplot_api.html#matplotlib.pyplot.hexbin) for the value (`C`) and reduction function `reduce_C_function`.

A brief overview:

by default (`C=None`), compute a histogram in each bin. Essentially each `(x, y)` point in a bin has a value of 1, and the reduction function is `sum` (i.e. `count` since every value is 1.)

If you specify `C` (another column of `df` for pandas), then each point is a 3 tuple `(x, y, c)`: `(x, y)` still specify what bin you end up in, and `c` determines the value passed to the reduction function `reduce_C_function`

What's the policy on respecting other libraries' function arguments. Other pandas plotting functions seem to use `colormap` instead of matplotlib's `cmap`.

If we're willing to change, I'd suggest renaming `C` to `value` and `reduce_C_function` to `reduce` or `reduce_func`

Here's an example to play with if that helps:

``` python
df = DataFrame({"A": np.random.uniform(size=20),
                "B": np.random.uniform(size=20),
                "C": np.arange(20) + np.random.uniform(size=20)})

ax = df.plot(kind='hexbin', x='A', y='B', gridsize=10)  # histogram by default
ax = df.plot(kind='hexbin', x='A', y='B', C='C', reduce_C_function=np.max)
```

So in the second case the color of a bin will be determined by the maximum value of `C` (from column `C` in `df`) in that bin.

While cubehelix solves the saturation problem, it doesn't solve the issue that the [rainbow colormap is harmful](http://www.jwave.vt.edu/~rkriz/Projects/create_color_table/color_07.pdf) and does not help with interpretation of the heatmap. The linked paper gives great examples how the changing colors introduce patterns in the data that don't actually exist and can lead to over-interpretation of color changes.

I don't think using different colormaps for different data ranges is that complicated, in `prettyplotlib` it's a simple check of the data's `max` and `min` and using a divergent heatmap like blue-red if there's negative _and_ positive values, or a sequential heatmap if there's only positive _or_ negative values. These colormaps are built into `matplotlib`

I'm still working on my `matplotlib` install. I have some code for a clustered heatmap as well (like `heatmap.2` in R)  that would use but it depends on `scipy`, so I'm guessing I'd have to implement hierarchical clustering myself to reduce `pandas`'s dependencies. Plus as a data science/bioinformatics purist I'd want it to have [optimal leaf ordering](http://bioinformatics.oxfordjournals.org/content/17/suppl_1/S22.long), too. This is the [default in MATLAB](http://www.mathworks.com/help/stats/optimalleaforder.html), but not in R or `scipy`.

As for renaming, I agree that the `matplotlib` arguments are cryptic but I personally prefer consistency over introducing new standards: http://xkcd.com/927/

I set the matplotlib default colormap to cubehelix in seaborn because I wanted to banish jet and that seemed like a reasonably non-crappy alternative that is at least somewhat adaquate for most data. I don't think the hue shifts are as big a problem in cubehelix as in rainbow maps and jet because they're pretty gradual and accompanied by a shift in lightness/saturation.

But in general I agree that there's no good one-size-fits-all solution to colormaps, and it's better to adapt to the data. For the [corrplot function](http://stanford.edu/~mwaskom/software/seaborn/linear_models.html#plotting-correlation-heatmaps) the default colormap is `coolwarm`, which I tend to prefer a bit to `RdBu` as the extreme values aren't quite as dark.

Thanks for the feedback! I'll think on this, but adapting to the data would be complicated by the business with `C` and `reduce_C_function`. That's all handled on the matplotlib side, so I won't know the range of the values when I hand everything off to matplotlib's `hexbin`.

The default behavior is to just do counts, so things will always be positive in that case. It might make sense just to pick the default with that case in mind, and assume that people doing fancier things can choose an appropriate color palette.

@olgabot `scipy` is a soft dependency for pandas, so that should be OK. I think the Gaussian KDE plot also depends on `scipy`.

Ah, yeah, if the data are counts then it probably makes sense to use a sequential map; any of the [colorbrewer](http://colorbrewer2.org/) ones should work.

I think this is ready to go. Thanks for the suggestions on the colormaps. I'm going with `BuGr` as the default since the default aggregation is by counts, which will always be nonnegative. If you're using a different aggregation then you should choose the appropriate colormap.

One thing I'm a tad concerned about is the docstring on df.plot.  I thew in a `Notes` with things specific to `hexbin`. I wonder if we should put explanations in a separate function like `scipy.integrate.quad_explain()` does. Or I could remove that and just point people to the html docs.

Let me know when your ready and I'll rebase and squash.

@TomAugspurger you want to throw in 0.13?

pls change the release notes, rebase, and squash...

if you think the API might change, then just label it experimental (if you want)

does this have an associated issue?

No issue associated. I just put the PR number in the release notes. I'm pretty confident the API won't need adjusting, but I guess labeling it experimental is the safe thing to do.

Ready when you and Travis say it's good!

perfect

@cpcloud @jtratner @y-p any comments?

This looks nice. 

@TomAugspurger, do you know if binhex plots are available via [yhat/ggplot](http://github.com/yhat/ggplot) ?
If not, it'd be great to help them grow the library, they're building it on top of pandas already.

looks ready to merge...any objections? @TomAugspurger @y-p @jtratner ?

Merging new features between RC and final? 

seems non-invasive to me

I'm not in a rush to get this in if that's a problem.

@TomAugspurger push this to 0.14, @y-p?

Fine by me. Once .13 is out I'll make the doc changes and ping you when I get that done (could be a little while).

Yup, and to stay on track we should be more aggressive about tagging 0.13 final when it's time
and moving on to 0.14, release channels can be async to that. Just a headsup to pandas-dev
when bug queue is empty and green across platforms.

@TomAugspurger, so is this plot available out of the box with seaborn which is built on top of pandas?

Not yet:
http://stanford.edu/~mwaskom/software/seaborn/plotting_distributions.html

---

Olga Botvinnik
PhD Program in Bioinformatics and Systems Biology
Gene Yeo Laboratory | Sanford Consortium for Regenerative Medicine
University of California, San Diego
olgabotvinnik.com
blog.olgabotvinnik.com
github.com/olgabot

On Mon, Dec 16, 2013 at 10:38 AM, y-p notifications@github.com wrote:

> @TomAugspurger https://github.com/TomAugspurger, so is this plot
> available out of the box with seaborn which is built on top of pandas?
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/5478#issuecomment-30687094
> .

I resisted other PRs for more sophisticated plots but this is closer to home,  should be ok.

I've been a tad worried about inconsistency in what vis PRs are accepted as well, so I haven't wanted to push to get this in.  That said I think this should go in since I see hexbin plots as a drop-in replacement for scatter plots when you have a bunch of points.

Plus Wes [tweeted](https://twitter.com/wesmckinn/statuses/404809949726797824) about it, and we can't make him a liar...

hahah.!

ok....

pls rebase and put in release notes (0.13.1)..thxs

I still think 0.14.0... don't make me out to be a softie.

I threw everything in .14, so this should be good to sit until we start on that.

I'm having trouble building the docs right now (on master too, not just this branch). I'll try to track down what's wrong with my environment and then I'll let you know that everything builds correctly.