API: support `c` and `colormap` args for DataFrame.plot with kind='scatter'
State: closed

`matplotlib.pyplot.scatter` supports the argument `c` for setting the color of
each point. This patch lets you easily set it by giving a column name (currently
you need to supply an ndarray to make it work, since pandas isn't aware of it):

```
df.plot('x', 'y', c='z', kind='scatter')
```

vs

```
df.plot('x', 'y', c=df['z'].values, kind='scatter')
```

While I was at it, I noticed that `kind='scatter'` did not support the `colormap`
argument that some of the other methods support (notably `kind='hexbin'`). So
I added it, too.

This change should be almost entirely backwards compatible, unless folks are naming
columns in their data frame valid matplotlib colors and using the same color name
for the `c` argument.

A colorbar will also be added automatically if relevant.


Labels:
name: Enhancement
name: Visualization
description: plotting
name: API Design

Comments:
can you add a test?

How do we test plotting code? I can't find any tests for `pandas.tools.plotting` (not that I'm opposed to adding some!) but maybe I'm looking in the wrong place.

plotting tests are in a file called `test_graphics.py`, don't remember exactly where it is

Is there any reason to use different colormap default from normal and `hexbin`?

And overriding `MPLPlot._get_colors` makes your logic more readable (`hexbin` should have done the same manner...)

I picked a different colormap default because [the rainbow colormap is terribly misleading](http://people.renci.org/~borland/pdfs/RainbowColorMap_VisViewpoints.pdf). But that's just a fact of life with matplotlib, which people who care about can already easily work around. My preference would be to use a slightly saner colormap as the default, but I don't feel strongly about it.

Thanks for the pointers on testing and `MPLPlot._get_colors`, I will definitely do that in the next patch.

OK, added a commit with tests (all passing).

@sinhrks I figured out why `MPLPlot._get_colors` wasn't used for hexbin and why it won't work here. That method (and the `_get_standard_colors` function) is designed for using a discrete set of colors. It doesn't support using color maps.

I see. Maybe `_get_colors` and `_maybe_add_color` should be refactored to cover colormaps (will do in separate PR).

Regarding tests, there is a function `TestPlotBase._check_colors` which checks the target is actually has intended colors.

@sinhrks OK, I was able to use `_check_colors` for one test but it doesn't really do colormaps, either. See the latest commit.

I think this patch is ready for review.

can you add a release note in v0.15.0 (ref this PR as it doesn't have an issue), API section. I don't think this needs addtl docs (as you updated the doc-string), right?

also, pls squash the commits together

I agree that we should accept `c` as a synonym for `colormap`, since matplotlib does, but I don't think we should document it or encourage it in anyway. Especially since you can end up with something like

``` python
df.plot(kind='hexbin', c='col1', C=np.mean)
```

two arguments differing by capitalization is no good.

@TomAugspurger glad to see you back!

@shoyer ok, then maybe just put a note in the code to that effect (and not modify the doc-string)

A couple of weeks still, then I'll be back :)

Just happened to have some (bad) wifi and a bit of free time.

RE: `c` vs `colormap`: I don't think these are valid synonyms. The matplotlib function `plt.scatter` uses both of these arguments for different uses, `c` for numeric colors or a single color for all points and `cmap` for the colormap used in the case of numeric colors. The synonyms would be `cmap`/`colormap`. 

commits squashed and release notes updated

so `c` is an argument to `plt.scatter`? that's why you think we should accept it (and not doc-it)?

Yes, `c` is an argument to `plt.scatter`.

I only documented it at the end because that was what I noticed for the `hexbin` specific arguments (e.g., the similar but distinct argument `C`). I'd be happy to add them into the explicit parameter list in the docstring if that is your preference.

why are we accepting it at all though? isn't it more confusing? esp since u have to see if it's a column and such

@jreback Here is the sort of scatter plot I want to be able to make:

![image](https://cloud.githubusercontent.com/assets/1217238/3709373/65107f30-1454-11e4-957c-c5e4bd103b41.png)

In this case `x`, `y` and `z` would be columns in my dataframe.

Actually, it occurs to me now that this patch should also label the colorbar (z) automatically with the column name if possible... it's not doing that yet but I'll add another commit to do so.

ok looks good
maybe add a graph like that in the docs then 

we don't seem to have a sub-section on scatter plots (in the first part of plotting docs)

One more question, can `c` and `cmap` work with `xerr` and `yerr`? Maybe errorbars are not colorized properly?

OK, finally got around to fixing this up. Colorbars are now automatically labeled.

After rebasing, I discovered that somebody beat me to the punch for writing the docs section on scatter plots :). I added in an example of providing the colors by name.

@sinhrks The color of errorbars appears to be controlled by the `color` keyword argument (which will also color the points if `c` is not provided). We could change this to match the colors of points, but I doubt colored errorbars along with the points are what anyone wants.

Unfortunately, this means the error bars default to blue when `c` is provided with is pretty ugly (I would rather default to black if `c` is provided):

```
%matplotlib inline
import numpy as np
import pandas as pd

df = pd.DataFrame(np.random.randn(10, 5), columns=['a', 'b', 'c', 'd', 'e'])
df.plot('a', 'b', c='c', s=100, xerr='d', yerr='e', kind='scatter')
```

![download 4](https://cloud.githubusercontent.com/assets/1217238/3977378/c6c98ed2-2837-11e4-8f20-757469c4274a.png)

I tried fixing that myself but a `color` argument seems to be filled in to `ScatterPlot.kwds` even when not given explicitly in the method call. If you understand the coloring logic, it would be awesome if you could fix that, maybe in a follow-on PR.

Actually, when I try running the errorbar example again (it worked before rebasing) I get this error:

```
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-7-658a11055925> in <module>()
      4 
      5 df = pd.DataFrame(np.random.randn(10, 5), columns=['a', 'b', 'c', 'd', 'e'])
----> 6 df.plot('a', 'b', c='c', s=100, xerr='d', yerr='e', kind='scatter')

/Users/shoyer/dev/pandas/pandas/tools/plotting.pyc in plot_frame(frame, x, y, subplots, sharex, sharey, use_index, figsize, grid, legend, rot, ax, style, title, xlim, ylim, logx, logy, xticks, yticks, kind, sort_columns, fontsize, secondary_y, layout, **kwds)
   2250                              secondary_y=secondary_y, layout=layout, **kwds)
   2251 
-> 2252     plot_obj.generate()
   2253     plot_obj.draw()
   2254     return plot_obj.result

/Users/shoyer/dev/pandas/pandas/tools/plotting.pyc in generate(self)
    911         self._compute_plot_data()
    912         self._setup_subplots()
--> 913         self._make_plot()
    914         self._add_table()
    915         self._make_legend()

/Users/shoyer/dev/pandas/pandas/tools/plotting.pyc in _make_plot(self)
   1424 
   1425         errors_x = self._get_errorbars(label=x, index=0, yerr=False)
-> 1426         errors_y = self._get_errorbars(label=y, index=1, xerr=False)
   1427         if len(errors_x) > 0 or len(errors_y) > 0:
   1428             err_kwds = dict(errors_x, **errors_y)

/Users/shoyer/dev/pandas/pandas/tools/plotting.pyc in _get_errorbars(self, label, index, xerr, yerr)
   1362                         err = None
   1363                 elif index is not None and err is not None:
-> 1364                     err = err[index]
   1365 
   1366                 if err is not None:

IndexError: index 1 is out of bounds for axis 0 with size 1
```

It works if I remove the argument `yerr`.

Was there a recent refactoring of error bars?

@shoyer Thanks to confirm. I think no problem if it isn't easy.

Following works with current master. Thus using column name with `c` keyword causes a problem?

(And this shows scatter in green but errorbars in blue. Thus color problem with `c` keyword should be existing bug...)

```
df = pd.DataFrame(np.random.randn(10, 5), columns=['a', 'b', 'c', 'd', 'e'])
df.plot(x='a', y='b', c='green', s=100, xerr='d', yerr='e', kind='scatter')
```

@sinhrks Actually, your example is broken for me on master -- no column name with `c` keyword necessary. I just made a new issue: #8081

@shoyer this looks cool. I'll try to review today / this weekend. You'll need to rebase now that #8183 is merged. The example from @sinhrks should work.

A question: we already have the `color` keyword. E.g. if you want all points to have one color, use `df.plot(x='x', y='y', kind='scatter', color='g')`. Why not use that keyword instead of adding `c`?

Update: actually, already both `color` and `c` work, both for standard line plot as scatter plot. Is there any difference or are it just aliases?

@jorisvandenbossche That's a good question. I choose `c` because it was already the keyword in the matplotlib API. In matplotlib, only `c` argument works for indicating colors as an array with `scatter`:

``` python
plt.scatter(np.random.randn(100), np.random.randn(100), c=np.random.randn(100)) # works
plt.scatter(np.random.randn(100), np.random.randn(100), color=np.random.randn(100)) # does not work
```

I don't really have a unifying theory here. I was most copying the precedent I noticed for `hexbin`, with uses a `C` argument.

Rebased on master.