ENH: categorical scatter plot
State: closed

- [ ] closes #12380 (this idea was mentioned as a comment here). Also, closes #31357 (I hadn't noticed found that issue when I first opened this)
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Here's what this would look like:

![image](https://user-images.githubusercontent.com/33491632/82671464-cd9ce500-9c36-11ea-9f1d-eb901d209ab9.png)


Was just hoping to get initial feedback on whether this would be welcome

Somethings I still need to do:
- [ ] add docstring / annotations to `_plot_colorbar`
- [x] `test_fash.sh` passes locally but there are some CI failures, need to reproduce / figure out why (those tests pass now, failure was probably unrelated)
- [ ] add an example of this functionality to the docs
- [ ] if ints are passed, treat same way as categorical

Labels:
name: Enhancement
name: Visualization
description: plotting

Comments:
Is passing a string argument to `c` non-ambiguous? 
> Is passing a string argument to `c` non-ambiguous?

It's already supported, see [this example](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.plot.scatter.html) from the docs:
![image](https://user-images.githubusercontent.com/33491632/82579286-7be85200-9b85-11ea-8340-10582601dcd0.png)

However, currently, it's required that the column be numeric. This PR would add support for the categorical case.
emm, I think `c` is used to assign values for color points, the usage here looks more like showing labels for scatter points to me? and I am not sure if this will bring confusions to users.

what do you think?
So this covers unordered well - do we need to make any considerations for ordered categoricals as well?
> emm, I think `c` is used to assign values for color points, the usage here looks more like showing labels for scatter points to me? and I am not sure if this will bring confusions to users.
> 
> what do you think?

This functionality already exists though, see the last example from [here](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.plot.scatter.html)  where they pass
```python
ax2 = df.plot.scatter(x='length',
                      y='width',
                      c='species',
                      colormap='viridis')
```

The only difference is that currently, you can only pass `c='species'` if `species` is numeric. This PR (once I sort out the errors) would allow you to do `c='species'` if `species` is categorical, and then you'd get a discrete legend instead of a continuous colormap.

> So this covers unordered well - do we need to make any considerations for ordered categoricals as well?

This should work with both, but in the issue linked above (#12380) they mention using discrete colorbars for ordered categoricals. That's a possibility, though that would make this PR bigger - perhaps it's best to have one PR deal with categoricals, and then another one to split between ordered and unordered?

----

Motivation for working on this comes from how in `ggplot2` you can do
```R
ggplot(data=Iris,aes(x=SepalWidthCm, y=SepalLengthCm,color=Species)) + geom_point()
```
and it works even if `Species` isn't numeric
> This functionality already exists though

yeah, I know the functionality does exist, but based on the definition:
> A column name or position whose values will be used to color the marker points according to a colormap.

Using numeric values is because they are used to select colors from colormap. but if uses categorical values, i _think_ it will only use the first N colors from colormap (N is the number of categories). So this doc should be changed and well documented to avoid confusion I think.

And @WillAyd has a good point on sorted categorical values, and in this case, the colors selected from colormap are not the first N, but based on the order, so would be nice to test for those cases.
@charlesdong1991 ah I see what you mean now, thanks for clarifying. Yes, that's a good point, will think about this
you are welcome @MarcoGorelli 

~I _think_ if to keep PR small (and get merged faster ^^) and to support categorical values for `c`~, I would suggest focusing on supporting sorted categorical values first, because it's very natural thinking that the order can be considered as a way to select differnt colors from colormap.

sorry, nvm, this PR has supported unordered case, so above does not make sense, sorry again for the noise
Here's what I'm currently working on: if we start with
```python
import matplotlib as mpl
import numpy as np

import pandas as pd

CMAP = "viridis"
df = pd.DataFrame(
    [[5.1, 3.5], [4.9, 3.0], [7.0, 3.2], [6.4, 3.2], [5.9, 3.0]],
    columns=["length", "width"],
)
```
then we get the following:

- unordered categorical
![image](https://user-images.githubusercontent.com/33491632/82658305-aa663b80-9c1e-11ea-8045-8906b3ceded1.png)

- ordered categorical
![image](https://user-images.githubusercontent.com/33491632/82658357-bf42cf00-9c1e-11ea-8584-ba74b69cad9f.png)

- numeric (no change to current behaviour)
![image](https://user-images.githubusercontent.com/33491632/82658385-cd90eb00-9c1e-11ea-9aae-e966f237a59f.png)

Will try to get a commit in with tests by the end of the day, but if the above screenshots indicate I'm on the wrong path please do let me know
I _think_, the second plot looks pretty good, and should be what it looks like if supporting categorical values. 

And I have mixed feeling about the first one, from a user perspective, this will bring confusions to users because the behaviour of unsorted categorical values are different than others for `c` arguement, and also, the color bar is gone (currently, they should expect to see colorbar if they explicitly assign values). 

So IMHO, for categorical features, sorted and unsorted, they should both have the second plot look. And the difference between those two, will be what colors the categories will represent in colormap based on the order.
I think it would in general be very nice to support "categorical" data (whether it is actually categorical dtype, or with non-numerical dtype) in a scatter plot. 

But some comments:

- IMO, we should not use a colorbar, as this is typically for a numerical scale (whether continuous or discrete colors). So I think we should rather use a legend than a colorbar (like the first plot in https://github.com/pandas-dev/pandas/pull/34293#issuecomment-632619052). But @charlesdong1991 do I understand you correctly that you would prefer the colorbar in all cases?
- We should also not use a numerical colormap (like viridis), but rather the discrete color cycle from matplotlib in this case, I think?
@jorisvandenbossche  I agree that it's also an option to have legend than colorbar for categorical values. 

But the main concern is consistency of the meaning and expectation of `c`, right now, they assign values (either array or column name or other allowed input), it will look for colors from colormap or specific colors and has colorbar to show out. But would be weird to users if the categorical features, they see legend.

So, if using legend, this change will need to be well documented for `c` argument to reduce confusion, and will also probably impact `colormap`. And maybe should also think if we should assign it to `c` or not, because it is plotting legends, if so, probably assigning to `label` sounds more aligned with other plots api.
emm, sorry, i think maybe legend looks a bit better in case of categorical values. i just tried out to use specific discrete colors to assign to `c` using the example here, and actually the plot does not look nice. Not sure if it is better to have legends or to have discrete colors in color bar here.

![Screen Shot 2020-05-22 at 4 48 26 PM](https://user-images.githubusercontent.com/9269816/82680107-1dd27200-9c4c-11ea-8e4b-bbc71e06b29d.png)

The color bar is there but does not make sense, so probably we should make changes for such cases too (another PR though).
>  i think maybe legend looks a bit better in case of categorical values

Is that for both ordered and unordered cases?

> The color bar is there but does not make sense, so probably we should make changes for such cases too (another PR though).

OK, have opened #34316.
> Is that for both ordered and unordered cases?

i really don't know, sorry, somehow though i still prefer discrete color in color bar like your second plot for both cases. i think @jorisvandenbossche  @WillAyd @TomAugspurger could decide which one is the better option

> OK, have opened #34316.

very nice! 
@charlesdong1991 your example is actually still a somewhat different use case, I think (but relevant for this discussion).

In your last example, the column passed to `c` is a column with color names, and we use those to color the points, apparently? (didn't know we supported that) 

But if that is the case, we actually need to distinguish the case where the passed "categorical" values are 1) color names or 2) discrete non-numeric values that we want to use default colors for (like the iris species categories example)
> In your last example, the column passed to c is a column with color names, and we use those to color the points

yeah, right now, `c` accepts quite wide range of cases, and color referred by names is one of the allowed cases. It's allowing color name, or color code, or a list of them, or column name/loc whose numeric values which points to the loc of colors in the colormap.

> distinguish the case where the passed "categorical" values are 1) color names or 

I think color names and color codes are natively and widely supported in matplotlib, so not sure if we even need to put efforts distinguishing it, in other words, no matter if it is already categorical type or general object type, they should be plotted correctly. Therefore, I _think_, even without this PR, the code below should also work (haven't tried out though, might wrong):

```python
df = pd.DataFrame(
    [[5.1, 3.5], [4.9, 3.0], [7.0, 3.2], [6.4, 3.2], [5.9, 3.0]],
    columns=["length", "width"],
)

df['specifics'] = pd.Categorical(['r', 'b', 'b', 'r', 'g'])
df.plot.scatter(x=0, y=1, c='specifics')
```

> 2) discrete non-numeric values that we want to use default colors for

I, as a user, the `c` is about colors, and that's what we want to use to color the points, that's why I had some initial concerns about what we should accept here.

I think the feature @MarcoGorelli wants to implement here is when categorical values are not color names which matplotlib cannot recognise, e.g. random category names. And we need to figure out how they could map to colors. Therefore, to me, supporting sorted categorical values are more intuitive, because order can be viewed as discrete numbers, and we could use them to find color in colormap (that's how they use numeric values to find colors, but difference is now in this case, the numeric values are discrete, so its kind of a special case of numeric values). Thus, I would prefer to see discrete names in color bar to be plotted (as in his second plot), which could presents the link between the random category name and the colormap. And for unsorted categorical values, we could just treat them as the order is the order of appearance, like how we do to unsorted objects.

So above is the main reason why I would prefer to have discrete values in colormap for categorical cases than having them as legends (which also might bring in ambiguity to users) although they might not look as nice as legends in the plot

Sorry to put so many words, and will be happy to discuss if you have further questions. @jorisvandenbossche 
> I think the feature @MarcoGorelli wants to implement here is when categorical values are not color names which matplotlib cannot recognise, e.g. random category names. And we need to figure out how they could map to colors.

Yes, exactly - I was thinking that the mapping could be done using the `codes` attribute.

Anyway, I'll wait for further direction about whether a legend or a discrete colorbar would be preferred before adding further commits - thanks for the discussion/input so far!
Hi all - any update on whether pandas would like to go ahead with this, and whether a discrete colorbar or a legend would be preferable?
cc @charlesdong1991 @ivanovmg @TomAugspurger  if any comments.
Thanks for reviews, have updated.

Looking through the thread, I think there's still no consensus on what to plot in the unordered categorical case.

Legend (as in the first plot from https://github.com/pandas-dev/pandas/pull/34293#issuecomment-632619052), or colorbar (so, no difference to the unordered categorical case)? The current set of commits has the latter, i.e.
![image](https://user-images.githubusercontent.com/33491632/104839091-eab5b080-58b6-11eb-9a1b-b87fba954c3e.png)

---

IMO a discrete colorbar in both the ordered and unordered cases minimises the risk of confusion, and I think it looks alright
+1 on the colorbar option for both ordered and unordered.
I guess it will be more clear to the users.
Cool, thanks

> also may pay to add an example in visualization.rst

Agreed, though it's the kind of thing people like to contribute, so if it's OK I'd rather open it as a good first issue (once this PR's in) and mentor someone through it
Sure, added - here's what the new part in `visualisation.rst` looks like:

![image](https://user-images.githubusercontent.com/33491632/105906258-5e745d80-601b-11eb-88ae-a3a56e843cec.png)

thanks @MarcoGorelli