ENH: option to plot data points on top of boxplot
State: closed

- [x ] tests added / passed 
- [x ] passes `black pandas`
- [x ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry - not sure how to do this. I guess I have to wait until pull request is complete.

This change is a a feature I often want, which is the ability to add raw data on top of a boxplot to get a better idea of data distribution.


Labels:
name: Stale

Comments:
You're right. That comment was just leftover from my hacking. I removed it.

Here's an example below.  It's inspired by a few examples here:
https://stackoverflow.com/questions/29779079/adding-a-scatter-of-points-to-a-boxplot-using-matplotlib
Use of seaborn's swarmplot on top of boxplot.
https://seaborn.pydata.org/generated/seaborn.boxplot.html


The .04 is the standard deviation of the noise added to the x value so that
the points don't end up on top of each other. The underlying x tick
apparently start at 1 (I can't claim to understand exactly how/why).
[image: image.png]


On Fri, Nov 6, 2020 at 4:30 AM Maxim Ivanov <notifications@github.com>
wrote:

> *@ivanovmg* requested changes on this pull request.
>
> @mikessut <https://github.com/mikessut> thank you for the PR.
>
> I guess that for this kind of PR it is necessary to show how the
> visualization works before/after,
> since it is rather complicated to infer that from the code alone.
> Can you add a visual example?
> ------------------------------
>
> In pandas/plotting/_matplotlib/boxplot.py
> <https://github.com/pandas-dev/pandas/pull/37656#discussion_r518691753>:
>
> >          keys = [pprint_thing(x) for x in keys]
>          values = [np.asarray(remove_na_arraylike(v), dtype=object) for v in values]
>          bp = ax.boxplot(values, **kwds)
> +
> +        if plot_points:
> +            # if plot_points
> +            for x, ys in enumerate(values):
> +                ax.plot(
> +                    x + np.random.normal(size=len(ys)) * 0.04 + 1, ys, "ro", alpha=0.5
>
> Can you explain what this 0.04 and + 1 mean here?
> I suggest to extract a function and use named variables there for clarity.
> ------------------------------
>
> In pandas/plotting/_matplotlib/boxplot.py
> <https://github.com/pandas-dev/pandas/pull/37656#discussion_r518692006>:
>
> >          keys = [pprint_thing(x) for x in keys]
>          values = [np.asarray(remove_na_arraylike(v), dtype=object) for v in values]
>          bp = ax.boxplot(values, **kwds)
> +
> +        if plot_points:
> +            # if plot_points
>
> I think that the comment here is not required.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/37656#pullrequestreview-525070001>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AARFNBBN2WJSFSZIRMPYN33SOPM3TANCNFSM4TL6SBSQ>
> .
>

Try that again to get example...
![image](https://user-images.githubusercontent.com/2250372/98373857-16860180-1ffd-11eb-9bf4-97db1b42ca47.png)

I guess that the feedback from the core team on the plotting is necessary here.

It seems to me that three points require attention: 
 - Public API (new parameter)
 - Hard-coded color. Should it be optional? If so, then should it be selected from ``color``?
 - Hard-coded width of the distribution. Will it work for small and large (say 10) number of boxes? Would not they overlap?
IMHO, i don't think this should be implemented on pandas for now, this might be for matplotlib.

IIUC this implementation is adding points using ax.plot and this should be separated apart from boxplot, and the significance of boxplot is not plotting individual points but for plotting them through quartiles (more from a statistical perspective). 

So if users want to plot those points, they could use the code you have in PR and put it after df.boxplot(ax=ax), and so that users also have the flexibility to change color or shape or size of those points. Therefore, i would suggest creating an issue to collect some feedbacks first before diving into the detailed implementation (and hard-coded color and floats need to be reconsidered)

FWIW, my longing for this feature comes from many years of using Minitab, which has a convenient option to show [individual data on boxplots](https://support.minitab.com/en-us/minitab/18/help-and-how-to/graphs/how-to/general-graph-options/display-options/boxplot/).  As a result, I often find myself wanting to be able to get a better visual idea of the data distribution.  

You are absolutely correct, that the data can be plotted by other means.  However, I haven't found those other means particularly intuitive and every time I want to show the underlying data, I find myself digging through stack overflow trying to remember how to do it.  The last time I found myself doing this, I decided I'd create the pull request.

Seaborn appears to have quite a few of the features that I'm looking for -- both boxplot and categorical scatter plots.  Maybe that's a better workflow for what I'm trying to accomplish. E.g.:
```
df = pd.DataFrame(np.random.randn(10, 3))
df2 = df.melt()

sns.boxplot(x='variable', y='value', data=df2)
sns.stripplot(x='variable', y='value', data=df2)
```

That get's pretty close, but certainly not as convenient as the following when trying to quickly inspect the data:
```
df = pd.DataFrame(np.random.randn(10, 3))
df.boxplot(plot_points=True)
```
I think this is a reasonable suggestion, to make the inspection of the data easier.
However I agree that it is better to move this discussion to a dedicated enhancement-related issue and collect feedback from those concerned.
thanks for your comments!
 
As I said, the usage of boxplot is _not_ plotting individual points, but I do understand allowing this option might make the inspection easier in certain circumstances. However, there will be many issues to think of, especially if there are large amount of data points, I can imagine plotting individual points can be very messy (in your example there are 10, but i think 10000 is enough to make the visual not readable at all). And having this is also an API change, so definitely need to collect feedbacks first.

Could you please create an issue first for this? @mikessut 
Created issue:
https://github.com/pandas-dev/pandas/issues/37759
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
closing, see https://github.com/pandas-dev/pandas/issues/37759#issuecomment-774693593