Repository: matplotlib/matplotlib

Issue:
Boxplotting Masked Arrays
<!--To help us understand and resolve your issue, please fill out the form to the best of your ability.-->
<!--You can feel free to delete the sections that do not apply.-->

### Bug report
When I try to boxplot several np.arrays of masked values, ax.boxplot includes the masked values. This throws off the analysis of the data and gives me an almost unusable boxplot. 

**Bug summary**
I masked several -9999 values (equipment error values) in order to look at the variances in the rest of our good data (range between 0~500). I masked the -9999 values using a np.ma.masked_less to give an array of good data with several masked values in the array. Once I called these data into the boxplot, it included the masked -9999 values. I patched this bug by switching out the masked_less call with an inequality statement that replaced the -9999 values with erroneous values (i.e. swapped out -9999 with a random number instead of removing it from the data set).

<!--A short 1-2 sentences that succinctly describes the bug-->
Boxplot function including masked values in a np.array where it should be ignoring said masked values.

**Code for reproduction**

<!--A minimum code snippet required to reproduce the bug.
Please make sure to minimize the number of dependencies required, and provide
any necessary plotted data.
Avoid using threads, as Matplotlib is (explicitly) not thread-safe.-->

```python
# Paste your code here
#
#
import numpy as np
import matplotlib.pyplot as plt

a = np.ma.array([229.5,374.0,536.5,-9999,-9999,-9999,-9999,-9999,182.0,42.5,49.0])
b = np.ma.array([363.0,118.5,159.0,-9999,311.0,516.0,380.0,338.5,223.0,211.5,128.5])
c = np.ma.array([205.5,277.5,141.5,278.0,302.0,251.0,299.0,250.0,315.5,92.1,211.9])

a = np.ma.masked_less( a, -9000 )
b = np.ma.masked_less( b, -9000 )
c = np.ma.masked_less( c, -9000 )

data_to_plot = [ a , b, c ]
labels = [ 'a', 'c', 'c' ]

fig = plt.figure( 3, figsize = ( 8, 10 ) )
ax = fig.add_subplot( 111, frameon = False )
bp = ax.boxplot( data_to_plot, patch_artist = True, widths = .85, 
                labels = labels, vert = True)
plt.setp( bp[ 'boxes' ], color = 'black', alpha = .8, linewidth = 3 )
plt.setp( bp[ 'whiskers' ], color = 'black', linewidth = 3 )
plt.setp( bp[ 'fliers' ], markeredgecolor = 'black', alpha = 1, markersize = 30, marker = '.' )
plt.setp( bp[ 'medians' ], color = 'orange', linewidth = 3 )
plt.setp( bp[ 'means' ], color = 'black' )
plt.setp( bp[ 'caps' ], color = 'black', linewidth = 4 )

colors = ['r', 'b', 'g', ]
for b in ( bp ):
    for patch, color in zip( bp[ 'boxes' ], colors ):
        patch.set_facecolor( color )
plt.show()
```

**Actual outcome**

![bp_masked_array](https://user-images.githubusercontent.com/48063090/53507494-0676fa00-3a86-11e9-87c2-5c9878243670.png)

<!--The output produced by the above code, which may be a screenshot, console output, etc.-->

```
# If applicable, paste the console output here
#
#
```

**Expected outcome**

<!--A description of the expected outcome from the code snippet-->
<!--If this used to work in an earlier version of Matplotlib, please note the version it used to work on-->

Expected a boxplot with masked values ignored so only shows unmasked values above -9000

**Matplotlib version**
<!--Please specify your platform and versions of the relevant libraries you are using:-->
  * Operating system: macOS High Sierra 10.13.6
  * Matplotlib version: 2.0.2
  * Matplotlib backend (`print(matplotlib.get_backend())`): module://ipykernel.pylab.backend_inline
  * Python version: 3.6
  * Jupyter version (if applicable):
  * Other libraries: Spyder 3.1.2

<!--Please tell us how you installed matplotlib and python e.g., from source, pip, conda-->
<!--If you installed from conda, please specify which channel you used if not the default-->

Unsure how installed. Think it was through pip


Discussion:
its possible that boxplot is not mask-aware.  However, I don't see any reason why it can't be made aware.

Did you mean version 3.0.2?  


@phobson, are you the boxplot guru?  If not, feel free to un-assign yourself!
@jklymak matplotlib.__version__ says it's "2.0.2" which could be part of the issue.
This is still reproducible in master, so I don't think its a version thing.  Thanks for checking!
@jklymak thanks for the heads up. i'll take a look 
The crux of the matter is here:
https://github.com/matplotlib/matplotlib/blob/master/lib/matplotlib/cbook/__init__.py#L1280

Paraphrased:
```python
for x in data:
    x = np.asarray(x)
```

So we're explicitly throwing away the mask. 

For an immediate work around:
```python
data_to_plot = list(map(lambda X: [x for x in X if x >= -9000], [a, b, c]))
```
May or may not scale with your real data
![masked_array_bp](https://user-images.githubusercontent.com/48063090/53514125-f87ca580-3a94-11e9-950d-662978e930ce.png)
Thanks @phobson ! The code is running well. Here's the new figure
I thought we had a standard way to regularize input like this; certainly `np.asarray` is not quite safe enough.  
perhaps this:
https://github.com/matplotlib/matplotlib/blob/master/lib/matplotlib/cbook/__init__.py#L996

That could work...
We've noticed all data on numpy masks is deleted after https://github.com/matplotlib/matplotlib/blob/8137b4c5459c9362e704853914fbbd74ea6e3199/lib/matplotlib/cbook/__init__.py#L1243 

Specifically here in _reshape_2D https://github.com/matplotlib/matplotlib/blob/8137b4c5459c9362e704853914fbbd74ea6e3199/lib/matplotlib/cbook/__init__.py#L1418

Thinking of two approaches for this fix using `delete_masked_points` (https://github.com/matplotlib/matplotlib/blob/master/lib/matplotlib/cbook/__init__.py#L996) 
1. Have a function to filter out masked values in `X` using `delete_masked_points` before passing `X` to `_reshape_2D`
2. Run the filter function within  `_reshape_2D` (avoids some duplicate code for violin plots that also depend on `_reshape_2D`, but changes the overall behaviour of this function)

We're going ahead with solution 2 unless there is an objection for it, either way it seems to fix this particular issue.
@dasrirez 

Who is "we" in this case? Does this comment mean that you're going to submit a PR?
@phobson yep, me and a group of other students are working on understanding and contributing to this project. Just wanted check in here on the best approach for our PR (please let me know if this is the right place for these sorts of discussions). Sorry for the ambiguity!
@dasrirez Thanks for clearing that up. 

This is the right place for that discussion. When you have a PR ready, ping me on it and I'll happily review it.
This issue has been marked "inactive" because it has been 365 days since the last comment. If this issue is still present in recent Matplotlib releases, or the feature request is still wanted, please leave a comment and this label will be removed. If there are no updates in another 30 days, this issue will be automatically closed, but you are free to re-open or create a new issue if needed. We value issue reports, and this procedure is meant to help us resurface and prioritize issues that have not been addressed yet, not make them disappear.  Thanks for your help!
This is still a bug in 3.7.

The analysis above still seems correct.  Anyone picking this up should review #13651 .  On a quick skim it is not clear to me if starting from those commits or starting from scratch is better.



Patch:
diff --git a/doc/api/next_api_changes/behavior/27605-DS.rst b/doc/api/next_api_changes/behavior/27605-DS.rst
new file mode 100644
index 000000000000..a4bc04ccfb04
--- /dev/null
+++ b/doc/api/next_api_changes/behavior/27605-DS.rst
@@ -0,0 +1,4 @@
+Boxplots now ignore masked data points
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+`~matplotlib.axes.Axes.boxplot` and `~matplotlib.cbook.boxplot_stats` now ignore
+any masked points in the input data.
diff --git a/lib/matplotlib/cbook.py b/lib/matplotlib/cbook.py
index b0cde743482b..ff6b2a15ec35 100644
--- a/lib/matplotlib/cbook.py
+++ b/lib/matplotlib/cbook.py
@@ -1274,7 +1274,8 @@ def _compute_conf_interval(data, med, iqr, bootstrap):
             continue
 
         # up-convert to an array, just to be safe
-        x = np.asarray(x)
+        x = np.ma.asarray(x)
+        x = x.data[~x.mask].ravel()
 
         # arithmetic mean
         stats['mean'] = np.mean(x)


Test patch:
diff --git a/lib/matplotlib/tests/test_axes.py b/lib/matplotlib/tests/test_axes.py
index d8d7cb0144f1..0557ac9941ec 100644
--- a/lib/matplotlib/tests/test_axes.py
+++ b/lib/matplotlib/tests/test_axes.py
@@ -3388,6 +3388,20 @@ def test_boxplot():
     ax.set_ylim((-30, 30))
 
 
+@check_figures_equal(extensions=["png"])
+def test_boxplot_masked(fig_test, fig_ref):
+    # Check that masked values are ignored when plotting a boxplot
+    x_orig = np.linspace(-1, 1, 200)
+
+    ax = fig_test.subplots()
+    x = x_orig[x_orig >= 0]
+    ax.boxplot(x)
+
+    x = np.ma.masked_less(x_orig, 0)
+    ax = fig_ref.subplots()
+    ax.boxplot(x)
+
+
 @image_comparison(['boxplot_custom_capwidths.png'],
                   savefig_kwarg={'dpi': 40}, style='default')
 def test_boxplot_custom_capwidths():