Repository: DistrictDataLabs/yellowbrick

Issue:
Matplotlib 3.1.1 impacts some visual tests
Matplotlib 3.1.1 (released on July 1, 2019) broke some of our tests, particularly the Rank2D tests with very big RMSE (~40) and very different diff images. For now in in #907 I have regenerated the baseline images for Rank2D so that our tests will pass on Travis and Appveyor, but I have not changed the `tests/requirements.txt` (except to note this strange behavior). Running the tests with `matplotlib==3.1.0` will result in image comparison failures for Rank2D. We should investigate what the behavior is for other matplotlib versions and determine what has changed in 3.1.1 that changed the Rank2D images (but not, say, the Confusion Matrix or Classification Heatmap) so drastically.



Discussion:
I looked at plots on Windows conda under 3.1.0 and 3.0.3 - both are different from 3.1.1. baseline in the same way (~40).

The difference is most likely due to do to a recent fix for the conversions between types for stacked bar charts (bar and barh). It had a 3.1 milestone:
https://github.com/matplotlib/matplotlib/issues/10788#issuecomment-373177134
The heights of rows are no longer consistent within 3.1.1, as compared to 3.1.0 and 3.0.3. Exactly where this would need to be corrected, if it is a bug in matplotlib, or a needed change in yb, is not clear to me as yet.

This is across the 6 Rank2D visualizers, but here is the rank2d_integrated_numpy visualizer:

**Integrated NumPy 3.0.3**
![rank2d_integrated_numpy_303](https://user-images.githubusercontent.com/4530874/60759074-8db01d80-9fe4-11e9-8e39-3b2bd966fa7d.png)

**Integrated NumPy 3.1.0**
![rank2d_integrated_numpy_310](https://user-images.githubusercontent.com/4530874/60759077-94d72b80-9fe4-11e9-8705-2ca961f8462a.png)

**Integrated NumPy 3.1.1**
![rank2d_integrated_numpy_311](https://user-images.githubusercontent.com/4530874/60759081-a4567480-9fe4-11e9-95d8-dc82fdf5df26.png)

Even when reverting to the yb commit https://github.com/DistrictDataLabs/yellowbrick/commit/1bb171cafb468ea6867668099c1b4a0f23db2b79 from prior to the release of 3.1.1, but using 3.1.1, the heights of the rows are inconsistent. 
I would recommend that matplotlib!=3.1.1 be added to requirements.txt and tests/requirements.txt to avoid users and contributors installing with 3.1.1. I can file a PR for this, include the previous Rank2D images in it, and create a matplotlib issue, pointing to this yb issue for illustration. Does this sound reasonable?
>I would recommend that matplotlib!=3.1.1 be added to requirements.txt and tests/requirements.txt to avoid users and contributors installing with 3.1.1. I can file a PR for this, include the previous Rank2D images in it, and create a matplotlib issue, pointing to this yb issue for illustration. Does this sound reasonable?

Thanks for all this great research you've been doing into this issue @nickpowersys! If you wouldn't mind holding off on the PR for just a bit longer as the maintainers are still powering through [some of those contagious issues](https://github.com/DistrictDataLabs/yellowbrick/pull/894#issuecomment-507370010). In the meantime though, If you could do a bit more digging into this, that would be great! A few open questions seem to be:

1. Does the same behavior occur between Matplotlib 3.1.0 and 3.1.1 using a different dataset? E.g. credit or bikeshare? Perhaps there is some data type checking we should be doing from inside the visualizer to ensure the proper dimensions?
2. Specifically what changed in Matplotlib between 3.1.0 and 3.1.1 that is causing this discrepancy in the Rank2D plots? Maybe @tacaswell or @jklymak can help point us in the right direction? 
3. Why is the behavior manifesting in Rank2D but not in any of the other visualizers?

This was a bug and should be fixed 3.1.2.  Sorry about that.  https://github.com/matplotlib/matplotlib/pull/14677
Thanks for the speedy response @jklymak; btw the mpl team are total badasses &mdash; thank you, thank you, thank you for all you do! And thanks @nickpowersys for offering to do the PR, but I'll go ahead and open one now since I already have a branch on my local with the repairs to the images 😉. Stay tuned!

Resolving patch:
diff --git a/yellowbrick/features/rankd.py b/yellowbrick/features/rankd.py
--- a/yellowbrick/features/rankd.py
+++ b/yellowbrick/features/rankd.py
@@ -17,14 +17,17 @@
 ## Imports
 ##########################################################################
 
+import warnings
 import numpy as np
+import matplotlib as mpl
+
 from scipy.stats import shapiro
 from scipy.stats import spearmanr
 from scipy.stats import kendalltau as sp_kendalltau
 
 from yellowbrick.utils import is_dataframe
 from yellowbrick.features.base import MultiFeatureVisualizer
-from yellowbrick.exceptions import YellowbrickValueError
+from yellowbrick.exceptions import YellowbrickValueError, YellowbrickWarning
 
 
 __all__ = ["rank1d", "rank2d", "Rank1D", "Rank2D"]
@@ -189,7 +192,16 @@ def finalize(self, **kwargs):
             generic keyword arguments
 
         """
-        # Set the title
+        # There is a known bug in matplotlib 3.1.1 that affects RankD plots
+        # See #912 and #914 for details.
+        if mpl.__version__ == "3.1.1":
+            msg = (
+                "RankD plots may be clipped when using matplotlib v3.1.1, "
+                "upgrade to matplotlib v3.1.2 or later to fix the plots."
+            )
+            warnings.warn(msg, YellowbrickWarning)
+
+        # Set the title for all RankD visualizations.
         self.set_title(
             "{} Ranking of {} Features".format(
                 self.ranking_.title(), len(self.features_)


Test patch:
diff --git a/tests/baseline_images/test_features/test_rankd/test_rank1d_integrated_numpy.png b/tests/baseline_images/test_features/test_rankd/test_rank1d_integrated_numpy.png
Binary files a/tests/baseline_images/test_features/test_rankd/test_rank1d_integrated_numpy.png and b/tests/baseline_images/test_features/test_rankd/test_rank1d_integrated_numpy.png differ
diff --git a/tests/baseline_images/test_features/test_rankd/test_rank1d_integrated_pandas.png b/tests/baseline_images/test_features/test_rankd/test_rank1d_integrated_pandas.png
Binary files a/tests/baseline_images/test_features/test_rankd/test_rank1d_integrated_pandas.png and b/tests/baseline_images/test_features/test_rankd/test_rank1d_integrated_pandas.png differ
diff --git a/tests/baseline_images/test_features/test_rankd/test_rank1d_orientation.png b/tests/baseline_images/test_features/test_rankd/test_rank1d_orientation.png
Binary files a/tests/baseline_images/test_features/test_rankd/test_rank1d_orientation.png and b/tests/baseline_images/test_features/test_rankd/test_rank1d_orientation.png differ
diff --git a/tests/baseline_images/test_features/test_rankd/test_rank1d_shapiro.png b/tests/baseline_images/test_features/test_rankd/test_rank1d_shapiro.png
Binary files a/tests/baseline_images/test_features/test_rankd/test_rank1d_shapiro.png and b/tests/baseline_images/test_features/test_rankd/test_rank1d_shapiro.png differ
diff --git a/tests/baseline_images/test_features/test_rankd/test_rank2d_covariance.png b/tests/baseline_images/test_features/test_rankd/test_rank2d_covariance.png
Binary files a/tests/baseline_images/test_features/test_rankd/test_rank2d_covariance.png and b/tests/baseline_images/test_features/test_rankd/test_rank2d_covariance.png differ
diff --git a/tests/baseline_images/test_features/test_rankd/test_rank2d_integrated_numpy.png b/tests/baseline_images/test_features/test_rankd/test_rank2d_integrated_numpy.png
Binary files a/tests/baseline_images/test_features/test_rankd/test_rank2d_integrated_numpy.png and b/tests/baseline_images/test_features/test_rankd/test_rank2d_integrated_numpy.png differ
diff --git a/tests/baseline_images/test_features/test_rankd/test_rank2d_integrated_pandas.png b/tests/baseline_images/test_features/test_rankd/test_rank2d_integrated_pandas.png
Binary files a/tests/baseline_images/test_features/test_rankd/test_rank2d_integrated_pandas.png and b/tests/baseline_images/test_features/test_rankd/test_rank2d_integrated_pandas.png differ
diff --git a/tests/baseline_images/test_features/test_rankd/test_rank2d_kendalltau.png b/tests/baseline_images/test_features/test_rankd/test_rank2d_kendalltau.png
Binary files a/tests/baseline_images/test_features/test_rankd/test_rank2d_kendalltau.png and b/tests/baseline_images/test_features/test_rankd/test_rank2d_kendalltau.png differ
diff --git a/tests/baseline_images/test_features/test_rankd/test_rank2d_pearson.png b/tests/baseline_images/test_features/test_rankd/test_rank2d_pearson.png
Binary files a/tests/baseline_images/test_features/test_rankd/test_rank2d_pearson.png and b/tests/baseline_images/test_features/test_rankd/test_rank2d_pearson.png differ
diff --git a/tests/baseline_images/test_features/test_rankd/test_rank2d_random.png b/tests/baseline_images/test_features/test_rankd/test_rank2d_random.png
new file mode 100644
Binary files /dev/null and b/tests/baseline_images/test_features/test_rankd/test_rank2d_random.png differ
diff --git a/tests/baseline_images/test_features/test_rankd/test_rank2d_spearman.png b/tests/baseline_images/test_features/test_rankd/test_rank2d_spearman.png
Binary files a/tests/baseline_images/test_features/test_rankd/test_rank2d_spearman.png and b/tests/baseline_images/test_features/test_rankd/test_rank2d_spearman.png differ
diff --git a/tests/requirements.txt b/tests/requirements.txt
--- a/tests/requirements.txt
+++ b/tests/requirements.txt
@@ -1,5 +1,5 @@
 # Library Dependencies
-matplotlib>=1.5.1,!=3.0.0 # Note: rank2d image tests require mpl==3.1.1
+matplotlib>=1.5.1,!=3.0.0,!=3.1.1
 scipy>=1.0.0
 scikit-learn>=0.20
 numpy>=1.13.0