Repository: DistrictDataLabs/yellowbrick

Issue:
Error from KElbowVisualizer's distortion_score method
`KElbowVisualizer` is generating a `ValueError` from Sklearn.

### Issue

Using the Yellowbrick `hobbies` corpus and vectorizing with TFIDF, the KElbowVisualizer is generating an error from Sklearn: `ValueError: Found array with dim 3. check_pairwise_arrays expected <= 2`. The `distortion_score` method appears to be causing the error when calling Sklearn's `pairwise_distances` method.

### Code
```
corpus = load_corpus('hobbies')
tfidf  = TfidfVectorizer()
docs   = tfidf.fit_transform(corpus)
visualizer = KElbowVisualizer(KMeans(), k=(4, 8))
visualizer.fit(docs)
visualizer.poof()
```

### Error
```
Traceback (most recent call last):
  File "elbows.py", line 82, in <module>
    visualizer.fit(docs)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/yellowbrick/cluster/elbow.py", line 245, in fit
    self.scoring_metric(X, self.estimator.labels_)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/yellowbrick/cluster/elbow.py", line 87, in distortion_score
    distances = pairwise_distances(instances, [center], metric=metric)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sklearn/metrics/pairwise.py", line 1247, in pairwise_distances
    return _parallel_pairwise(X, Y, func, n_jobs, **kwds)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sklearn/metrics/pairwise.py", line 1090, in _parallel_pairwise
    return func(X, Y, **kwds)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sklearn/metrics/pairwise.py", line 223, in euclidean_distances
    X, Y = check_pairwise_arrays(X, Y)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sklearn/metrics/pairwise.py", line 112, in check_pairwise_arrays
    warn_on_dtype=warn_on_dtype, estimator=estimator)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sklearn/utils/validation.py", line 451, in check_array
    % (array.ndim, estimator_name))
ValueError: Found array with dim 3. check_pairwise_arrays expected <= 2.
```


Discussion:
I've discovered the source of the problem but not the cause. 

In [`distortion_score`](https://github.com/DistrictDataLabs/yellowbrick/blob/develop/yellowbrick/cluster/elbow.py#L84) we take the mean of the instances along `axis=0` to find the centroid. We expect `instances` to be a 2D array, therefore the mean should be a 1D array. In order to pass this to the `pairwise_distances` function, we then covert the center into a single 2D point.

For some reason (still undetermined), in this particular case, the mean is returning a 2D array with only one row in it. It happens that the input is a sparse matrix, not a dense matrix, however, it should still only output a 1D array. Some testing:

```python
import numpy as np 
from scipy.sparse import csc_matrix

def amake(ndims=1, s=10, sparse=False):
    cube = [s]*ndims
    a = np.random.normal(loc=10.0, scale=1.8, size=cube)

    if sparse:
        return csc_matrix(a)
    return a

for i in range(1,6):
    mshape = amake(ndims=i).mean(axis=0).ndim 
    print("{}D -> {}D mean".format(i, mshape))
```

output:

```
1D -> 0D mean
2D -> 1D mean
3D -> 2D mean
4D -> 3D mean
5D -> 4D mean
```

So I'm a bit flummoxed, but still looking into it. 

**UPDATE** 

Apparently sparse matrices always return a 2D matrix, for any input dimension, although they won't accept over 2D input ... 

``` 
for i in range(0, 3):
    mshape = amake(ndims=i, sparse=True).mean(axis=0).ndim 
    print("{}D sparse -> {}D mean".format(i, mshape))
```

output:

```
0D sparse -> 2D mean
1D sparse -> 2D mean
2D sparse -> 2D mean
```

Resolving patch:
diff --git a/yellowbrick/cluster/elbow.py b/yellowbrick/cluster/elbow.py
--- a/yellowbrick/cluster/elbow.py
+++ b/yellowbrick/cluster/elbow.py
@@ -19,6 +19,8 @@
 ##########################################################################
 
 import time
+import numpy as np
+import scipy.sparse as sp
 
 from .base import ClusteringScoreVisualizer
 from ..exceptions import YellowbrickValueError
@@ -83,8 +85,16 @@ def distortion_score(X, labels, metric='euclidean'):
         # Compute the center of these instances
         center = instances.mean(axis=0)
 
+        # NOTE: csc_matrix and csr_matrix mean returns a 2D array, numpy.mean
+        # returns an array of 1 dimension less than the input. We expect
+        # instances to be a 2D array, therefore to do pairwise computation we
+        # require center to be a 2D array with a single row (the center).
+        # See #370 for more detail.
+        if not sp.issparse(instances):
+            center = np.array([center])
+
         # Compute the square distances from the instances to the center
-        distances = pairwise_distances(instances, [center], metric=metric)
+        distances = pairwise_distances(instances, center, metric=metric)
         distances = distances ** 2
 
         # Add the mean square distance to the distortion


Test patch:
diff --git a/tests/test_cluster/test_elbow.py b/tests/test_cluster/test_elbow.py
--- a/tests/test_cluster/test_elbow.py
+++ b/tests/test_cluster/test_elbow.py
@@ -18,19 +18,27 @@
 ##########################################################################
 
 import pytest
-import unittest
 import numpy as np
 import matplotlib.pyplot as plt
 
 from ..base import VisualTestCase
 from ..dataset import DatasetMixin
 
+from scipy.sparse import csc_matrix, csr_matrix
+from numpy.testing.utils import assert_array_almost_equal
+
 from sklearn.datasets import make_blobs
 from sklearn.cluster import KMeans, MiniBatchKMeans
+from sklearn.feature_extraction.text import TfidfVectorizer
+
 from yellowbrick.cluster.elbow import distortion_score
 from yellowbrick.cluster.elbow import KElbowVisualizer
 from yellowbrick.exceptions import YellowbrickValueError
-from numpy.testing.utils import assert_array_almost_equal
+
+try:
+    import pandas as pd
+except ImportError:
+    pd = None
 
 
 ##########################################################################
@@ -61,7 +69,7 @@
 y = np.array([0, 0, 1, 1, 0, 0, 1, 1, 0, 1, 0, 0, 1, 1, 1, 1, 0, 0])
 
 
-class KElbowHelperTests(unittest.TestCase):
+class TestKElbowHelper(object):
     """
     Helper functions for K-Elbow Visualizer
     """
@@ -71,19 +79,40 @@ def test_distortion_score(self):
         Test the distortion score metric function
         """
         score = distortion_score(X, y)
-        self.assertEqual(score, 7.6777850157143783)
+        assert score == 7.6777850157143783
+
+    @pytest.mark.parametrize("Xs", [
+        csc_matrix(X), csr_matrix(X),
+    ], ids=["csc", "csr"])
+    def test_distortion_score_sparse_matrix_input(self, Xs):
+        """
+        Test the distortion score metric on a sparse array
+        """
+        score = distortion_score(Xs, y)
+        assert score == pytest.approx(7.6777850157143783)
+
+    @pytest.mark.skipif(pd is None, reason="pandas is required")
+    def test_distortion_score_pandas_input(self):
+        """
+        Test the distortion score metric on pandas DataFrame and Series
+        """
+        df = pd.DataFrame(X)
+        s = pd.Series(y)
+
+        score = distortion_score(df, s)
+        assert score == pytest.approx(7.6777850157143783)
 
 
 ##########################################################################
 ## KElbowVisualizer Test Cases
 ##########################################################################
 
-class KElbowVisualizerTests(VisualTestCase, DatasetMixin):
+class TestKElbowVisualizer(VisualTestCase, DatasetMixin):
     """
     K-Elbow Visualizer Tests
     """
 
-    @pytest.mark.skip("images not close due to timing lines")
+    @pytest.mark.xfail(reason="images not close due to timing lines")
     def test_integrated_kmeans_elbow(self):
         """
         Test no exceptions for kmeans k-elbow visualizer on blobs dataset
@@ -92,12 +121,12 @@ def test_integrated_kmeans_elbow(self):
 
         # Generate a blobs data set
         X,y = make_blobs(
-            n_samples=1000, n_features=12, centers=6, shuffle=True, random_state=42
+            n_samples=1000, n_features=12, centers=6,
+            shuffle=True, random_state=42
         )
 
         try:
-            fig = plt.figure()
-            ax = fig.add_subplot()
+            _, ax = plt.subplots()
 
             visualizer = KElbowVisualizer(KMeans(random_state=42), k=4, ax=ax)
             visualizer.fit(X)
@@ -105,9 +134,9 @@ def test_integrated_kmeans_elbow(self):
 
             self.assert_images_similar(visualizer)
         except Exception as e:
-            self.fail("error during k-elbow: {}".format(e))
+            pytest.fail("error during k-elbow: {}".format(e))
 
-    @pytest.mark.skip("images not close due to timing lines")
+    @pytest.mark.xfail(reason="images not close due to timing lines")
     def test_integrated_mini_batch_kmeans_elbow(self):
         """
         Test no exceptions for mini-batch kmeans k-elbow visualizer
@@ -120,37 +149,57 @@ def test_integrated_mini_batch_kmeans_elbow(self):
         )
 
         try:
-            fig = plt.figure()
-            ax = fig.add_subplot()
+            _, ax = plt.subplots()
 
-            visualizer = KElbowVisualizer(MiniBatchKMeans(random_state=42), k=4, ax=ax)
+            visualizer = KElbowVisualizer(
+                MiniBatchKMeans(random_state=42), k=4, ax=ax
+            )
             visualizer.fit(X)
             visualizer.poof()
 
             self.assert_images_similar(visualizer)
         except Exception as e:
-            self.fail("error during k-elbow: {}".format(e))
+            pytest.fail("error during k-elbow: {}".format(e))
+
+    @pytest.mark.skip(reason="takes over 20 seconds to run")
+    def test_topic_modeling_k_means(self):
+        """
+        Test topic modeling k-means on the hobbies corpus
+        """
+        corpus = self.load_corpus("hobbies")
+
+        tfidf  = TfidfVectorizer()
+        docs   = tfidf.fit_transform(corpus.data)
+        visualizer = KElbowVisualizer(KMeans(), k=(4, 8))
+
+        visualizer.fit(docs)
+        visualizer.poof()
+
+        self.assert_images_similar(visualizer)
 
     def test_invalid_k(self):
         """
         Assert that invalid values of K raise exceptions
         """
 
-        with self.assertRaises(YellowbrickValueError):
+        with pytest.raises(YellowbrickValueError):
             KElbowVisualizer(KMeans(), k=(1,2,3,4,5))
 
-        with self.assertRaises(YellowbrickValueError):
+        with pytest.raises(YellowbrickValueError):
             KElbowVisualizer(KMeans(), k="foo")
 
     def test_distortion_metric(self):
         """
         Test the distortion metric of the k-elbow visualizer
         """
-        visualizer = KElbowVisualizer(KMeans(random_state=0), k=5, metric="distortion", timings=False)
+        visualizer = KElbowVisualizer(
+            KMeans(random_state=0), k=5, metric="distortion", timings=False
+        )
         visualizer.fit(X)
 
         expected = np.array([ 7.677785,  8.364319,  8.893634,  8.013021])
-        self.assertEqual(len(visualizer.k_scores_), 4)
+        assert len(visualizer.k_scores_) == 4
+
         visualizer.poof()
         self.assert_images_similar(visualizer)
         assert_array_almost_equal(visualizer.k_scores_, expected)
@@ -159,11 +208,14 @@ def test_silhouette_metric(self):
         """
         Test the silhouette metric of the k-elbow visualizer
         """
-        visualizer = KElbowVisualizer(KMeans(random_state=0), k=5, metric="silhouette", timings=False)
+        visualizer = KElbowVisualizer(
+            KMeans(random_state=0), k=5, metric="silhouette", timings=False
+        )
         visualizer.fit(X)
 
         expected = np.array([ 0.691636,  0.456646,  0.255174,  0.239842])
-        self.assertEqual(len(visualizer.k_scores_), 4)
+        assert len(visualizer.k_scores_) == 4
+
         visualizer.poof()
         self.assert_images_similar(visualizer)
         assert_array_almost_equal(visualizer.k_scores_, expected)
@@ -172,15 +224,19 @@ def test_calinski_harabaz_metric(self):
         """
         Test the calinski-harabaz metric of the k-elbow visualizer
         """
-        visualizer = KElbowVisualizer(KMeans(random_state=0), k=5, metric="calinski_harabaz", timings=False)
+        visualizer = KElbowVisualizer(
+            KMeans(random_state=0), k=5,
+            metric="calinski_harabaz", timings=False
+        )
         visualizer.fit(X)
+        assert len(visualizer.k_scores_) == 4
 
         expected = np.array([
             81.662726256035683, 50.992378259195554,
             40.952179227847012, 35.939494
         ])
 
-        self.assertEqual(len(visualizer.k_scores_), 4)
+
         visualizer.poof()
         self.assert_images_similar(visualizer)
         assert_array_almost_equal(visualizer.k_scores_, expected)
@@ -189,28 +245,33 @@ def test_bad_metric(self):
         """
         Assert KElbow raises an exception when a bad metric is supplied
         """
-        with self.assertRaises(YellowbrickValueError):
+        with pytest.raises(YellowbrickValueError):
             KElbowVisualizer(KMeans(), k=5, metric="foo")
 
     def test_timings(self):
         """
         Test the twinx double axes with k-elbow timings
         """
-        visualizer = KElbowVisualizer(KMeans(random_state=0), k=5, timings=True)
+        visualizer = KElbowVisualizer(
+            KMeans(random_state=0), k=5, timings=True
+        )
         visualizer.fit(X)
 
         # Check that we kept track of time
-        self.assertEqual(len(visualizer.k_timers_), 4)
-        self.assertTrue(all([t > 0 for t in visualizer.k_timers_]))
+        assert len(visualizer.k_timers_) == 4
+        assert all([t > 0 for t in visualizer.k_timers_])
 
         # Check that we plotted time on a twinx
-        self.assertTrue(hasattr(visualizer, "axes"))
-        self.assertEqual(len(visualizer.axes), 2)
+        assert hasattr(visualizer, "axes")
+        assert len(visualizer.axes) == 2
 
         # delete the timings axes and
         # overwrite k_timers_, k_values_ for image similarity Tests
         visualizer.axes[1].remove()
-        visualizer.k_timers_ = [0.01084589958190918, 0.011144161224365234, 0.017028093338012695, 0.010634183883666992]
+        visualizer.k_timers_ = [
+            0.01084589958190918, 0.011144161224365234,
+            0.017028093338012695, 0.010634183883666992
+        ]
         visualizer.k_values_ = [2, 3, 4, 5]
 
         # call draw again which is normally called in fit