Repository: DistrictDataLabs/yellowbrick

Issue:
MissingValuesBar does not work on categorical columns
## Describe the solution you'd like
Currently, I am unable to get MissingValuesBar to work on data that has categorical columns. It is internally trying to convert these columns to a floating point value and producing an error at this step. I am hoping this can be handles internally in the class itself since it is just checking for missing values so it should not care about the type of column.

## Is your feature request related to a problem? Please describe.

Described above, but this is the error that I get on categorical data. One of the columns has continent values and it looks like it is producing an error at that point.

```python
# Instantiate the visualizer
plt.figure(figsize=(15,10))
visualizer = MissingValuesBar(features=features)

visualizer.fit(X=data[features], y=data[target].values) # Supply the targets via y
_ = visualizer.show() # Finalize and render the figure
```

**Traceback**
```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-7-2b58032dfe22> in <module>
      3 visualizer = MissingValuesBar(features=features)
      4 
----> 5 visualizer.fit(X=data[features], y=data[target].values) # Supply the targets via y
      6 _ = visualizer.show() # Finalize and render the figure

~\.conda\envs\ds7337_cs3\lib\site-packages\yellowbrick\contrib\missing\base.py in fit(self, X, y, **kwargs)
     70         self.y = y
     71 
---> 72         self.draw(X, y, **kwargs)
     73         return self
     74 

~\.conda\envs\ds7337_cs3\lib\site-packages\yellowbrick\contrib\missing\bar.py in draw(self, X, y, **kwargs)
    137         target values.
    138         """
--> 139         nan_col_counts = self.get_nan_col_counts()
    140 
    141         # the x locations for the groups

~\.conda\envs\ds7337_cs3\lib\site-packages\yellowbrick\contrib\missing\bar.py in get_nan_col_counts(self, **kwargs)
    110 
    111         else:
--> 112             nan_matrix = self.X.astype(np.float)
    113 
    114         if self.y is None:

ValueError: could not convert string to float: 'euorpe'
```

**If I just take the numeric features, it works well**

```python
# Instantiate the visualizer
plt.figure(figsize=(15,10))
visualizer = MissingValuesBar(features=features_numerical)

visualizer.fit(X=data[features_numerical], y=data[target].values) # Supply the targets via y
_ = visualizer.show() # Finalize and render the figure
```

**See output below**

![image](https://user-images.githubusercontent.com/33585645/100137410-0cb02b00-2e52-11eb-8a0f-04a94c07dfdd.png)




Discussion:
Hi @ngupta23 thank you so much for using Yellowbrick! Sorry it's taken us so long to reply to your issue, it just got buried in the stream of GitHub messages. I apologize for that. Thank you for mentioning the MissingValuesBar visualizer - right now this visualizer is in our `contrib` package since I don't think we fully fleshed it out for prime time use in the package. I believe @ndanielsen  was working on it - @ndanielsen do you have any comments?

If this visualizer is useful to you, we'd love to have you contribute to our project and help make the visualizer ready for categorical variables and moved out of contrib! Let us know if you're interested and how we can advise you to get started! 
@ngupta23 thanks for the detailed bug report. I'll take a look at this. My suggestion for the time being is that you encode categorical columns into numeric.

For example utilizing the [LabelEncoder in scikit-learn](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html) would resolve this:

```
from sklearn import preprocessing

le = preprocessing.LabelEncoder()
df['label'] = le.fit_transform(df['label'])

```


Resolving patch:
diff --git a/yellowbrick/cluster/elbow.py b/yellowbrick/cluster/elbow.py
--- a/yellowbrick/cluster/elbow.py
+++ b/yellowbrick/cluster/elbow.py
@@ -130,6 +130,9 @@ def distortion_score(X, labels, metric="euclidean"):
     "calinski_harabasz": chs,
 }
 
+DISTANCE_METRICS = ['cityblock', 'cosine', 'euclidean', 'haversine',
+                    'l1', 'l2', 'manhattan', 'nan_euclidean', 'precomputed']
+
 
 class KElbowVisualizer(ClusteringScoreVisualizer):
     """
@@ -182,6 +185,12 @@ class KElbowVisualizer(ClusteringScoreVisualizer):
         - **silhouette**: mean ratio of intra-cluster and nearest-cluster distance
         - **calinski_harabasz**: ratio of within to between cluster dispersion
 
+    distance_metric : str or callable, default='euclidean'
+        The metric to use when calculating distance between instances in a 
+        feature array. If metric is a string, it must be one of the options allowed
+        by sklearn's metrics.pairwise.pairwise_distances. If X is the distance array itself,
+        use metric="precomputed".
+
     timings : bool, default: True
         Display the fitting time per k to evaluate the amount of time required
         to train the clustering model.
@@ -250,6 +259,7 @@ def __init__(
         ax=None,
         k=10,
         metric="distortion",
+        distance_metric='euclidean',
         timings=True,
         locate_elbow=True,
         **kwargs
@@ -263,11 +273,18 @@ def __init__(
                 "use one of distortion, silhouette, or calinski_harabasz"
             )
 
+        if distance_metric not in DISTANCE_METRICS:
+            raise YellowbrickValueError(
+                "'{} is not a defined distance metric "
+                "use one of the sklearn metric.pairwise.pairwise_distances"
+            )
+
         # Store the arguments
         self.scoring_metric = KELBOW_SCOREMAP[metric]
         self.metric = metric
         self.timings = timings
         self.locate_elbow = locate_elbow
+        self.distance_metric = distance_metric
 
         # Set the values of the colors
         self.colors = {
@@ -331,7 +348,11 @@ def fit(self, X, y=None, **kwargs):
 
             # Append the time and score to our plottable metrics
             self.k_timers_.append(time.time() - start)
-            self.k_scores_.append(self.scoring_metric(X, self.estimator.labels_))
+            if self.metric != 'calinski_harabasz':
+                self.k_scores_.append(self.scoring_metric(X, self.estimator.labels_,
+                                      metric=self.distance_metric))
+            else:
+                self.k_scores_.append(self.scoring_metric(X, self.estimator.labels_))
 
         if self.locate_elbow:
             locator_kwargs = {
@@ -465,6 +486,7 @@ def kelbow_visualizer(
     ax=None,
     k=10,
     metric="distortion",
+    distance_metric='euclidean',
     timings=True,
     locate_elbow=True,
     show=True,
@@ -504,6 +526,12 @@ def kelbow_visualizer(
                           distance
         - **calinski_harabasz**: ratio of within to between cluster dispersion
 
+    distance_metric : str or callable, default='euclidean'
+        The metric to use when calculating distance between instances in a 
+        feature array. If metric is a string, it must be one of the options allowed
+        by sklearn's metrics.pairwise.pairwise_distances. If X is the distance array itself,
+        use metric="precomputed".
+
     timings : bool, default: True
         Display the fitting time per k to evaluate the amount of time required
         to train the clustering model.
@@ -542,6 +570,7 @@ def kelbow_visualizer(
         ax=ax,
         k=k,
         metric=metric,
+        distance_metric='euclidean',
         timings=timings,
         locate_elbow=locate_elbow,
         **kwargs
diff --git a/yellowbrick/contrib/missing/bar.py b/yellowbrick/contrib/missing/bar.py
--- a/yellowbrick/contrib/missing/bar.py
+++ b/yellowbrick/contrib/missing/bar.py
@@ -100,17 +100,16 @@ def __init__(self, width=0.5, color=None, colors=None, classes=None, **kwargs):
         self.colors = color_palette(kwargs.pop("colors", None), n_colors)
 
     def get_nan_col_counts(self, **kwargs):
-        # where matrix contains strings, handle them
-        if np.issubdtype(self.X.dtype, np.string_) or np.issubdtype(
-            self.X.dtype, np.unicode_
+        if np.issubdtype(self.X.dtype, np.floating) or np.issubdtype(
+                self.X.dtype, np.integer
         ):
-            mask = np.where(self.X == "")
+            nan_matrix = self.X.astype(np.float64)
+        else:
+            # where matrix contains strings, handle them
+            mask = np.where((self.X == "") | (self.X == 'nan'))
             nan_matrix = np.zeros(self.X.shape)
             nan_matrix[mask] = np.nan
 
-        else:
-            nan_matrix = self.X.astype(np.float64)
-
         if self.y is None:
             nan_col_counts = [np.count_nonzero(np.isnan(col)) for col in nan_matrix.T]
             return nan_col_counts
@@ -126,7 +125,6 @@ def get_nan_col_counts(self, **kwargs):
                     [np.count_nonzero(np.isnan(col)) for col in target_matrix.T]
                 )
                 nan_counts.append((target_value, nan_col_counts))
-
             return nan_counts
 
     def draw(self, X, y, **kwargs):
diff --git a/yellowbrick/contrib/missing/base.py b/yellowbrick/contrib/missing/base.py
--- a/yellowbrick/contrib/missing/base.py
+++ b/yellowbrick/contrib/missing/base.py
@@ -66,6 +66,9 @@ def fit(self, X, y=None, **kwargs):
                 self.features_ = X.columns
         else:
             self.X = X
+            if self.features_ is None:
+                n_columns = X.shape[1]
+                self.features_ = np.arange(0, n_columns)
 
         self.y = y
 


Test patch:
diff --git a/tests/baseline_images/test_cluster/test_elbow/test_distance_metric.png b/tests/baseline_images/test_cluster/test_elbow/test_distance_metric.png
new file mode 100644
Binary files /dev/null and b/tests/baseline_images/test_cluster/test_elbow/test_distance_metric.png differ
diff --git a/tests/baseline_images/test_contrib/test_missing/test_bar/test_missingvaluesbar_numpy_no_features_passed.png b/tests/baseline_images/test_contrib/test_missing/test_bar/test_missingvaluesbar_numpy_no_features_passed.png
new file mode 100644
Binary files /dev/null and b/tests/baseline_images/test_contrib/test_missing/test_bar/test_missingvaluesbar_numpy_no_features_passed.png differ
diff --git a/tests/baseline_images/test_contrib/test_missing/test_bar/test_missingvaluesbar_numpy_with_mixed_dtypes.png b/tests/baseline_images/test_contrib/test_missing/test_bar/test_missingvaluesbar_numpy_with_mixed_dtypes.png
new file mode 100644
Binary files /dev/null and b/tests/baseline_images/test_contrib/test_missing/test_bar/test_missingvaluesbar_numpy_with_mixed_dtypes.png differ
diff --git a/tests/baseline_images/test_contrib/test_missing/test_bar/test_missingvaluesbar_numpy_with_string_and_bool_cols.png b/tests/baseline_images/test_contrib/test_missing/test_bar/test_missingvaluesbar_numpy_with_string_and_bool_cols.png
new file mode 100644
Binary files /dev/null and b/tests/baseline_images/test_contrib/test_missing/test_bar/test_missingvaluesbar_numpy_with_string_and_bool_cols.png differ
diff --git a/tests/baseline_images/test_contrib/test_missing/test_bar/test_missingvaluesbar_pandas_no_features_passed.png b/tests/baseline_images/test_contrib/test_missing/test_bar/test_missingvaluesbar_pandas_no_features_passed.png
new file mode 100644
Binary files /dev/null and b/tests/baseline_images/test_contrib/test_missing/test_bar/test_missingvaluesbar_pandas_no_features_passed.png differ
diff --git a/tests/baseline_images/test_contrib/test_missing/test_bar/test_missingvaluesbar_pandas_with_mixed_dtypes.png b/tests/baseline_images/test_contrib/test_missing/test_bar/test_missingvaluesbar_pandas_with_mixed_dtypes.png
new file mode 100644
Binary files /dev/null and b/tests/baseline_images/test_contrib/test_missing/test_bar/test_missingvaluesbar_pandas_with_mixed_dtypes.png differ
diff --git a/tests/baseline_images/test_contrib/test_missing/test_bar/test_missingvaluesbar_pandas_with_string_and_bool_cols.png b/tests/baseline_images/test_contrib/test_missing/test_bar/test_missingvaluesbar_pandas_with_string_and_bool_cols.png
new file mode 100644
Binary files /dev/null and b/tests/baseline_images/test_contrib/test_missing/test_bar/test_missingvaluesbar_pandas_with_string_and_bool_cols.png differ
diff --git a/tests/test_cluster/test_elbow.py b/tests/test_cluster/test_elbow.py
--- a/tests/test_cluster/test_elbow.py
+++ b/tests/test_cluster/test_elbow.py
@@ -296,6 +296,29 @@ def test_calinski_harabasz_metric(self):
         self.assert_images_similar(visualizer)
         assert_array_almost_equal(visualizer.k_scores_, expected)
 
+    @pytest.mark.xfail(sys.platform == "win32", reason="images not close on windows")
+    def test_distance_metric(self):
+        """
+        Test the manhattan distance metric of the distortion metric of the k-elbow visualizer
+        """
+        visualizer = KElbowVisualizer(
+            KMeans(random_state=0),
+            k=5,
+            metric="distortion",
+            distance_metric='manhattan',
+            timings=False,
+            locate_elbow=False,
+        )
+        visualizer.fit(self.clusters.X)
+        assert len(visualizer.k_scores_) == 4
+        assert visualizer.elbow_value_ is None
+
+        expected = np.array([189.060129, 154.096223, 124.271208, 107.087566])
+
+        visualizer.finalize()
+        self.assert_images_similar(visualizer)
+        assert_array_almost_equal(visualizer.k_scores_, expected)
+
     @pytest.mark.xfail(
         IS_WINDOWS_OR_CONDA,
         reason="computation of k_scores_ varies by 2.867 max absolute difference",
@@ -347,6 +370,13 @@ def test_bad_metric(self):
         with pytest.raises(YellowbrickValueError):
             KElbowVisualizer(KMeans(), k=5, metric="foo")
 
+    def test_bad_distance_metric(self):
+        """
+        Assert KElbow raises an exception when a bad distance metric is supplied
+        """
+        with pytest.raises(YellowbrickValueError):
+            KElbowVisualizer(KMeans(), k=5, distance_metric="foo")
+
     @pytest.mark.xfail(
         IS_WINDOWS_OR_CONDA,
         reason="font rendering different in OS and/or Python; see #892",
diff --git a/tests/test_contrib/test_missing/test_bar.py b/tests/test_contrib/test_missing/test_bar.py
--- a/tests/test_contrib/test_missing/test_bar.py
+++ b/tests/test_contrib/test_missing/test_bar.py
@@ -70,6 +70,31 @@ def test_missingvaluesbar_pandas(self):
         viz = MissingValuesBar(features=features)
         viz.fit(X_)
         viz.finalize()
+        
+        self.assert_images_similar(viz, tol=self.tol)
+
+    @pytest.mark.skipif(pd is None, reason="pandas is required")
+    def test_missingvaluesbar_pandas_no_features_passed(self):
+        """
+        Integration test of visualizer with pandas
+        """
+        X, y = make_classification(
+            n_samples=400,
+            n_features=20,
+            n_informative=8,
+            n_redundant=8,
+            n_classes=2,
+            n_clusters_per_class=4,
+            random_state=854,
+        )
+
+        # add nan values to a range of values in the matrix
+        X[X > 1.5] = np.nan
+        X_ = pd.DataFrame(X)
+
+        viz = MissingValuesBar()
+        viz.fit(X_)
+        viz.finalize()
 
         self.assert_images_similar(viz, tol=self.tol)
 
@@ -97,6 +122,29 @@ def test_missingvaluesbar_numpy(self):
 
         self.assert_images_similar(viz, tol=self.tol)
 
+    def test_missingvaluesbar_numpy_no_features_passed(self):
+        """
+        Integration test of visualizer with numpy without target y passed in
+        """
+        X, y = make_classification(
+            n_samples=400,
+            n_features=20,
+            n_informative=8,
+            n_redundant=8,
+            n_classes=2,
+            n_clusters_per_class=4,
+            random_state=856,
+        )
+
+        # add nan values to a range of values in the matrix
+        X[X > 1.5] = np.nan
+
+        viz = MissingValuesBar()
+        viz.fit(X)
+        viz.finalize()
+
+        self.assert_images_similar(viz, tol=self.tol)
+
     def test_missingvaluesbar_numpy_with_y_target(self):
         """
         Integration test of visualizer with numpy without target y passed in
@@ -146,3 +194,136 @@ def test_missingvaluesbar_numpy_with_y_target_with_labels(self):
         viz.finalize()
 
         self.assert_images_similar(viz, tol=self.tol)
+
+    def test_missingvaluesbar_numpy_with_string_and_bool_cols(self):
+        """
+        Integration test of visualizer with numpy array with string and boolean columns
+        """
+        X, y = make_classification(
+            n_samples=400,
+            n_features=10,
+            n_informative=2,
+            n_redundant=3,
+            n_classes=2,
+            n_clusters_per_class=2,
+            random_state=854
+        )
+
+        # add nan values to a range of values in the matrix
+        X[X > 1.5] = np.nan
+
+        rng = np.random.default_rng(2021)
+        fruit_choices = np.array(['apples', 'pears', 'peaches', "", np.nan, 'bananas'])
+        fruits = rng.choice(fruit_choices, (400, 1))
+
+        bool_choices = np.array([np.nan, False, True])
+        booleans = rng.choice(bool_choices, (400, 1))
+
+        X = np.append(X, fruits, axis=1)
+        X = np.append(X, booleans, axis=1)
+
+        features = [str(n) for n in range(12)]
+        viz = MissingValuesBar(features=features)
+        viz.fit(X, y)
+        viz.finalize()
+
+        self.assert_images_similar(viz, tol=5)
+
+    @pytest.mark.skipif(pd is None, reason="pandas is required")
+    def test_missingvaluesbar_pandas_with_string_and_bool_cols(self):
+        """
+        Integration test of visualizer with pandas dataframe with string and boolean columns
+        """
+        X, y = make_classification(
+            n_samples=400,
+            n_features=10,
+            n_informative=2,
+            n_redundant=3,
+            n_classes=2,
+            n_clusters_per_class=2,
+            random_state=854
+        )
+
+        # add nan values to a range of values in the matrix
+        X[X > 1.5] = np.nan
+
+        rng = np.random.default_rng(2021)
+        fruit_choices = np.array(['apples', 'pears', 'peaches', "", np.nan, 'bananas'])
+        fruits = rng.choice(fruit_choices, (400, 1))
+
+        bool_choices = np.array([np.nan, False, True])
+        booleans = rng.choice(bool_choices, (400, 1))
+
+        X = np.append(X, fruits, axis=1)
+        X = np.append(X, booleans, axis=1)
+
+        X_ = pd.DataFrame(X)
+
+        features = [str(n) for n in range(12)]
+        viz = MissingValuesBar(features=features)
+        viz.fit(X_, y)
+        viz.finalize()
+
+        self.assert_images_similar(viz, tol=5)
+
+    def test_missingvaluesbar_numpy_with_mixed_dtypes(self):
+        """
+        Integration test of visualizer with numpy array with mixed dtypes in a single column
+        """
+        X, y = make_classification(
+            n_samples=400,
+            n_features=10,
+            n_informative=2,
+            n_redundant=3,
+            n_classes=2,
+            n_clusters_per_class=2,
+            random_state=854
+        )
+
+        # add nan values to a range of values in the matrix
+        X[X > 1.5] = np.nan
+
+        rng = np.random.default_rng(2021)
+        mixed_dtype_choices = np.array(['apples', 'pears', 'peaches', "", np.nan, 'bananas', 1, 2.4, 5.6, False, True])
+        mixed_dtypes = rng.choice(mixed_dtype_choices, (400, 1))
+
+        X_with_mixed_dtypes = np.append(X, mixed_dtypes, axis=1)
+
+        features = [str(n) for n in range(11)]
+        viz = MissingValuesBar(features=features)
+        viz.fit(X_with_mixed_dtypes, y)
+        viz.finalize()
+
+        self.assert_images_similar(viz, tol=5)
+
+    @pytest.mark.skipif(pd is None, reason="pandas is required")
+    def test_missingvaluesbar_pandas_with_mixed_dtypes(self):
+        """
+        Integration test of visualizer with pandas dataframe with mixed dtypes in a single column
+        """
+        X, y = make_classification(
+            n_samples=400,
+            n_features=10,
+            n_informative=2,
+            n_redundant=3,
+            n_classes=2,
+            n_clusters_per_class=2,
+            random_state=854
+        )
+
+        # add nan values to a range of values in the matrix
+        X[X > 1.5] = np.nan
+
+        rng = np.random.default_rng(2021)
+        mixed_dtype_choices = np.array(['apples', 'pears', 'peaches', "", np.nan, 'bananas', 1, 2.4, 5.6, False, True])
+        mixed_dtypes = rng.choice(mixed_dtype_choices, (400, 1))
+
+        X_with_mixed_dtypes = np.append(X, mixed_dtypes, axis=1)
+        X_ = pd.DataFrame(X_with_mixed_dtypes)
+
+        features = [str(n) for n in range(11)]
+        viz = MissingValuesBar(features=features)
+        viz.fit(X_, y)
+        viz.finalize()
+
+        self.assert_images_similar(viz, tol=5)