Repository: DistrictDataLabs/yellowbrick

Issue:
Demonstration/walkthrough proposal on how to use ClassBalance (and variations) to bin continuous values for classification
I. Title 
Better Binning for Classification Problems: 
Creating Categorical Values from Continuous Values

II. Premise
- A lot of machine learning problems in the real world suffer from the curse of dimensionality; you have fewer training instances than you’d like, and predictive signal is distributed (often unpredictably!) across many different features. 
- One example is when your target is continuously-valued, but there aren’t enough instances to predict these values to the precision of regression.
- What if we transform the regression problem into a classification problem? We can try to do this by binning the continuous values into buckets for classification. But how do we pick the bins?

III. Dataset Intro
- About the Pitchfork album reviews corpus - funny! snarky! sentiment analysis??
- Download the data from https://www.kaggle.com/nolanbconaway/pitchfork-data/data
- Custom CorpusReader to access the text and scores
- Custom TextNormalizer to lemmatize and remove stop words 
- use Numpy digitize method to naively bin the continuous target values

IV. Preliminary Text Analytics Pipeline
- Build using Scikit-Learn Pipeline
- use ConfusionMatrix to visually evaluate
- use ClassBalance to visualize imbalance
- talk through selection bias - why initial bins didn’t work

V. Tuning Bins with ClassBalance
- redo with better distributed bins for target values
- (hopefully) show better results

VI. Conclusion/Teaser for New ClassBalanceHeatmap Visualizer
- how to combine insight from ConfusionMatrix with interpretability of ClassBalance?

Reviewers:
@marskar
@lwgray
@yzyzy


Discussion:
You beat me to it... But here is my attempt at transcribing your notes
[Sane_Binning.pdf](https://github.com/DistrictDataLabs/yellowbrick/files/1769942/Sane_Binning.pdf)

Has the coding for the corpus reader, TextBormalizer, and Pipeline been completed?
Hey there @lwgray - yes; here's the `TextNormalizer`:

```
class TextNormalizer(BaseEstimator, TransformerMixin):

    def __init__(self, language='english'):
        self.stopwords  = set(nltk.corpus.stopwords.words(language))
        self.lemmatizer = WordNetLemmatizer()

    def is_punct(self, token):
        return all(
            unicodedata.category(char).startswith('P') for char in token
        )

    def is_stopword(self, token):
        return token.lower() in self.stopwords

    def normalize(self, document):
        return [
            self.lemmatize(token, tag).lower()
            for sentence in document
            for (token, tag) in sentence
            if not self.is_punct(token)
               and not self.is_stopword(token)
        ]

    def lemmatize(self, token, pos_tag):
        tag = {
            'N': wn.NOUN,
            'V': wn.VERB,
            'R': wn.ADV,
            'J': wn.ADJ
        }.get(pos_tag[0], wn.NOUN)

        return self.lemmatizer.lemmatize(token, tag)

    def fit(self, documents, y=None):
        return self

    def transform(self, documents):
        return [
            ' '.join(self.normalize(doc)) for doc in documents
        ]
```

I should have some time to work on the draft tomorrow, so I'll post the corpus reader and preprocessor then!
If you need any help with the coding, just let me know 😄 


I know you have done all the analysis but I produced this so I could better understand the data and possible workflow

### Confusion Matrix and Class Balance: Before Class Adjustment
![confusionmatrix-before](https://user-images.githubusercontent.com/1000117/36933177-c0eb9734-1ea2-11e8-9827-7ef89ca64a12.png)
![classbalance-before](https://user-images.githubusercontent.com/1000117/36933180-d1936210-1ea2-11e8-845b-3772dcddb7f6.png)

### Confusion Matrix and Class Balance: After Class Adjustment
![confusionmatrix-after](https://user-images.githubusercontent.com/1000117/36933185-d7feda8a-1ea2-11e8-9dde-1e0f605eca49.png)
![classbalance-after](https://user-images.githubusercontent.com/1000117/36933187-e74418c0-1ea2-11e8-987c-09ad8c97a198.png)

@lwgray nice! Would you be interested in taking a crack at pulling the prototype code that @bbengfort wrote into a new `ClassBalanceHeatMap` visualizer using the [Yellowbrick API](http://www.scikit-yb.org/en/latest/contributing.html#visualizer-api)? It would be awesome to be able to reference the work-in-progress in my post, and then maybe you could do a follow-up post on creating a new Yellowbrick visualizer? 

Here's the prototype code:

```python
import numpy as np 
import matplotlib.pyplot as plt 

from matplotlib import cm 
from sklearn.utils.multiclass import unique_labels 
from sklearn.metrics.classification import _check_targets


def plot_class_balance_preds(y_true, y_pred, labels=None, ax=None):
    # Use Sklearn tools to validate the target 
    # Note y_true and y_pred should already be label encoded 
    y_type, y_true, y_pred = _check_targets(y_true, y_pred)
    indices = unique_labels(y_true, y_pred)
        
    # Create a 2D numpy array where each row is the count of 
    # the predicted classes and each column is the true class 
    data = np.array([
        [(y_pred[y_true==label_t] == label_p).sum() for label_p in indices]
        for label_t in indices 
    ])
    
    # Ensure that the number of elements in data matches y_pred and y_true 
    # Not necessary but used as a sanity check
    assert data.sum() == len(y_pred) == len(y_true)
    
    # labels_present is the indices of the classes, labels is the string names 
    # Another sanity check, this will not prevent missing classes, which is bad
    labels = labels if labels is not None else indices
    assert len(labels) == len(indices)
    
    # Create a matplotlib axis 
    if ax is None:
        _, ax = plt.subplots()
      
    # Create a unique color for each predict class 
    colors = [cm.spectral(x) for x in np.linspace(0, 1, len(indices))]
    
    # Track the stack of the bar graph 
    prev = np.zeros(len(labels))
    
    # Plot each row 
    for idx, row in enumerate(data):
        ax.bar(indices, row, label=labels[idx], bottom=prev, color=colors[idx])
        prev += row 
    
    # Make the graph pretty 
    ax.set_xticks(indices)
    ax.set_xticklabels(labels)
    ax.set_xlabel("actual class")
    ax.set_ylabel("number of predicted class")

    # Put the legend outside of the graph 
    plt.legend(bbox_to_anchor=(1.04,0.5), loc="center left")
    plt.tight_layout(rect=[0,0,0.85,1])
    
    return ax


## Usage 
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split as tts
from sklearn.ensemble import GradientBoostingClassifier

digits = load_digits()
X_train, X_test, y_train, y_true = tts(digits.data, digits.target, test_size=0.33)

model = GradientBoostingClassifier() 
model.fit(X_train, y_train)

y_pred = model.predict(X_test)
g = plot_class_balance_preds(y_true, y_pred, labels=digits.target_names)

plt.show()
```

If you're looking for a model for the Yellowbrick PR process, you can check out [the one @bbengfort did](https://github.com/DistrictDataLabs/yellowbrick/pull/317) that @ndanielsen and I are reviewing now.
FYI @marskar @lwgray @yzyzy - Still have some sections to flesh out, but you can see my draft in development [here](https://github.com/rebeccabilbro/rebeccabilbro.github.io/blob/master/_drafts/critical-binning.md). 
I will give it a shot.  However, this is my first implementation of a visualizer.  I might need your assistance if I get stuck.   😟  
Awesome! Sure thing @lwgray -- definitely start by checking out the [Visualizer API description](http://www.scikit-yb.org/en/latest/contributing.html#visualizer-api) in the docs. The best strategy is to open a pull request early that describes and scopes the task so that us maintainers can be ready and available to assist as needed. Look forward to seeing what you come up with!
@rebeccabilbro I read through your drafts 3 times.... I think it is fun and fairly thorough.  The only question I have is, will you discuss changing your binning so that the classes are better balanced?

Also, do you think we could talk for 10 minutes at the March 14th meeting, I would like to understand some of your coding decisions.
@rebeccabilbro I read through the draft and think it's great. My question here is it seems to me that you are trying to see if they can build a ClassBalanceHeatmap visualizer so you can see where your prediction goes wrong with different classes in a bar chart. However, this doesn't seem like solve your problem of the non-balanced sample from the beginning when you try to assign them from different scores to classes. 
@rebeccabilbro I created a visualizer to help with balanced binning. You can create a balanced binning based on the referenced value created from the visualizer. You can check it via the following link: 
https://github.com/Juan0001/yellowbrick-balanced-bin-reference/blob/master/balanced_binning.ipynb

I was trying to put the package on yellowbrick, but don't have a permission. Could you help me? Thank you.
@Juan0001 very nice! We'd love to review your work and add it to Yellowbrick. The way we do this is to have you fork the Yellowbrick repository into your own GitHub account; once you've done that you can create a pull request so that we can go over your additions, and once approved we merge them into Yellowbrick. 

Detailed instructions are here: [Contributing to Yellowbrick](http://www.scikit-yb.org/en/develop/contributing.html#getting-started-on-github) but of course we're happy to go over it with you tonight. 
Ok, draft now published [here](https://rebeccabilbro.github.io/better-binning/).
@rebeccabilbro @Juan0001 
To me it isn't obvious what the reader can get out of visiting Juan's jupyter notebook.  Maybe say something like "If you are looking for an automated way to create balanced binning then checkout this visualizer in the works"
@lwgray - I believe @Juan0001's notebook is in draft form and she's still working on fleshing out some of the possible use cases.
@rebeccabilbro Thank you very much!
@lwgray You are right, the notebook version is not very clear yet. It's first version of the draft. In the later versions I will explain why I created this function, what can it do and how to use it in more details. Please let me know if there's anything else I need to improve, thank you very much.

Resolving patch:
diff --git a/yellowbrick/target/__init__.py b/yellowbrick/target/__init__.py
--- a/yellowbrick/target/__init__.py
+++ b/yellowbrick/target/__init__.py
@@ -20,3 +20,4 @@
 
 # Hoist visualizers into the top level of the target package
 from .class_balance import ClassBalance, class_balance
+from .binning import BalancedBinningReference, balanced_binning_reference
diff --git a/yellowbrick/target/binning.py b/yellowbrick/target/binning.py
new file mode 100644
--- /dev/null
+++ b/yellowbrick/target/binning.py
@@ -0,0 +1,177 @@
+
+# yellowbrick.target.binning
+# Implementations of histogram with vertical lines to help with balanced binning.
+#
+# Author:   Juan L. Kehoe (juanluo2008@gmail.com)
+# Author:   Prema Damodaran Roman (pdamo24@gmail.com)
+
+# Created:  Tue Mar 13 19:50:54 2018 -0400
+#
+# Copyright (C) 2018 District Data Labs
+# For license information, see LICENSE.txt
+#
+# ID: binning.py
+
+"""
+Implements histogram with vertical lines to help with balanced binning.
+"""
+
+##########################################################################
+## Imports
+##########################################################################
+import matplotlib.pyplot as plt
+import numpy as np
+
+from .base import TargetVisualizer
+from yellowbrick.exceptions import YellowbrickValueError
+
+##########################################################################
+## Balanced Binning Reference
+##########################################################################
+
+class BalancedBinningReference(TargetVisualizer):
+    """
+    BalancedBinningReference allows to generate a histogram with vertical lines
+    showing the recommended value point to bin your data so they can be evenly
+    distributed in each bin.
+
+    Parameters
+    ----------
+    ax : matplotlib Axes, default: None
+        This is inherited from FeatureVisualizer and is defined within
+        BalancedBinningReference.
+    target : string, default: "Frequency"
+        The name of the y variable
+    bins : number of bins to generate the histogram, default: 4
+    kwargs : dict
+        Keyword arguments that are passed to the base class and may influence
+        the visualization as defined in other Visualizers.
+        
+    Attributes
+    ----------
+    bin_edges : binning reference values
+
+    Examples
+    --------
+    >>> visualizer = BalancedBinningReference()
+    >>> visualizer.fit(y)
+    >>> visualizer.poof()
+
+
+    Notes
+    -----
+    These parameters can be influenced later on in the visualization
+    process, but can and should be set as early as possible.
+    """
+
+    def __init__(self, ax=None, target=None, bins=4, **kwargs):
+
+        super(BalancedBinningReference, self).__init__(ax, **kwargs)
+
+        self.target = target
+        self.bins = bins
+
+    def draw(self, y, **kwargs):
+        """
+        Draws a histogram with the reference value for binning as vetical lines
+        Parameters
+        ----------
+        y : an array of one dimension or a pandas Series
+        """
+
+        # draw the histogram
+        hist, bin_edges = np.histogram(y, bins=self.bins)
+        self.bin_edges_ = bin_edges
+        self.ax.hist(y, bins=self.bins, color=kwargs.pop("color", "#6897bb"), **kwargs)
+
+        # add vetical line with binning reference values
+        plt.vlines(bin_edges,0,max(hist),colors=kwargs.pop("colors", "r"))
+
+    def fit(self, y, **kwargs):
+        """
+        Sets up y for the histogram and checks to
+        ensure that y is of the correct data type
+        Fit calls draw
+        Parameters
+        ----------
+        y : an array of one dimension or a pandas Series
+        kwargs : dict
+            keyword arguments passed to Scikit-Learn API.
+        """
+
+        #throw an error if y has more than 1 column
+        if y.ndim > 1:
+            raise YellowbrickValueError("y needs to be an array or Series with one dimension") 
+
+        # Handle the target name if it is None.
+        if self.target is None:
+            self.target = 'Frequency'
+
+        self.draw(y)
+        return self
+
+
+    def poof(self, **kwargs):
+        """
+        Creates the labels for the feature and target variables
+        """
+
+        self.ax.set_xlabel(self.target)
+        self.finalize(**kwargs)
+
+    def finalize(self, **kwargs):
+        """
+        Finalize executes any subclass-specific axes finalization steps.
+        The user calls poof and poof calls finalize.
+        Parameters
+        ----------
+        kwargs: generic keyword arguments.
+        """
+
+        for tk in self.ax.get_xticklabels():
+            tk.set_visible(True)
+            
+        for tk in self.ax.get_yticklabels():
+            tk.set_visible(True)
+        
+        
+##########################################################################
+## Quick Method
+##########################################################################
+        
+def balanced_binning_reference(y, ax=None, target='Frequency', bins=4, **kwargs):
+    
+    """
+    BalancedBinningReference allows to generate a histogram with vertical lines
+    showing the recommended value point to bin your data so they can be evenly
+    distributed in each bin.
+
+    Parameters
+    ----------
+    y : an array of one dimension or a pandas Series
+    
+    ax : matplotlib Axes, default: None
+        This is inherited from FeatureVisualizer and is defined within
+        BalancedBinningReference.
+    target : string, default: "Frequency"
+        The name of the y variable
+    bins : number of bins to generate the histogram, default: 4
+    kwargs : dict
+        Keyword arguments that are passed to the base class and may influence
+        the visualization as defined in other Visualizers.
+
+    """
+    
+    # Initialize the visualizer
+    visualizer = BalancedBinningReference(ax=ax, bins=bins, target=target, **kwargs)
+    
+    # Fit and poof the visualizer
+    visualizer.fit(y)
+    visualizer.poof()
+    
+    
+
+    
+    
+    
+


Test patch:
diff --git a/tests/baseline_images/test_target/test_binning/test_balancedbinningreference.png b/tests/baseline_images/test_target/test_binning/test_balancedbinningreference.png
new file mode 100644
Binary files /dev/null and b/tests/baseline_images/test_target/test_binning/test_balancedbinningreference.png differ
diff --git a/tests/test_target/test_binning.py b/tests/test_target/test_binning.py
new file mode 100644
--- /dev/null
+++ b/tests/test_target/test_binning.py
@@ -0,0 +1,39 @@
+# tests.test_target.test_binning
+# Tests for the BalancedBinningReference visualizer
+#
+# Author:   Juan L. Kehoe (juanluo2008@gmail.com)
+# Author:  Prema Damodaran Roman (pdamo24@gmail.com)
+# Created: Thu Jul 20 10:21:49 2018 -0400
+#
+# ID: test_binning.py
+
+from tests.base import VisualTestCase
+from tests.dataset import DatasetMixin
+from yellowbrick.target.binning import *
+
+##########################################################################
+## BalancedBinningReference Tests
+##########################################################################
+
+class TestBalancedBinningReference(VisualTestCase, DatasetMixin):
+	"""
+	Test the BalancedBinningReference visualizer 
+	"""
+
+	def test_balancedbinningreference(self):
+		"""
+		Test Histogram on a real dataset
+		"""
+		# Load the data from the fixture
+		dataset = self.load_data('occupancy')
+
+		# Get the data
+		y = dataset["temperature"]
+
+		
+		visualizer = BalancedBinningReference()
+		visualizer.fit(y)
+		visualizer.poof()
+		self.assert_images_similar(visualizer, tol=0.5)
+			
+		
\ No newline at end of file