Introducing the “Banana Test” for Near-Perfect Text Classification Models
By: Sj Porter
Tags: Machine Learning, Testing, Text Classification, Deep Learning, Towards Data Science

Introducing the “Banana Test” for Near-Perfect Text Classification Models

Combine spelling correction, term filtering, and the “Banana Test” to build bullet-proof text classification models

Introduction

If your goal is maximizing accuracy for a new text classification model, you should consider using the Banana Test.

There are millions of instances in which businesses have collected free-form text in their systems. In order to automate business processes that utilize this data, the free-form text often needs to be bucketed into higher-level categories. Text classification models are capable of classifying such free-form text quickly and effectively… for the most part.

Regardless of the reported validation/test accuracy, there are a number of gotchas that can cause even the most well-trained text classification model to fail miserably at making an accurate classification. In particular, text classification models tend to fail when faced with text they have never seen seen before.

If your goal is maximizing accuracy for a new text classification model, you should consider using the Banana Test. Despite the silly name, it’s a very real technique that you can use to improve the quality of your models.

What is the Banana Test?

The Banana Test is intended to be performed on new data based on the terms that a model was trained on. It should be applied as one of the primary deciding factors (usually in combination with the confidence interval) for whether a given classification can be trusted. Since a machine learning model will always classify data by default and the confidence interval it provides is little more than a measure of probability (rather than accuracy), the Banana Test ensures that a model only classifies new data when it has more than enough training data to make a confident decision. How it works and why it’s important are both very simple concepts.

To explain the name of the test, the word “banana” has never been relevant to any company I’ve worked for. Due to my prior experience as a product owner before branching into data science, I came to believe that defining proper acceptance criteria is essential to developing a good product. So, as a form of acceptance criteria for any new text classification model being built, I enforce that my models should never confidently classify a text string that has the word “banana” in it.

Why not? Well…

An ideal text classification model would be trained on a vast text corpus containing just about every term that shows up in production. If a model has been trained on a really big heap of data, then any terms that are not present in the text corpus are almost certain to be odd or unusual. This indicates the term(s) might be associated with an untrained class, misspelled, or completely discrepant/erroneous data (e.g. text entered into the wrong field). In the interest of not making mistakes, a model would only classify text when it is well-trained on the data it is looking at and very confident in the answer. If a model has never seen one or more term(s) in a provided text string, then there’s no way to guarantee it can be confident in making a classification. There may be many instances in which it will still get the answer correct, but there’s also a significant chance that the untrained term(s) may change the answer. In the interest of striving for perfection, the model should self-report that it doesn’t know the answer when faced with text data it has never seen before. Ideally, the model would also self-report on the term(s) that it failed to recognize.

So, the Banana Test is simply the practice of comparing the list of terms in the input data you want to classify against the list of terms that the model has been trained on and determining whether or not there is full coverage. If there are terms that aren’t contained in the training data, then the input data fails the Banana Test and should probably not be classified by the model. This can be done extremely easily in Python using a set intersection (or a number of other solutions); the specific implementation is up to you.

This strategy has led to extremely accurate and reliable text classification models. The trade-off is a slight reduction to the percentage of records that these models will classify — however, a significant amount of trust can be placed in the models when they do make a confident classification. Also, with enough training data, models can still classify 90%+ of records even with a conservative configuration of the Banana Test (see the “Banana Split” section below). This helps ensure production-readiness even in business-critical use cases.

Practical Example

Let’s say you’re training a tensorflow model to distinguish beers from wines, and the input data is provided as free-form text. Let’s also say that your model accepts (5) terms as a padded array based on a tokenizer and word index.

After preprocessing and tokenizing some training data, your word index may look something like this:

{

"beer": 1,

"wine": 2,

"pinot": 3,

"noir": 4,

"chardonnay": 5

"lager": 6,

"cabernet": 7,

"cab": 8,

"rose": 9,

"ipa": 10,

"pilsner": 11,

"zinfandel": 12,

"chianti": 13,

"porter": 14,

"stout": 15,

"sauvignon": 16,

"red": 17,

"blend": 18,

"merlot": 19,

"ale": 20,

"mead": 21

}

Here are some basic examples of input data:

If you provided the text "pinot noir" to the model, the resulting array would be [3, 4, 0, 0, 0] .

to the model, the resulting array would be . If you provided the text "red blend" , the resulting array would be [17, 18, 0, 0, 0] .

, the resulting array would be . If you provided the text "ipa" , the resulting array would be [10, 0, 0, 0, 0] .

However, there are many values you could provide to such a model that would result in mistakes. These examples demonstrate why the Banana Test is effective:

If you provided the text "banana" , the resulting array would be [0, 0, 0, 0, 0] . The term didn’t appear in the word index at all, but the array still got created based on the function you’re using to pad the input sequences. The resulting classification would show a high confidence interval for whichever of the two classes (beer or wine) show up most frequently in your training data. One idea to work around this problem would be to kick out all of the empty arrays before classifying them and call it a day, but…

, the resulting array would be . The term didn’t appear in the word index at all, but the array still got created based on the function you’re using to pad the input sequences. The resulting classification would show a high confidence interval for whichever of the two classes (beer or wine) show up most frequently in your training data. One idea to work around this problem would be to kick out all of the empty arrays before classifying them and call it a day, but… If you provided the text "neither wine nor beer" , the resulting array would be [0, 2, 0, 1, 0] . Assuming you decided to kick out empty arrays, the model would still classify this record. The confidence interval of the classification may also be surprisingly high depending on how much training data you have for each of the two classes (i.e. if you have 300 records of training data for wine and only 20 for beer, the model would be highly confident that this input belongs to the wine class).

, the resulting array would be . Assuming you decided to kick out empty arrays, the model would still classify this record. The confidence interval of the classification may also be surprisingly high depending on how much training data you have for each of the two classes (i.e. if you have 300 records of training data for wine and only 20 for beer, the model would be highly confident that this input belongs to the wine class). If you provided the text "this is a beer" , the resulting array would be [0, 0, 0, 1, 0] . The model would still likely classify this record correctly, but it’s clear that the input could’ve said "definitely not a beer" and the resulting array would’ve been exactly the same.

, the resulting array would be . The model would still likely classify this record correctly, but it’s clear that the input could’ve said and the resulting array would’ve been exactly the same. If you provided the text "bourbon" , the resulting array would be [0, 0, 0, 0, 0] . This should probably be a brand new class (liquor).

Based on the examples above, wouldn’t it be best if your model were able to self-report that it saw new terms that it was never trained on and avoid classifying these discrepant records?

By applying the Banana Test before making a prediction, your model would be able to kick out each of the records above and provide an array of terms that failed the test such as bourbon , banana , definitely , etc. This approach would allow you to create a very meaningful feedback loop. It also means you could put your model into production and know that it will only classify records when it recognizes each of the terms.

Prerequisites for the Banana Test

There are a few issues that can occur when applying the Banana Test. Namely, the Banana Test may flag terms that were filtered out during the training process and/or fail to identify misspellings.

The following steps are what I would consider to be prerequisites for applying the Banana Test so that you can avoid such issues. No matter how you decide to preprocess your data, the exact same steps should be applied during both the training process and the classification process.

Case normalization

You should almost always normalize text into a single case (typically lowercase) before training a text classification model or feeding it new data to classify. I say “almost always” because I’m sure there’s some exception out there in which casing may be important.

Term filtering

Term filtering is not always a requirement, but typically recommended. Some terms, even stopwords, may still be important for providing context to something such as a Long Short Term Memory (LSTM) layer. There’s a fine line that you will need to walk between filtering too few or too many terms from the text corpus. Good luck with that.

Spelling correction

Spelling correction helps normalize variations of terms so that the Banana Test (and the model in general) are as effective as possible. I recommend using a string distance function to compare the most frequently-spotted terms in the text corpus to every other term in the text corpus. By filtering the results based on the string distance score, many misspellings that are present in the data will come to light. The output of such a technique requires manual vetting, but often yields great results in my experience.

The Banana Split

The Banana Split is an optional enhancement to the Banana Test (as you can tell by now, I enjoy goofy names for serious things). To put it simply, the Banana Split is the practice of splitting off the top N percent of the total terms from the text corpus (based on TF/IDF) before performing the Banana Test.

The Banana Split is important because the least frequent terms in the text corpus presumably have the least amount of training data associated with them. By applying the Banana Split to your implementation of the Banana Test, you can control (via configuration or otherwise) how conservative your model behaves when classifying data. Terms that the model has seen (but perhaps very infrequently) can be kicked out as if the model has never seen them before.

Conclusion

Despite the funny name and simple approach, this technique has been critical for making text classification algorithms that can handle new data effectively with minimal mistakes. It has helped myself and my team substantially improve the accuracy and reliability of dozens of models so far, and has become a standard component for implementations of text classification. I hope that it benefits you as well, and maybe one day I’ll hear about someone else using the Banana Test.