[Part 2] Ticket Tagging with BERT — NLP Text Classification
By: Julián Gutiérrez Ostrovsky
Tags: NLP, Bert, Keras, Hugging Face, Text Classification

Motivation

In this previous post, we took a dataset with all the tickets loaded to our IT Support Team and our goal was to create two context-free linear models to predict two different targets. One target was binary, the other one was a multilabel class. We got some decent results, but also taking a deeper look into data, we’ve visualized some problems. So…

What if we try with a contextual-based, more powerful and specially built for NLP purposes model? Will we get better results or just overfit our model? Will this powerful model be able to deal with the problems that we found in our data, or it’s just not possible without an special custom made cleaning?

what if we try BERT?

A little bit about BERT

BERT was open-sourced in 2018 as a new deep learning technique for NLP. It has two main keys:

First, it’s bidirectional (or actually, non-directional). Before BERT, a language model would have looked at every text sequence during training from either left-to-right or combined left-to-right and right-to-left. This is a one-directional approach and it does perform well. It predicts next word in a sentence given its previous words. Ie, for the sentence “I want to play guitar with my musician friends”, if we would want to predict the word guitar, we could only consider “I want to play”. Bert, on the other hand, will use full sentence for prediction. This target word it’s called [MASK] and that’s why it’s non directional.

“I want to play [MASK] with my musician friends”.

Second, it’s based on Transformers. A Transformer works by performing a small, constant number of steps. In each step, it applies an attention mechanism to understand relationships between all words in a sentence, regardless of their respective position. For example, given the sentence, “I arrived at the bank after crossing the river”, to determine that the word “bank” refers to the shore of a river and not a financial institution, the Transformer can learn to immediately pay attention to the word “river” and make this decision in just one step.

Now, we’re in condition to understand that BERT is short for Bidirectional Encoder Representations from Transformers.

Off course, this is just an overview about BERT since we’re not going to dive into its architecture. For more details about BERT check the References section at the bottom.

Keeping it simple with Hugging Face

To develop our models, we’re going to use pretrained models based on BERT.

These models are all trained from BERT for specific NLP purposes and with different corpuses. This is really helpful for many reasons. First, we don’t have to invent the wheel again to tackle common problems. Also, probably we don’t have such big corpuses to achieve similar results. Not to mention computing power and time (!) to create this huge models.

Hugging face provides community transformers that are state of the art for NLP purposes. These transformers are backed by PyTorch and TensorFlow and provide a single interface for a quick implementation.

We can surf through all of its models, and we should choose the one that fits better to our problem. This will be the biggest challenge for us using hugging face, so it’s really important to know all the features of our corpus in order to find the best transformer. Even though it’s not exactly straight forward for all cases, it’s really easy to switch between transformers. Once we have our pretrained model, we will fine tune it to adapt our dataset. For this, we will add special bert tokens to indicate where our text begins and ends, and provide correct tag. By training this new model we will learn our dataset in order to generate predictions.

To start working with hugging face transformers

Training With TensorFlow and Hugging Face

Let’s recap first and remember a little bit about about our dataset. First, it’s written mainly in spanish, but it has some english words. And some spanglish words as well, to be fair. It has near 14k training rows and it’s really dirty with special characters, HTML tags, accents, etc.