Automatic Ticket Tagging with NLP Text Classification
By: Julián Gutiérrez Ostrovsky
Tags: Machine Learning, Text Classification, Scikit Learn, NLP

Getting to the root

The first step towards training a classifier with machine learning is to transform each text into a numerical representation in the form of a vector. In order to do that, we must first keep just relevant words from our model. Then, it would be nice to have just the root of each word. Meaning that “I broke my PC” and “My PC is broken” is equal to “Break PC”. Here “Break” is the root word, we want both cases to be same.

type_pipeline_preprocess = Pipeline([

('clean', htmlCleanerPipeStep()),

('lower', lowerCasesPipeStep()),

('stopwords', stopWordsPipeStep('utils/stopwords.txt')),

('lemmatize', lemmatizePipeStep())

]) component_pipeline_preprocess = Pipeline([

('clean', htmlCleanerPipeStep()),

('lower', lowerCasesPipeStep()),

('stopwords', stopWordsPipeStep('utils/stopwords.txt')),

('stemmize', stemmizePipeStep())

])

These are our preprocess pipelines. First we’ve created some straightforward regex to remove HTML tags, and some other special characters. Then we convert every word to lower, filter not relevant words for our model such as connectors, prepositions, specific words from our business which are not important to determine ticket category, etc.

In the last step, for token normalization we choose lemmatization for one case and stemming for other. Why is that? Basically with lemmatization you get the actual root of words, meaning its inflected form. For example lemma of forms “studies” and “studying” is “study”.

With stemming you just get the root prefix for every word, and those words are not necessary within the language. For example stemme of “studies” is “studi”. Stemme of “studying” is “study”. For other words it could match with actual root of word.

In our case it just worked better this way, since component is a little harder to predict, with stemming we have more information about the word. We recommend you to try both.

Unprocess words at left and different pipes applied on the right

So far, we applied these transformations, now we have to vectorize our words to have a numerical matrix to train on.

Vectorization

By doing this, we will transform all of our processed sentences into numerical vectors representing original words. We used two Word Embedding techniques based on our domain: Word2vec for Type prediction and Tf-Idf for Component, and we added it to each pipeline. With these configuration we’ve got better results. Nevertheless we could have done a bit more fine-tuning over word2vec to achieve more accuracy in vector similarity. We note it as a future work. We advise you to try both and choose the one that suits better to your model.

Training Time

Right now we have a “simple” classification problem, where we have a matrix with a target column to train and predict. We follow an straightforward approach in which we choose some candidates classifiers from scikit-learn library that tend to work well for NLP such as LinearSVC, SGDClassifier, LogisticRegression, RandomForestClassifier, AdaBoostClassifier, LGBMClassifier, MLPClassifier. We give a quick train for all this classifiers, get the score and choose the best. Then, we’ve performed a GridSearch to help us find better parameters, and fine tune by hand until we got the best results.

If you need more details on how to do this, you should read Choosing between ML models using pipes for code reuse.

This is our final model for Type prediction:

type_pipeline_fe = Pipeline([

('preprocess', type_pipeline_preprocess),

('word2vec', word2vecPipeStep())

]) type_pipeline_fe.fit(X_train,y_train)

X_train_transform = type_pipeline_fe.transform(X_train)

y_train_transform = y_train from sklearn.neural_network import MLPClassifier

clf_t_nn = MLPClassifier(

hidden_layer_sizes=(100,75),

activation='relu',

random_state=42,

tol=0.001,

alpha=1.3,

early_stopping=True,

n_iter_no_change=20,

validation_fraction=0.1,

verbose=False,

warm_start=False

) _ = clf_t_nn.fit(X_train_transform,y_train_transform)

We got better score values using this network for type prediction. Second classifier was Logistic Regression, which is faster, but overfit our training.

For Component Prediction:

component_pipeline_fe = Pipeline([

('preprocess', component_pipeline_preprocess),

('tfidf', tfIdfVectorizerPipeStep({

'stop_words':utils.get_stopwords(),

'strip_accents':'unicode',

'use_idf':True,

'ngram_range':(1,3)

}))

]) component_pipeline_fe.fit(X_train,y_train)

X_train_transform = component_pipeline_fe.transform(X_train)

le = preprocessing.LabelEncoder()

y_train_transform = le.fit_transform(y_train) from sklearn.linear_model import SGDClassifier

clf_c = SGDClassifier()

clf_c.set_params(alpha=0.0005,learning_rate='optimal',penalty='l2',

random_state=42,tol=0.0005,loss='modified_huber') _= clf_c.fit(X_train_transform,y_train)

Here are the results:

Component Model is Overfitted

This closes the full training cycle, and we are ready to start cycling over and over again, trying to improve our model. From now on, we did several things which won’t worth telling because are tightly coupled to our problem. Nevertheless we leave some ideas below: