Sentence classification using Bi-LSTM | by akshay uppal | Towards Data Science

So there are various ways for sentence classification like a bag of words approach or neural networks etc. In this article, I would be discussing mainly the sentence classification task using deep learning model (specifically Bi-LSTM)
This article mainly is concerned with some basic introduction and jumps right into implementation. If you need in-depth information, I have included the links in the references.
I have organized the articles in various categories,
introductionimplementation: (preprocessing and baseline model)Bi-LSTM modelresults and conclusion
introduction
implementation: (preprocessing and baseline model)
Bi-LSTM model
results and conclusion
Feel free to jump in a specific category.
I. INTRODUCTION
For sentence classification we have mainly two ways:
Bag of words model (BOW)Deep neural network models
Bag of words model (BOW)
Deep neural network models
The BOW model works by treating each word separately and encoding each of the words. For BOW approach we can use TF-IDF methods but it doesn’t preserve the context of each word in the sentences.
So to achieve better performance for the task like named entity extraction, sentiment analysis, we use deep neural networks.
Dataset:
In this article I have used the Reddit -dataset[2] which is based on four emotion categories like rage, happy, gore and creepy.
For the deep neural models, we need embeddings for the text. Embeddings capture the representation of the word in higher dimensional plane. Through embeddings, we create a vector representation of the word which is learned by understanding the context of words. We can either use pre-trained embeddings like the glove, fasttext which are trained on billion of documents or we can create our own embeddings (trained on our own corpus) using gensim package etc.
In this article, I have used pre-trained glove-twitter embeddings which is suitable in our context of the social network data. Also, I choose 100-Dimensional embeddings which performs pretty good without taking too much time to train. You can choose other (25, 50, 300 D as well).
embedding_path = "~/glove.twitter.27B.100d.txt" ## change # create the word2vec dict from the dictionarydef get_word2vec(file_path):    file = open(embedding_path, "r")    if (file):        word2vec = dict()        split = file.read().splitlines()        for line in split:            key = line.split(' ',1)[0] # the first word is the key            value = np.array([float(val) for val in line.split(' ')[1:]])            word2vec[key] = value        return (word2vec)    else:        print("invalid fiel path")w2v = get_word2vec(embedding_path)
Preprocessing the text:
So their data has four files representing four different emotions so we need to merge the files for the multi-category classification task.
df_rage = pd.read_csv(os.path.join(dir_path,'processed_rage.csv'))df_happy =  pd.read_csv(os.path.join(dir_path,'processed_happy.csv'))df_gore =  pd.read_csv(os.path.join(dir_path,'processed_gore.csv'))df_creepy =  pd.read_csv(os.path.join(dir_path,'processed_creepy.csv'))# create a random balances dataset of all of the categorieslength = np.min([len(df_rage),len(df_happy),len(df_creepy),len(df_gore)])df_final = pd.concat([df_rage[:length], df_happy[:length], df_gore[:length], df_creepy[:length]], ignore_index=True)
Tokenizing :
To break the sentences into simpler tokens or words we tokenize the text. In here we will be using nltk Tweet tokenizer as it works great with social network data.
import nltkfrom nltk.corpus import stopwordsstopwords = set(stopwords.words('english'))nltk.download('wordnet')nltk.download('stopwords')from nltk.tokenize import TweetTokenizerfrom nltk.corpus import wordnet as wntknzr = TweetTokenizer()def get_tokens(sentence):#     tokens = nltk.word_tokenize(sentence)  # now using tweet tokenizer    tokens = tknzr.tokenize(sentence)    tokens = [token for token in tokens if (token not in stopwords and len(token) > 1)]    tokens = [get_lemma(token) for token in tokens]    return (tokens)def get_lemma(word):    lemma = wn.morphy(word)    if lemma is None:        return word    else:        return lemmatoken_list = (df_final['title'].apply(get_tokens))
Preparing the input variable
# integer encode the documentsencoded_docs = t.texts_to_sequences(sentences)# pad documents to a max length of 4 wordsmax_length = max_lenX = pad_sequences(encoded_docs, maxlen=max_length, padding='post')
Output variable:
from sklearn import preprocessingle = preprocessing.LabelEncoder()Y_new = df_final['subreddit']Y_new = le.fit_transform(Y_new)
Splitting the data to training and testing
## now splitting into test and training datafrom sklearn.model_selection import train_test_splitX_train,X_test, Y_train, Y_test =  train_test_split(X, y,test_size =0.20,random_state= 4 )
Baseline models
Before getting the scores with the LSTM model I have got some metrics from our baseline models:
For the baseline models, we can calculate simply the mean of the word2vec embeddings.
# the object is a word2vec dictionary with value as array vector,# creates a mean of word vecotr for sentencesclass MeanVect(object):    def __init__(self, word2vec):        self.word2vec = word2vec        # if a text is empty we should return a vector of zeros        # with the same dimensionality as all the other vectors        self.dim = len(next(iter(word2vec.values())))            # pass a word list    def transform(self, X):        return np.array([            np.mean([self.word2vec[w] for w in words if w in self.word2vec]                    or [np.zeros(self.dim)], axis=0)            for words in (X)        ])
SVM
def svm_wrapper(X_train,Y_train):    param_grid = [    {'C': [1, 10], 'kernel': ['linear']},    {'C': [1, 10], 'gamma': [0.1,0.01], 'kernel': ['rbf']},]    svm = GridSearchCV(SVC(),param_grid)    svm.fit(X_train, Y_train)    return(svm)
Metrics
# svmsvm = svm_wrapper(X_train,Y_train)Y_pred = svm.predict(X_test)score = accuracy_score(Y_test,Y_pred)print("accuarcy :", score)0.70
For baselines, you can further apply other classifiers (like random forest etc)but I got the best F1 score with SVM.
For the neural models in the language context, most popular are LSTMs (Long short term memory) which are a type of RNN (Recurrent neural network), which preserve the long term dependency of text. I have included the links in references which seem to explain LSTM’s in great detail.
III. Bidirectional LSTM:
For the bidirectional LSTM we have an embedding layer and instead of loading random weight we will load the weights from our glove embeddings
# get the embedding matrix from the embedding layerfrom numpy import zerosembedding_matrix = zeros((vocab_size, 100))for word, i in t.word_index.items(): embedding_vector = w2v.get(word) if embedding_vector is not None:  embedding_matrix[i] = embedding_vector
We also want to calculate the vocab size for neural model.
from keras.preprocessing.text import Tokenizerfrom keras.preprocessing.sequence import pad_sequences# prepare tokenizert = Tokenizer()t.fit_on_texts(token_list)vocab_size = len(t.word_index) + 1# integer encode the documentsencoded_docs = t.texts_to_sequences(sentences)# pad documents to a max length of 4 wordsmax_length = max_lenX = pad_sequences(encoded_docs, maxlen=max_length, padding='post')y = Y_new
Final Model
# main modelinput = Input(shape=(max_len,))model = Embedding(vocab_size,100,weights=[embedding_matrix],input_length=max_len)(input)model =  Bidirectional (LSTM (100,return_sequences=True,dropout=0.50),merge_mode='concat')(model)model = TimeDistributed(Dense(100,activation='relu'))(model)model = Flatten()(model)model = Dense(100,activation='relu')(model)output = Dense(3,activation='softmax')(model)model = Model(input,output)model.compile(loss='sparse_categorical_crossentropy',optimizer='adam', metrics=['accuracy'])
The max_len above have to be fixed for our neural model, which could be the sentence with max no of words or could be a static value. I defined it as 60.
model summary — -
Layer (type)                 Output Shape              Param #   =================================================================input_32 (InputLayer)        (None, 60)                0         _________________________________________________________________embedding_34 (Embedding)     (None, 60, 100)           284300    _________________________________________________________________bidirectional_29 (Bidirectio (None, 60, 200)           160800    _________________________________________________________________time_distributed_28 (TimeDis (None, 60, 100)           20100     _________________________________________________________________flatten_24 (Flatten)         (None, 6000)              0         _________________________________________________________________dense_76 (Dense)             (None, 100)               600100    _________________________________________________________________dense_77 (Dense)             (None, 3)                 303       =================================================================Total params: 1,065,603Trainable params: 1,065,603Non-trainable params: 0_________________________________________________________________
So in the paper for neral architecture for ner model [1] they use a CRF layer on top of Bi-LSTM but for simple multi categorical sentence classification, we can skip that.
Fit the training data to the model:
model.fit(X_train,Y_train,validation_split=0.25, nb_epoch = 10, verbose = 2)
IV: RESULTS
Evaluating the mode
# evaluate the modelloss, accuracy = model.evaluate(X_test, Y_test, verbose=2)print('Accuracy: %f' % (accuracy*100))Accuracy: 74.593496
Classification report
from sklearn.metrics import classification_report,confusion_matrixY_pred = model.predict(X_test)y_pred = np.array([np.argmax(pred) for pred in Y_pred])print('  Classification Report:\n',classification_report(Y_test,y_pred),'\n')Classification Report:               precision    recall  f1-score   support           0       0.74      0.72      0.73       129           1       0.75      0.64      0.69       106           2       0.76      0.89      0.82       127           3       0.73      0.72      0.72       130   micro avg       0.75      0.75      0.75       492   macro avg       0.75      0.74      0.74       492weighted avg       0.75      0.75      0.74       492
Conclusion:
So we saw for the neural models that preserve the sequence of text and their context, we get better scores as compared BOW models but it depends on the context and application. So in some cases, it might be beneficial to have simple models as compared to complex neural models. Also, we had a smaller dataset so the training time was quite low but for larger datasets ( > 100k ), it might take more than 1 hour of training.
Hope you enjoyed, if you have any doubts or comments please feel free to add to the comment section. Thanks.
References:
[1]: Lample, Guillaume, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. “Neural architectures for named entity recognition.” arXiv preprint arXiv:1603.01360 (2016).
[2] Duong, Chi Thang, Remi Lebret, and Karl Aberer. “Multimodal Classification for Analysing Social Media.” The 27th European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML-PKDD), 2017