Emotion Detection from Hindi Text Corpus Using ULMFiT
Tags: Deep Learning, Sentiment Analysis, Text Classification, Developer, Naturallanguageprocessing

Figure 1: Source- Impact

Written by-Ankit Singh, Dhairya Patel, Kaustumbh Jaiswal

Introduction

Deep Learning has charged up the space of Image recognition and Speech processing for some time now.

We are witnessing a similar trend in Natural Language Processing.

Deep Learning for NLP was less impressive at first, but with the introduction of techniques like ULMFiT, ELMo,Transformers, BERTetc., it has become an impact driver, yielding state-of-the-art (SOTA) results for common NLP tasks.

Named entity recognition (NER), part of speech (POS) tagging, Sentiment analysis, etc., are some of the problems where neural network models have outperformed traditional approaches. The progress in machine translation is perhaps the most remarkable amongst all.

In this blog we will showcase a ULMFiT model and use it for Emotion Detection. ULMFiT is the technique of using transfer learning for text classification task.

Let’s begin!

Transfer Learning

Transfer learning is the technique of using weights from a pre-trained deep neural network and tweaking them a bit to suit our application. In other words, it is applying the knowledge of an already trained model to a different but related problem.

Figure 2: Source- EverythingAi

It is suited to applications having a small dataset and also reduces computation time.

What is ULMFiT?

ULMFiT stands for Universal Language Model Fine-tuning for Text Classification, a technique introduced by Jeremy Howardand Sebastian Ruder. It is a technique to incorporate transfer learningin NLP tasks.

USPs of ULMFiT is-

Discriminative fine-tuning

Slanted triangular learning rates

Gradual unfreezing

Discriminative Fine-Tuning

Figure 3: Source- towardsdatascince

Different layers of a neural network capture different types of information so they should be fine tuned to different extents. Instead of using the same learning rates for all layers of the model, discriminative fine-tuning allows us to tune each layer with different learning rates.

Slanted Triangular Learning Rates

Figure 4: Source- ULMFiT

The model should quickly converge to a suitable region of the parameter space in the beginning of training and then later refine its parameters. Using a constant learning rate throughout training is not the best way to acheive this behaviour. Instead Slanted Triangular Learning Rates (STLR) linearly increases the learning rate at first and then linearly decays it.

Gradual Unfreezing

Gradual unfreezing is the concept of unfreezing the layers gradually which avoids catastrophic loss of knowledge possessed by the model. It first unfreezes the top layer and fine-tunes all the unfrozen layers for 1 epoch. It then unfreezes the next lower frozen layer and repeats until all the layers have been fine-tuned until convergence at the last iteration.

For a detailed explanation on ULMFiT we strongly suggest you to go through this paper.

Let’s Code!

Installation

For running the code explained in the subsequent sections make sure fastai version 0.7 is installed in your system. To install fastai follow the instructions given here.

from fastai.text import *

import html

Getting Started

We start up by creating different folders for classification and language models.

PATH = Path('') # path to the data CLAS_PATH=Path('emotion_hindi_clas/')

CLAS_PATH.mkdir(exist_ok=True) LM_PATH=Path('emotion_hindi_lm/')

LM_PATH.mkdir(exist_ok=True)

Dataset

The dataset is created manually as there’s no pre-existing dataset for Hindi Emotion Detection. It comprises of 5 labels Angry, Happy, Neutral, Sad and Excited.