Grammar Error Correction with Deep Learning | by PUSHAP GANDHI | Towards Data Science

IntroductionProblem DefinitionPrerequisitesData SourceUnderstanding the dataExploratory Data AnalysisData PreprocessingData Preparation and Data PipelineBenchmark Solution(Vanilla Encoder-Decoder Model)Attention MechanismMonotonic AttentionInference (Greedy Search VS BEAM Search)Results and Model ComparisonModel Deployment and PredictionsFuture WorkGithub Repository and LinkedinReferences
Introduction
Problem Definition
Prerequisites
Data Source
Understanding the data
Exploratory Data Analysis
Data Preprocessing
Data Preparation and Data Pipeline
Benchmark Solution(Vanilla Encoder-Decoder Model)
Attention Mechanism
Monotonic Attention
Inference (Greedy Search VS BEAM Search)
Results and Model Comparison
Model Deployment and Predictions
Future Work
Github Repository and Linkedin
References
Grammatical Error Correction as the name suggests is the process by which the detection and correction to an error in the text are done. The problem seems easy to understand but is actually tough due to the diverse vocabulary and set of rules in a language. In addition, we are not only going to identify the mistake but correction is also required.
There are immense applications to this problem, the reason being writing is a very common means to share ideas and information. This could help the writer to speed up their work with very minimal chance of error.
Moreover, there could be many individuals who are not fluent in a particular language. Therefore, these types of models make sure that language is not a barrier in communication.
Now we shall define the task at hand into a machine learning problem. The problem that we are dealing is a type of NLP (Natural Language Processing) problem. NLP is the field of Machine Learning which deals with the interaction between human languages and computers. I would recommend going through this paper to know the progress of approaches used to solve the problem.
The approach that we are going to look into is the Sequence to Sequence model. In short out, the deep learning model is going to receive a sequence (incorrect text in this case) and it will output another sequence(corrected text in this case).
Now we have defined our problem into a machine learning problem, there is another very integral idea that needs to be dealt with that is performance metric. Performance Metric is a mathematical measure that helps to understand the performance of our machine learning model.
A very popular performance metric for the NLP problems is BLEU (Bilingual Evaluation Understudy) Score, hence we are going to use it for our model also. You can refer to this video to know more about the BLEU scores.
Before we go into the details of this case study, I assume that the readers know the concepts for Machine Learning and Deep Learning. To be specific, the ideas of LSTMs, Encoder-Decoders, Attention Mechanism should be familiar.
I will try to provide some good references on the way.
As I mentioned above the problem is tough to handle to which one of the reasons is the nonavailability of a good quality dataset. In my research, I found some of them useful and considered the following two datasets.
Lang-8 DatasetNUS Social Media Text Normalization and Translation Corpus
Lang-8 Dataset
NUS Social Media Text Normalization and Translation Corpus
Both these datasets are publicly available if you are interested to work with them.
This is a Japanese website that works towards language learning. Users are able to post in the language(s) they are learning and that post will appear to native speakers of that language for correction. The dataset from this website has data in the following column separated by \t :
Number of correction
Serial number
Sentence number
0 is the title
Sentence written by a learner of English
Corrected Sentences (If exists)
Out of this we have extracted input sentence and corrected sentence pairs for those data points for which both these sentences exists.
This is the dataset by the National University of Singapore (NUS) which is social media data text data. The dataset size is 2000 data points. The data in its raw form has three rows for each datapoint.
The first row is the social media text
The second row is the correct formal English translated text
The third row is the Chinese translated data.
Out of these, we can use the first two rows of each data point for our problem.
Once we have extracted the data in the required form now is a very integral step which is Exploratory Data Analysis which is an opportunity to understand and develop a good sense of the dataset.
I have extracted and preprocessed the dataset in a required form which will be discussed in the next section.
First, we will gather overall information about the dataset like the number of data points, datatypes, mean or median values of numerical data, etc.
# TOTAL DATAPOINTSdf.shape# CHECK FOR NULL VALUESdf.info()
So there are around 500k data points and no null values are present.
We have two columns in the dataset one is input and the other is output or target. Let's see one by one the analysis for each column.
The input is a text feature we can analyze the distribution for the length of the text and the distribution of the number of words.
The distribution for the length of characters and the number of words in the input is highly skewed.
Both the distributions almost follow the same pattern.
The number of characters in the input could go as high as 2000 and words are up to 400 words.
The ratio of the higher length of words or characters is significantly less.
More than 99 percent of inputs have a number of characters less than 200 and a number of words less than 50.
The same is done for the output text also.
And surprisingly the observations for the input text hold true for output text also.
Word Cloud is a great visualization to know the most frequently occurring words in the total text.
The most common words in both the text are:
think
want
today
Japan
English
Here wrong words are those words that are present in input but not present in target sentences and Corrected words are those that are present in target sentences but not present in input sentences. I have extracted all these words and done a word cloud analysis on them.
After observing the incorrect and corrected words images we can observe the change in the form of the verb that is being used.
go ==> goinggo ==> wentstudy ==> studyinguse ==> using
In this part of the analysis, we will associate the words with their parts of speech and plot the top 10 most occurring parts of speech. Following are the plots:
This shows part of speech being used a maximum number of times is Noun(NN) followed by Preposition(IN).
The overall numbers of parts of speech remain the same for input and output sentences but the numbers vary in NNS(Noun, plural) and PRP(personal pronouns)
Now we will see how to remove the unwanted data from our dataset. We know that the data is in text format therefore all the special characters, unwanted spaces need to be removed and the decontraction of contracted words is to be done. We are also going to convert all the text into lower to reduce the complexity of the problem. The following code is followed to do all the above-mentioned operations:
Following this, we can also follow some other operations is removing null values and deduplication which are available in the pandas library.
Before we feed our data to the DL model it needs to be converted to a form which the machine can understand. So for this problem, we are going to use tokenization and padding to convert the dataset into a sequence of integers of the same length.
While forming the data pipeline for the model we are going to pad the sequence to form the length of all the data points equal. The maximum length used for the padding is the 99th percentile of the distribution of the number of words.
The data has to be converted into batches so as to input it into the deep learning model. The following code is used to form the pipeline for the dataset
For the benchmark solution, I am going to the vanilla sequence to sequence model which is also known as Encoder-Decoder Model. This model takes input in a sequence form and predicts another sequence as output. Due to this, it has many applications in Machine Translation Problems.
The following code is used to form Encoder-Decoder layers in the model.
This is a really good blog to get an insight into the Encoder-Decoder models.
I have tried many variations of Vanilla Encoder-Decoder models like using pre-trained embeddings of Word2Vec and FastText, you can check them out at my GitHub profile.
After experimenting with these models, I found that for my case trainable embeddings are working better than pre-trained ones, therefore it would be used for the advanced models.
The BLUE score achieved for Encoder-Decoder for our dataset is 0.4603.
Attention Mechanism is a very ingenious idea in Machine Learning which clones the humanistic way of grasping information. Moreover, there are also certain disadvantages of the simple encoder-decoder model which the Attention Model overcomes. Some of the popular attention mechanism techniques used today are Bahdanau Attention Mechanism, Loung Attention Mechanism.
Let me give a brief idea about the steps of the Loung Attention Mechanism which I have used extensively in my case study. To be specific this idea is known as the Global Attention Mechanism.
The encoder part remains the same as the vanilla encoder-decoder, which outputs the hidden states of the input sequence. Now in the decoder part for every time step we have to compute something called a Context Vector. This context vector holds the relevant information from the encoder about the words which is to be predicted in that time step. It may sound complex but once you know how it is computed everything will fall in place.
First, when we achieve all encoder hidden states(ht) and decoder previous hidden(hs) from an RNN we find the alpha values().
In the paper, Loung and others provided three alternatives for computing the score.
Once the alpha values are computed the context vector is calculated as the weighted average over all the encoder hidden states with the weights being the alpha values.
Given the target hidden state and the source side context vector simple concatenation layers if used to combine the information from both vectors to produce attention hidden state as follows:
The attentional vector h ̃t is then fed through the softmax layer to produce the next word in sequence.
This type of attention mechanism is called Global Attention Mechanism because all the hidden states of the encoder are considered at every time step of decoder to produce the context vector.
In this case study, I have implemented Loung Attention Mechanism from scratch through Model Subclassing. Below is the code for the Attention layer for Dot type scoring. For full code, you refer to my GitHub profile.
The performance metric for Attention models is significantly better than our benchmark models.
                   BLEU ScoreDot Scoring         0.5055General Scoreing    0.5545Concat Scoreing     0.5388
Before we get into the details for Monotonic Attention let us see some drawbacks of simple attention mechanism and what was the need of monotonic attention.
We know in attention mechanisms at each decoder time step all the encoder hidden states need to be referred to. This creates a quadratic time complexity which hinders its use in online settings.
Therefore to overcome this disadvantage the concept of Monotonic Attention is introduced which simply says that there is no need to inspect all the attention weights at each time step. We are going to inspect the hidden states in a specific order (from left to right in English) and one of the hidden states is selected as the context vector at every time step. Once a particular entry is inspected then it will not be inspected in the next time step.
The most important advantage for this approach is the linear time complexity and thus could be used for online settings.
Here we are going to implement a very simple type of monotonic attention as explained in this blog and paper. This has to be kept in mind that the changes would only occur in the attention layer but the encoder and decoder layer will remain the same. These are the steps for monotonic attention:
Given the previous hidden states, we are going to compute the score or energy for which we are going to use the same methods as used above in the Loung attention mechanism.
After the score is computed it is converted to probabilities through with sigmoid function.
The attention for the present time step is computed by this formula:
Once we get the attention weights we follow the similar steps of computing the weighted sum to get the context vector and concatenate it to the hidden state of the decoder to pass through the softmax layer.
In this case study, I have implemented only one variation of the monotonic attention layer. For more information refer to the blog and code by Colin Raffel where he has given the complete implementation of this layer.
The performance of monotonic attention is comparable to simple attention but as mentioned in the paper it provides some advantage in the time complexity.
If you know about the sequence to sequence models then you would be aware of the fact that we have to make certain changes with the model to predict the required output at the inference time.
Again for that, we have two options one is known as Greedy Search and the other is Beam Search. Let us see both of them one by one.
In greedy search for every time step, we consider the token with maximum probability and ignore all the remaining tokens even though they have comparable values of probabilities after the softmax layer.
In the code, you can see the argmax function on the output of the dense softmax activation layer.
This has a disadvantage because if any of the predicted words at a time step is wrong the result of future time steps will also get affected.
A better way of prediction is BEAM Search in which at each time step there is a choice provided for topmost probable tokens which have numbers equal to with BEAM Width and the total score for each of the predicted sequences is equal to the product of the probabilities for each word in the sequence.
For this case study, I have taken the beam width equal to 3. For more insights, you can go through this blog.
For this case study, I have tried a total of 12 models with certain variations. Below are the results.
The results of the attention mechanism are better than the simple encoder-decoder and among the attention model, the performance does not differ much.
We can also observe that the BLEU score for beam search is better than the greedy search as expected.
I have done the Model Deployment for the Monotonic Attention model which is computing the score with the dot product. You can check out the model at this link.
Some of the predictions for the above model are as follows :
For future work, I would love to work with other variations of the Monotonic Attention mechanism.
If a better dataset is available publically that could really improve the model performance.
I have tried to give a brief on the work I have done in this project. If you want to see the full detailed code where I have commented on each and every line, check out my Github repository. Feel free to connect on Linkedin.
If you have reached it so far, help me to improve the content by leaving a comment. That would mean a lot to me :)
https://colinraffel.com/blog/online-and-linear-time-attention-by-enforcing-monotonic-alignments.html
Project under the guidance of Hina Sharma at Applied AI Course: https://www.appliedaicourse.com/course/11/Applied-Machine-learning-course
Loung Attention Paper: https://arxiv.org/pdf/1508.04025.pdf
Monotonic Attention Paper: https://arxiv.org/pdf/1409.0473.pdf
Implementation of Monotonic Attention: https://github.com/craffel/mad/blob/b3687a70615044359c8acc440e43a5e23dc58309/example_decoder.py#L22
Blog by Rishang Prashnani