Natural Language Processing Advancements By Deep Learning: A Survey
By: Rimzim Thube
Tags: NLP, Recurrent Neural Network, Convolutional Network, Machine Learning, Deep Learning

Introduction

Natural Language Processing (NLP) is a discipline of computer science involving natural languages and computers. It helps machines to understand, process, and analyse human language. NLP is easy in these times due to advanced computational power, greater availability of large datasets and deep learning. This article covers the role of deep learning in the area of NLP and its broad range of categories. This article explains the challenges, opportunities, and evaluations of the impact of applying deep learning to NLP problems.

Background

Deep learning is using deep neural networks to massive amounts of data to learn a problem aimed at handling a task. It can solve any problem provided large and relevant dataset is available. It also detects and analyses the important features of the problem statement.

Deep Learning Architectures

Numerous deep learning architectures have been developed in different research areas.

Multi Layer Perceptron

A multilayer perceptron (MLP) has at least three layers — input, hidden, and output layers. A layer transfers information from the previous layer to the next layer using neurons. Neurons do not communicate with each other and uses nonlinear activation functions. Every node forms fully connected network by being connected to other neurons of next layer. MLPs are simplest form of Feed-Forward Neural Networks (FNNs) which do not create any cycle.

2. Convolutional Neural Networks

CNNs internally use convolution as their mathematical function which is a measure of the interoperability of its input functions. It is mainly used when data is to be represented as 2D or 3D map. The data points in the map represent information correlation. In image CNNs, the image pixels of the data map are highly correlated to their neighbouring pixels. CNNs have width, height, and depth. A CNN takes an image represented as an array of numeric values. After feature extraction, the key image content is captured and represented in output. These features can be used for different tasks like image classification or object detection. In the case of utilising CNNs for NLP, the inputs are sentences or documents represented as matrices. Each row of the matrix is associated with a language element such as a word or a character. CNN architectures are used for various classification tasks such as Sentiment Analysis and Topic Categorisation.

Fig. 1. A typical CNN architecture for object detection

3. Recurrent Neural Network

When we feed the output of each FNN as an input to the next one, a recurrent neural network (RNN) will be constructed. Like FNNs, layers in an RNN can be categorised into input, hidden, and output layers. Sequences of input vectors are fed as the input, in discrete time frames, one vector at a time. Thus, as shown in Fig. 2, at each time step, we use parameters of the current hidden layer as input to the next time step and make predictions.

Fig. 2. Recurrent Neural Network (RNN)

Hidden layers in RNNs carry information from the past or memorise the information. This is especially useful in inputs of sequence type. Long Short Term Memory Network (LSTM) is most widely used class of RNNs. LSTMs capture long time dependencies between inputs from different time steps. Speech Recognition often rely on LSTMs.

4. Auto-encoders

Auto-encoders implement unsupervised methods in deep learning and consist of sequence to sequence modelling. They are used in dimensionality reduction or NLP applications. Fig. 3 illustrates the schematic of an Auto-encoder. They aim to learn a code representation for each input since there is no label corresponding to each input. The decoder operates similar to the encoder, but in reverse, i.e., constructing an output based on the encoded input.

Fig 3. Schematic of an Auto-encoder

5. Generative Adversarial Networks

GAN is a combination of two neural networks, a discriminator and a generator and is iterative in nature. First, the generator network generates a fake sample. Then the discriminator network tries to decide whether this sample is real or fake. The goal of the generator is to fool the discriminator such that the discriminator believes the generated samples created by the generator are real. This iterative process continues until the generator produces samples that are indistinguishable by the discriminator. In GAN, once the training phase is finished, there is no need for the discrimination network, so we solely can work with the generation network. In the NLP domain, GANs often are used for text generation.

Core Concepts of NLP

Feature Representation

For NLP systems, due to issues related to the atomic representation of the symbols, it is imperative to learn word representations. The encoded input features can be characters, words or sentences. It is more preferable to provide a compact representation of the words than a sparse one. Word2vec and doc2vec approaches were proposed as an unsupervised algorithm and are called Paragraph Vector (PV). Doc2vec overcomes the drawbacks of BoW and is efficient in text classification and sentiment analysis.

One-Hot Representation

In one-hot encoding, each unique element that needs to be represented has its dimension which results in a very high dimensional, very sparse representation. In this method, there is no meaningful connection between different words in the feature space. Some research efforts present promising results using one-hot encoding.

Continuous Bag of Words

CBOW tries to predict a word given its surrounding context. CBOW is neither dependent on the sequential order of words or on probabilistic characteristics. This model is mainly used as a pre-trained model for more sophisticated tasks. In weighted CBOW (WCBOW), vectors get different weights depending on their relative importance in context. TF-IDF scores is an example of CBOW.

Word-Level Embedding

Word embedding is a learned representation in which words with related semantics become highly correlated in the representation space. It has high generalisation power as opposed to sparse, higher dimensional representations. Learning a distributed representation has advantage of word usage in context and provides similar representations for semantically correlated words.

Character-Level Embedding

CNNs have successfully been utilised for generating character-level embeddings and has been used in NLP. It can use small model sizes and represent words with lower-level language elements. It also uses out of-vocabulary word (OOV) issue which is usually encountered when, for the given word, there is no equivalent vector in the word embedding. This approach suffers from a weak correlation between characters and semantic and syntactic parts of the language.

Seq2Seq Framework

Seq2seq models the input and the output is represented as a sequence. This is used in applications like machine translation, text summarisation , speech-to-text, and text-to speech. It comprises of an encoder and a decoder. The encoder takes the sequence of input data and generates an intermediate output which is consumed by the decoder to produce the series of final outputs. This is implemented using Recurrent Neural Networks or LSTM. In different applications, the decoder could take advantage of more information such as a context vector or intra-attention vectors to generate better outputs. Teacher Forcing is a common approach to train the model. Once the model is optimised using the cross-entropy loss, it can generate an entire sequence as follows. In NLP applications, the output can be improved by using beam search to find a reasonably good output sequence. During beam search, rather than using argmax for selecting the best output, we choose the top K outputs at each step, generate K different paths for the output sequence, and finally choose the one that provides better performance as the final output.

Reinforcement Learning in NLP

Seq2seq models has two issues: (1) exposure bias and (2) inconsistency between training time and test time measurements. In teacher forcing, during the training of the model, the decoder utilizes two inputs, the previous decoder output state and the real input, to determine its current output state. However, during testing , the decoder fully relies on the previously created token from the model distribution. As the ground-truth data is not available, such a step is necessary to predict the next action. Henceforth, in training, the decoder input is coming from the ground truth, while, in the test phase, it relies on the previous prediction. This creates error growth. To solve this, remove the ground-truth dependency in training by solely relying on model distribution to minimise the cross-entropy loss. The second problem with seq2seq models is, when training is completed using the cross-entropy loss, it is typically evaluated using ROUGE or METEOR. This will form an inconsistency between the training objective and the test evaluation metric. This can be solved using reinforcement learning. The model employs its predictions to produce a sequence of actions (words sequences). Then, at each time step, a greedy search algorithm is employed to learn the optimal action, and the policy will be trained to predict that particular action.

In Actor-Critic training, the actor is usually the same neural network used to generate the output, while the critic is a regression model that estimates how the actor performed on the input data. The actor later receives the feedback from the critic and improves its actions. A major challenge is the massive action space in NLP applications, which causes difficulty for the right action selection and makes the training process very slow. This makes the process of finding the best Actor-Critic model very complicated and model convergence usually requires a lot of tweaks to the models.

Datasets

Benchmarking is the assessment of methods and algorithms to test their capability to learn specific patterns. Benchmarking helps validation of a new approach, compared to other existing methods. Benchmark datasets typically take one of three forms. 1) The first is real-world data 2) The second is synthetic data which is artificially generated. It is mainly used when the data required is more than available data or data privacy is a concern. 3) The third type are toy datasets used for demo and visualisation purposes. They are artificially generated. The effectiveness of the deep learning model depends on the quality of the data. The model needs to be trained on the data associated with the desired task. Hence, for different machine domains such as NLP, creating new datasets is crucial. However, that is not easy.Data should have 3 properties — the right data to train on, sufficient for the evaluation, and accurate to work on. The training data set is used to train a model to know how to find the connections between the inputs and the associated outputs. The test data set is used to assess the intelligence of the machine. Data preparation to make sure it is simple and understandable for human experts. Distribution of data may need to have specific authorisations, especially if we are dealing with sensitive or private data. Creating proper datasets is complicated and of great importance. That’s why few datasets are frequently chosen by the researchers and developers for benchmarking.

Deep learning for NLP tasks

Part-Of-Speech Tagging

Part-of-Speech tagging is one of the basic tasks in Natural Language Processing. It is the process of labelling words with their part of speech categories. WSJ corpus7 dataset contains over a million tokens and has been widely used as a benchmark dataset tagging systems. A wide variety of neural network based models have been proposed for sequence tagging tasks, e.g., LSTM networks, bidirectional LSTM networks, LSTM networks with a CRF8 layer, etc. Sequence tagging itself includes part of speech tagging, chunking, and named entity recognition.

Parsing

Parsing is assigning a structure to a recognised string. Constituency Parsing refers in particular to assigning a syntactic structure to a sentence. A greedy parser performs a syntactic and semantic summary of content using vector representations. Recently, deep neural network models outperformed traditional algorithms. Dependency Parsing shows the structural relationships between the words in a targeted sentence. In dependency parsing phrasal elements and phrase-structure rules do not contribute to the process. Neural networks have shown their superiority regarding generalisability and reducing the feature computation cost. Bidirectional-LSTMs have been used in dependency parsers for feature representation. A new control structure has been introduced for sequence-to-sequence neural networks based on the stack LSTM and has been used in transition-based parsing.

Semantic Role Labelling

Semantic Role Labelling (SRL) is the process of identification and classification of text arguments. It is done to characterise elements to determine “who” did “what” to “whom” as well as “how,” “where,” and “when.” The goal of SRL is to extract the semantic relations between the predicate and the related arguments.

Recently, deep learning approaches have achieved the SRL state-of-the-art without taking the explicit syntax representation into consideration. Current state-of-the-art methods employ joint prediction of predicates and arguments, novel word representation approaches, and self-attention models.

Text Classification

The primary objective of text classification is to assign predefined categories to text parts for preliminary classification purposes. A simple example is the categorisation of given documents as to political or non-political news articles. The use of CNNs for sentence classification, in which training the model on top of pre-trained word-vectors through fine-tuning, has resulted in considerable improvements in learning task-specific vectors. RNNs have been used for text classification. An LSTM-RNN architecture has been utilised in for sentence embedding with particular superiority in a defined web search task. A recurrent architecture in addition to max-pooling with an effective word representation method, and demonstrates superiority compared to simple window-based neural network approaches.

Information Extraction

Information extraction identifies structured information from “unstructured” data such as social media posts and online news. Information extraction has subtasks such as Named Entity Recognition, Relation Extraction, Coreference Resolution, and Event Extraction.

Named Entity Recognition

Named Entity Recognition (NER) aims to locate and categorize named entities in context into pre-defined categories such as the names of people and places.

Relation Extraction

Relation Extraction aims to find the semantic relationships between entity pairs. The recursive neural network (RNN) model has been proposed for semantic relationship classification by learning compositional vector representations

Coreference Resolution

Coreference resolution includes identification of the mentions in a context that refer to the same entity. For instance, the mentions “car,” “Camry,” and “it” could all refer to the same entity.

Event Extraction

Extraction of events recognising trigger words related to an event and assigning labels to entity mentions that represent event triggers.

Sentiment analysis

Sentiment analysis is sometimes called opinion mining, as its primary goal is to analyse human opinion, sentiments, and even emotions regarding products, problems, and varied subjects. Sentiment analysis is generally divided into three categories/levels: document level, sentence level, and aspect level.

1) Document-level Sentiment Analysis: The task is to determine whether the whole document reflects a positive or negative sentiment about exactly one entity. The Gated Recurrent Neural Network architecture has been utilised for this,

2) Sentence-level Sentiment Analysis: Sentiment analysis determines the positivity, negativity, or neutrality regarding an opinion expressed in a sentence. Recursive auto-encoders have been employed for sentence-level sentiment label prediction

3) Aspect-level Sentiment Analysis: Aspect-level sentiment analysis directly targets an opinion, with the assumption of the existence of the sentiment and its target. Aspect-level sentiment analysis usually involves Aspect Sentiment Classification and Aspect Extraction. The former determines opinions on different aspects (positive, neutral, or negative) while the latter identifies the target aspect for evaluation in context. Attention-based LSTMs are proposed for this.

Machine Translation (MT)

It is one of the areas of NLP that has been profoundly affected by the advances in deep learning.

Traditional Machine Translation

The BLEU score was introduced as a new evaluation metric, allowing more rapid improvement than when the only approach involved using human labor for evaluation.

2. Neural Machine Translation

Unlike traditional statistical machine translation, NMT is based on an end-to-end neural network. This implies that there is no need for extensive preprocessing and word alignments. A sequence of input tokens is fed into the network. Once it reaches an end-of-sentence (EOS) token, it starts generating the output sequence. The output sequence is generated in the same recurrent manner as the input sequence until it reaches an end-of-sentence token. One major advantage of this approach is that there is no need to specify the length of the sequence; the network takes it into account automatically.

3. Question Answering

Question answering (QA) is a fine-grained version of Information Retrieval (IR). In IR a desired set of information has to be retrieved from a set of documents. In QA specific answers are sought, typically ones that can be inferred from available documents.

Rule-based Question Answering: Baseball is one of the early works on QA The baseball system consists of (1) question read-in, (2) dictionary lookup for words in the question, (3) syntactic (POS) analysis of the words in question, (4) content analysis for extracting the input question, and (5) estimating relevance regarding answering the input question. Question answering in the era of deep learning: Smartphones (Siri, Ok Google, Alexa, etc.) are common examples of QA systems with which many interact on a daily basis. While earlier such systems employed rule-based methods, today their core algorithm is based on deep learning. The system tries to pick a sentence in the database that has an answer to a question, and a feature vector represents each question-sentence pair. Convolutional neural networks in order to encode Question-Answer sentence pairs in the form of fixed length vectors regardless of the length of the input sentence. Instead of using distance measures like cosine correlation, they incorporate a non-linear tensor layer to match the relevance between question and answer. Visual Question Answering: Given an input image, Visual Question Answering (VQA) tries to answer a natural language question about the image. VQN addresses multiple problems such as object detection, image segmentation, sentiment analysis.

Fig. 4. Neural Image Question Answering

Document Summarisation

Document summarisation refers to a set of problems involving generation of summary sentences given one or multiple documents as input.

Extractive Summarisation where the goal is to identify the most salient sentences in the document and return them as the summary. Extractive summarisation is prone to generate long and sometimes overlapping summary sentences; however, the result reflects the author’s mode of expression.

Abstractive Summarisation where the goal is to generate summary sentences from scratch; they may contain novel words that do not appear in the original document. Abstractive methods generate a shorter summary but they are hard to train.

Dialogue Systems

Dialogue Systems are quickly becoming a principal instrument in human-computer interaction, due in part to their promising potential and commercial value. Due to the high cost of knowledgeable human resources, companies frequently turn to intelligent conversational machines. Dialogue systems are usually task-based or non-task based. Despite useful statistical models employed in the backend of dialogue systems, most deployed dialogue systems rely on expensive hand-crafted and manual features for operation. Deep learning facilitates the creation of end-to-end task-oriented dialogue systems, which enriches the framework to generalise conversations beyond annotated task-specific dialogue resources.

Fig. 4. The framework of a dialogue system.

Task-based Systems: The structure of a task-based dialogue system usually consists of the following elements:

Natural Language Understanding (NLU): This component deals with understanding and interpreting user’s spoken context by assigning a constituent structure to the spoken utterance

Dialogue Manager (DM): The generated representation by NLU would be handled by the dialogue manager, which investigates the context and returns a reasonable semantic-related response.

Natural Language Generation (NLG): The natural language generation (NLG) component produces an utterance based on the response provided by the DM component.

The general pipeline is as follows: NLU module (i.e., semantic decoder) transforms the output of the speech recognition module to some dialogue elements. Then the DM processes these dialogue elements and provides a suitable response which is fed to the NLG for response generation. Recent task-oriented dialogue systems have been designed based on deep reinforcement learning, which provided promising results regarding performance, domain adaptation, and dialogue generation.

Non-task-based Systems

As opposed to task-based dialogue systems, the goal behind designing and deploying non task-based dialogue systems is to empower a machine with the ability to have a natural conversation with humans. Typically, chatbots are of one of the following types: retrieval based methods and generative methods. Retrieval-based models have access to information resources and can provide more concise, fluent, and accurate responses. Generative models, on the other hand, have the advantage of being able to produce suitable responses when such responses are not in the corpus. However, as opposed to retrieval-based models, they are more prone to grammatical and conceptual mistakes arising from their generative models. Retrieval-based methods select an appropriate response from the candidate responses. Retrieval-based methods usually employ either Single-turn Response Matching or Multi-turn Response Matching. In the first type, the current query (message) is solely used to select a suitable response. The model tries to choose a response which considers the whole context to guarantee conversation consistency. An LSTM-based model has been proposed for context and response vectors creation. Generative models don’t assume the availability of predefined responses. New responses are produced from scratch and are based on the trained model. Generative models are typically based on sequence to sequence models and map an input query to a target element as the response. In general, designing and implementing a dialogue agent to be able to converse at the human level is very challenging. For this goal, the machine is generally trained on large corpora of conversations. For quantitative evaluation, adversarial evaluation was initially used for quality assessment of sentence generation and employed for quality evaluation of dialogue systems. Despite remarkable advancements in AI and much attention dedicated to dialogue systems, in reality, successful commercial tools, such as Apple’s Siri and Amazon’s Alexa, still heavily rely on handcrafted features. It still is very challenging to design and train data-driven dialogue machines given the complexity of the natural language, the difficulties in framework design, and the complex nature of available data sources.

Conclusion

In this article, we presented a comprehensive survey of the most distinguished works in Natural Language Processing using deep learning. We provided a categorised context for introducing different NLP core concepts, aspects, and applications, and emphasised the most significant conducted research efforts in each associated category. Deep learning and NLP are two of the most rapidly developing research topics nowadays. Due to this rapid progress, it is hoped that soon, new effective models will supersede the current state-of-the-art approaches.

This article is overview of the below IEEE paper —

https://arxiv.org/pdf/2003.01200.pdf