Neural Networks, at Attention!
By: Davide Salvaggio
Tags: Bert, Deep Learning, Attention, Gpt 3, NLP

THE LANGUAGE MODELING PATH: CHAPTER 2

Neural Networks, at Attention!

The mechanism that makes language models focus on the important things

Hi and welcome to the second step of the Language Modeling Path; a series of articles from Machine Learning Reply aimed to cover the most important milestones that brought to life the huge language models able to imitate and (let’s say) understand the human language like BERT and GPT-3.

In this article we are going to talk about attention layers. This kind of architectural trick was first applied to the computer vision field [1] but in this article we will focus only on the Neural Natural Language Processing application and in particular on the sequence to sequence applications for Neural Machine Translation (NMT). While this article is based on two papers ([2] and [3]), attention is a widespread technique that you can find explained everywhere if you need to go deeper.

To better understand this chapter, it is strongly suggested to have a good comprehension of encoder-decoder sequence-to-sequence models. If you need a good refresh of the key concept of those architectures, you can start from the first chapter of our Language Modeling Path.

Attention in the real world

The introduction of attention mechanism in common seq2seq applications allows the elaboration of longer and more complex sentences. As we anticipated the basic insight of this trick was born inside the computer vision framework and then developed around natural language for Neural Machine Translation (NMT) applications. In this article we will focus on NMT simply because this is the natural habitat of such algorithm but nevertheless the family of attention-based models (models that rely on this particular architectural pattern) counts among its ranks many state of the art models in most of Natural Language Processing application fields.

This is because attention mechanism is the key that makes Google Assistant and Amazon Alexa understand our intentions even if we use more than a simple sentence to express it.

It can give a boost of accuracy in all the applications that require text embedding. Here it is a brief (incomplete) list of topics where we were able to experience its improvements with respect to not attention-based models.

Document retrieval

Text classification

Text clustering

Text similarity

Personalized search engines

Text generation

Moreover, the attention mechanism gives a more detailed insight on what part of the input had the highest impact on the decision made by our model, and this is a huge advantage in a production environment making the black box neural network a little less black.

A problem in standard Sequence-to-Sequence NMT

In the previous chapter we discussed one of the most effective architectures still used nowadays for the NMT problem, the sequence-to-sequence model. In those kind of networks an input sequence, made of the single words that compound the sentence in the starting language, is fed into a recurrent neural network, like for example an LSTM network. This network tries to collect all the information from each single input word and stores it inside a fixed length array. This first part of the model is called the encoder. The encoded array is then passed as initial hidden state of a second recurrent neural network that tries to generate the correct translation of one word at the time, starting at each time from the previous hidden state and the previously generated word; this second part is called decoder.

This kind of approach allows the input and the output sequence to have a great flexibility in length and opened up the way to deep learning inside the automatic translation field of research. However this approach still presents some intuitive downsides.

From [2] we can in fact read:

“A potential issue with this encoder–decoder approach is that a neural network needs to be able to compress all the necessary information of a source sentence into a fixed-length vector. This may make it difficult for the neural network to cope with long sentences, especially those that are longer than the sentences in the training corpus. Cho et al.(2014b) showed that indeed the performance of a basic encoder–decoder deteriorates rapidly as the length of an input sentence increases.”

Even if LSTM recurrent neural networks are able to persist information taken from the beginning of the input sentence up to the end i.e. up to the encoded array, the available space where this information can be stored is limited by the fixed dimension of the encoded array. If the input sentence is too long with respect to this fixed dimension, then we cannot avoid the loss of some important information and the translation will be affected from it.

The attention-based model

The idea behind the attention mechanism is the following. During encoding phase, at each step of the RNN I store the hidden state related to each one of the input sentence’s words. Then during the decoding, at each step of the RNN, in addition to the previous decoder’s hidden state and generated word, I take also into account a weighted average of the previously stored encoder’s hidden states. The weights of this average depend directly from the previous decoder’s hidden state and are learned during training in order to give more attention to the encoder’s hidden states related to the words that are most “aligned” (semantically, syntactically, etc..) with the translated word I am trying to generate at the current step.