FROM Pre-trained Word Embeddings TO Pre-trained Language Models — Focus on BERT
By: Adrien Sieg
Tags: Deep Learning, NLP, Artificial Intelligence, Data Science, Machine Learning

Here is the result :

Dropout, Add & Norm

https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1184/lectures/lecture12.pdf

Before this layer, there is always a layer for which inputs and outputs have the same dimensions (Multi-Head Attention or Feed-Forward). We will call that layer Sublayer and its input x.

After each Sublayer, dropout is applied with 10% probability. Call this result Dropout(Sublayer(x)). This result is added to the Sublayer’s input x, and we get x + Dropout(Sublayer(x)).

Observe that in the context of a Multi-Head Attention layer, this means adding the original representation of a token x to the representation based on the relationship with other tokens. It is like telling the token:

“Learn the relationship with the rest of the tokens, but don’t forget what we already learned about yourself!”

Finally, a token-wise/row-wise normalization is computed with the mean and standard deviation of each row. This improves the stability of the network.

We compute the mean and variance used for normalization from all of the summed inputs to the neurons in a layer on a single training case.

Position-wise Feed-Forward Network

In addition to attention sub-layers, each of the layers in our encoder and decoder contains a fully connected feed-forward network, which is applied to each position separately and identically. This consists of two linear transformations with a ReLU activation in between.

While the linear transformations are the same across different positions, they use different parameters from layer to layer. Another way of describing this is as two convolutions with kernel size 1. The dimensionality of input and output is dmodel=512, and the inner-layer has dimensionality dff=2048.

Couches de neurones formels avec une ReLU comme fonction d’activation, formalisé par :

Ici W1 a pour dimensions [dimension des embeddings]x[dimmension d’entrée du FFN — au choix] et W2 [dimmension d’entrée du FFN — au choix] x [dimension des embeddings]. Source ici.

3.→Decoder Block

Each decoder layer consists of sublayers:

Masked multi-head attention (with look ahead mask and padding mask) Multi-head attention (with padding mask). V (value) and K (key) receive the encoder output as inputs. Q (query) receives the output from the masked multi-head attention sublayer. Point wise feed forward networks

Each of these sublayers has a residual connection around it followed by a layer normalization. The output of each sublayer is LayerNorm(x + Sublayer(x)) .

There are N decoder layers in the transformer.

As Q receives the output from decoder’s first attention block, and K receives the encoder output, the attention weights represent the importance given to the decoder’s input based on the encoder’s output. In other words, the decoder predicts the next word by looking at the encoder output and self-attending to its own output.

The Decoder consists of:

Output Embedding Positional Encoding N decoder layers

The target is put through an embedding which is summed with the positional encoding. The output of this summation is the input to the decoder layers. The output of the decoder is the input to the final linear layer.

What are the inputs of Transformer?

We feed it both the input and output sentences at the same time. The outputs initially can be filled with anything, the model ignores whatever you fill into that. It uses the entire input sentence and output sentence to predict the next word in a single go. Once we predict the word, we replace that in output sequence, and model only considers output till that point and ignores what is ahead of it. We continue to do that till we have a complete sentence.

Multi Head Masked Self Attention

In encoder, self-attention layers process input queries, keys and values that comes from the output of previous layer. Each position in encoder can attend to all positions from previous layer of the encoder.

In decoder, self-attention layer enable each position to attend to all previous positions in the decoder, including the current position.

To prevent positions from attending to subsequent position (http://www.peterbloem.nl/blog/transformers)

In other words, the self-attention layer is only allowed to attend to earlier positions in the output sequence. Masking multi-head attention is done by masking future positions (setting them to -∞) before the softmax step in the self-attention calculation. This step ensures that the predictions for position i can depend only on the known outputs at positions less than i. Since we want these elements to be zero after the softmax, we set them to −∞.

With RNNs — there is no issue like that, since they cannot look forward into the input sequence: output i depends only on inputs 0 to i. With a transformer, the output depends on the entire input sequence, so prediction of the next words/characters becomes vacuously easy, just retrieve it from the input.

To use self-attention as an autoregressive model, we’ll need to ensure that it cannot look forward into the sequence. We do this by applying a mask to the matrix of dot products, before the softmax is applied. This mask disables all elements above the diagonal of the matrix.

After we’ve handicapped the self-attention module like this, the model can no longer look forward in the sequence.

The “Decoder Attention” layer works just like multiheaded self-attention, except it creates its Queries matrix from the layer below it, and takes the Keys and Values matrix from the output of the encoder stack.

Self-attention layers in the decoder allow each position in the decoder to attend to all positions in the decoder up to and including that position. We need to prevent leftward information flow in the decoder to preserve the auto-regressive property. We implement this inside of scaled dot- product attention by masking out (setting to −∞) all values in the input of the softmax which correspond to illegal connections.

4.→The Final Linear and Softmax Layer

The decoder stack outputs a vector of floats. How do we turn that into a word? That’s the job of the final Linear layer which is followed by a Softmax Layer.

The Linear layer is a simple fully connected neural network that projects the vector produced by the stack of decoders, into a much, much larger vector called a logits vector. This space is the size of vocabulary (all words). We just project the matrix of weights (provided by the decoder block) into a “vocabulary space”.

Mathematically speaking, what does it mean?

Appelons S la matrice en sortie du décodeur. On la multiplie par une matrice de poids (qui peuvent apprendre) W1. C’est une couche totalement connectée qui projette simplement la sortie précédente dans un espace de la taille de notre vocable. W1 est la matrice qui va permettre d’extraire un mot dans notre dictionnaire de vocabulaire. Elle aura donc pour dimensions [dimension des embeddings, i.e. dmodel] x [nombre de mots dans notre vocable].

Let’s assume that our model knows 10,000 unique English words (our model’s “output vocabulary”) that it’s learned from its training dataset. This would make the logits vector 10,000 cells wide — each cell corresponding to the score of a unique word. That is how we interpret the output of the model followed by the Linear layer.

The softmax layer then turns those scores into probabilities (all positive, all add up to 1.0). The cell with the highest probability is chosen, and the word associated with it is produced as the output for this time step. Softmax provides us the most likely word to predict (we take the word of the column which give us the highest probability).

This figure starts from the bottom with the vector produced as the output of the decoder stack. It is then turned into an output word.

5.→ Residual connection

A residual connection is basically just taking the input and adding it to the output of the sub-network, making training deep networks easier in the field of computer vision. Layer normalization is a normalization method in deep learning that is similar to batch normalization. In layer normalization, the statistics are computed across each feature and are independent of other examples. The independence between inputs means that each input has a different normalization operation.

7. → Model Training — How BERT is trained?

A — Masked Language Modeling (MLM)

“The masked language model randomly masks some of the tokens from the input, and the objective is to predict the original vocabulary id of the masked word based only on its context. Unlike left-to-right language model pre-training, the MLM objective allows the representation to fuse the left and the right context, which allows us to pre-train a deep bidirectional Transformer.”

The Google AI researchers masked 15% of the words in each sequence at random. The task? To predict these masked words. A caveat here — the masked words were not always replaced by the masked tokens [MASK] because the [MASK] token would never appear during fine-tuning.

So, the researchers used the below technique:

80% of the time the words were replaced with the masked token [MASK]

10% of the time the words were replaced with random words

10% of the time the words were left unchanged

XLNet

B- Next Sentence Prediction

Generally, language models do not capture the relationship between consecutive sentences. BERT was pre-trained on this task as well.

For language model pre-training, BERT uses pairs of sentences as its training data. The selection of sentences for each pair is quite interesting. Let’s try to understand it with the help of an example.

Imagine we have a text dataset of 100,000 sentences and we want to pre-train a BERT language model using this dataset. So, there will be 50,000 training examples or pairs of sentences as the training data.

For 50% of the pairs, the second sentence would actually be the next sentence to the first sentence

For the remaining 50% of the pairs, the second sentence would be a random sentence from the corpus

The labels for the first case would be ‘IsNext’ and ‘NotNext’ for the second case

Applications?

A- Real Applications : Pre-trained vs. Fine-tuned

A review of BERT based models (https://towardsdatascience.com/a-review-of-bert-based-models-4ffdc0f15d58)

Models pretrained on domain/application specific corpus are Pre-trained models. Training on domain specific corpus has shown to yield better performance when fine-tuning them on downstream NLP tasks like NER etc. for those domains, in comparison to fine tuning BERT (which was trained on BooksCorpus and Wikipedia).

BioBERT (biomedical text)

SciBERT (scientific publications)

ClinicalBERT (clinical notes)

G-BERT (medical/diagnostic code representation and recommendation)

M-BERT from 104 languages for zero-shot cross lingual model transfer (task specific annotations in one language is used to fine tune model for evaluation in another language)

ERNIE (knowledge graph) + ERNIE (2) incorporates knowledge into pre-training but by masking entities and phrases using KG.

TransBERT — unsupervised, followed by two supervised steps, for a story ending prediction task

videoBERT (model that jointly learns video and language representation learning) by representing video frames as special descriptor tokens along with text for pretraining. This is used for video captioning.

Fine tuned models. Models fine tuned for a specific task using a pretrained model :

B- Case study

BONUS :

1. Why divide by square(dk) ?

2. Fine tuning

From https://yashuseth.blog/2019/06/12/bert-explained-faqs-understand-bert-working/

What is the fine-tuning procedure for sequence classification tasks?

What is the fine-tuning procedure for sentence pair classification tasks?

What is the fine-tuning procedure for Question Answering tasks?

What is the fine-tuning procedure for single sentence tagging tasks?

3. BERT as a service

The final hidden states (the transformer outputs) of the input tokens can be concatenated and / or pooled together to get the encoded representation of a sentence. bert-as-a-service is an open source project that provides BERT sentence embeddings optimized for production. Serving Google BERT in Production using Tensorflow and ZeroMQ.

4. Kullback Leibler divergence