What is XLNet and why it outperforms BERT | by Xu LIANG | Towards Data Science

Not less a week after the release, it seems everyone around me in the NLP field is talking about XLNet.
Yes, the “improves upon BERT on 20 tasks” did attract our eyes. But the more important thing is to understand how it works and why it outperforms BERT. So I write this blog to share my thoughts after reading the paper.
The content is structured as follows.
What is XLNet?
What are the differences between XLNet and BERT?
How XLNet works?
If you are interested in the Two-Stream Self-Attention in XLNet, you can look up my another post, What is Two-Stream Self-Attention in XLNet.
First of all, XLNet is a BERT-like model instead of a totally different one. But it is a very promising and potential one. In one word, XLNet is a generalized autoregressive pretraining method.
So what is the autoregressive(AR) language model?
AR language model is a kind of model that using the context word to predict the next word. But here the context word is constrained to two directions, either forward or backward.
The GPT and GPT-2 are both AR language model.
The advantages of AR language model are good at generative NLP tasks. Because when generating context, usually is the forward direction. AR language model naturally works well on such NLP tasks.
But AR language model has some disadvantages, it only can use forward context or backward context, which means it can't use forward and backward context at the same time.
Unlike the AR language model, BERT is categorized as autoencoder(AE) language model.
The AE language model aims to reconstruct the original data from corrupted input.
The corrupted input means we use [MASK] to replace the original token into in the pre-train phase. And the objective is to predict into to get the original sentence.
The advantages of AE language model is that it can see the context on both forward and backward direction.
But the AE language model also has its disadvantages. It uses the [MASK] in the pretraining, but this kind of artificial symbols are absent from the real data at finetuning time, resulting in a pretrain-finetune discrepancy. Another disadvantage of [MASK] is that it assumes the predicted (masked) tokens are independent of each other given the unmasked tokens. For example, we have a sentence “It shows that the housing crisis was turned into a banking crisis”. We mask “banking” and “crisis”. Attention here, we know the masked “banking” and “crisis” contains implicit relation to each other. But AE model is trying to predict “banking” given unmasked tokens, and predict “crisis” given unmasked tokens separately. It ignores the relation between “banking” and “crisis”. In other words, it assumes the predicted (masked) tokens are independent of each other. But we know the model should learn such correlation among the predicted (masked) tokens to predict one of the tokens.
What the author wants to emphasize is that the XLNet propose a new way to let the AR language model learn from bi-directional context to avoid the disadvantages brought by the MASK method in AE language model.
The AR language model only can use the context either forward or backward, so how to let it learn from bi-directional context?
Language model consists of two phases, the pre-train phase, and fine-tune phase. XLNet focus on pre-train phase. In the pre-train phase, it proposed a new objective called Permutation Language Modeling. We can know the basic idea from this name, it uses permutation.
Here we take an example to explain. The sequence order is [x1, x2, x3, x4]. All permutations of such sequence are below.
So for this 4 tokens (N) sentence, there are 24 (N!) permutations.
The scenario is that we want to predict the x3. So there are 4 patterns in the 24 permutations, x3 is at the 1st position, 2nd position, 3rd position, 4th position.
[x3, xx, xx, xx][xx, x3, xx, xx][xx, xx, x3, xx][xx, xx, xx, x3]
Here we set the position of x3 ast-th position and t-1 tokens are the context words for predicting x3.
The words before x3 have every possible words and length in the sequence. Intuitively, the model will learn to gather information from all positions on both sides.
The implementation is far complicated than the above explanation, I won’t talk it here. But you should get the most basic and most important idea about XLNet. If you are interested in the Two-Stream Self-Attention in XLNet, you can look up my another post, What is Two-Stream Self-Attention in XLNet.
Like BERT brought the MASK method to the public, XLNet showed permutation method is a good choice as the language model objective. It is predictable that there are more works to explore the language model objective in the future.
Check out my other posts on Medium with a categorized view!GitHub: BrambleXuLinkedIn: Xu LiangBlog: BrambleXu
paper: https://arxiv.org/abs/1906.08237
code: pytorch_transformers/modeling_xlnet.py
What is Two-Stream Self-Attention in XLNet