Most Powerful NLP Transformer - ELECTRA | Towards Data Science

Transformers dominate the field of Natural Language Processing (NLP). The 2017 paper ‘Attention is All You Need’ introduced the first transformer — no other architecture has since dethroned these NLP powerhouses.
Fast-forward a year to 2018, and we find BERT (Bidirectional Encoder Representations from Transformers). A huge pre-trained model that could be taken and applied to a wide range of language tasks without further training.
BERT’s approach has become the standard in NLP — train a massive transformer-based architecture on an unimaginably large dataset, test it on a few standardized benchmarks — and release it to the public.
Several models have since been introduced that have improved on the success of BERT — like RoBERTa and XLNet — but all of them rely on bigger networks and larger datasets.
All but one — the 2020 ELECTRA model — which has introduced a new approach to training that produces a model with comparable (and often better) performance than even the best transformers — and requires just a fraction of the compute power.
> How to Train a Transformer> A Different Pre-Training Approach> Performance Compared to Other Models
BERT makes use of masked language modeling (MLM) during pre-training. MLM consists of taking the input sequences and replacing some of those tokens with a mask token [MASK].
That means that:
["the", "chef", "cooked", "the", "meal"]
Becomes:
["[MASK]", "chef", "[MASK]", "the", "meal"]
After masking these inputs, the model is trained to replace these [MASK] tokens with the correct word.
The MLM approach works brilliantly and has produced genuinely astonishing results. BERT is being applied as a key component of solutions across many industries — by thousands of ML engineers and data scientists. And for every BERT use-case, we see another for XLNet, and another for GPT-2/3.
These models have been hugely successful. But they all rely on and huge amounts of training data. The compute cost for training these models is another hugely influential factor — GPT-3 costs an estimated $4.6M to train [1].
$4.6M to train a model is huge — and this is just one training session. No research team is going to successfully design and train a model on their first attempt. These things take time and a whole lot of training and retraining.
Heading into the future, the trend for these huge transformer models is to add more training data and more parameters — pushing performance higher and higher. This naturally results in longer training times and more compute.
ELECTRA breaks from this trend. Unlike its predecessors that relied on MLM pre-training — ELECTRA uses a more efficient approach called ‘replaced token detection’.
Rather than masking a random selection of input tokens — this approach uses another neural network that attempts to trick our model by replacing random tokens with fake tokens.
The approach is similar to that employed by GANs. When training GANs, we pit two networks (the generator and discriminator) against each other. The generator is optimized to ‘trick’ the discriminator with increasingly convincing (but nonetheless fake) data.
Our discriminator is then left to identify what is true and what is false data generated by the generator model.
ELECTRA applies a similar approach. Our model is the discriminator, which is tasked with identifying which tokens are genuine and which are fakes that have been placed there by the generator.
The reason that this is not a GAN-type architecture is that the generator is not optimized to increase the loss of the discriminator — it is instead trained like a typical MLM model — where it must best guess the [MASK] values.
According to the ELECTRA paper, this approach to pre-training is more efficient than MLM because the model must consider every single token in every single sample it sees — whereas MLM only requires the model to focus on [MASK] tokens [2].
Because the model focuses and training on every token, the model produces an improved comprehension of context.
Side note: using a smaller model for the generator improves results — specifically a generator with 25–50% the number of parameters as the discriminator [2].
After pre-training is complete, the generator model is discarded, leaving us with our new ELECTRA transformer model. We can then apply ELECTRA to the typical language tasks handled by transformers, such as text generation or question-and-answering.
Overall, ELECTRA looks to be one of the highest performing transformer models while being significantly smaller than all other models on par with its performance.
In the 2020 paper, the model was tested on both GLUE and SQuAD benchmarks.
GLUE is a classification task where the language model must accurately predict the correct words for given sentences. The benchmark is designed to deeply measure the model’s ability in Natural Language Understanding (NLU) with nine tasks that measure different linguistic phenomena [3].
For the GLUE benchmarking, the model is fine-tuned with additional linear classifiers on-top of ELECTRA.
SQuAD is a question-and-answering benchmark created at Stanford — of which there are two versions, SQuAD and SQuAD 2. Here, ELECTRA is benchmarked against both — and is tasked with selected the span of a passage that answers a given question.
For the SQuAD benchmark, the Q&A module from XLNet is added to ELECTRA.
Score is calculated as the average of CoLA, SST, MRPC, STS, QQP, MNLI, RTE, and WNLI metrics
On the GLUE test set, Electra outperformed every other transformer model (when taking an average score of several metrics) — despite requiring less compute to train than all other models (except Bert).
SQuAD values are F1 scores
Comparing Electra to other models on the SQuAD benchmark shows that Electra is a clear winner both in performance and training compute requirements — with the exception of Bert in compute and XLNet in performance on the SQuAD 1.1 dev set.
To address the title — in reality, Bert is still very relevant — but Electra shows what can be done with a little out-of-the-box thinking, and it will be incredible to see what other fascinating models appear in the future.
If you haven’t already — I would definitely recommend trying Electra in any transformer-based NLP tasks you’ve been working on.
If this is all new to you, but you’d love to give Electra a go, take a look at HuggingFace’s Transformers library and their documentation on Electra itself.
I hope you enjoyed the article! If you have any suggestions or questions, feel free to reach out via Twitter or in the comments below! If you’re interested in seeing more content like this — I post on YouTube too.
Thanks for reading!
[1] L. Chuan, OpenAI’s GPT-3 Language Model: A Technical Overview (2020), Lambda Labs
[2] K. Clark, et al., ELECTRA: Pre-Training Tect Encoders as Discriminators Rather Than Generators (2020), ICLR
[3] A. Wang et al., GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (2018), EMNLP
🤖 NLP With Transformers Course
Interested in learning more about transformers? I’ve written about them a lot — but this is easily my favorite intro:
towardsdatascience.com
*All images are by the author except where stated otherwise