user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: MobileBERT: Task-Agnostic Compression of BERT by Progressive Knowledge Transfer

Paper content:
The recent development of Natural Language Processing (NLP) has achieved great success using large pre-trained models with hundreds of millions of parameters.
However, these models suffer from the heavy model size and high latency such that we cannot directly deploy them to resource-limited mobile devices.
In this paper, we propose MobileBERT for compressing and accelerating the popular BERT model.
Like BERT, MobileBERT is task-agnostic; that is, it can be universally applied to various downstream NLP tasks via fine-tuning.
MobileBERT is a slimmed version of BERT-LARGE augmented with bottleneck structures and a carefully designed balance between self-attentions and feed-forward networks.
To train MobileBERT, we use a bottom-to-top progressive scheme to transfer the intrinsic knowledge of a specially designed Inverted Bottleneck BERT-LARGE teacher to it.
Empirical studies show that MobileBERT is 4.3x smaller and 4.0x faster than original BERT-BASE while achieving competitive results on well-known NLP benchmarks.
On the natural language inference tasks of GLUE, MobileBERT achieves 0.6 GLUE score performance degradation, and 367 ms latency on a Pixel 3 phone.
On the SQuAD v1.1/v2.0 question answering task, MobileBERT achieves a 90.0/79.2 dev F1 score, which is 1.5/2.1 higher than BERT-BASE.
The NLP community has witnessed a revolution of pre-training self-supervised models.
These models usually have hundreds of millions of parameters.
They are trained on huge unannotated corpus and then fine-tuned for different small-data tasks (Peters et al., 2018; Radford et al., 2018; Devlin et al., 2018; Radford et al., 2019; Yang et al., 2019) .
Among these models, BERT (Devlin et al., 2018) , which stands for Bidirectional Encoder Representations from Transformers (Vaswani et al., 2017) , shows substantial accuracy improvements compared to training from scratch using annotated data only.
However, as one of the largest models ever in NLP, BERT suffers from the heavy model size and high latency, making it impractical for resource-limited mobile devices to deploy the power of BERT in mobile-based machine translation, dialogue modeling, and the like.
There have been some works that task-specifically distill BERT into compact models (Turc et al., 2019; Tang et al., 2019; Sun et al., 2019; Tsai et al., 2019) .
To the best of our knowledge, there is not yet any work for building a task-agnostic lightweight pre-trained model, that is, a model that can be fine-tuned on downstream NLP tasks just like what the original BERT does.
In this paper, we propose MobileBERT to fill this gap.
In practice, task-agnostic compression of BERT is desirable.
Task-specific compression needs to first fine-tune the original large BERT model into task-specific teachers and then distill.
Such a process is way more complicated and costly than directly fine-tuning a task-agnostic compact model.
At first glance, it may seem straightforward to obtain a task-agnostic compact version of BERT.
For example, one may just take a narrower or shallower architecture of BERT, and then train it with a prediction loss together with a distillation loss (Turc et al., 2019; Sun et al., 2019) .
Unfortunately, empirical results show that such a straightforward approach results in significant accuracy loss (Turc et al., 2019) .
This may not be that surprising.
It aligns with a well-known observation that shallow networks usually do not have enough representation power while narrow and deep networks are difficult to train.
Our MobileBERT is designed to be as deep as BERT LARGE while each layer is made much narrower via adopting bottleneck structures and balancing between self-attentions and feed- MobileBERT is trained by progressively transferring knowledge from IB-BERT.
forward networks (Figure 1 ).
To train MobileBERT, we use a bottom-to-top progressive scheme to transfer the intrinsic knowledge of a specially designed Inverted Bottleneck BERT LARGE (IB-BERT) teacher to it.
As a pre-trained NLP model, MobileBERT is both storage efficient (w.r.t model size) and computationally efficient (w.r.t latency) for mobile and resource-constrained environments.
Experimental results on several NLP tasks show that while being 4.3× smaller and 4.0× faster, MobileBERT can still achieve competitive results compared to BERT BASE .
On the natural language inference tasks of GLUE, MobileBERT can have only 0.6 GLUE score performance degradation with 367 ms latency on a Pixel 3 phone.
On the SQuAD v1.1/v2.0 question answering task, MobileBERT obtains 90.3/80.2 dev F1 score which is 1.5/2.1 higher than BERT BASE .
2 RELATED WORK 2.1 BERT BERT takes the embedding of source tokens as input.
Each building block of BERT contains one Multi-Head self-Attention (MHA) module (Vaswani et al., 2017) and one Feed-Forward Network (FFN) module, which are connected by skip connections.
The MHA module allows the model to jointly attend to information from different subspaces, while the position-wise FFN consists of a two-layer linear transformation with gelu activation (Hendrycks & Gimpel, 2016) , which increase the representational power of the model.
Figure 1 (a) illustrates the original BERT architecture.
In the pre-training stage, BERT is required to predict the masked tokens in sentences (mask language modeling task), as well as whether one sentence is the next sentence of the other (next sentence prediction task).
In the fine-tuning stage, BERT is further trained on task-specific annotated data.
We perform an ablation study to investigate how each component of MobileBERT contributes to its performance on the dev data of a few GLUE tasks with diverse characteristics.
To accelerate the experiment process, we halve the original pre-training schedule in the ablation study.
We conduct a set of ablation experiments with regard to Attention Transfer (AT), Feature Map Transfer (FMT) and Pre-training Distillation (PD).
The operational OPTimizations (OPT) are removed in these experiments.
Moreover, to investigate the effectiveness of the proposed novel architecture of MobileBERT, we compare MobileBERT with two compact BERT models from Turc et al. (2019) .
For a fair comparison, we also design our own BERT baseline BERT SMALL* , which is the best model setting we can find with roughly 25M parameters under the original BERT architecture.
The detailed model setting of BERT SMALL* can be found in Table 2 .
Besides these experiments, to verify the performance of MobileBERT on real-world mobile devices, we export the models with Tensorflow Lite 5 APIs and measure the inference latencies on a single large core of a Pixel 3 phone with a fixed sequence length of 128.
The results are listed in Table 5 .
We first can see that the propose Feature Map Transfer contributes most to the performance improvement of MobileBERT, while Attention Transfer and Pre-training Distillation also play positive roles.
As expected, the proposed operational OPTimizations hurt the model performance a bit, but it brings a crucial speedup of 1.68×.
In architecture comparison, we find that although specifically designed for progressive knowledge transfer, our MobileBERT architecture alone is still quite competitive.
It outperforms BERT SMALL * and BERT SMALL on all compared tasks, while outperforming the 1.7× sized BERT MEDIUM on the SST-2 task.
Finally, we can L1 H1  L1 H2  L1 H3  L1 H4  L12 H1  L12 H2  L12 H3  L12 H4 MobileBERT ( find that although augmented with the powerful progressive knowledge transfer, our MobileBERT still degrades greatly when compared to the IB-BERT LARGE teacher.
We visualize the attention distributions of the 1 st and the 12 th layers of a few models in Figure  3 for further investigation.
The proposed attention transfer can help the student mimic the attention distributions of the teacher very well.
Surprisingly, we find that the attention distributions in the attention heads of "MobileBERT(bare)+PD+FMT" are exactly a re-order of those of "Mobile-BERT(bare)+PD+FMT+AT" (also the teacher model), even if it has not been trained by the attention transfer objective.
This phenomenon indicates that multi-head attention is a crucial and unique part of the non-linearity of BERT.
Moreover, it can explain the minor improvements of Attention Transfer in ablation table 5, since the alignment of feature maps lead to the alignment of attention distributions.
We have presented MobileBERT which is a task-agnostic compact variant of BERT.
It is built upon a progressive knowledge transfer method and a conjugate architecture design.
Standard model compression techniques including quantization (Shen et al., 2019) and pruning (Zhu & Gupta, 2017) can be applied to MobileBERT to further reduce the model size as well as the inference latency.
In addition, although we have utilized low-rank decomposition for the embedding layer, it still accounts for a large part in the final model.
We believe there is a big room for extremely compressing the embedding table (Khrulkov et al., 2019; May et al., 2019) .

assistant: text: We develop a task-agnosticlly compressed BERT, which is 4.3x smaller and 4.0x faster than BERT-BASE while achieving competitive performance on GLUE and SQuAD.