user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Extreme Language Model Compression with Optimal Subwords and Shared Projections

Paper content:
Pre-trained deep neural network language models such as ELMo, GPT, BERT and XLNet have recently achieved state-of-the-art performance on a variety of language understanding tasks.
However, their size makes them impractical for a number of scenarios, especially on mobile and edge devices.
In particular, the input word embedding matrix accounts for a significant proportion of the model's memory footprint, due to the large input vocabulary and embedding dimensions.
Knowledge distillation techniques have had success at compressing large neural network models, but they are ineffective at yielding student models with vocabularies different from the original teacher models.
We introduce a novel knowledge distillation technique for training a student model with a significantly smaller vocabulary as well as lower embedding and hidden state dimensions.
Specifically, we employ a dual-training mechanism that trains the teacher and student models simultaneously to obtain optimal word embeddings for the student vocabulary.
We combine this approach with learning shared projection matrices that transfer layer-wise knowledge from the teacher model to the student model.
Our method is able to compress the BERT-BASE model by more than 60x, with only a minor drop in downstream task metrics, resulting in a language model with a footprint of under 7MB.
Experimental results also demonstrate higher compression efficiency and accuracy when compared with other state-of-the-art compression techniques.
Recently, contextual-aware language models such as ELMo (Peters et al., 2018) , GPT (Radford et al., 2019) , BERT (Devlin et al., 2018) and XLNet have shown to greatly outperform traditional word embedding models including Word2Vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014) in a variety of NLP tasks.
These pre-trained language models, when finetuned on downstream language understanding tasks such as sentiment classification (Socher et al., 2013) , natural language inference (Williams et al., 2018) and reading comprehension (Rajpurkar et al., 2016; Lai et al., 2017) , have achieved state-of-the-art performance.
However, the large number of parameters in these models, often above hundreds of millions, makes it impossible to host them on resource-constrained tasks such as doing real-time inference on mobile and edge devices.
Besides utilizing model quantization techniques (Gong et al., 2014; Lin et al., 2016) which aim to reduce the floating-point accuracy of the parameters, significant recent research has focused on knowledge distillation (Ba & Caruana, 2014; Hinton et al., 2015) techniques.
Here, the goal is to train a small-footprint student model by borrowing knowledge, such as through a soft predicted label distribution, from a larger pre-trained teacher model.
However, a significant bottleneck that has been overlooked by previous efforts is the input vocabulary size and its corresponding word embedding matrix, often accounting for a significant proportion of all model parameters.
For instance, the embedding table of the BERT BASE model, comprising over 30K WordPiece tokens (Wu et al., 2016b) , accounts for over 21% of the model size.
While there has been existing work on reducing NLP model vocabulary sizes (Sennrich et al., 2016) , distillation techniques cannot utilize these, since they require the student and teacher models to share the same vocabulary and output space.
This profoundly limits their potential to further reduce model sizes.
We present two novel ideas to improve the effectiveness of knowledge distillation, in particular for BERT, with the focus on bringing down model sizes to as much as a few mega-bytes.
Our model is among the first to propose to use a significantly smaller vocabulary for the student model learned during distillation.
In addition, instead of distilling solely on the teacher model's final-layer outputs, our model leverages layer-wise teacher model parameters to directly optimize the parameters of the corresponding layers in the student model.
Specifically, our contributions are:
• Dual Training: Our teacher and student models have different vocabularies and incompatible tokenizations for the same sequence.
To address this during distillation, we feed the teacher model a mix of teacher vocabulary-tokenized and student vocabulary-tokenized words within a single sequence.
Coupled with the masked language modeling task, this encourages an implicit alignment of the teacher and student WordPiece embeddings, since the student vocabulary embedding may be used as context to predict a word tokenized by the teacher vocabulary and vice versa.
• Shared Variable Projections: To minimize the loss of information from reducing the hidden state dimension, we introduce a separate loss to align the teacher and student models' trainable variables.
This allows for more direct layer-wise transfer of knowledge to the student model.
Using the combination of dual training and shared variable projections, we train a 12-layer highlycompressed student BERT model, achieving a maximum compression ratio of ∼61.94x (with 48 dimension size) compared to the teacher BERT BASE model.
We conduct experiments for measuring both generalized language modeling perspective and for downstream tasks, demonstrating competitive performance with high compression ratios for both families of tasks.
Shared projections and model performance: We see that for downstream task performance, dual training still consistently improves upon the direct fine-tuning approach for virtually all experiments.
The effect of shared variable projection, however, is less pronounced, with consistent improvements visible only for MRPC and for the 48-dimensional models i.e. the smallest dataset and models respectively in our experiments.
This aligns with our intuition for variable projection as a more direct way to provide a training signal from the teacher model internals, which can assume more importance for a low-data or small-model scenario.
However, for larger models and more data, the linear projection of parameters may be reducing the degrees of freedom available to the model, since linear projection is a fairly simple function to align the teacher and student parameter spaces.
A related comparison of interest is between up-projection and down-projection of the model variables: we note up-projection does visibly better on the language modeling task and slightly better on the downstream tasks.
The parameters of a well-trained teacher model represent a high-quality local minimum in the teacher space, which may be easier to search for during up-projection.
Vocabulary size tradeoffs: Issues with input vocabulary size are peculiar to problems in natural language processing: they do not always apply to other areas such as computer vision, where a small fixed number of symbols can encode most inputs.
There has been some work on reducing input vocabulary sizes for NLP, but typically not targeting model compression.
One concern with reducing the vocabularies of NLP models is it pushes the average tokenized sequence lengths up, making model training harder.
In this work, however, we consider classification tasks on shorter texts, which are not as affected by input sequence lengths as, say, tasks such as machine translation are.
Furthermore, many real-world applications revolve around short text inputs, which is why a better trade-off between vocabulary size and sequence lengths may be worthwhile for such applications.
Order of distillation and fine-tuning: Most of the existing work on distilling language models such as BERT and reporting results on downstream tasks, including some of the baselines in this work, first fine-tune a teacher model on the downstream tasks, and then distill this model.
Our goal in this work, however, is to explore the limits to which BERT's language modeling capacity itself, and how much of it is driven by its large WordPiece vocabulary.
We leave experiments on distilling fine-tuned teacher models, potentially yielding better results on downstream tasks, to future work.
We proposed two novel ideas to improve the effectiveness of knowledge distillation for BERT, focusing on using a significantly smaller vocabulary, as well as smaller embedding and hidden dimensions for the student BERT language models.
Our dual training mechanism encourages implicit alignment of the teacher and student WordPiece embeddings, and shared variable projection allows for the faster and direct layer-wise transfer of knowledge to the student BERT model.
Combining the two techniques, we trained a series of highly-compressed 12-layer student BERT models.
Experiments on these models, to evaluate both generalized language perspective and four standardized downstream tasks, demonstrate the effectiveness of our proposed methods on both model accuracy and compression efficiency.
One future direction of interest is to combine our approach with existing work to reduce the number of layers in the student models and explore other approaches such as low-rank matrix factorization to transfer model parameters from the teacher space to the student space.
In addition, taking into account the frequency distribution of the WordPiece tokens while training embeddings may help optimize the model size further.

assistant: text: We present novel distillation techniques that enable training student models with different vocabularies and compress BERT by 60x with minor performance drop.