Cross-Architecture Distillation Using Bidirectional CMOW Embeddings
Authors: Lukas Paul Achatius Galke
Isabelle Cuber
Christoph Meyer
Henrik Ferdinand Nölscher
Angelina Sonderecker
Ansgar Scherp
Venue: ICLR 2022 Submitted (2022)
Keywords: natural language processing
word embedding
knowledge distillation
model compression
efficient methods
transfer learning

Abstract: Large pretrained language models (PreLMs) are revolutionizing natural language processing across all benchmarks. However, their sheer size is prohibitive for small laboratories or deployment on mobile devices. Approaches like pruning and distillation reduce the model size but typically retain the same model architecture. In contrast, we explore distilling PreLMs into a different, more efficient architecture CMOW, which embeds each word as a matrix and uses matrix multiplication to encode sequences. We extend the CMOW architecture and its CMOW/CBOW-Hybrid variant with a bidirectional component, per-token representations for distillation during pretraining, and a two-sequence encoding scheme that facilitates downstream tasks on sentence pairs such as natural language inferencing. Our results show that the embedding-based models yield scores comparable to DistilBERT on QQP and RTE, while using only half of its parameters and providing three times faster inference speed. We match or exceed the scores of ELMo, and only fall behind more expensive models on linguistic acceptability. Still, our distilled bidirectional CMOW/CBOW-Hybrid model more than doubles the scores on linguistic acceptability compared to previous cross-architecture distillation approaches. Furthermore, our experiments confirm the positive effects of bidirection and the two-sequence encoding scheme.

Under review as a conference paper at ICLR 2022
CROSS-ARCHITECTURE DISTILLATION USING BIDIRECTIONAL CMOW EMBEDDINGS
Anonymous authors Paper under double-bling review
ABSTRACT
Large pretrained language models (PreLMs) are revolutionizing natural language processing across all benchmarks. However, their sheer size is prohibitive for small laboratories or deployment on mobile devices. Approaches like pruning and distil- lation reduce the model size but typically retain the same model architecture. contrast, we explore distilling PreLMs into a different, more efficient architecture CMOW, which embeds each word as a matrix and uses matrix multiplication to en- code sequences. We extend the CMOW architecture and its SMOW/CBOW-Hyyeee variant with a bidirectional component, per-token representations for distillation during pretraining, and two-sequence encoding scheme that facilitates down- stream tasks on sentence pairs such as natural language inferencing. Our results show that the embedding- baandd models yield scores comparable to DistilBERT QQP and RTE, while using only half of its parameters and providing three times faster inference speed. We match or exceed the scores of ELMo, and only fall behind more expensive models on linguistic acceptability. Still, our distilled bidi- rectional KMOW/CBOW-HyMme
model more than doubles the scores on linguistic acceptability compared to previous cross- crssttrchite distillation approaches. Fur- thermore, our experiments confirm the positive effects of bidirection and the two-sequence encoding scheme.
INTRODUCTION
Large pretrained language models (Devlin et al., 2019 Raffel et al. 2020 de-facto standard methods for natural language processing (Wang et al.
have emerged as
2018 2019). The common strategy is to pretrain models on enormous amounts of unlabeled text before fine-tuning them for downstream tasks. However, the drawback of PreLMs is that the models are becoming larger and larger with up to several billions of parameters (Brown et al. (020). This comes with high environmental and economic costs (Strubell et al., 2019) and shifts the development and research in the hands of a few global players only (Bommasani et al. 2022 pp. 10-12). Even though a single pretrained model can be reused for multiple downstream tasks, the sheer model size is often prohibitive. The immense resource requirements prevent using those models in small-scale laboratories and on mobile devices, which is tied to privacy concerns (Sanh et al. 2020b
There a need for more efficient models or compressed versions of large models make AI research more inclusive and energy-friendly, while fostering deployment in applications. Reducing the size of PreLMs using knowledge distillation (Hinton al. 2015) model compression (Bucila al. 2006) is an active area of research (Sanh et al. 2020a] Jiao et al. 2020 Sun et al. 2020). Both know ledge distillation and model compression can described as teacher-student setups. The student is trained to imitate the predictions of the teacher while using less resources. Typically, large PreLM takes the role of the teacher while the student a smaller version of the same architecture, Sharing the same architecture between student and teacher enables using dedicated distillation techniques, e. e.g..
aligning the representations of intermediate layers (Sanh et al. 2020al Sun et al. 2020).
However, using more efficient architectures student has already shown promising results such as the task-specific distillation approaches by Tang et al. (2019) and Wasserblat et al. (2020). In their works, the student models are LSTMs (Hochreiter & Schmidhuber 1997) or models based
continuous bag-oft ofrwodds representation (CBOW) (Collobert & Weston 2008 Mikolov et al., 2013).
On the hand, LSTMs are difficult to parallelize they need least ((n) sequential steps to

---

Under review as a conference paper at ICLR 2022
encode sequence of length On the other hand, CBOW-based models are not order-aware, e.e., cannot distinguish sentences with the same words but in different order ("cat cats mouse" vs. "mouse eats cat" are treated equivalent). There are, however, more efficient models such as Mai et al. (2019)'s continual multiplication of words (CMOW) that do capture word order by representing each token as matrix, instead of vector in CBOW. A sequence in CMOW is modeled by the non-commutative matrix multipiccaiinn, which makes the encoding of a sequence becomes dependent on the word order. We denote such models as matrix embeddings.
The present work investigates how order-aware matrix embeddings can be used as student models in architecthreee distillation from large PreLM teachers. Thus, we complement the existing body of works that focused predominantly on sume-architecturs distillation. All previous acchttrcc approaches are task-specifie, whereas we also explore general distillation. We aim to understand to which extent order-aware embeddings are suited to capture the teacher signal of large PreLM model, which is
such as BERT, To this end, we extend Mai et al. (2019)'s 's CMOW/CBOW- hybrid variant unifying the strength of CBOW and CMOW, with bidirectional representation of the sequences. Furthermore, we add the ability to emit per-token representations to facilitate using a modern masked language model objective (Devlin et al. 2019). We investigate both general distillation, e., the distillation is applied during pretraining on unlabeled text, as well as task-specific distillation, when an already fine-tuned PreLM is distilled on per-task basis. We further introduce two-sentence encoding scheme to CMOW such that it can deal with similarity and natural language inferencing tasks. This improves the performance by 32% compared
naive joint encoding.
Our results show that large PreLMs can be distilled into efficient order-aware embedding models and achieve performance comparable ELMo (Peters et al. 2018) on the GLUE benchmark. On certain tasks, embedding-bell models even challenge other size-reduced BERT models such as DistilIBERT. In summary, our contributions are:
We extend order-aware embedding models with bidirection and make them amenable for masked language model pretraining.
We explore using order-aware embedding models as student models in a cross-architecture distillation setup with BERT as teacher and compare general and task task-specific distillation.
We introduce the first encoding scheme that enables CMOW/CBOW- two-sentence tasks (32% increase over the naive approach).
to deal with
Our results show that the best distilled embedding models can be on-par with more expensive models such as ELMo or DistilBERT on certain tasks of the GLUE benchmark, while having a much higher encoding speed (thrice high as DistiBBERT).
Below, we introduce our embedding models, our cross-architecture distillation setup, and our two- sequence encoding scheme. The experimental procedure described in Section The results are reported in Section 4 and discussed in Section 5 where we also relate our work to the literature.
2
METHODS
First, we introduce our bidirectional extension to the CMOW/BBOW-Hybrid model. Subsequently, introduce approach for aresitecche distillation that use during the pretraining and fine-tuning stages. Finally, we introduce a two-sentence encoding scheme for awaer embedding models that is crucial for fine-tuning on downstream tasks with paired sentences.
2.1 EXTENDING THE ORDER-AWARE EMBEDDING MODELS
We extend the CMOW/CBOW embeddings of Mai et al. 2019) with bidirection and the ability to emit per-token representations foundation for cross architecture distillation. CMOW/CBOW. Hybrid embeddings, our baseline model, are a combination of matrix embeddings and vector embeddings. Compared vector-only embeddings, the word order can be captured because matrix multiplication non-commutative. Given a sequence of n tokens, matrix-space embeddings X; E Rdxd for each different token j, and vector-space embeddings x; e Rdvee, the CMOW/CBÓW. Hybrid embedding of sequence of length is the multiplication of embedding matrices X;
2

---

Under review as a conference paper at ICLR 2022
concatenated (symbol -||-) to the summation of the embedding vectors Zş (CMOW X
(CMOW) X2 x/CMOW)
X (CMOW)
h (CBOW) Σ
(CBOW)
15i5"
h (yyril = flatten
(H((MOW))
|| (CBOW)
where flatten collapses the matrix into vector. The original work on CMOW (Mai et al. 2019) has extensively analyzed the CMOW and CBOW components individually and found that joint training generally preferable. The CMOW and CBOW components can have different dimensionalities, as they are combined via concatentation. We initialize each matrix X, as identity plus Gaussian noise Ia NO, Pist)) with Oinit 0.01.
H
dog
Left-to-right Matrix Multiplication
H!hw
HDWI
Masked Language
Modeling Head
fax
xfh
xfflg quick
((h))
lazy
the
quick
brown
[MASK]]
jumps
over
the
[MASK]
dog
Xiee
X(tmlgog
Right--toleftt Multiplication
Teacher Signal
H(Ow)
Figure The bidirectional CMOW component of our proposed architecture during pretraining. In this example, the model predicts the masked token at position 4 by concatenating forward and backward matrix embeddings, which are then fed into a masked language modeling head.
Proposed Model: Bidirectional CMOW/BBOW-Hybrl Inspired by the success of bidirection RNNs (Schuster & Paliwal 1997), LSTMs (Peters et al. 2018), and Transformers (Devlin et al. 2019) extend CMOW by bidirectional component. Hence, introduce second set of matrix- space embeddings that are multiplied in reverse order. We then have one matrix embedding for the forward direction (fw) E [R Roocb xdxd and one for the backward direction X(bw) E mvoce xdxd Then, we concatenate forward and backward directions. Figure T Iillustrates bidirectional CMOW.
Furthermore, we emit one representation per token position i, which allows training with a masked language model objective (Devlin et al. 2019). Thus, we are able to make use the BERT teacher signal for pretraining. Since we can reuse computations, (nn) matrix multiplications are sufficient to encode a sequence of length n. For these intermediate representations, we also modify the CBOW component in way that it yields partial sums for the forward and backward directions. Formally, compute the CMOW/CBOW-Hybri representation as follows:
H (Biii. CMOW) X (fw)
x((w)
X (fw)
X (fw)
(bw). x (bw)) ...X! (bw)
1
(Bidi. CBOW) h
(CBOW)
(CBOW)
x
h
(Bidi. Hybrid) flatten
H
Bidi. (otos)
h
(Bidi. CBOW)
3

---

Under review as a conference paper at ICLR 2022
For fine-tuning tasks with full sentences as imput, g., natural language inferencing, do not need per-token representations. In this case, we compute the representation of a token sequence as:
H (Bidi CMOw) == X (fw)
(fw)
X (tw)
X (bw).
X
(bw)
X (bw)
n-1
h (CROW)
(CBOW)
j=1
h (Bidi. Hybrid) flatten
H
Bidi. ) (CBOW)
h
Note that the forward and backward directions of the embedding vectors (CBOW) conflate to equivalent formulas when we encode entire sequences. Thus, we only need to include single CBOW representation along with the two CMOW components that do yield different results for the forward and backward direction. At inference time, the model is parallelizable along the sequential dimension.
For regularization. we apply a mild dropout (p=0.1) on both the embeddings and their aggregated representations during pretraining. Then, we feed them into a linear masked language modeling head, see Figure [] or an MLP classification head to tackle the downstream tasks. We have also experimented with linear, LSTM, and CNN classifiers, which are described in Appendix A We chose an MLP because adds nonlinearity to the model without introducing further complexity. The MLP has led to the best average performance across all GLUE tasks (we report the results in Appendix C
2.2
DISTILLATION
central question of our research is whether we can distill a large PreLM, e. g.. BERT, into more efficient, non-transformer architectures such the proposed bidirectional CMOW/CBOW-Hyyrid model. This requires a cross-architecture distillation approach, which we describe below.
In general, the idea of knowledge distillation is to compress the knowledge of large teacher model
into a smaller student model (Hinton et al. 2015 Bucila et al. 2006) It involves a loss function L that is combination of loss terms, i. e., L a Lhard + (1 ax) Looft with weighting parameter a. Liart denotes the cross-entropp loss with respect to the ground truth and Lsott = E,t, log(s;) )) is the cross-entropy between student logits and the teacher signal Optionally, the softmax within Loon is flattened by temperature parameter T.. We distinguish general distillation, where BERT's teacher signal is only used during pretraining, and task task-specific distillation, where the BERT teacher signal is used during fine-tuning for the downstream (see related work).
Considering our goal to design cross-architecture distillation, the general distillation approach has the conceptual benefit that the teacher model is not needed for fine-tuning. Thus, the student model capable of tackling downstream tasks without supervision of the large teacher. This has the benefit that does not need to carry around the BERT model for adaption every downstream task. Above, we have introduced the ability to emit per-token representations with lightweight (bidirectional) CMOW/CBOW-HYyle teacher signal during pretraining together with a masked language modeling objective. With other words, this enables us perform cross-architecture distillation with matrix embeddings.
embedding models. This enables now
BERT's
We consider three variants of cross-architecture distillation in our experiments: a) When using general distillation, depicted in Figure [ BERT acts as teacher during pretraining and the model is fine-tuned to downstream tasks on its own b) For task-specific distillation, BERT acts as teacher during fine-tuning as shown in Figure 2 For this case, we have the option of either bl) starting with pretrained embeddings (from general distillation, i. e.. the a) variant), or b2) starting from scratch with randomly initialized embeddings.
2.3 TWO-SEQUENCE ENCODING WITH MATRIX EMBEDDINGS
When fine-tuning our matrix embeddings downstream tasks, we can deviate from BERT's input processing, even if BERT is used as a teacher. This is because the distillation loss is computed per sentence (pair) and not per token. The input processing of BERT encodes two sequences by joining them into one sequence. For example, in a natural language inferencing task, there is a sentence A that potentially entails sentence B, which is encoded as one sequence using special separator token.
4

---

Under review as a conference paper at ICLR 2022
This encoding scheme is less useful matrix embeddings without any attention component since the order-aware matrix multiplications would blend the representation of the two sequences.
To develop appropriate two-sequence encoding scheme for matrix embeddings, take inspiration from the pre-transformer cra, e. g.g. Mou et al. (016)), and from SentenceBERT (Reimers & Gurevych 2019). The key idea is to encode two sentences A and B separately before combining them. As combination operation, we use the absolute elementwise difference and concatenate it to the representations of and B, which we denote as DiftCat:
h (DDiiCa) (() || [ (B) h
CMOW Forward
CMOW Forward [A-B]
CMOW Forward (B)
Backward
Backward
Backward
(A)
(A)
|A-B]
(B)
Classification Head
CBOW (A)
CBOW JA BI
CBOW (B)
Contradict
Entail
Neutral
CMOW Forward (A)
CMOW Backward (A)
CMOW Forward (B)
CMOW Backward (B)
CBOW
CBOW
[CLS]
Sentence A
[SEP]
Sentence B
[EOS]
Teacher Signal
Figure 2: Seperate encoding (DiffCat) for sequence pairs using a Bidirectional CMOW/CBOW- Hybrid model during fine-tuning, optionally, with task-specific distillation with a BERT teacher.
We illustrate this separate encoding scheme during task-specific distillation in Figure 2 The rationale for using concatenation of both sequence representations along with their difference is that we add a component for the similarity of the two sequence representations, without loss of expressive power.
EXPERIMENTAL PROCEDURE
The experimental procedure is divided in pretraining on unlabeled text and fine-tuning the downstream tasks. We provide the details for these two stages and close the section by outlining the downstream tasks and evaluation measures.
3.1 PRETRAINING AND GENERAL DISTILLATION
In the pretraining stage, shown in Figure I we train our proposed bidirectional CMOW/CBOW. Hybrid model with masked language modeling objective (MLM) (Devlin et al.) 2019) on large amounts of unlabeled text. The MLM objective is to predict left-out words from their context. We put equal weights on the MLM objective and the teacher signal from BERT (a 0.5). As suggested by] Liu et al. (2019) and Sanh et al. (2020a), we do not use the next-sentence prediction peteseetnnceeerreiitiiioon objective of BERT, but only the MLM objective.
As datasets for pretraining, we use a combination of English Wikipedia and Toronto Books (Zhu al. al., 2015), as used original BERT. To reduce the environmental footprint of our experiments, have only pretrained single bidircctional CMOW/CBOW-HYE Horc--bbo---iidd model with BERT-basc teacher on the full unlabeled training data, after pre-experiments on 10% of the training data showed that the selected SMOW/CBOW-Hyyre with distillation exceeded the performance of the baseline.
We use matrix embeddings of size 20 x 20 = 400 (d = 20) for both the CMOW directions (forward and backward) and vector embeddings of size dvec 400. We use BERT's tokenizer and its vocabulary for both, the teacher and student. The BERT tokenizer relies primarily on the WordPiece algorithm (Wu al. 2016), which yields a high coverage while maintaining small vocabulary.
5

---

Under review as a conference paper at ICLR 2022
3.2 FINE-TUNING AND TASK-SPECIFIC DISTILLATION
In the fine-tuning stage, as shown in Figure 2 the pretrained model is adapted for each downstream task individually. The training objective for fine-tuning is either cross-entropy with the ground truth (in general distillation) or mixture of the ground truth loss and cross-entrope with respect to the teacher's $ logits (in task-specific distillation). Again we put equal weight on ground truth and teacher signal along (a = 0.5) with unit temperature. To facilitate distillation regression tasks, we follow Raffel et al. 2020) and cast STS-B from regression to classification by binning the scores into intervals of 0.2. To encode the inputs for two-sequence tasks, we use a sequential encoding similar to BERT and the proposed DiffCat encoding (see Section 2.3
For task-specific distillation, we employ uncased BERT-base mode from the Huggingface repository that has already been fine-tuned for each task of the GLUE benchmark. We have fine-tuned the BERT model ourselves Huggingface model led to subpar results. We use the same fine-tuned BERT model as a teacher for all reported results with task-specific distillation.
the tasks STS-B, where applied binning, and MNLI, since the
We seek a fair comparison between unidirectional CMOW/CBOW baseline model and our bidirectional model. As such, we allow both models equally benefit from the BERT's teacher signal during fine-tuning. For this comparison, we use random initialization for both models because the Mai al. (2019)'s pretrained embeddings would come with different vocabulary that covers only 53% of the one of BERT. Throughout the other experiments, we initialize our bidirectional CMOW/CBOW-Hy Hybrid with the pretrained embeddings from general distillation, while we isolate the effect of task-specific distillation in a dedicated experiment.
For hyperparameter optimization, we tune learning rates in the range [10-3, 10-0) In total, we have conducted 306 training and evaluation runs for hyperparameter optimization of each GLUE task. To determine the best model, we use each task's evaluation measure on the development set. We run each model for 20 epochs with early stopping (5 epochs patience). We select appropriate batch sizes on the basis of preliminary experiments and training data sizes. For a more detailed discussion, we refer to Appendix B The hyperparameters of the persfrrrforming models are reported in Appendix C
We have further experimented with data augmentation and using exclusively the teacher signal during task-specific distillation (a 1). In some tasks, could further increase the results by small margin, but found no consistent improvement. These results are also reported in Appendix
3.3 DOWNSTREAM TASKS AND MEASURES
We use the GLUE benchmark (Wang et al. 2018) to evaluate our models. The GLUE benchmark consists of nine tasks for English language comprehension (Wang al., 2018). These tasks comprise natural language inference (MNLI-m, QNLI, WNLI, RTE), sentence similarity (QQP, STS-B, MRPC), linguistic acceptibility (CoLA), and sentiment analysis (SST-2). All tasks are based pairs of sentences except for CoL.A and SST-2, which are single-sentence tasks. The GLUE benchmark explicitly encourages using different fine-tuning strategies for different tasks. For our evaluation, we use the GLUE development set along with its task-specific measures. As such, the performance on all four NLI tasks as well as SST-2 is measured with accuracy. CoLA is evaluated by Matthews correlation coefficient. Similarity tasks are measured by the average of Pearson and Spearman correlation for the STS-B task, and as the average of accuracy and F -scre for MRPC and QQP.
4 RESULTS
We present the results along the design choices introduced in Section 2 namely bidirection, cross- architecture distillation approaches, and two-sequence encoding scheme. Finally, we compare our best embedding methods with ELMo and BERT distillates from the literature. For brevity, we focus on reporting key results. The detailed results can be found in Appendix C
Bidirectional CMOW/CBOW-Hyyree versus Baseline To isolate the effect of the bidirectional component, we compare unidirectional (MOW/CBOW-Hybrid with bidirectional CMOW/CBOW-
httta:///ogginffacccco//eetttttt

---

Under review as a conference paper at ICLR 2022
Hybrid under equal conditions. We train both variants from scratch for the downstream tasks, while using a BERT's teacher signal. embeddings with the proposed bidirectional Hybrid embeddings. Bidirection helps on the tasks MNLI, MRPC, QNLI, SST-2, STS-B, B,, and WNLL On the other tasks, the difference is marginal. We have average improvement of 1 1% of the bidirectional model over the unidirectional model across all tasks of the GLUE benchmark.
the results of comparing unidirectional Hybrid
Table I: Comparison of bidirectional CMOW/CBO--Hyyrrd versus unidirectional and bag-of-words baselines under task speciiic distillation with DifiCat encoding and MLP classifier.
Model Type
Score
CoLA
MNLI-m
MRPC
QNLI
QQP
RTE
SSI-2
STS-B
WNLI
Hybrid, rand. init.
62,5
13.1
62.5
74,3
71.5
86.6
58.1
83.1
58.6
56.3
Bidirectional Hybrid, rand. init.
63.2
13.0
75.7
72.6
86.1
57.4
83.3
59.7
57.7
General Distillation vs. Task-specific Distillation Next, we compare general distillation with task- sask-specific distillation. As shown in Table using general distillation leads better results for five tasks (MNLI, MRPC, QQP, STS-B, and RTE) compared to task susk-specific distillation. For the other four tasks (CoL.A, QNLI, SST-2, and WNL.), task-specific distillation achieves higher scores. The average score of general distillation is higher than with task-specific distillation in both pretrained and randomly initialized cases.
Table 2: Comparison of task-specific general distillation using bidirectional CMOW/CBOW- Hybrid embeddings and MIP classifier. In 5 out of 9 tasks, general distillation performs best.
Distillation Type
Score
CoLAA
MNLI-m
MRPC
QNLI
QQP
KTE
SST-2
STS-B
WNLI
General
66.6
16.7
66.6
79.7
71.7
87.2
61.0
82.9
76.9
56.3
Task-specific, rand. init
63.2
13.0
63.3
75.7
72.6
86.1
5744
83.3
59.7
57.7
Task-specifie,, pretrained
23.3
75.0
72.0
86.3
82.9
57.7
DiffCat Encoding vs. Joint Encoding Now, we compare the two encoding schemes, whose results are presented in Table 3 The best scores were achieved with DiffCat encoding. Most remarkable is the improvement from 18.1 56.2 on the sentence similarity task STS-B when encoding the sentence pair input via DiffCat. The average improvement DiffCat encoding is 32% across the two-sentence GLUE tasks.
Table 3: Comparison of DiffCat encoding vs. joint BERT-like encoding. Both variants use pretrained bidirectional CMOW/COWW-Hybri embeddings with MLP under task-specific distillation. DiffCat encoding improves the results in all cases except for RTE with the largest margin on STS-B.
Two-Sentence Encoding
MNLI-m
MRPC
QNLI
QQP
RTE
STS-B
WNLI
BERT-like Encoding
47.4
71.6
60.0
79.5
57.8
18.1
56.3
DiffCat Encoding
62.6
74.5
68.6
85.7
56.3
56.2
56.3
Comparing bidirectional CMOW/CHOW-Hybid to the Literature Finally, Table Isoows the results of the berf-pnen bidirectional CMOW/CBOW- considered distillation methods. As described by Wasserblat et al. (2020), a model needs to capture context and linguistic structure to perform well on CoLA. We doubled the results for CoLA and SST-2 compared to the best previously reported cross- architercee distillation approaches by Wasserblat (et al. (2020). Our best models scored higher than ELMo (Peters et al., 2018) on the tasks MRPC, QNLI, QQP, RTE, and WNLI. We achieve higher scores than DistilBERT on RTE and WNLI.
variants using any of the three
Runtime Performance and Parameter Count To compare runtime, we generate 1,024 batches with 256 random sequences of length 64 and measure the time encode the sequences with gradient computation disabled. As shown in Table CMOW/CBOW --ybidddd is more than 3 times faster than the fastest competitor, DistilBERT, and only uses about half of its parameters. The inference speed of CMOW/CBOW Wob-d could be increased even further because the ((n) steps to encode a sequence of length can be parallelized to Oloo n) sequential steps, since matrix multiplication is associative.
7

---

Under review as a conference paper at ICLR 2022
Table 4: Comparison of best embedding based methods (in bold) with methods from the literature on the validation set of the GLUE benchmark
Method
Score
CoLA
MNLI-m
MRPC
QNLI
QQP
RTE
SSI-2
STS-B
WNLI
ELMo Poters al. 2018
68.7
68.6
71.1
86.2
53.4
91.5
70.4
56.3
DistBBETT Sama et 2020g
77.0
51.3
82.2
87.5
89.2
88.5
59.9
91.3
86,9
56.3
MohileBERT Sun et
2020
51.1
84.3
88.8
91.6
70.5
70.4
92.6
84.8
CBOW Waaserblat al.
2020
10.0
79.1
BiLSTM [Wasserbaa et 2020
10.0
80.7
Hybrid (Mar 6 2019
79.6
63.4
WordZrate [Phun et al. 2021
65.7
53.1
Bidi. Hybrid + MLP (ours)
68.0
23,3
66.6
80.9
72.6
87.2
61.0
84.0
76,9
59.2
Table 5: Number of parameters and inference time of the models. Inference time is measured as encoding speed without gradient computation NVIDIA 1000SXXM4400BB card
Parameters
Encoding speed (seniences secund)
ELMo
94M
1.1k
BERT- hase
109M
4.6k
DistiBBERT-buse
66M
9.2k
MobileBERT
25M
5.5k
Tiny BERT (4 layer)
14M
30.0k
Bidi. CMOW/CBOW Hlybri
37M
30.0k
DISCUSSION AND RELATED WORK
Key Results We have shown that BERT can be distilled into efficient matrix embedding models during pretraining by emitting intermediate representations. We have further introduced a bidirectional component and a separate two-sequence encoding scheme for CMOW-style models. We have observed that the general distillation approach, i. e., using the BERT teacher only during pretraining, leads to results that are oftentimes even better than those achieved with task-specific distillation. This is interesting result because all previous works on cross- arciteccue distillation relied on ssec-fpe distillation. Our proposed model offers an encoding speed at inference time that is three times faster than DistiIBERT and more than five times faster than MobileBERT.
Limitations Currently, matrix embeddings still fall behind other BERT distillates, most notably, MobileBERT, on many of the downstream tasks. In particular, detecting linguistic acceptability remains challenge for non-transformer methods, even though improve upon previous cross- architecture distillation approaches. So far, we have not analyzed the trade-off between embedding size and downstream performance. We rely general arguments for the benefits of increased dimensionality (Wieting & Kiela 2019).
Reflection w.r.t. Distillation and Size-reduction Approaches In general distillation, a PreLM is distilled into student model during pretraining. DistilBERT (Sanh et al. 2020a) is such general- purpose language model that has been distilled from BERT. Apart from masked language modeling and distillation objectives, the authors also introduce cosine loss term align the student's and teacher's 's hidden states (layer transfer). Furthermore, the student is initialized with selected layers of the teacher. MobileBERT (Sun et al. 2020) introduced a bottleneck BERT such that layers can be transferred to student models with smaller dimensions.
In task-specific distillation, the teacher signal is used during fine-tuning, Sun al. (2019) use layer-wise distillation objectives and initialize with teacher weights to train BERT students with fewer layers. TinyBERT (Jiao et al. 2020) applies knowledge distillation in both stages, pretraining and fine-tuning. TinyBERT (Jiao al. 2020 yields high throughput rates comparable to CMOW/CBOW. Hybrid. Yet, Tiny BERT requires keeping the teacher model for fine-tuning. We have shown that CMOW/CBOW-Hyyrri is even better with general distillation only than under task-specific distillation. Tiny BERT also augments the training data, which we have also considered but found no consistent improvement. Ture et al. (2019) analyzes the interaction between pre-training and fine-tuning with BERT models and find that pretrained distillation works well, which agrees with our findings on the importance of pretraining with CMOW-style models. LadaBERT (Mao al. 2020
8

---

Under review as a conference paper at ICLR 2022
combines knowledge distillation with pruning and matrix factorization. Other approaches consider distillation in multi-lingual (Tsai et al. 2019) or multi-task settings (Yang et al. 2019p.
The works described above assume that the teacher and student share the same architecture. However, the student model does not need to have the same architecture as the teacher, what we then call crosstrrcite distillation. Wasserblat et al. (2020) use simple feed forward network with CBOW embeddings and a bidirectional LSTM model as students. Both models perform well in several downstream tasks. Tang et al. (2019) explore distilling BERT into a single layer BiLSTM without using any additional training data or modifications to the teacher architecture. Their distillation-baa approach yields improvements compared
plain BiLSTM without teacher signal: about 4 points on all reported tasks (QQP, MNLI, and SST-2). This has motivated us to investigate whether even simpler models can be used as students of a BERT teacher.
Other techniques for reducing the size of a model are pruning and quantization. Pruning approaches such as [Snnh et al. (2020b) reduce the number of parameters. Still, the resulting smaller models use the same architecture as their larger counterparts and, thus, pruning does not necessarily improve inference speed. Quantization a common post-processing step to reduce model size by decreasing
the floating point precision of the weights (Gupta et al. 2015 Wu et al.] (020). Pruning and quantization can be applied in conjunction with knowledge distillation (Sanh et al., 2020b; Sun
al. 2020). Aside from techniques for reducing the model size, there is also a tremendous effort to improve the efficiency of the transformer architecture in the first place TTy et al. 2020
The literature focuses reducing the size of PreLMs via distillation, pruning, and quantization. Specialized techniques depend on teacher and student sharing the same architecture, and thus, are not applicable for cros-archhitcctue distillation. So far, only few recent works consider distilling PreLMs into other architectures like LSTMs and CBOW models. In this work, we show that cross-architecture distillation with order-aware embedding models as students outperform previous archttrrch distillation approaches and achieve scores comparable to ELMo, while using less computational resources. Finally, all previous arssitercie distillation approaches are task-specific, while we could show that general distillation can lead to even higher scores than task-specific distillation.
CONCLUSION AND FUTURE WORK
We have introduced three extensions to the CMOW/BBOW-Hybrde model: a bidirectional component, separate two-sequence encoding scheme, and the ability to emit per-token representations. These per-token representations allow us to distill BERT into SMOW/CBOW-Hybre pretraining with a masked language modeling objective. Our results show that separate encoding scheme improves the performance of CMOW/CBOW-Hybhn more than 30%, while bidirection improves the performance by 1% compared the unidirectional model. Furthermore, we have shown that general distillation seems to be sufficient and task-specific distillation is not necessary for most GLUE tasks. In comparison with more expensive models from the literature, our embedding-basee approach achieves scores that match or exceed the scores of ELMo and are competitive to DistilBERT on QQP and RTE with only half of its parameters and thrice its encoding speed. While linguistic acceptibility remains a challenge for non-transformer models, our approach yields notably higher scores than previous cross-architecture distillation approaches.
already during
on two-sentence GLUE tasks by
In future work, could further improve the efficiency by applying pruning (Sanh al. 2020b) and/or quantization (Wu et al., 2020) techniques on the learned matrices to allow sparse matrix multiplication during encoding. Future work could explore what components would be necessary to improve the scores on particularly challenging downstream tasks such as detecting linguistic acceptability. This might include the introduction of a small attention module like the one of gMLP (Liu et al., 2021) to ZMOW/CBOW-Hybrit Since we have shown how to emit (and train) per-token representations with CMOW, an interesting direction of future work would be to explore whether CMOW representations are suited as order-aware embeddings in transformer models, which might alleviate the need for a dedicated position embedding. Similarly, the per-token representations enables tackling question answering tasks with CMOW-style models, in which span of the context ought to be predicted. Finally, matrix embeddings are technically not limited to natural language processing. Therefore, another interesting path of future work would be to explore the use of matrix embeddings in other domains where sequences of discrete elements need to be encoded, e. g., event sequences, gene sequences, or item sequences in recommender systems.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
There has been criticism about large-scale language models and the amplification of bias through the reduction of model size. This is particularly a problem when the model is not explainable. In proposed method, we distill large-scale pretrained language models into models that do not have nonlinearity (except for the final classification head). Since nonlinearities make explainabilty more difficult, we hope to contribute to more explainable language models with our work, even though explainability is not the main focus of the present work.
Another concern with large language models is their immense consumption of resources. In our work, we have only conducted the expensive pretraining step once on the full unlabeled training data. Only afterwards, we conduct multiple "cheap" runs to optimize hyperparameters for downstream tasks. Thus, our experiments themselves have been conducted resource efficient. Furthermore, this research contributes the development of simpler and more efficient models for natural language processing.
REPRODUCIBILITY STATEMENT
To ensure that our experiments are reproducible, we have tracked all hyperparameters throughout experimentation. We have described steps for data processing, i. e., tokenization (we use the BERT tokenizer) and binning (for STS-B) in the main part. The datasets from the GLUE benchmark are well-known to the community, because of which we have omitted a detailed description, except for the metrics that we have reported.
We make the code and pretrained embeddings available upon acceptance of the paper. For now, we provide the code for experiments supplementary material.
REFERENCES
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, and others. On the opportumities and risks of foundation models. ArXiv, abs/2108.07258, 2021. (ttpp/
7/arxiv. org/abs/ 0
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, and others. Language models are few-shot learners. ArXiv, abs/2005. 14165, 2020. URL
abs/2005.1 14165
Cristian Bucila, Rich Caruana, and Alexandru Niculesuu Mizil. Model compression. In KDD, pp. 535-541. ACM, 2006. URL/attps::/aao..rrgg 11..:....11111111 1150464
Guobin Chen, Wongun Choi, Xiang Yu, Tony X. Han, and Manmohan Chandraker. Learning efficient object detection models with knowledge distillation. NIPS, pp- 742-751, 2017. URL hntpps://proeeddnnss ctt.://dddiaeeeeeueps....o.oo/ee wttp:psscccppppppeee wttppaperr//1///aalll e22222236eee Abst html
Ronan Collobert and Jason Weston. A unified architecture for natural language processing: deep neural networks with multitask learning. In ICML,, volume 307 of ACM International Conference Proceeding Series, pp. 160-167. ACM, 2008
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT (1), pp. 4171- 4186. Association for Computational Linguistics, 2019. URLhttps: anthologyy/N19-111
URhttp::/w.....eeeee wwwwbb.bwwwwwww...oo...oo....... org/
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In /CML, volume 37 of JMLR Workshop and Conference Proceedings, pp. 1737-1746. JMLR.org. 2015. URL http:://ppooeeedgg...ttt mlr press/ v37/ 337//upta15..m 1tttttal5mmmmmmm..mm
Z. Hameed and B. Garcia-Zapiraal Sentiment classification using sayreel bilstm model. JEEE Access, 1773992-74001 2020. URL htt.:ttppe//do...///10010.000
2020
2988550
10

---

Under review as a conference paper at ICLR 2022
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. ArXiv, abs/1503.02531, 2015. http:::p////att /ttp:pppp////rrtt.ooor/..././// 02533
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory, Neural Comput., 978):1735-
1780, 1997. URL /ttAA//////////..000000000111//1
9.. 1735
Xiaoqi Jiao, Yichun Yin, Lifeng Shang. Xin Jiang. Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. TinyBERT: Distilling BERT for natural language understanding. In EMNLP (Findings), pp. 4163 4174. Association for Computational Linguistics, 2020. URL httpp:://do...or//1111 I8653/v1/2022 wind:oiiiiiiinnnniiiii
372
Hanxiao Liu, Zihang Dai, David R. So, and Quoc V. Le. Pay attention mips. ArXiv, abs/2105.080500 BRRpptts: org/ /ttp://bb/22110005533
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11699 1692, 2019. URL htttht:/:///xiii
abs//ab///// 11692
Florian Mai, Lukas Galke, and Ansgar Scherp. CBOW is not all you need: Combining CBOW with the compositional matrix space model. In ICLR (Poster). OpenReview.net, 2019. URL httpps://ppenrevier net. www.fforrmiiorrmee 2o
Yihuan Mao, Yujing Wang, Chufan Wu, Chen Zhang, Yang Wang, Quanlu Zhang, Yaming Yang, Yunhai Tong, and Jing Bai. Ladabert: Lightweight adaptation of BERT through hybrid model compression. In COLING, pp. 3225 3234. International Committee on Computational Linguistics, 2020. URL URL wtt.tttps//// pettps://doi..rgge
coling main. 287
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representa- tions in vector space. ArXiv, abs/1301.3781, 2013.
3781
Asit K. Mishra and Debbie Marr. Apprentice: Using knowledge distillation techniques to improve low-precision network accuracy. In ICLR (Poster). OpenReview.net, 2018. URL httpp::/
Lili Mou, Rui Men, Ge Li, Yan Xu, Lu Zhang, Rui Yan, and Zhi Jin. Natural language inference by tree-based convolution and heuristic matching. In ACL (2). The Association for Computer /olg/PP1...0 2022/
Linguisties, 2016. URL Wtlhttps: https/:/aaaaaaahhooaooe
Subhabrata Mukherjee and Ahmed Hassan Awadallah. Distilling bert into simple neural networks with unlabeled transfer data. ArXiv, abs/1910.01769, 2020. URL
1910. 01769
Matthew F. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In NAACL-HLT, pp. 2227-2237. Association for Computational Linguistics, 2018. URL hattip:://ddi.crg1111118555
n18-1200 1202
Gary Phua, Shaowei Lin, and Dario Poletti. Word2rate: training and evaluating multiple word embeddings as statistical transitions. ArXiv, abs/2104.08173, 2021. URL pattps://arxxe
org/abs/2104. 08173
Antonio Polino, Razvan Pascanu, and Dan Alistarh. Model compressioo via distillation and quamtiza- tion. In ICLR (Poster). OpenReview.net. 2018.
id=$1
Xx 1dd=S1Xo10bal 1
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. ArXiv, abs/19 Mtss1990.116833 2020. URL
abs/ / / 1910. 0. 0683
Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using siamese bert- networks. In EMNLP/JCNNP (1), pp. 2019. https::/t/// http:///.......11111///oo1
Association for Computational Linguistics,

---

Under review as a conference paper at ICLR 2022
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01108, 2020a. URL https
7/arxiv. otg/ htth/bbb/199900 01108
Victor Sanh, Thomas Wolf, and Alexander M. Rush. Movement pruning: Adaptive sparsity by fine-
tuning. In NeurlPS, 2020b. URL /tppst/pppcpceedaasssss neurips. ct..cc//ppp/ sc/pappe/2020/
hash/eaelaaapeee 78 wtt.ae/aaaaaaaaataaaaaaaaaaaaaae Abst html
Mike Schuster and Kuldip K. Paliwal. Bidirectional recureent neural networks. IEEE Trans. Signal
Process., 45((1)):673--2688 1997. URL/attp:://oi.or/// www.:///..............1 1110006669988669999
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NLP. In ACL((1), pp. 3645 3650. Association for Computational Linguistics, 2019.
URL
1355
Siqi Sun, Yu Cheng. Zhe Gan, and Jingjing Liu. Patient knowledge distillation for BERT model compression. In EMNLP/JJCNLP (1), pp. 4322 4331. Association for Computational Linguistics, 2019. URL /ttL 1ttp:///010011113////111111-1991
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang. and Denny Zhou. MobileBERT: a compact task-agnostic BERT for essource-limites devices. In ACL, pp. 2170. Associa- tion for Computational Linguisties, 2020. URL
ac main.nn.. 195
Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. Distilling task- specific knowledge from bert into simple neural networks. ArXiv, abs/ 1st/1993.11336 2019. URL
https://arxiv. org/
www.aas//////aaooooaooo.a/eora.a abs 903. 12136
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. ArXiv, abs/2009.06732, 2020. URL http:://erxxx..rrr//a//00.....
Henry Tsai, Jason Riesa, Melvin Johnson, Naveen Arivazhagan, Xin Li, and Amelia Archer. Small and practical BERT models for sequence labeling. In EMNLP//JCNLLP()) pp. 3630-3634. Association for Computational Linguistics, 2019. URL https: httpss//acclaanttoooo...oo 0rg1// 1374/
Julia Ture, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better: The impact of student initialization on knowledge distillation. Arxiv, abs/ 009662, 2019, URL
https://arxiv. org/ wabprr//bb//// abs/ 1908. 08962
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In BlackboxNLP@ EMNLP, pp. 353-355. Association for Computational Linguistics, 2018. URL
https://dos .og 18w.3...66611111
5446
Alex Wang, Yada Pruksachatkun. Nikita Nangia, Amanpreet Singh, Julian Michael, Fe- lix Hill, Omer Levy, and Samuel R. Bowman. SuperGI.UF: A stickier bench- mark for general-purpose language understanding systems. In NewrlPS, pp. 3261- 3275, 2019. URL lattps://proceedngss wwwpiiinnnnn.nuussee.eeemm www.ips httpa//sssseuppppppppaa////////e Abst rabs html
Moshe Wasserblat, Oren Pereg, and Peter Izsak. Exploring the boundaries of low-resource BERT distillation. In Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing, pp. 35-40. Association for Computational Linguistics, November 2020. URL https //aclanthology wwwp:hoooooooorog2220e0/00 ort.oo///222/
sustainip-1.5
John Wieting and Douwe Kiela. No training required: Exploring random encoders for sentence classification. In ICLR (Poster). OpenReview.net, 2019.
Hao Wu, Patrick Judd, Xiaojie Zhang, Mikhail Isaev, and Paulius Micikevicius. Integer quantization for deep learning inference: Principles and empirical evaluation. ArXiv, abs/2004.09602, 2020. URL htRl:stts:://// org/
httgg/ab//ae abs 200020000022222.
12

---

Under review as a conference paper at ICLR 2022
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, and others. Google's neural machine translation system: Bridging the gap between human and machine translation. ArXiv, abs/1609.08144, 144, 2016.
08144
G. Xu, Y. Meng. X. Qiu, Z. Yu, and X. Wu. Sentiment analysis commentt texts based bilstm. IEEE Access, 7:51522-5153,, 2019. UKL.ttpps.//01..rrr/1111111/55.. 2019 29C9919
Ze Yang, Linjun Shou, Ming Gong, Wutao Lin, and Daxin Jiang. Model compression with multi-task knowledge distillation for web-scale question answering system. Arxiv, abs/1904.096366 2019.
URL httttsss//axxi..rrppp.. barrg//bbs/in 1904. 09636
Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In ICCV, pp. 19-27. IEEE Computer Society, 2015. URL https;
7/ieeexplore. ieee. org/ ///////ooomme
13

---

Under review as a conference paper at ICLR 2022
APPENDIX
A OVERVIEW OF ARCHITECTURAL CHOICES
Figure provides overview of the architectural choices explored in this paper. We pretrained BERT Devlin et al. 2019) as well as the embeddings from Mai et al. (2019) as teacher for general distillation. Additionally, pretrain model CMOW/CHOW-Hyyrid with extension of masked language model training and bidirection on the same English Wikipedia + Toronto Books dataset. These pretrained embeddings may serve as initialization for the downstream classification models. We also evaluate downstream classification models that have been initialized randomly. For the downstream classification models, we consider three types of embeddings along with four types of classifiers.
Embedding Model Downstread Classifler
Wordovec-style Pretraining (Mai 2019)
CMOW
CNN (only with para CMOW)
LITT
Pretrained Embeddings (sptional)
CMOWICBOW-tyyhri
Linear
Fine-Tuning
Language Model Pretraining
Dovinatram Tacks
CNOWICBOW-Hybrid
General Distiliation
(Devlin 2018)
Task-specific Distilation (optionat
Figure All considered for embeddings and downstream classifiers, pretraining and fine-tuning.
For training on the downstream task, we use once again BERT as teacher for task-specific distillation, while our experiments on general distillation only benefit from the initalization of the MLM-pretraing CMOW/CBOW-HYye
model. Throughout the main part of the paper, have reported scores with an MLP downstream classifier, which achieved the highest average scores. We have further experimented with a linear downstream classifier, LSTM, and 2D-CNN, which we briefly describe below.
LSTM We have further experimented with pooling the sequence of embeddings with an LSTM. In the past, BiLSTM models have been successfully used in sentiment analysis tasks Xu et al. (2019):
;
(2020). In an LSTM network, the information at hand is propagated Hameed & Barcia-Zapiran the forward direction. Thus, each state & depends on its predecessor t 1. BiLSTM are LSTM networks, in which the inputs are processed twice: once in the forward direction and once in the backward direction, generating a set of two outputs. In order to generate the output vectors, the output of single BiL.STM block is fed into an MLP, consisting of two consecutive linear layers with ReLU activation functions. Note that the BiLSTM operates on a sequence of token embeddings, instead of operating on pooled sentence embeddings like the other student models. We apply dropout of 0.5 after the first linear layer.
We also explore a 2D-CNN classifier that induces a bias for learning two-dimensional struc- tures within the (aggregated) embedding matrices. The CNN consists of one transposed convolution, which increases the matrix dimensions by a factor of four. Following that, we employ a block of three convolutional layers, the first one having single filter (or two, for hybrid variants) and a kernel size of four, with the remaining two layers having 3 (4) kernels with stride 2. To avoid distorting the input embeddings, no padding is applied. ReLU is used for all activation functions. We apply BatchNorm for regularization before the last convolutional layer's 'ss output is flattened and passed into linear layer, which produces the predictions. We add dropout of 0.4 before the last linear layer.
CNN
14

---

Under review as a conference paper at ICLR 2022
Table 6: Hyperparameter search space and optimization method
Hyperparameter
Range
Opt. method
General Distillation
Learning rate
{10-3 3
5 10
10 -.00--5-170 5- 50-10055 ,10
grid scarch grid search
Warmup steps Embedding dropout
{0, 500)
{0. 0.1]
grid search fixed
Hidden unit dropout Batch size
{0.2)
11,8,22,641128,255
manual
Task-specific Distillation (10 3, 10 10 100 5 10 10 10.0100..000 10
Learning
grid search
Embedding type
Hybrid, CMOW, CBOW random, pretrained true, false true, false
grid search
Embedding initialization
grid search
DiffCat
grid search
Bidirectional
grid search
Classifier
Linear Probe, MLP, CNN, BiLSTM
grid search
B DISCUSSION OF HYPERPARAMETERS AND LOss FUNCTIONS FOR DISTILLATION
We list hyperparameter search spaces along with their optimization methods in Table 6 For the experiments on data augmentation and using only soft loss, we keep the configurations of the best models (See Table
7 and tune the learning rate, again. We optimize over all six initial learning rates, namely {10 3 10 10-4, 10-4, 10-5, and 10 10-"). All initial learning rates decay linearly over the course of training. Note, we also experimented with using warmup steps versus no warmup for the learning rate schedule. As the warmup did not improve the results, we did not use it.
For the softmax temperature, we find that T = 1 is often used Hinton et al. (2015); Mao et al. (200)) Jiao et al. (2000;] Mishra & Marr ): Polino et al. (2018). Since higher temperature also flattens the curve over all predictions, it could add too much noise and it is therefore better to use smaller temperature Chen et al. (207)). Setting the weight a 1 corresponds to only using hard loss and 0 only using soft loss. Since do not want to discard any information stemming from the hard loss, we do not follow the approach of Wasserblat et al. who only use the soft loss Wasserblat al. (2020) but instead, we employ a vanilla knowledge distillation approach following Hinton
al. Hinton et al. 2015).
Hinton al. (0155) state, that using cross-entropy loss the softmax-tem boftmax-temperature with large temperature, for example T = 20, corresponds to only using the Mean Square Error (MSE) loss on the raw student and teacher logits. Therefore it is also common use this loss for the soft distillation loss Tang et al. (2019); ; Wasserblat et al. (2020): Mukherjee & Awadal lah (202)). While Tang al. Tang al. (2019 used the weighted hard cross-entropy loss in the overall loss calculation, Wasserblat et al. and Mukherjee et al. only used the soft loss Tang et al. (2019); Wasserblat et al.) (2020); Mukherjee Awadallah (2020). disadvantage of MSE loss, that every error has huge effect on the overall loss, since it is squared. Another point is, that Hinton et al. found it beneficial to use small temperature values if the teacher way bigger than the student Hinton et al. (2015). Since using MSE loss corresponds to using big T values, this loss does not apply to small students for lower bound knowledge distillation, but with cross-entropy loss, we still have the possibility to achieve the behavior of the MSE loss by setting the value of T to big value.
of using
C DETAILED RESULTS
In the following, we provide detailed results for task-specific distillation including the different downstream classifiers, unidirectional SMOW/CBOW-Hyyrtt and joint two-sequence encoding. The best performing model per task are marked bold. We abbreviate CMOW/CBOW-Hyynn as 'Hybrid'.
15

---

Under review as a conference paper at ICLR 2022
For the unidirectional baseline model CMOW/CBOW-Hyrrln we initialize with pretrained embed- dings provided by Mai et al. (2019 which cover 54% of BERT's vocabulary. As initialization for the newly developed bidirectional HlOW/CBOW-Hyyoe models, we
own pretrained model
obtained by general distillation with BERT, as described in Section 3
Table 7 summarizes the best performing models along with their hyperparameter configurations for each task. Note that we have chosen to use an MLP downstream classifier for the results reported in the main part of this work. Using an MLP downstream classifier has led the highest average scores across all GLUE tasks.
In Table & report an extended version of the comparison with the literature. Here, also include our BERT- base teacher model, aswell as TinyBERT Jiao et al. (2020) and Tang et al. (2019)'s distilled BiLSTM. Note that TinyBERT and BiLSTM are not fully comparable, because those numbers are reported on the official GLUE test set, while we have used the validation set for our experiments.
In Table 9 we report the results for all downstream classifiers without the DiffCat aggregation but with a sequential BERT-like two-sequence encoding. It is interesting to see that the CMOW-only variant with 2D-CNN classifier leads to best scores on sentiment analysis task SST-2, Note that all CMOW variants reported in this table are unidirectional and use task-specific distillation.
In Table 10 we report the results for all downstream classifiers with DiffCat two-sequence encoding. Here we observe, that pretrained CBOW with an MLP classifier leads to the best results on sentence similarity (STS-B). Again, all CMOW variants reported in this table are unidirectional and use task-specific distillation.
In Table TT| we report the results for bidirectional models with DiffCat two-sequence encoding.
From all tables combined, we see that Bidirectional CMOW/CBOW-Hybre model leads to the highest scores on average, even though, on individual tasks, some other variations of the approach lead to higher scores. Thus, we regard bidrectional CMOW/CBOW-Hybri scores we have reported in the main paper, while isolating the effect of the individual components (bidirection, DiffCat encoding, distillation strategies).
our primary model, whose
We list the number of parameters in Tables 12 and [3] While the absolute numbers might seem high, it is important to note that have also counted the parameters of the embeddings. As show in the tables, the number of parameters in the classification models is much lower.
We have performed further experiments with the best performing model for each task: data augmen- lation and using only soft loss.
Using Only Soft Loss We study the influence of the alpha value used in the loss function, based on the best results obtained with the initial 0.5. The goal is to investigate whether using only soft loss, i. e., setting a = 0.0 leads different results. As Table shows, using only soft loss improves only the MRPC task by small margin.
Data Augmentation We conduct a further study with data augmentation as in TinyBERT (Jiao al. 2020). We employ their technique of replacing words by similar word embeddings and nearest predictions from BERT to augment the GLUE training datasets. The results are shown in Table 8 We find that the effect of data augmentation small. An improvement was only observed on SST-2 (+1.2 points) and STS-B (+3.6).
Downloaded from Zenodo: http://///aecooco/////////////// Bww.:/444KK_XX_XXXXX__ XJ__xXXXJUU
16

---

Under review as a conference paper at ICLR 2022
Task
Score
Classifier
Emb. type
Emb. initialization
DiffCat
Bidirectional
Learning
CoLAA
CMOWCCOOW Hyhrid
pretrained
trae
1.0.44
MNLI-m
63.3
MLP
CMOW/CBOW-Hylue
not pretrained
trae
tr00
1.0E-4
MRPC
78.2
MLP
CBOW
pretrained
true
false
1.E:33
QNLI
72.6
MLP
CMOW/CBOW Iyyrid
pretrained not pretrained
Irue
5.0E:5
QQP
86.6
ML.P
NOOWCBOW-Hyyheee
Inse
falke
1.0E.4
CMOW/CBOW Hyybid
pretrained not pretrained
SST-2
86.8
CNN
CMOW
false
false
5.0E-4
66.0
CBOW
pretrained
trae
false
1.0E.4
WNLI
69.0
CNN
CMOW
pretrained
false
false
1.0E-5
Table 7: Hyperparameter configurations for best-performing models by GLUE task
Table 8: Scores the GLUE development set. Our best performing general distillation and task- specific distillation models are highlighted in bold font per task. References indicate sources of scores. The symbol indicates numbers the official GLUE test set. CMOW/CROW-HYybrt is abbreviated as 'Hybrid'.
Score
CoLA
MNLI-m
MRPC
QNLI
QQP
RTE
SST-2
STS-B
WNLI
large-scale pre-srained language models
ELMo Sanh et al 2020a
68.7
44.1
68.6
76.6
71.1
86.2
53.4
91.5
70.4
56,3
BERT-Base 20200
91.7
BERT huse our leacher minie
78.9
$7.9
84.2
84.6
91.4
89.7
67.9
91.7
88.0
54.9
Word2rate Hyhrid (Phun
general distillation buselines
DistiBERT MobileBERT
89.2
59.9
Sun al 2020
$1.1
84.3
88.8
91.6
70.5
70.4
92.6
84.8
fask-specific distillation baselines
TTnyBEKT
Jia et al.
2020
54.0
84.5
90.6
91.1
88.0
70.4
93.0
90.1
*Bil STM
2019
73.0
78.2
90.7
CBOW-FFN Wasserbiat et
2020
10.0
79.1
BILSTM
geneval distiflation (owrs)
Bidi. Hybrid Linear
65.1
15.0
63.6
80.9
70.7
84.3
56.7
84.0
71.1
59.2
Bidi. Hybrid ML.P
66.6
16.7
66.6
79.7
71.7
87.2
61.0
82.9
76.9
56.3
task specific distillation (ours)
CMOW (rand, CMOW CNN (pretrained)
56.3
56.2
18.3
50.1
71.8
60.5
80.6
57.0
85.0
13.2
69.0
CBOW MLP (pretrained)
63.8
14.0
61.7
78.2
70.8
86.2
37.4
83.8
56.3
Hybrid MLP (rand. init.)
62.5
13.1
62.5
74.3
71.5
86.6
58.1
83,1
58.6
56.3
Bidi. Hybrid + MI. (rand. init.)
63.2
13.0
63.3
75.7
72.6
86.
57.4
83.3
59.7
57.7
Bidi. Hybrid MLP (pretrained)
64.6
23.3
72.0
86.3
$9.9
82.9
57.7
further experiments best-performing task-specific distiflation models
Only soft loss
64.0
19.9
62.3
78.7
72.4
68.5
56.3
86.6
62.4
69.0
Data augmentation
63.5
21.2
47.3
76.2
72.1
86.6
52.7
88.0
69.6
57.7
17

---

Under review as a conference paper at ICLR 2022
Table 9: Scores on the GLUE development set without DiffCat encoding
Task-Specific Distillation
Score
CoLA
MNLI-m
MRPC
QNLI
QQP
RTE
SST-2
STS-B
WNLI
task mask-specifie finetuning (ours)
Teacher BERT-base
78.9
57.9
84.2
84.6
91.4
89.7
67.9
91.7
88.0
$4.9
task task-speete specific distillation (ours) CBOW not pretrained
Linear probe
52.8
12.2
43.0
72.3
60.1
74.8
55.6
82.8
17.7
56.3
MIP
53.2
13.0
46.3
71.3
59.7
76.9
54.5
82.9
17.5
56.3
CNN
52.8
11.7
43.0
72.1
60.1
77.5
54.5
82.7
17.2
56.3
BiL.TM
52.1
10.9
44.9
70.8
59.8
78.1
54.5
81.3
12.3
56.3
lask-specific distillation (ours) CBOW pretrained
Linear probe
52.4
11.0
43.2
72.1
58.8
74.8
54.9
82.5
14.0
60.6
54.0
14.3
71.3
60.1
76.9
58.5
83.1
14.8
60.6
CNN
53.0
12.0
43.5
71.6
59.2
77.5
55.2
82.6
18.8
56.3
BiLSTM
50.8
0
44.9
71.3
59.4
78.0
54.0
81.0
12.0
$6.3
speciic distillation (ours) CMOW not pretrained
Linear probe
53.7
13.8
45.3
72.1
62.5
80.9
53.4
84.1
15.2
56.3
MLP
54.8
15.1
45.6
72.8
60.6
82.6
55.6
84.3
20.0
56.3
CNN
54.6
13.4
45.6
61.2
82.6
56.3
86.8
15.0
57.8
BiLSTM
53,2
16.7
44.9
72.1
64.8
80.6
54.2
82.9
7.9
54.9
task-specific distillation (ours)
CMOW pretrained
Linear probe
54.3
20.8
48.6
71.3
60.3
78.4
54.9
84.5
13.8
56.3
MLP
55.4
18.9
50.4
72.3
61.3
79.3
55.2
83.0
17.9
60.6
CNN
56.2
18.3
50.1
71.8
60.5
80.6
$7.0
85.0
13.2
69.0
BiLSTM
51.4
44.2
68.4
59.8
81.1
55.2
82.3
15.0
56.3
task-specific distillation (ours) CMOW/CBOW-H
pretrained
Linear probe
54.4
17.0
72.6
61.1
$3.4
84.5
15.1
$7.8
MLP
54.4
13.8
50.0
73.0
60.4
78.6
53.8
84.9
18.5
56.3
CNN
53.6
12.0
42.1
72.6
60.9
79.6
52.7
85.7
16.3
60.6
BiLSTM
52.4
O
43.2
72.1
61.2
80.0
57.4
83.0
18.1
56.3
saec-fpe distillation (ours)
CMOW/CBOW Hybrid pretrained
Linear probe
53,9
19.1
41.0
71.8
57.6
78.7
57.8
83.7
16.2
59.2
MLP
55.3
47.4
71.6
60.0
79.5
18.1
56.3
CNN
54.0
20.7
71.8
59.9
79.7
54.9
85.9
9.9
SILSTM
53.7
17.0
40.6
71.8
61.3
80.3
57.4
82.5
14.0
59.2
18

---

Under review as a conference paper at ICLR 2022
Table 10: Scores on the GLUE development set with DiffCat two-sequence encoding
Task- Task-Specific Distillation
Score
CoLA
MNLI-m
MRPC QNLI
QQP
RTE
SST-2
STS-B
WNLI
task-specific finetuning (ours)
Teacher BERT-base
78.9
57.9
84.2
84.6
91.4
89.7
67.9
91.7
88.0
54.9
lask-specific distillation (ours) CBOW not pretrained
Linear probe
53.8
11.5
46.6
72.8
62.2
76.7
52.7
83.5
22.0
56.3
MLP
61.0
14.3
57.8
77.2
70.3
86.0
56.7
82.3
47.0
57.7
CNN
53.8
11.2
51.5
75.0
65.8
81.3
82.3
7.2
56.3
BiLSTM
48.4
11.5
31.8
68.3
66.8
63.2
56.7
83.5
1.5
56.3
task-specific distillation (ours) CBOW pretrained
Linear prohe
56.3
9.0
47.1
72.8
64.8
77.1
53.4
82.5
43.4
56.3
MLP
63.8
14.0
61.7
78.2
70.8
86,2
57.4
83.8
66.0
56.3
CNN
53.7
10.9
55.0
73.8
66.2
82.1
53.1
82.2
3.8
56.3
BiLSTM
47.7
32.7
68.4
69.6
63.2
55.6
82.5
1.3
56.3
lask-specific distillation (ours) CMOW pretruined
Linear probe
55.1
10.9
54.3
71.8
62.7
80.9
56.0
85.2
17.6
MLP
63.2
14.2
61.9
75.5
72.4
86.3
55.2
83.7
62.7
56.3
CNN
55,4
12.4
45.3
72.3
61.5
82.6
57.4
84.3
26.1
56.3
BiLSTM
47.5
0
31.8
70.3
49.5
81.0
55.6
83,4
0
56.3
task-specific distillation (ours)
CMOW pretrained
Linear probe
56.3
22.4
48.4
72.5
61.3
81.9
54.5
83.9
24.2
57.7
MLP
61.2
20.9
73.8
64.6
85.9
54.9
84.4
49.4
56.3
CNN
53.4
18.5
40.6
71.8
58.2
68.3
54.9
85.4
26.9
56.3
BiLSTM
49.7
0
32.7
68.3
67.2
82.9
57.0
82.5
0
56.3
lask-specific distillation (ours) Hybrid not pretruined
Linear probe
51.7
11.2
39.0
71.1
49.5
81.8
56.0
85,2
14.3
57.7
MLP
62.5
13.1
62.5
74.3
71.5
86.6
58.1
83.1
58.6
56.3
CNN
52.8
11.9
45.3
71.6
61.4
84.8
55.2
85.4
2.9
56.3
BiLSTM
50.9
42.6
70.1
60.3
79.3
56.0
84.4
9.3
56.3
task-specific distillation (ours) Hybrid pretrained
Linear probe probe
54.0
19.6
45.7
71.3
63.4
80,9
54.2
84.1
11.7
54.9
MLP
62.7
20.9
62.6
74.5
68.6
85.7
56.3
83.1
56.2
56.3
CNN
57.9
19.6
37.6
75.7
62.0
85.4
54.9
82.3
48.5
54.9
BiLSTM
52.1
0
48.0
68.4
71.9
85.3
56.7
82.5
0
56.3
Table 11: Scores on the GLUE development with DiffCat encoding and the bidirectional CMOW/CBOW-HYbri model
Task Bisk-Specific Distillation
Score
CoLA
MNLI-m MRPC QNLI
QQP
RTE
SST-2
STS-B
WNLI
tpec-specific finetuning (ours)
Teacher BERT-base
78.9
57.9
84.2
84.6
91.4
89.7
67.9
91.7
88.0
54.9
spec-sp distillation (ours) Bidirectional Hybrid, pretrained
Linear probe
53.5
11.6
71.6
64.3
82.5
56.3
85.0
56.3
MLP
63.2
13.0
63.3
75.7
72.6
86.1
57.4
83.3
59.7
57.7
CNN
52.7
14.5
37.3
71.3
60.8
86.4
55.2
85.8
6.6
56.3
lask-specific distillation (ours) Bidirectional Hybrid, pretruined
Linear probe
55.5
18.1
42,4
72.1
64.9
81.2
56.7
85,2
22.5
56.3
MLP
64.6
23.3
61.8
75.0
72.0
86.3
59,9
82.9
62.9
57.7
CNN
55.1
20.5
39.3
73.8
61.3
85.9
56.3
85.5
15.9
57.7
19

---

Under review as a conference paper at ICLR 2022
Table 12: Number of parameters without DiffCat encoding
CoLA, MRPC, QNLI QQP. SST-2, RTE, WNLI
MNLI
STS-B
spsk-specific distillation (ours) CBOW
Linear probe
47,861,634
47,862,419
47,876.549
only classifier
3,138
3,923
18,053
MLP
48,6 48,647,498
48,648,499
48, 48,666,517
only classifier
789,002
790,003
808,021
CNN
47,862,708
47,864,737
47,901,22 ,559
only classifier
4,212
6,241
42,763
BiLSTM
53,704,002
53,705,027
53,723,477
only classifier
5,845,506
5,846,531
5,8649981
spsk-sp distillation (ours) CMOW
Linear probe
23,932,386
23,933,171
23,947,301
only classifier
3,138
3,923
18,053
MLP
24,718,250
24.719,251
24.737.269
only classifier
789,002
790,003
808,021
CNN
23,933,460
23,935,489
23,972.011
only classifier
4,212
6,241
42,763
BiLSTM
24,853,978
24,854,371
35,022,869
only classifier
924,730
925,123
110,936,21
task sesk--pecificc distillation (ours) Hybrid
Linear probe
24,420,802
24,421,603
24,436,021
only classifier
3,202
4,003
18,421
MLP
25,222,602
25,223,603
25,241,621
only classifier
805,002
806,003
824,021
CNN
24,420,558
24.421.855
24,445,201
only classifier
2,958
4,255
27,601
BiLSTM
30,328,642
30,329,667
30,348.117
only classifier
5,911,042
5,912,067
5,930,517
20

---

Under review as a conference paper at ICLR 2022
Table 13: Number of parameters with DiffCat two-sequence encoding
CoLA, MRPC, QNLI, QQP, SST-2, RTE, WNLI
MNLI
STS-B
task-specific distillation (ours) CBOW
Linear probe
47,867,906
47,870,259
47,912,613
only classifier
9,410
11,763
54,117
MLP
50,215,498
50.216,499
50,234,517
only classifier
2,357,002
2,358,003
2,376,021
CNN
47, 47,865,932
47,869,313
47,930.171
only claxsifier
7,436
10,817
71,675
BiLSTM
147,4 147,477,458
147, 111,179.811
147,522,
only classifier
99,618,922
99,621,315
99,663,669
task tasc-specific distillation (ours) CMOW
Linear probe
23,938,658
23,941,011
23,983,365
only claxsifier
9,410
11,763
54,117
MI.P
26,286,250
26,287,251
26,305.269
only classifier
2,357,002
2,358,003
2,376,021
CNN
23,936.684
23,940,065
24,000,923
only classifier
7,436
10,817
71,675
BiLSTM
123,548.2U1
123,550,563
123,592,917
only claxsifier
99,618,962
99,621,315
99,663,669
task- saek-specific distillation (ours) Hybrid
Linear probe
24,427,202
24,429,603
24,472.821
only classifier
9,602
12,003
55,221
MLP
26,822,602
26,823,603
26,841,621
only classifier
2,405,002
2,406,003
2,424.021
CNN
24,424,990
24,427,583
24,474,257
only classifier
7,390
9,983
56,657
BiL
128, 143,202
128, 145,603
188,888,821
only classifier
103,725.602
103.728,003
103,771,221
speciffc distillation (ours) Hybrid bidirectional
Linear probe
36,640,80 802
46,644,403
36,709,221
only claxsifier
14402
18003
82,821
MLP
40,231,402
40,232,403
4,025,0421
only classifier
3,605,0 ,002
0.606,003
3,624,021
CNN
36,638, 164
36,641,729
36,705,899
only classifier
11,764
15,329
79,499
BiLSTM
451,437,602
451,528, 821
only classifier
414,811,202
414,902,421
21


Reviews:
[{"review_id": "xOwQ6qJCfDJ", "reviewer": "Reviewer_AXE2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I am curious how it performs with a single CBOW or a single CMOW, not a hybrid one. \n\nDid you try other sizes of matrix embeddings of size? How does it perform? Should the CBOW and CMOW embeddings have the same dimension size? What are their individual functions for CBOW and CMOW? Are they complementary or just redundant? I know CMOW is from another paper, but as a reader, we might want to read a self-contained paper.\n\n", "summary_of_the_paper": "The paper proposes to distill a BERT model to a smaller one,  a CMOW/CBOW-Hybrid. Its performance is better than ELMo, but worse than the typical distillation method using transformer-based students like DistillBERT. The authors claim that CMOW/CBOW-Hybrid is much faster. The paper is clearly written.", "summary_of_the_review": "\nTo distill a better model to a weaker model, one can try different student models, either using shallower/narrower Transformer-based models or other models. If we could tolerate a big performance drop (as in this paper, the authors were satisfied that it exceeds ELMo), there are much more choices, of which CMOW/CBOW-Hybrid might be a good one. My concern is that using CMOW/CBOW-Hybrid is not that interesting, one might also try CBOW with convolutions, for example. It may help if the authors could provide more insights to choose CMOW/CBOW-Hybrid,  or why the investigation of  CMOW/CBOW-Hybrid is important, as we might have so many lightweight models as the student model.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "worx0ExiHWu", "reviewer": "Reviewer_M3tk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main contribution of this work mainly lies in the bidirectional CMOW and a two-stage knowledge distillation method. However, the technical contribution in bidirectional CMOW is quite limited, the related extension has been widely used in existing works, like bidirectional LSTM. Furthermore, the two-stage knowledge distillation method was originally proposed in TinyBERT, a well-known work on BERT distillation, which is directly used in this work. The proposed method has poor performances on several GLUE tasks, e.g., CoLA and MRPC, and many related baselines are not included, e.g. TinyBERT, BERT-PKD, MiniLM. The proposed method even does not have advantages on #parameters and inference time with comparison to 4-layer TinyBERT.      ", "summary_of_the_paper": "This paper presents a knowledge distillation method for BERT, where BERT-base acts as a teacher network and  CMOW is used as a student network. To get a relatively strong student network, the authors extend the vanilla CMOW to bidirectional CMOW. The authors also adopt a two-stage distillation method consisting of general distillation and task-specific distillation to further improve the performances. Finally, the authors evaluate the proposed method on GLUE dataset.     ", "summary_of_the_review": "The bidirectional extension of CMOW is straightforward, and the two-stage knowledge distillation method is originally  proposed by TinyBERT. The final performances of the proposed method performs worse than most baselines, e.g., TinyBERT, DistilBERT, MiniLM, BERT-PKD. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "T7m0J4s_zGl", "reviewer": "Reviewer_mHTJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The proposed idea is novel: the CMOW model can only be used in sentence-level tasks. The authors extend CMOW using the bidirectional variant to extract token-level representations, so now it can be used for the masked language modeling training objective.\n2. DiffCat is designed specifically for two-sequence tasks. It improves the performance by 32% compared to naive joint encoding.\n3. The proposed method can achieve comparable results to ELMo, while having a much higher encoding speed.\n\nWeaknesses:\n1. The evaluation of the proposed method is focused on the GLUE task, which is a collection of classification tasks. I encourage the authors to test the model on other tasks as well. For example, DistilBERT can also perform well on question answering datasets like SQuAD. It's interesting to see whether this shallow model can perform appropriately on more complex tasks.\n2. How can we trade-off between this fast inference speed and good performance? It seems that this model is very fast at inference speed, but the performance was compromised. It would be better if we are allowed to find a balance between speed and performance according to our demand. For example, TinyBERT provides a 4-layer version and a 6-layer version, so we can find a balance between the inference time and performance. In this paper, it seems that the author only provides 20x20 matrix embeddings for the CMOW model.", "summary_of_the_paper": "The authors of this paper try to distill large pretrained language models into a bidirectional CMOW/CBOW-Hybrid model. The proposed architecture was designed to output per-token representations, so as to perform knowledge distillation on the masked language modeling pretraining task. A two-sequence encoding scheme (DiffCat) was designed for the downstream task on sentence pairs. Empirical results show that the proposed method can achieve comparable results to ELMo while using only half of the parameters and providing three times faster inference speed.", "summary_of_the_review": "The paper presents a novel and useful idea: distill a complex pretraining model into a simple CMOW model. The initial result is promising while some limitations can be further explored. I think this paper is worth accepting because it can motivate the research on smaller distilled PreLMs.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a method for distilling pretrained models (such as BERT) into a different student architecture (CMOW), and extend the CMOW architecture with a bidirectional component.  On a couple of datasets, results are comparable to DistilBERT a previous baseline. This paper is nice, but can be stronger with more empirical experiments on non-GLUE tasks (TriviaQA, Natural Questions, SQUAD for example).  Furthermore, I agree with Reviewer M3tk that there are many empirical comparisons with baselines such as TinyBERT missing and the argument of not needing the teacher model to be super convincing.
Decision: Reject