SlowGuess/NAACL_2022_OCR huggingface dataset:
Marzieh S. Tahaei
marzieh.tahaei@huawei.com
Ella Charlaix
charlaixe@gmail.com
vahid/partovinia@huawei.com
Actuarial Science, University of Waterloo
ali.ghodsi@uwaterloo.com
Mehdi Rezagholizadeh
Mehdi.rezagholizadeh@huawei.com
The development of over-parameterized pretrained language models has made a significant contribution toward the success of natural language processing. While over-parameterization of these models is the key to their generalization power, it makes them unsuitable for deployment on low-capacity devices. We push the limits of state-of-the-art Transformer-based pre-trained language model compression using Kronecker decomposition. We present our KroneckerBERT, a compressed version of the  $\mathrm{BERT}_{\mathrm{BASE}}$  model obtained by compressing the embedding layer and the linear mappings in the multi-head attention, and the feed-forward network modules in the Transformer layers. Our KroneckerBERT is trained via a very efficient two-stage knowledge distillation scheme using far fewer data samples than state-of-the-art models like MobileBERT and TinyBERT. We evaluate the performance of KroneckerBERT on well-known NLP benchmarks. We show that our KroneckerBERT with compression factors of  $7.7\times$  and  $21\times$  outperforms state-of-the-art compression methods on the GLUE and SQuAD benchmarks. In particular, using only  $13\%$  of the teacher model parameters, it retain more than  $99\%$  of the accuracy on the majority of GLUE tasks.
In recent years, the emergence of Pre-trained Language Models (PLMs) has led to a significant breakthrough in Natural Language Processing (NLP). The introduction of Transformers and unsupervised pre-training on enormous unlabeled data are the two main factors that contribute to this success.
Transformer-based models (Devlin et al., 2018; Radford et al., 2019; Yang et al., 2019; Shoeybi et al., 2019) are powerful yet highly overparameterized. The enormous size of these models
does not meet the constraints imposed by edge devices on memory, latency, and energy consumption. Therefore there has been a growing interest in developing new methodologies and frameworks for the compression of these large PLMs. Similar to other deep learning models, the main directions for the compression of these models include low-bit quantization (Gong et al., 2014; Prato et al., 2019), network pruning (Han et al., 2015), matrix decomposition (Yu et al., 2017; Lioutas et al., 2020) and Knowledge distillation (KD) (Hinton et al., 2015). These methods are either used in isolation or in combination to improve compression-performance trade-off.
Recent works have been relatively successful in compressing Transformer-based PLMs to a certain degree (Sanh et al., 2019; Sun et al., 2019; Jiao et al., 2019; Sun et al., 2020; Xu et al., 2020; Wang et al., 2020; Kim et al., 2021); however, moderate and extreme compression of these models (compression factors  $>5$  and 10 respectively) is still quite challenging. In particular, several works (Mao et al., 2020; Zhao et al., 2019a, 2021) that have tried to go beyond the compression factor of 10, have done so at the expense of a significant drop in performance.
Following the classical assumption that matrices often follow a low-rank structure, low-rank decomposition methods have been used for compression of weight matrices in deep learning models (Yu et al., 2017; Swaminathan et al., 2020; Winata et al., 2019) and especially Transformer-based models (Noach and Goldberg, 2020; Mao et al., 2020). However, low-rank decomposition methods only exploit redundancies of the weight matrix in the horizontal and vertical dimensions and thus limit the flexibility of the compressed model. Kronecker decomposition on the other hand exploits redun
Human Language Technologies, pages 2116 - 2127
Figure 1: An example of Kronecker product of two 2 by 2 matrices.
dancies in predefined patches and hence allows for more flexibility in their representation. Recent works prove Kronecker product to be more effective in retaining accuracy after compression than SVD (Thakker et al., 2019).
This work proposes a novel framework that uses Kronecker decomposition for compression of Transformer-based PLMs and provides a very promising compression-performance trade-off for medium and high compression levels, with  $13\%$  and  $5\%$  of the original model parameters respectively. We use Kronecker decomposition for the compression of both Transformer layers and the embedding layer. For Transformer layers, the compression is achieved by representing every weight matrix both in the multi-head attention (MHA) and the feed-forward neural network (FFN) as a Kronecker product of two smaller matrices. We also propose a Kronecker decomposition for compression of the embedding layer. Previous works have tried different techniques to reduce the enormous memory consumption of this layer (Khrulkov et al., 2019; Li et al., 2018). Our Kronecker decomposition method can substantially reduce the amount of required memory while maintaining low computation.
Using Kronecker decomposition for large compression factors leads to a reduction in the model expressiveness. This is due to the nature of the Kronecker product and the fact that elements in this representation are tied together. To address this issue, we propose to distill knowledge from the intermediate layers of the original uncompressed network to the Kronecker network during training.
Training of the state-of-the art BERT compression models (Zhao et al., 2019a,b; Sun et al., 2020, 2019) involve an extensive training which requires vast computational resources. For example in (Sun et al., 2020), first a specially designed teacher, i.e IB-BERT $_{\text{LARGE}}$  is trained from scratch on the en
tire English wikipedia and Book Corpus. The student is then pretrained on the same corpus via KD while undergoing an additional progressive KD phase. Another example is TinyBERT(Jiao et al., 2019) which requires pretraining on the entire English Wikipedia and also uses extensive data augmentation  $(20\times)$  for fine-tuning on the downstream tasks. We show that our Kronecker BERT can out perform state-of-the-art with significantly less training requirements. More precisely, our Kronecker-BERT model undergoes a very light pretraining on only  $10\%$  of the English Wikipedia for 3 epochs followed by finetuning on the original downstream data.
Note that, while our evaluations in this work are limited to BERT, this proposed compression method can be directly used to compress other Transformer-based NLP models. The main contributions of this paper are as follows:
- Compression of the embedding layer using the Kronecker decomposition with very low computational overhead.
- Deploying the Kronecker decomposition for the compression of Transformer modules.
- Efficient training the compressed model via an intermediate-layer KD that uses only  $10\%$  of English Wikipedia in the pretraining stage.
- Evaluating the proposed framework for compression of  $\mathrm{BERT}_{\mathrm{BASE}}$  model on well-known NLP benchmarks
In this section, we first go through some of the most related works for BERT compression in the literature and then review the few works that have used Kronecker decomposition for compression of CNNs and RNNs.
2.1 Pre-trained Language Model Compression
In recent years, many model compression methods have been proposed to reduce the size of PLMs while maintaining their performance on different tasks. KD, which was first introduced by (Bucilua et al., 2006) and then later generalized by (Hinton et al., 2015), is a popular compression method where a small student network is trained to mimic the behavior of a larger teacher network. Recently, using KD for the compression of PLMs has gained a growing interest in the NLP community. BERTPKD (Sun et al., 2019), uses KD to transfer knowl
edge from the teacher's intermediate layers to the student in the fine-tuning stage. TinyBERT (Jiao et al., 2019) uses a two-step distillation method applied both at the pre-training and at the fine-tuning stage. MobileBERT (Sun et al., 2020) also uses an intermediate-layer knowledge distillation methodology, but the teacher and the student are designed by incorporating inverted-bottleneck structure. Authors in (Zhao et al., 2019a) use a mixed-vocabulary training method to train models with a smaller vocabulary. They combine this method with intermediate layer KD through shared projection matrices. In (Mao et al., 2020), the authors present LadaBERT, a lightweight model compression pipeline combining SVD-based matrix factorization with weight pruning while using KD for training to achieve a high compression factor.
2.2 Kronecker Decomposition
Kronecker products have previously been utilized for the compression of CNNs and small RNNs. Zhou and Wu 2015 was the first work that utilized Kronecker decomposition for NN compression. They used a summation of multiple Kronecker products to replace weight matrices in the fully connected and convolution layers in simple CNN architectures like AlexNet. Thakker et al., 2020 used Kronecker product for the compression of very small language models for deployment on IoT devices. To reduce the amount of performance drop after compression, they propose a hybrid approach where the weight matrix is decomposed into an upper part and lower part. The upper part remains un-factorized, and only the lower part is factorized using the Kronecker product. More recently, Thakker et al. 2020 tried to extend the previous work to non-IoT applications. Inspired by robust PCA, they add a sparse matrix to Kronecker product factorization and propose an algorithm for learning these two matrices together.
To the best of our knowledge, this work is the first attempt to compress Transformer-based language models using Kronecker decomposition. Unlike prior arts, we use a simple Kronecker product of two matrices for the representation of linear layers and uses KD framework to improve the performance.
In this section, we first introduce the background of Kronecker decomposition and then explain our
Figure 2: Illustration of our proposed method for the compression of the embedding layer. Left: conventional embedding stored in a lookup table. Right: Our proposed compression method where the original embedding matrix is represented as a Kronecker product of a matrix and a row vector. The matrix is stored in a lookup table to minimize computation overhead.
compression method in detail.
3.1 Kronecker Product
Kronecker product is an operation that is applied on two matrices resulting in a block matrix. Let  $\mathbf{A}$  be a matrix  $\in \mathbb{R}^{m_1\times n_1}$ , and let  $\mathbf{B}$  be a matrix  $\in \mathbb{R}^{m_2\times n_2}$ , then the Kronecker product of  $\mathbf{A}$  and  $\mathbf{B}$  denoted by  $\otimes$  is a block matrix, where each block  $(i,j)$  is obtained by multiplying the element  $\mathbf{A}_{i,j}$  by matrix  $\mathbf{B}$ . Therefore, the resulting matrix  $\mathbf{A}\otimes \mathbf{B}$  is  $\in \mathbb{R}^{m\times n}$  where  $m = m_{1}m_{2}$  and  $n = n_{1}n_{2}$ . Figure 1 illustrates the Kronecker product between two small matrices. See (Graham, 2018) for more detailed information on Kronecker products. Replacing matrix product with Kronecker product replaces the projection of the original linear space by a more constrained linear space in which the projection angle is defined by the core tensors, see Figure 5 in the appendix.
3.2 Kronecker Decomposition
Given a shape for  $\mathbf{A}$  and  $\mathbf{B}$ , i.e.  $(m_1, n_1, m_2, n_2)$ , any matrix  $\mathbf{W} \in \mathbb{R}^{m \times n}$ , can be approximated as a summation of Kronecker product of matrices  $\mathbf{A}_r \in \mathbb{R}^{m_1 \times n_1}$  and  $\mathbf{B}_r \in \mathbb{R}^{m_2 \times n_2}$ :
$$
\mathcal {W} \approx \sum_ {i = 1} ^ {I} \mathcal {A} _ {i} \otimes \mathcal {B} _ {i} \tag {1}
$$
we can obtain exact representation of  $\mathbf{W}$  by setting the number of Kronecker summations  $I$  equal to  $\min(m_1n_1, m_2n_2)$ . However, in order to achieve
compression, a much smaller value of  $I$  is often used. In fact prior arts show promising results using a single Kronecker product (Thakker et al., 2019, 2020). When decomposing a matrix  $\mathbf{W} \in \mathbb{R}^{m \times n}$ , as  $\mathbf{A} \otimes \mathbf{B}$ , there are different choices for the shapes of  $\mathbf{A}$  and  $\mathbf{B}$ . The dimensions of  $\mathbf{A}$  i.e  $m_1$  and  $n_1$  can be any factor of  $m$  and  $n$  respectively, the dimensions of  $\mathbf{B}$  will subsequently be equal to  $m_2 = m / m_1$  and  $n_2 = n / n_1$ .
3.2.1 The Nearest Kronecker Product
The nearest Kronecker problem is defined as finding matrices  $\mathbf{A}$  and  $\mathbf{B}$  that their Kronecker product best approximate a given  $\mathbf{W}$  (for a given shape of  $\mathbf{A}$  and  $\mathbf{B}$ ):
$$
\min  _ {\mathbf {A}, \mathbf {B}} \| \mathbf {W} - \mathbf {A} \otimes \mathbf {B} \| _ {F}. \tag {2}
$$
(Van Loan and Pitsianis, 1993) show that this problem can be solved using rank-1 SVD approximation of rearranged  $\mathbf{W}$ :
$$
\left. \min  _ {\mathbf {A}, \mathbf {B}} \left\| \mathcal {R} _ {n _ {1}, m _ {1}} (\mathbf {W}) - \mathcal {V} (\mathbf {A}) \mathcal {V} (\mathbf {B}) ^ {\top} \right\| _ {F}. \right. \tag {3}
$$
Here,  $\mathcal{V}$  is an operation that transforms a matrix to a vector (vectorizes) by stacking its columns and  $\mathcal{R}_{m_2,n_2}$  is a rearrangement operation that extracts patches of size  $m_{2} \times n_{2}$ , vectorizes the resulting patches and finally concatenates them together to form a matrix of size  $m_{2}n_{2} \times m_{1}n_{1}$ . The rearrangement operation turns the Kronecker product into a matrix of rank one while retaining the Frobenius norm making the minimizations in Eq.3 and Eq.2 equivalent. Hence, the rank-one SVD solution  $\mathbf{U}(:,\mathbf{1})\sigma \mathbf{V}(:,\mathbf{1})^T$  can be used to obtain the optimum  $\mathbf{A}$  and  $\mathbf{B}$  as:
$$
\mathbf {A} = \mathcal {V} _ {m _ {1}, n _ {1}} ^ {- 1} (\sqrt {\sigma} \mathbf {U} (:, 1)) \tag {4}
$$
$$
\mathbf {B} = \mathcal {V} _ {m _ {2}, n _ {2}} ^ {- 1} (\sqrt {\sigma} \mathbf {V} (:, 1)) \tag {5}
$$
Here,  $\mathcal{V}_{m_1,n_1}^{-1}(\mathbf{x})$  is an operation that transforms a vector  $\mathbf{x}$  to a matrix of size  $m_{1}\times n_{1}$  by dividing the vector to columns of size  $m_{1}$  and concatenating the resulting columns together. Similarly, rank-  $r$  SVD decomposition can be used to approximate summation of Kronecker products. We use this method for the initialization of Kronecker layers from the non-compressed model.
3.2.2 Relation to SVD
By choosing  $n_1 = 1$  and  $m_2 = 1$ ,  $\mathbf{A}$  becomes a column vector of size  $\in \mathbb{R}^{m \times 1}$  and  $\mathbf{B}$  becomes a
row vector of size  $\mathbb{R}^{1\times n}$ , then the Kronecker decomposition becomes equivalent to rank-1 SVD decomposition. Therefore rank-1 SVD is a special case of Kronecker product decomposition and rank- $r$  SVD is a special case of Kronecker product summation decomposition. This indicates that with Kronecker product one can achieve more flexibility than low rank decomposition.
3.2.3 Memory and Computation Reduction
When representing  $\mathbf{W}$  as  $\mathbf{A} \otimes \mathbf{B}$ , the number of elements is reduced from  $mn$  to  $m_1n_1 + m_2n_2$ . Moreover, using the Kronecker product to represent linear layers can reduce the required computation. In fact, a linear projection of any vector  $\mathbf{x}$  can be performed efficiently without explicit reconstruction of  $\mathbf{A} \otimes \mathbf{B}$  using the following popular property of Kronecker product:
$$
(\mathbf {A} \otimes \mathbf {B}) \mathbf {X} = \mathcal {V} (\mathbf {B} \mathcal {V} _ {n _ {2}, n _ {1}} ^ {- 1} (\mathbf {X}) \mathbf {A} ^ {\top}) \tag {6}
$$
where  $\mathbf{A}^{\top}$  is  $\mathbf{A}$  transpose. The consequence of performing multiplication in this way is that it reduces the number of FLOPs from  $(2m_{1}m_{2} - 1)n_{1}n_{2}$  to:
$$
\begin{array}{l} \min  \left((2 n _ {2} - 1) m _ {2} n _ {1} + (2 n _ {1} - 1) m _ {2} m _ {1}, \right. \\ (2 n _ {1} - 1) n _ {2} m _ {1} + (2 n _ {2} - 1) m _ {2} m _ {1}) \tag {7} \\ \end{array}
$$
3.3 Kronecker Embedding Layer
The embedding layer in large language models is a very large lookup table  $\mathbf{X} \in \mathbb{R}^{v \times d}$ , where  $v$  is the size of the dictionary and  $d$  is the embedding dimension. In order to compress  $\mathbf{X}$  using Kronecker decomposition, the first step is to define the shape of Kronecker factors  $\mathbf{A}^E$  and  $\mathbf{B}^E$ . We define  $\mathbf{A}^E$  to be a matrix of size  $v \times \frac{d}{n}$  and  $\mathbf{B}^E$  to be a row vector of size  $n$ . There are two reasons for defining  $\mathbf{B}^E$  as a row vector. 1) it allows disentangled embedding of each word since every word has a unique row in  $\mathbf{A}^E$ . 2) the embedding of each word can be obtained efficiently in  $\mathcal{O}(d)$ . More precisely, the embedding for the  $i$ 'th word in the dictionary can be obtained by the Kronecker product between  $\mathbf{A}_i^E$  and  $\mathbf{B}^E$ :
$$
\mathbf {X} _ {i} = \mathbf {A} _ {i} ^ {E} \otimes \mathbf {B} ^ {E} \tag {8}
$$
where  $\mathbf{A}^E$  is stored as a lookup table. Note that since  $\mathbf{A}_i^E$  is of size  $1\times \frac{d}{n}$  and  $\mathbf{B}^{E}$  is of size  $1\times n$ , the computation complexity of this operation is  $\mathcal{O}(d)$ . Figure 2 shows an illustration of the Kronecker embedding layer.
Figure 3: Illustration of the proposed framework. Left: A diagram of the teacher BERT model and the student KroneckerBERT. Right: The two-stage KD methodology used to train KroneckerBERT.
3.4 Kronecker Transformer
The Transformer layer is composed of two main components: MHA and FFN. We use Kronecker decomposition to compress both. In the Transformer block, the self-attention mechanism is done by projecting the input into the Key, Query, and Value embeddings and obtaining the attention matrices through the following:
$$
\mathbf {O} = \frac {\mathbf {Q K} ^ {\top}}{\sqrt {d _ {k}}} \tag {9}
$$
Attention  $(\mathbf{Q},\mathbf{K},\mathbf{V}) = \mathrm{softmax}(\mathbf{O})\mathbf{V}$
where  $\mathbf{Q}$ ,  $\mathbf{K}$ , and  $\mathbf{V}$  are obtained by multiplying the input by  $\mathbf{W}^Q$ ,  $\mathbf{W}^K$ ,  $\mathbf{W}^V$  respectively. In a MHA module, there is a separate  $\mathbf{W}^{Q_l}$ ,  $\mathbf{W}^{K_l}$ , and  $\mathbf{W}^{V_l}$  matrix per attention head to allow for a richer representation of the data. In the implementation usually, matrices from all heads are stacked together resulting in 3 matrices  $\mathbf{W}^{\prime k}$ ,  $\mathbf{W}^{\prime Q}$  and  $\mathbf{W}^{\prime V}$ . Instead of decomposing the matrices of each head separately, we use Kronecker decomposition after concatenation:
$$
\mathbf {W} ^ {\prime K} = \mathbf {A} ^ {k} \otimes \mathbf {B} ^ {K} \tag {10}
$$
$$
\mathbf {W} ^ {\prime Q} = \mathbf {A} ^ {Q} \otimes \mathbf {B} ^ {Q}
$$
$$
\mathbf {W} ^ {\prime V} = \mathbf {A} ^ {V} \otimes \mathbf {B} ^ {V}
$$
By choosing  $m_2$  to be smaller than the output dimension of each attention head, matrix  $\mathbf{B}$  in the Kronecker decomposition is shared among all attention heads resulting in more compression. The result of applying Eq.9 is then fed to a linear mapping  $(\mathbf{W}^0)$  to produce the MHA output. We use
Kronecker decomposition for compressing this linear mapping as well the two weight matrices in the subsequent FFN block:
$$
\mathbf {W} ^ {0} = \mathbf {A} ^ {0} \otimes \mathbf {B} ^ {0} \tag {11}
$$
$$
\mathbf {W} _ {1} = \mathbf {A} _ {1} \otimes \mathbf {B} _ {1} \tag {12}
$$
$$
\mathbf {W} _ {2} = \mathbf {A} _ {2} \otimes \mathbf {B} _ {2} \tag {13}
$$
In the following section, we describe how KD is used to improve the training of the KroneckerBERT model.
3.5.1 Intermediate KD
Let  $S$  be the student, and  $\mathcal{T}$  be the teacher, then for a batch of data  $(\mathbf{X},\mathbf{y})$ , we define  $f_{l}^{\mathcal{S}}(\mathbf{X})$  and  $f_{l}^{\mathcal{T}}(\mathbf{X})$  as the output of the  $l^{th}$  layer for the student network and the teacher network respectively. The teacher here is the BERT<sub>BASE</sub> and the student is its corresponding Kronecker BERT that is obtained by replacing the embedding layer and the linear mappings in MHA and FFN modules with Kronecker factors (see Sections 3.3 and 3.4 for details). Note that like other decomposition methods, when we use Kronecker factorization to compress the model, the number of layers and the dimensions of the input and output of each layer remain intact. Therefore, when performing intermediate layer KD, we can directly obtain the difference in the output of a specific layer in the teacher and student networks without the need for projection. In the proposed framework, the intermediate KD from the teacher to student occurs at the embedding
Model	Compression Factor	FLOPS	WK, WQ, WV, WO	W1, W2T	WE	Number of sums
n, m	d
BERTBASE	1×	21.7B	768, 768	768, 3072	768	1
n1, m1	n
KroneckerBERT8	7.8×	5.2B	384, 384	2, 8	8	1
KroneckerBERT21	21×	1.4B	48, 384	2, 16	16	1
KroneckerBERT5	5.5 ×	9.4B	384, 384	384, 384	12	4
Table 1: Configuration of the Kronecker layers for the three KroneckerBERT models used in this paper.  $n$  and  $m$  are the input and output dimensions of the weight matrices  $\left( {\mathbf{W} \in  {\mathbb{{IR}}}^{m \times  n}}\right) .{m}_{1},{n}_{1}$  indicates the shape of the first Kronecker factor  $\left( {\mathbf{A} \in  {\mathbb{{IR}}}^{{m}_{1} \times  {n}_{1}}}\right)$  . For embedding layer we only need to set the size of the row vector  ${\mathbf{B}}^{E} \in  {\mathbb{{IR}}}^{1 \times  n}$  .
layer output, attention matrices and FFN outputs:
$$
\mathcal {L} _ {\text {E m b e d d i n g}} (\mathbf {X}) = \operatorname {M S E} \left(\mathbf {E} ^ {\mathcal {S}}, \mathbf {E} ^ {\mathcal {T}}\right)
$$
$$
\mathcal {L} _ {\text {A t t e n t i o n}} (\mathbf {X}) = \sum_ {l} \operatorname {M S E} \left(\mathbf {O} _ {l} ^ {\mathcal {S}}, \mathbf {O} _ {l} ^ {\mathcal {T}}\right)
$$
$$
\mathcal {L} _ {\mathrm {F F N}} (\mathbf {X}) = \sum_ {l} \operatorname {M S E} \left(\mathbf {H} _ {l} ^ {S}, \mathbf {H} _ {l} ^ {\mathcal {T}}\right)
$$
where  $\mathbf{E}^S$  and  $\mathbf{E}^T$  are the output of the embedding layer from the student and the teacher respectively.  $\mathbf{O}_l^S$  and  $\mathbf{O}_l^T$  are the attention matrices (Eq.9),  $\mathbf{H}_l^S$  and  $\mathbf{H}_l^T$  are the outputs of the FFN, of layer  $l$  in the student and the teacher respectively.
Our final loss is as follows:
$$
\mathcal {L} (x, y) = \sum_ {(x, y)} \mathcal {L} _ {\text {E m b e d d i n g}} (x) + \tag {14}
$$
$$
\mathcal {L} _ {\text {A t t e n t i o n}} (x) + \mathcal {L} _ {\text {F F N}} (x) +
$$
$$
\mathcal {L} _ {\text {L o g i t}} (x) + \mathcal {L} _ {\text {S t u d e n t}} (x, y),
$$
where  $\mathcal{L}_{\mathrm{Student}}(x)$  is the supervised loss of the student, e.g. the cross entropy loss when fine-tuning for sequence classification tasks.
3.5.2 KD at pre-training
Inspired by prior works we use KD at the pretraining stage to capture the general domain knowledge from the teacher. For the pre-training distillation, the pretrained  $\mathrm{BERT}_{\mathrm{BASE}}$  model is used as the teacher. Intermediate layer KD is then used to train the KroneckeBERT network in the general domain. KD at pre-training improves the initialization of the Kronecker model for the task-specific KD stage. The loss at the pre-training stage involves the intermediate KD loss as in Eq. 14 as well as the masked language modeling and next sentence prediction. Unlike other methods, we perform pretraining distillation only on a small portion of the dataset (10% of the English Wikipedia) for a few epochs (3 epochs) which makes our training far more efficient. See Table 10 in the Appendix for
a comparison of training requirements by various methods.
3.6 Model Settings