Year: 2021

Abstract:
We introduce DictFormer with efficient shared dictionary to provide a compact, fast, and accurate transformer model. DictFormer significantly reduces the redundancy in the transformer’s parameters by replacing the prior transformer’s parameters with compact, shared dictionary, few unshared coefficients and indices. Also, DictFormer enables faster computations since expensive weights multiplications are converted into cheap shared look-ups on dictionary and few linear projections. Training dictionary and coefficients are not trivial since indices used for looking up dictionary are not differentiable. We adopt a sparse-constraint training with l1 norm relaxation to learn coefficients and indices in DictFormer. DictFormer is flexible to support different model sizes by dynamically changing dictionary size. Compared to existing light-weight Transformers, DictFormer consistently improves performance on multiple tasks, e.g., machine translation, abstractive summarization, and language modeling. Extensive experiments show that DictFormer outperforms prior light-weight transformer by ∼ 2 BLEU for machine translation task and achieves ∼ 1.7 lower perplexity for the language modeling task, when matching or reducing the model size of the original transformer. Also, DictFormer reduces 4.9× to 8.9× model size with similar accuracy over multiple tasks, compared to Transformer.

heading: 1 INTRODUCTION
text: Transformer (Vaswani et al., 2017) has been widely used in natural language processing (NLP) for its superior capability in capturing long-distance dependencies. However, its good performance comes with the giant model size. For example, T5 (Raffel et al., 2019) with a hidden dimension of 65K and GPT-3 (Brown et al., 2020) with 96 transformer blocks have 11 billion and 175 billion parameters, respectively. These large Transformers suffer from multiple severe issues, such as complicated learning and difficult deployment on mobile/IoT devices. First, during the training of a big transformer model, large training corpora (Raffel et al., 2019; Brown et al., 2020; Mehta et al., 2021) or careful regularization (Merity et al., 2017; Mehta et al., 2021) are required. Furthermore, the trained model is over-parameterized (Reid et al., 2021). The large model size with 11 billion and 175 billion parameters is beyond the capabilities of many edge devices including mobile devices and IoTs. Therefore, there is an urgent need to design parameter-efficient and fast transformer model that eliminates redundant parameters and enables real-time NLP applications on the edge.
Weights sharing is a choice proved by ALBERT (Lan et al., 2020) for designing compact and efficient pre-trained transformer encoders like BERT on self-supervised learning task. However, directly sharing all weights in an encoder-decoder transformer on a sequence-to-sequence task like machine translation will dramatically decrease transformer’s performance and accuracy (Reid et al., 2021). Although a recent framework Universal Transformer (UT) (Dehghani et al., 2019) shows that a vanilla transformer’s accuracy can be improved by using recursive weight sharing, UT’s high accuracy comes at the cost of deeper blocks or wider hidden dimensions, which significantly enlarges the computational cost and does not necessarily reduce the model size.
This paper introduces a new compact, fast, and accurate transformer architecture, DictFormer, that can be easily trained and deployed on edge devices. DictFormer depends on dictionary sharing and unshared linear projection coefficients instead of weights sharing. Specifically, a shared dictionary among all encoder/decoder blocks can significantly reduce the parameter redundancy and
Under review as a conference paper at ICLR 2022
therefore compresses the model size. Few unshared linear projection with coefficients on the shared dictionary enable each encoder/decoder block to have distinct layer-wise feature representations, thus improving the representation abilities compared to prior weights sharing. Also, DictFormer provides a method to dynamically control the layer-wise representation abilities by using the groupwise shared dictionary for layers with large-dimension features, e.g., Feed-forward Network (FFN). Last but not least, we show that training dictionary and coefficients are not trivial since indices used for looking up dictionary are not differentiable. We convert coefficients and indices into a sparse matrix and train it with l1 norm relaxation and convert this sparse matrix into dense coefficients and indices during inference.
Extensive experiments demonstrate that our DictFormer provides significant improvements over existing transformers on three sequence-to-sequence tasks, (i) machine translation, (ii) abstractive summarization, and (iii) language modeling. For machine translation, on IWSLT 2014 GermanEnglish, DictFormer attains transformer performance with 8.9× fewer parameters and 2× fewer Multi-Adds, or achieves 2.2 higher BLEU score under similar model size; on WMT 2014 GermanEnglish, DictFormer brings about 4.9× model compression and 1.9× computation reduction with even 0.7 higher BLEU score; on WMT 2014 English-French, DictFormer obtains consistent performance improvements: 4.9× model compression and 1.9× less computation with a similar or better BLEU score, or 2.1 BLEU score improvement with similar model size. For abstractive summarization, DictFormer reduces the model size by more than 4.7× on CNN-DailyMail dataset. For language modeling, DictFormer matches the performance of transformer with 3.6× to 5.7× fewer parameters, or achieves ∼ 1.7 lower perplexity than transformer on WikiText-103 benchmark.

heading: 2 RELATED WORK AND MOTIVATION
text: Related Lite-weight Transformers. Several methods have been proposed to design lite-weight transformers. The first line of research is to reduce the transformer computation complexities by redesigning self-attention mechanism including (Katharopoulos et al., 2020; Zhou et al., 2021; Raganato et al., 2020; You et al., 2020; Correia et al., 2019; Kaiser et al., 2018). These methods cannot reduce model size. The second line of research is model compression, e.g., quantization (Prato et al., 2020), pruning (Behnke & Heafield, 2020), low-rank factorization (Ma et al., 2019), and knowledge distillation (Wang et al., 2020). These two directions of research can be combined with our Dictformer. The third line of research is efficient architecture design (So et al., 2019; Wu et al., 2020; Mehta et al., 2021) by improving the expressiveness of transformers. The forth line of research is weights sharing (Xia et al., 2019; Ma et al., 2019; Dehghani et al., 2019; Reid et al., 2021; Takase & Kiyono, 2021) by reusing parameters across transformer blocks. Weights sharing cannot reduce computations. Our Dictformer falls into the third and forth category. We show that our Dictformer with Dictionary sharing can reduce both model size and computations.
Transformers with Weights Sharing. Weight sharing is surprisingly effective to compress model size for discriminate NLP models based on Transformer encoders, e.g., BERT. For example, prior work ALBERT (Lan et al., 2020) shows that even sharing all parameters across layers does not introduce any accuracy reduction. However, for generative sequence-to-sequence models based on transformer’s encoders and decoders, sharing all parameters will significantly decrease accuracy on multiple standard machine translation or language modelling tasks (Reid et al., 2021; Takase & Kiyono, 2021). To match vanilla Transformer’s accuracy, multiple works (Reid et al., 2021; Takase & Kiyono, 2021; Xia et al., 2019; Ma et al., 2019) only share weights across partial layers instead of all layers. However, partial weights sharing remarkably brings down the model size compression effect of weights sharing. Also, how to decide which layers should be shared in partial weights sharing is difficult due to the large and dynamic search space that is dependent on the specific tasks.
Transformer with all-parameters sharing such as Universal Transformer (Dehghani et al., 2019) matches or improves transformer’s performance at the cost of a wider or deeper transformer architecture. A wider transformer with a larger embedding dimension enlarges the model size and brings larger computations (Mult-Adds). A deeper transformer with more encoder/decoder blocks does not only increase model size, but also introduces more computations. Importantly, weights sharing techniques cannot reduce Mult-Adds numbers and training/inference time. Figure 8(a) in Appendix shows the comparisons of Transformers with weights sharing and our dictionary sharing. Weights sharing techniques cannot solve the deployment challenges of transformers on resource-limited devices for real-time NLP applications.
Under review as a conference paper at ICLR 2022
Motivation of DictFormer with Dictionary Sharing. Figure 1 shows that although transformers with weights sharing including Universal Transformer (Dehghani et al., 2019), Tied Transformer (Xia et al., 2019), and Subformer (Reid et al., 2021) significantly reduce the model size over vanilla transformer and even one recent lightweight transformer DeLighT (Mehta et al., 2021), they cannot reduce the #Mult-Adds and even suffer from larger #Mult-Adds, e.g., Universal Transformer. Particularly, Tied Transformer (Xia et al., 2019) and Subformer (Reid et al., 2021) compress transformer model by 1.7× ∼ 2.6× but cannot reduce Mult-Adds. Universal Transformer (Dehghani et al., 2019) achieves ∼ 1 BLEU score improvement with 1.4× less parameters, but the Mult-Adds numbers are increased by ∼ 4.3×, thereby significantly prolonging the NLP inference latency and restraining the deployment of real-time NLP applications on edge devices.
To enable the deployment of transformer models on mobile devices, recent work Evolved Transformer (So et al., 2019) and Lite Transformer (Wu et al., 2020) try to design new lite-weight transformer architecture to meet the defined mobile settings, e.g., 10 million parameters, but their tiny architectures have to suffer from a large accuracy decrease. As Figure 1 shows, Evolved Transformer and Lite Transformer lose 2.9 and 2.4 BLEU score compared to base transformer, respectively.

heading: 3 DICTFORMER
text: Overview. We propose DictFormer with dictionary sharing to enable a fast, compact, and accurate transformer. When matching transformer’s accuracy, DictFormer can reduce more than 3.6× parameters and ∼ 3× Mult-Adds shown in Figure 1, which outperforms prior transformers with a higher BLEU score. Also, DictFormer is flexible to compress model size given an accuracy threshold. For instance, when DictFormer matches the accuracy of Lite Transformer, it could further compress the model of lite transformer by ∼ 2.2×. Given a N -layer transformer model, we can easily transform it into DictFormer by converting all the weights in N blocks into one shared dictionary and few unshared look-up coefficients, shown in Figure 2. For example, N -layer weights attention WAi and FFN WFi , where i ∈ [0, N − 1], in Transformer (Figure 2(a)) are represented by smaller dictionaries DA, DF that are shared by N blocks, and N -layer unshared coefficients CAi and C F i for DictFormer (Figure 2(b)). Meanwhile, the attention and FFN operations in vanilla Transformer are replaced with shared-dictionary attention and group-wise shared-dictionary FFN whose details are introduced in our following contents. Dictformer reduces Transformer’s #Params from O(d2N) to O(d(m + tN)). This is a O(dN/(m + tN))× model size reduction since dictionary size m < d, coefficient size t << d, where the embedding size is d. Dictformer also reduces #Mult-Adds from O(d2Nn) to O(dNn(m+ t)), where n is input sequence length. The details on how to define and calculate #Params and #Mult-Adds are shown in Appendix A.1.
Shared-dictionary Attention. Given a N -layer transformer model, we define that Qi, Ki, and Vi are the i-th layer query, key, and values. Same to Transformer (Vaswani et al., 2017), we also use equation 1 to calculate the attention scores once we have query, key, and values.
Attention(Qi,Ki, Vi) = softmax( Qi ·KTi√
d ) · Vi (1)
Our DictFormer utilizes equation 2 instead of MultiHead(Qi,Ki, Vi) = MHi · WOi used in Transformer (Vaswani et al., 2017) to compute multi-head values, where MHi is derived according
Under review as a conference paper at ICLR 2022
FFN Attention Embed Attention FFN Sh ar ed E nc od er T x
Embed Attention FFN Sh ar ed D ec od er T xAttention Shared Encoder Shared Encoder Embed ... t x t x

heading: N/t
text: Shared Decoder Shared Decoder Embed ... t x t x N/t
Embed Dict. Atten. N xSole Coeff. Dict. FFN Sole Coeff.
Embed Dict. Atten. Query & Scale Dict. FFN Query & Scale N x (a) Transformers with T-layer weight sharing (b) N/t t-layer partial weight sharing (c) N-layer dictionary sharing and sole coefficients Embed Attention FFN Sh ar ed E nc od er T x Embed Attention FFN Sh ar ed D ec od er T xAttention Shared Encoder Shared Encoder Embed ... t x t x N/t Shared Decoder Shared Decoder Embed ... t x t x N/t Embed Shared Dictionary N x Unique Query & Scale (a) Transformers with T-layer weight sharing (b) N/t t-layer partial weight sharing (c) N-layer dictionary sharing and sole coefficients Embed Shared Dictionary N x Unique Query & Scale ds d W QA ds 4ds WF Embed Encoder Block Ns x FFN Attention Embed Ns x Attention Decoder Block  Wider: ds > d  large model size & Mult-Adds#  Deeper: Ns > N  Mult-Adds# Dict-FFN Dict-Attention Embed Encoder Block N x Dict-FFN Dict-Attention Embed N x Dict-Attention Decoder Block d m DA t d CiA d m DF Ci F t 4d  m < d & tiny t & N blocks  Tiny model & Mult-Adds# (a) Transformer with weight sharing: wider or deeper (inefficient) (b) DictFormer: efficient transformer with dictionary sharing
FFN
Attention d d W Q i A
d
4d
WiF
Embed
Encoder Block
N x
FFN
Attention Embed N x Attention
Decoder Block #Params: (12d2N) #Mult-Adds: (12d2Nn)
Dict-FFN
Dict-Attention Embed
Encoder Block
N x
Dict-FFN
Dict-Attention Embed N x Dict-Attention
Decoder Block
d m DA t d CiA
d
m
DF Ci F
t 4d
#Params: (6dm+12dtN) #Mult-Adds: (6dmNn+12dtNn)
(a) Transformer (b) Our DictFormer
to equation 3, and DA, COi , and I O i are used to linearly project MHi, instead of W O i . The attention value headji for each head j in layer i is computed by equation 4 instead of head j i = Attention(Qi · W Qj i ,Ki ·W Kj i , Vi ·W Vj i ) used in existing transformers.
MultiHead(Qi,Ki, Vi) = SD(MHi, D A, COi , I O i ) (2)
MHi = Concat(head 1 i , ..., head h i ) (3)
headji = Attention(SD(Qi, D A, C Qj i , I Qj i ), SD(Ki, D A, C Kj i , I Kj i ), SD(Vi, D A, C Vj i , I Vj i )) (4)
The reason why we use the shared attention dictionary DA, indices Ii and coefficients Ci to replace the larger and unshared weights WQji , W Kj i , W Vj i and W Oj i used in previous transformers is that the linear projections of lookup ofDA with coefficients CXi can have similar representation abilities as WX j
i , where X represents inputs type, e.g., X = Q means input is query. To be specific, D A, Ii,
and Ci have size of d×mA, tA×d, and tA×d, respectively. mA and tA are the dictionary size and coefficient size in attention. The linear projections in transformer’s attention, e.g., WQji ·Qj , are replaced by our lightweight shared dictionary projection function (SD), e.g., SD(Qi, DA, C Qj i , I Qj i ), derived by equation 5 and equation 6. SD(Qi, DA, C Qj i , I Qj i ) replaces W Qj i · Qj since W Qj i can be replaced by looking up DA and few linear projections with coefficients CQji without accuracy decrease. The following paragraphs and Figure 3(a) introduce the details of look-up and scaling using equation 5 and equation 6.
In equation 5, the lookup of DA by indices IQji is defined by D A[:, I Qj i [t, id]] that can fetch the I Qj i [t, id]]-column vector from D A; Unshared linear projection that is used to enlarge the represen-
tation ability of shared dictionary is depicted by W̃Qji = ∑tA t=1 C Qj i [t, id] DA[:, I Qj i [t, id]], where
represents scaling a fetched vector from a dictionary with a scalar in coefficients. Therefore, linearly projecting Qj with Wi according to W Qj i · Qj is replaced by W̃ Qj i · Qj in equation 5 when
we find proper DA, IQji ,C Qj i to meet that W̃ Qj i and Wi Qj have similar representation abilities,
e.g., they have matched accuracy. Directly generating W̃iQj and multiplying it with Qj potentially increases the computations. To tackle this problem, we compute the multiplications between Qi and dictionary DA as Oi first according to equation 6, and reuse Oi for the following look-ups and
Under review as a conference paper at ICLR 2022
scaling with CQji .
SD(Qi, D A, C Qj i , I Qj i ) = Qi · tA∑ t=1 C Qj i [t, id] D A[:, I Qj i [t, id]]
= ( tA∑ t=1 Qi ·DA[:, I Qj i [t, id]]) C Qj i [t, id]
= tA∑ t=1 Oi[:, I Qj i [t, id]] C Qj i [t, id], id ∈ [0, d h ]
(5)
Oi[:, b] = Qi ·DA[:, b], b ∈ [1,mA] (6) We use Figure 3(a) to show an example of equation 5 about how to lookup dictionary with in-
dices and scaling it with coefficients. This process can create a N -layer weight WQ. For example, given the shared dictionary DA, to generate the first column of weight WQi , the first column of index matrix IQi is taken out, e.g., [1, 13, 35], and is used as indices to fetch the the corresponding columns from DA, e.g., DA[:][1], DA[:][13], DA[:][35]. Then the first column of coefficients CQi , e.g., [0.2, 0.3, 0.5], are multiplied with DA[:][1], DA[:][13], DA[:][35] respectively, and the sum of multiplication works as the first column of WQi . In this way, weights W in attention with 4d
2N parameters are compressed into mA, IQ, and CQ with size dmA + 8tAdN . For example, DictFormer can reduce 8.7× model size when d = 512, mA = 256, tA = 24, N = 6. More calculation details can be found at appendix A.1.
Group-wise Shared-dictionary FFN. The FFN in prior transformer includes two-layer computations: (i) F1 = max(0, Xi · WF1i + b1), (ii) F2 = F1 · W F2 i + b2. Instead of the regular linear projections Xi ·WF1i + b1 and F1 ·W F2 i + b2, DictFormer uses a new lite-weight projection called group-wise shared dictionary projection (GSD), i.e., GSD(Xi, D,CF1i , I F1 i , G) and GSD(F1, D,C F2 i , I F2 i , G), in equation 7 and equation 8 to compute the FFN. Here GSD projection is same as SD in shared-dictionary attention when G = 1. In SD projection, it works well to replace a N weights of d × d with a d × m dictionary where each column of dictionary is multiplied with a scalar in coefficients. However, when the column of dictionary is large, e.g., 4 × d in the second layer of FFN, SD projection is difficult to achieve an accurate model since an vector of dictionary with 4 × d, e.g. 2048, multiplied by a same scalar is not flexible enough. To increase the flexibility of SD projection and improve performance, GSD firstly divides each column of dictionary into G groups equally and assign different scalars to multiply the numbers in each group.
F1 = max(0, GSD(Xi, D,C F1 i , I F1 i , G) + b1) (7)
F2 = GSD(F1, D,C F2 i , I F2 i , G) + b2 (8)
Under review as a conference paper at ICLR 2022
Equation 9 and equation 10 show the computations ofGSD. DictionaryDF2 [:, :] is divided equally into G groups and the g-th group is defined as DF2 [Indexg, :], where Indexg = [(g − 1) 4dG , g 4d G ) and g ∈ [1, G]. Also, the multiplication result between g-th group dictionary and input is defined as Og[:, b] shown in equation 10, where b ∈ [1,mF2 ]. Different groups of dictionary share the same look-up indices IF2i . The t
F2 queried vectors in each group will be scaled by its corresponding coefficients CF g 2
i and the scaled results are then accumulated to derive the g-thGSD result shown in equation 9. In addition, G group-wise GSD are further summed to compute the final GSD result.
GSD(X,DF2 , CF2i , I F2 i , G) = G∑ g=1 tF2∑ t=1 Og[:, IF2i [t, id]] C F g 2 i [t, id] (9)
Og[:, b] = X[:, Indexg] ·DF2 [Indexg, b], g ∈ [1, G], b ∈ [1,mF2 ], Indexg = [(g − 1) 4d
G , g
4d G ) (10)
Figure 3(b) shows an example of group-2 shared dictionary for the the second layer of FFN. The dictionary DF2 is equally split into two parts that share the same indices IF2i but each group has unique coefficients, e.g., CF g=1 2 i and C F g=22 i . The first group and the second group use the corresponding coefficients, e.g., [0.3, 0.1, 0.6] and [0.1, 0.6, 0.3], to scale the queried dictionary vectors, respectively. The accumulation of scaled vectors can be used to represent the N -layer weight WF2 .
Training DictFormer via Constraints and Relaxation. DictFormer replaces transformer weights with linear projections of a dictionary, thereby having three-step computations including small projections, lookup and scale. DictFormer computes a small multiplication between input and dictionary, generating intermediate variable O, looking up O and scaling it with coefficients. To train the DictFormer, one should jointly optimize dictionary, index I and coefficients C. Directly training the Dictformer leads to a combinatorial optimization problem since index I is non-continuous. Although one could use AutoML, such as evolutionary method and Reinforcement learning to jointly learn dictionary, I and C, these methods suffer from a large training time with low performance. To workaround the training of index I , we reformulate the method to a regular linear projection with sparse constraints, which can efficiently train the DictFormer.
We convert the index Ii and coefficients Ci into sparse coefficients Z, so that the training of index Ii and coefficients Ci is replaced with training sparse coefficients Z. During the deployment phase, the sparse coefficients Z are reversely transformed into index Ii and coefficients Ci shown in Figure 4. In particular, the shape of coefficients C and index I is tA × d × N . The shape of sparse Z is mA × d × N . There are two steps to derive Z: Initializing all elements as 0 and copying C elements to Z according to the index values in Ii. For example, since the first column of I1 and C1 is [1, 13, 35] and [0.2, 0.3, 0.5], all entries of the first column in Z are zeros except the 1st entry, 13th entry, and the 35th entry are 0.2, 0.3, and 0.5. Therefore, the lookup and scale ofOi can be converted into the matrix multiplication betweenOi and Zi shown in equation 11. The new coefficient C has a sparsity constraint that requires that the non-zero elements (l0 norm) of each column is tA << mA shown in equation 12. Now we can train Z with ||ZQji [:, id]||l0 = tA, instead of training I and C. tA∑ t=1 Oi[:, I[t, id]] C Qj i [t, id] = Oi · Z Qj i (11)
||ZQji [:, id]||l0 = t A (12)
Considering the l0 norm sparsity constraint in equation 12 is non-differentiable, we loose this constraint to a l1 norm constraint shown in equation 13 to penalize the non-zero parameters. Therefore, the gradients of coefficients are calculated by equation 14, where parameter λ is used to control
Under review as a conference paper at ICLR 2022
the trade-off between loss δLδZ and l1 norm sparsity constraint. To improve training performance, dynamic-sparsity Z can be supported by enabling different columns to have various non-zero elements. For example, equation 15 can be used to globally change the near-zero values to zero given a ratio ρ, where value(ρ) derives the value at ratio ρ in the ascending order. One also can force that t of the elements in each column of Z are non-zero while the rest are set to 0 by letting value(ρ) is the t-th largest value of each column.
d∑ id ||ZQji [:, id]||l1 = ||Z||l1 (13)
δ(L+ λ||Z||l1) δZ = δL δZ + λsign(Z) (14)
µ(x) = { x, if |x| > value(ρ). 0 otherwise.
(15)

heading: 4 EXPERIMENTAL METHODOLOGY
text: Machine Translation Dataset. Three machine translation benchmarks are tested: IWSLT’14 German-English (De-En), WMT’14 English to German (En-De), and WMT’14 English to France (En-Fr). For IWSLT’14 De-En, we adopt the same settings in Wu et al. (2020) with 160K training sentence pairs and 10K joint byte pair encoding (BPE) vocabulary in lower case. For WMT’14 En-De, our models are trained with 4.5M sentence pairs, validated on newstest2013, and tested on newstest2014. Also, a 32K joint source and target BPE is used in the vocabulary. For WMT’14 EnFr, we follow the setup in Wu et al. (2020) by training models on 36M training sentence pairs from WMT’14, validating on newstest2012 and 2013, and testing on newstest2014. The vocabulary has a size of 40K and is based on a joint source and target BPE factorization. The evaluation setting used in Vaswani et al. (2017) is utilized with a beam size of 4 and a length penalty of 0.6. We replicate the same BLEU calculations in Wu et al. (2020) with case-sensitive tokenization. The last 10 model checkpoints are averaged for testing and the lowest-perplexity model is picked for validation.
Abstractive Summarization Dataset. We evaluate DictFormer on CNN-DailyMail dataset (Chen et al., 2016) that has 280K news articles with multi-sentence summaries. We follow the settings in (Wu et al., 2020), truncate the articles to 3000 tokens, use a 30K BPE vocabulary and F1-Rouge as the metric.
Language Modeling Dataset. We also evaluate DictFormer on WIKITEXT-103 (Merity et al., 2016) that has a 260K BPE vocabulary and contains 103M/217K/245K tokens for training, validation, and testing. Language modeling performance about perplexity (ppl) is reported.
Architecture and Evaluation. For machine translation tasks, we follow the sequence-to-sequence transformer (Vaswani et al., 2017) with encoder and decoder to develop our model. For IWSLT dataset, the general settings are same as (So et al., 2019). For WMT dataset, the setting is based on (Wu et al., 2020). Also, the same model for machine translation is used for summarization task. For language modelling, we use the transformer-based decoder architecture of (Baevski & Auli, 2019) but with 512 model dimensions and 12 blocks that is same to (Wu et al., 2020). Fairseq’s transformer implementation (Ott et al., 2019) is used as the backbone for the baseline model.
In our DictFormer architecture, encoder and decoder have three dictionaries, respectively, i.e., encoder attention dictionary DAenc, encoder F1 dictionary D F1 enc, encoder F2 dictionary D F2 enc, and decoder attention dictionary DAdec, decoder F1 dictionary D F1 dec, decoder F2 dictionary D F2 dec. Each of them is shared by all blocks in encoder or decoder. For example, DAenc is shared by all attention blocks in encoder. We study the effects of DictFormer with various dictionary size from 80 to 240 and different coefficient size from 12 to 84 shown in Figure 6. One can pick up a dictionary size and coefficient size to control the trade-off between accuracy and model size according to the requirements. For example, one could get the results in our tables when dictionary size is set as mA = d2.5 , mF1 = d2.5 , m F2 = d2 , and t A = d10 , t F1 = tF2 = d15 .
In this paper, #Params means parameter numbers of transformer without including embedding layers; #TParams shows parameter numbers of model and embedding layers. The Mult-Adds calculation is tested when a model is used to translating two sentences with the length of 30 in default (same to previous work Wu et al. (2020)).
Under review as a conference paper at ICLR 2022

heading: 5 RESULTS
text: Machine Translation. We first report the machine translation results and compare them with prior works. Our baseline transformer model settings are in line with Lite Trans. (Wu et al., 2020) which provides the best results under mobile settings. Our DictFormer generally outperforms the state-ofthe-art transformers on IWSLT’14 De-En, WMT’14 En-De, and WMT’14 En-Fr. Table 1 represents the quantitative results of our DictFormer. Compared to Transformer (Vaswani et al., 2017), DictFormer compresses 8.9× model size, reduces 1.9× #Mult-Adds, and improves 2.2 BLEU score on IWSLT’14 De-En, which achieves better trade-off between performance and model size than previous works. Similar to IWSLT’14 De-En, our experiments on larger corpora including WMT’14 En-De and En-Fr also show DictFormer obtains more compact and accurate model than existing Transformers. For instance, only our DictFormer can compress more than 6× model size over Transformer without decreasing any model performance under mobile settings.
Abstractive Summarization and Language Modeling. We then report the abstractive summarization and language modeling results and compare them with prior works. Abstractive summarization model follows the transformer model, but language modeling model follows the adaptive inputs model (Baevski & Auli, 2019). Our DictFormer consistently outperforms prior transformers on both summarization and language modeling tasks. Table 2 represents the quantitative results of our DictFormer. For abstractive summarization on CNN-DailyMail, DictFormer achieves similar F1Rouge score with Transformer but requires 4.7× less model size and ∼ 4× less computations. For language modeling on WIKITEXT103, compared to adaptive inputs model (Baevski & Auli, 2019), DictFormer compresses 3.8× model size with less computations and matched testing perplexity. Sensitivity Study. We study the DictFormer’s performance under different model size as Figure 5 shows. This is important since system-independent settings require scalable DictFormer. The performance of DictFormer improves with an increase in the number of model parameters, across different corpora. DictFormer achieves similar BLEU score with Transformer using 3× to 10× less model parameters. In Figure 6, we also report the hyper-parameter’s effects on DictFormer’s performance with WMT’14 En-Fr dataset with embedding size d = 512. Figure 6(a) shows the dictionary size study when the coefficient size is fixed to 60. All dictionaries share the same size, and all coefficients also share same size. The Figure 6(b) shows the coefficient size study when the dictionary size is fixed 240. Although DictFormer’s performance improves with an increase in the size of dic-
Under review as a conference paper at ICLR 2022
tionary and coefficients, the model size is also enlarged. So one should pick up a proper dictionary or coefficient size according to accuracy requirements.
Ablation Study. We also use table 3 to show the ablation study of DictFormer’s techniques. Transformer Vaswani et al. (2017) adopts the base architecture with hidden size 512. TransformerFlatten flattens the feed-forward hidden size of transformer from 2048 to 512, thus reducing the parameters from 44 millions to 25 millions without hurting the accuracy. Only-Dictionary is based on Transformer-Flatten and uses unshared dictionary architecture. This method achieves 33.2 BLEU with 9.7 million model parameters. The #TParams is 14.3 million and 27.4 million when adding embedding parameters on IWSLT’14 De-En and WMT’14 En-Fr, respectively. When sharing dictionaries in attention, Shared-Attention achieves similar BLEU score with 4.2 million model parameters. Meanwhile, Shared-FFN only shares FFN using group-wise shared dictionaries (G is group number). DictFormer that shares both FFN and attention with G=2, denoted by Shared-both, requires 2.6M model parameters and matches the accuracy of unshared DictFormer. Improved-Embed represents that the embedding layers of Shared-both are further compressed by using existing method Mehta et al. (2021) so that the #TParams is reduced to 5.1M from 7.1M.

heading: 6 CONCLUSION
text: We propose DictFormer with dictionary sharing to enable a fast, compact, and accurate transformer. DictFormer significantly reduces both model size and computation parameters of prior transformers with similar performance. Also, DictFormer can consistently improve the performance on multiple tasks, such as machine translation, abstractive summarization, and language modeling with similar model size. We will release code upon legal approval.
Under review as a conference paper at ICLR 2022

heading: A APPENDIX

heading: A.1 DICTFORMER BENEFITS WITH DICTIONARY SHARING
text: The block comparison of Transformer and our DictFormer.
Embed Attention FFN Sh ar ed E nc od er T x
Embed Attention FFN Sh ar ed D ec od er T xAttention
Shared Encoder Shared Encoder Embed ... t x t x N/t
Shared Decoder Shared Decoder Embed ... t x t x N/t
Embed Dict. Atten. N xSole Coeff. Dict. FFN Sole Coeff.
Embed Dict. Atten. Query & Scale Dict. FFN Query & Scale N x (a) Transformers with T-layer weight sharing (b) N/t t-layer partial weight sharing (c) N-layer dictionary sharing and sole coefficients Embed Attention FFN Sh ar ed E nc od er T x Embed Attention FFN Sh ar ed D ec od er T xAttention Shared Encoder Shared Encoder Embed ... t x t x N/t Shared Decoder Shared Decoder Embed ... t x t x N/t Embed Shared Dictionary N x Unique Query & Scale (a) Transformers with T-layer weight sharing (b) N/t t-layer partial weight sharing (c) N-layer dictionary sharing and sole coefficients Embed Shared Dictionary N x Unique Query & Scale
(a) Transformers block
(b) N/t t-layer partial weight sharing (c) N-layer dictionary sharing and sole coefficients
M ul
tihe ad A tte nt io n
FF N
Wi Qj Wi Kj Wi Vj Attention Concat
Add
Wi O
Wi F
Wi FFN
Add (b) Our Timer Block
DF d d d
d
d
d
dF = 4d
d
Paras #: 8Nd2 Ops: 8Nnd2
Paras #: 4Nd2 Ops: 4Nnd2 S D P At te nt io n G SD P FF N
D Attention Concat
Add
DF
Add
DFFN GSDP (Eq. 13)
SDP (Eq. 7)
SDP (Eq. 7)SDP (Eq. 7) Paras #: 8ds+md Ops: Nnd(m+s)
Paras #: 3dp+32ds
Ops: Nnd(3p+32s)
d m d
d
p
d
Figure 7 further depicts the comparison of transformer encoder block and DictFormer encoder block. The i-th Transformer block including attention and FFN has four weights WAi and W F i . W A i has four weights including query WQji , key W Kj i , value W Vj i , and output W Oj i , thereby having 4Nd 2 parameters and 4Nnd2 Mult-Adds given a sequence of size n. WFi has two parts: W F1 i and W F2 i . N -layer FFN blocks have 8Nd2 parameters and 8Nnd2 Mult-Adds operations. In contrast, DictFormer significantly reduces the parameters and Mult-Adds of Transformer. Specifically, each DictFormer block replaces the attention and FFN with shared dictionary (SD) attention and group-wise shared dictionary (GSD) FFN. Linear projection is performed by looking up dictionary DA using Eq.5. SD Attention has 8NdtA + mAd parameters and Nnd(mA + tA) operations. GSD FFN has 3dmF + 32NdtF parameters and Nnd(3mF + 32tF ) operations, where t, m are the size of coefficient and dictionary. To simplify the calculations of #params and #Mult-Adds, we can let tA = tF , mA = mF . Dictformer reduces Transformer’s #Params from O(d2N) to O(d(m+ tN)). This is a O(dN/(m + tN))× model size reduction since dictionary size m < d, coefficient size t << d, where the embedding size is d. Dictformer also reduces #Mult-Adds from O(d2Nn) to O(dNn(m+ t)).

heading: A.2 THE COMPARISON BETWEEN WEIGHT SHARING AND DICTIONARY SHARING
text: Weight sharing is surprisingly effective to compress model size for discriminate NLP models based on Transformer encoders, e.g., BERT. For example, prior work ALBERT (Lan et al., 2020) shows that even sharing all parameters across layers does not introduce any accuracy reduction. However, for generative sequence-to-sequence models based on transformer’s encoders and decoders, sharing all parameters will significantly decrease accuracy on multiple standard machine translation or language modelling tasks (Reid et al., 2021; Takase & Kiyono, 2021). To match vanilla Transformer’s accuracy, multiple works (Reid et al., 2021; Takase & Kiyono, 2021; Xia et al., 2019; Ma et al., 2019) only share weights across partial layers instead of all layers. However, partial weights sharing remarkably brings down the model size compression effect of weights sharing. Also, how to decide which layers should be shared in partial weights sharing is difficult due to the large and dynamic search space that is dependent on the specific tasks.
Transformer with all-parameters sharing such as Universal Transformer (Dehghani et al., 2019) matches or improves transformer’s performance at the cost of a wider or deeper transformer architecture. A wider transformer with a larger embedding dimension enlarges the model size and brings
Under review as a conference paper at ICLR 2022
Embed Attention FFN Sh ar ed E nc od er T x
Embed Attention FFN Sh ar ed D ec od er T xAttention Shared Encoder Shared Encoder Embed ... t x t x N/t
Shared Decoder Shared Decoder Embed ... t x t x N/t
Embed Dict. Atten. N xSole Coeff. Dict. FFN Sole Coeff.
Embed Dict. Atten. Query & Scale Dict. FFN Query & Scale N x (a) Transformers with T-layer weight sharing (b) N/t t-layer partial weight sharing (c) N-layer dictionary sharing and sole coefficients
Embed
Attention FFN
Sh ar
ed E nc od er
T x
Embed
Attention FFN
Sh ar
ed D ec od er T xAttention Shared Encoder Shared Encoder
Embed
...
t x t x N/t
Shared Decoder Shared Decoder
Embed
...
t x t x N/t
Embed
Shared Dictionary N x Unique Query & Scale
(a) Transformers with T-layer weight sharing (b) N/t t-layer partial weight sharing (c) N-layer dictionary sharing and sole coefficients
Embed
Shared Dictionary N x Unique Query & Scale
larger computations (Mult-Adds). A deeper transformer with more encoder/decoder blocks does not only increase model size, but also introduces more computations. Importantly, weights sharing techniques cannot reduce Mult-Adds numbers and training/inference time. Figure 8(a) shows the comparisons of Transformers with weights sharing and our DictFormer with dictionary sharing. Weights sharing techniques cannot solve the deployment challenges of transformers on resourcelimited devices and obtain real-time NLP applications. To reduce both model size and computations of Transformers, we introduce DictFormer with dictionary sharing instead of previous weights sharing. In particular, DictFormer shares dictionary across all layers so there is no need to decide which layers should be shared. Also, DictFormer with few unshared look-up coefficients does not require a wider embedding size or deeper encoder/decoder to improve accuracy. Rather than compressing model size by parameters sharing, DictFormer with dictionary sharing can also enable computations sharing to reduce running latency. Therefore, our DictFormer provides a compact, fast, and accurate transformer model for sequence-to-sequence NLP tasks.

heading: A.3 TRAINING SETTINGS
text: To fairly compare with our baselines, all of our training settings are in line with (Wu et al., 2020). For machine translation tasks, a dropout of 0.3 is used and the dropout ratio is linearly scaled down when we shrink the dimension of the embeddings for WMT datasets. The learning rate linearly warms up from 10−7 to 10−3 followed by a cosine annealing with a single cycle and Adam optimizer same as (So et al., 2019) and (Wu et al., 2020). Inverse linear square root learning rate scheduling is used for IWSLT De-En. The summarization training settings are the same as machine translation. To train models on language modeling task, we follow the training setting in (Baevski & Auli, 2019), but decrease the dropout rate by half in FFN layers. The training experiments of WMT, summarization, and language modeling are conducted on 8 NVIDIA Tesla V100 GPUs. IWSLT De-En is trained on two GPUs. Machine translations tasks are trained for 50K steps, but language modelling tasks are trained for 286K steps. We further describe deployment settings in Appendix.

heading: A.4 DEPLOYMENT SETTINGS.
text: (i) Mobile settings. Scaling up transformer models leads to higher machine translation performance, but those large architectures are not suitable for real-world applications, especially when we should
Under review as a conference paper at ICLR 2022
deploy them on edge devices where computation and memory size are highly constrained. Multiple prior works like (Wu et al., 2020) formalize the deployment of lite-weight transformers on mobile devices by defining the mobile settings regarding the amount of computation and parameter numbers. Particularly, given the ARM Cortex-A72 mobile CPU with computation performance of 48G FLOPS, the computation constraint for machine translation should be under 500M Mult-Adds and the parameter number should be around 10M with a sequence of 30 tokens that are the general length for machine translation. (ii) System-independent settings. Other than deployed on mobile devices, DictFormer also supports system-independent deployment. In this setting, we directly choose prior transformer architectures like DelighT (Mehta et al., 2021) and Subformer (Reid et al., 2021) as baselines and compare our DictFormer with them under the the same performance constraint.

Review summaries:
This paper proposes a compression method for Transformer-based encoder-decoder or language models. The key idea of the proposed method is to decompose the standard parameters into a much smaller shared parameter matrix and independent parameters for each original matrix. Then, the method can approximately recover the original Transformer models by simple additions and multiplications.  The experiments are conducted on three MT tasks, one summarization task, and one language modeling task. Experimental results show that the proposed method seems to reduce model sizes and computations successfully while preventing considerable performance degradation (in some cases, the proposed method appears to improve the performance).

This work proposes a modification of the original Transformer architecture by replacing attention layers and layers in its Feed-Forward Networks across all of its blocks with learned shared dictionaries. The proposed model, called DictFormer, has a smaller number of parameters and uses a smaller amount of computational operations when compared to the original Transformer and some of its variations. When evaluated against these models on popular machine translation, summarization, and language modeling benchmarks, DictFormer achieves comparable or better performance.

The authors proposed an efficient transformer layer based on a dictionary of shared parameters instead of standard self-attention.  The goal is to reduce redundant parameters in transformer models.  The main contributions are: a lite transformer model, modification of the self-attention parameters, and evaluation on language dowstream tasks. The proposed transformer model outperforms related work on the machine translation and language modelling tasks.