DictFormer: Tiny Transformer with Shared Dictionary
Authors: Qian Lou
Ting Hua
Yen-Chang Hsu
Yilin Shen
Hongxia Jin
Venue: ICLR 2022 Poster (2022)
Keywords: Transformer
Parameters Sharing
Tiny
On-device Transformer
Machine Translation
Attention
Dictionary Sharing

Abstract: We introduce DictFormer with the efficient shared dictionary to provide a compact, fast, and accurate transformer model. DictFormer significantly reduces the redundancy in the transformer's parameters by replacing the prior transformer's parameters with a compact, shared dictionary, few unshared coefficients, and indices. Also, DictFormer enables faster computations since expensive weights multiplications are converted into cheap shared look-ups on dictionary and few linear projections. Training dictionary and coefficients are not trivial since indices used for looking up dictionary are not differentiable. We adopt a sparse-constraint training with $l_1\,\,norm$ relaxation to learn coefficients and indices in DictFormer. DictFormer is flexible to support different model sizes by dynamically changing dictionary size. Compared to existing lightweight Transformers, DictFormer consistently reduces model size over Transformer on multiple tasks, e.g., machine translation, abstractive summarization, and language modeling. Extensive experiments show that DictFormer reduces $3.6\times$ to $8.9\times$ model size with similar accuracy over multiple tasks, compared to Transformer.   

Published as a conference paper at ICLR 2022
DICTFORMER: TINY TRANSFORMER WITH SHARED DICTIONARY
Qian Lou, Ting Hua, Yen-Chang Hsu, Yilin Shen, Hongxia Jin Samsung Research America (qian.lou, ting.hua, yenchang. hsu, yilin.shen, hongxia.jin] Btnggiaaiin@ssmmunngg.omm @samsung.com
ABSTRACT
We introduce DictFormer with efficient shared dictionary provide compact, fast, and accurate transformer model. DictFormer significantly reduces the redun- dancy the transformer's parameters by replacing the prior transformer's parame- ters with compact, shared dictionary. few unshared coefficients and indices. Also, DietFormer enables faster computations since expensive weights multiplications are converted into cheap shared look-ups dictionary and few linear projections. Training dictionary and coefficients are not trivial since indices used for looking up dictionary not differentiable. We adopt sparse-constraint training with li norm relaxation to learn coefficients and indices in DictFormer. DictFormer is flexible support different model sizes by dynamically changing dictionary size. Compared to existing lightweight Transformers, DictFormer consistently reduces model size over Transformer on multiple tasks, e.g., machine translation, abstrac- tive summarization, and language modeling. Extensive experiments show that DietFormer reduces 3.6x to 8.9x model size with similar accuracy over multiple tasks, compared to Transformer.
INTRODUCTION
Transformer /Vaswani et al. 2017) has been widely used in natural language processing (NLP) for its superior capability in capturing long-distance dependencies. However, its good performance comes with the giant model size. For example, T5 (Raffel et al., 2019) with hidden dimension of 65K and GPT-3 (Brown et al.) 2020) with 96 transformer blocks have 11 billion and 175 bil- lion parameters, respectively. These large Transformers suffer from multiple severe issues, such as complicated learning and difficult deployment on mobile/loT devices. First, during the training of big transformer model, large training corpora (Raffel et al. 2019; Brown et al. 2020) or careful regularization (Merity et al. 2017]] Mehta et al. (021) are required. Furthermore, the trained model oveo-vorraaaee pareraatttteeeeed (Reid et al. 202. Zhao et al. 20211. The large model size with 11 billion and 175 billion parameters is beyond the capabilities of many edge devices including mobile devices and IoT'. Therefore, there is urgent need to design parameter-efficient and fast transformer model that eliminates redundant parameters and enables real-time NLP applications on the edge.
Weights sharing choice proved by ALBERT (Lan al. 2020) for designing compact and ef- ficient pre-trained transformer encoders, like BERT, on self-supervised learning task. However, directly sharing all weights an encoder-decoder transformer for a sequence-to-sequenc task like machine translation will dramatically decrease performance and accuracy (Reid et al.) (02T). Al- though a recent framework Universal Transformer (UT) (Dehghani al. (019) shows that vanilla transformer's accuracy can be improved by using recursive weight sharing, UT's high accuracy comes at the cost of deeper blocks or wider hidden dimensions, which significantly enlarges the computational cost and does not necessarily reduce the model size.
This paper introduces a new compact, fast, and accurate transformer architecture, DictFormer, that can be easily trained and deployed on edge devices. DictFormer depends on dictionary sharing and unshared linear projection coefficients instead of weights sharing. Specifically, a shared dic- tionary among all encoder/decoder blocks significantly reduce the parameter redundancy and therefore compresses the model size. Few unshared linear projection with coefficients on the shared dictionary enable each encoder/decoder block to have distinct layer-wise feature representations,

---

Published as a conference paper at ICLR 2022
thus improving the representation abilities compared to prior weights sharing. Also, DictFormer provides a method to dynamically control the layer-wise representation abilities by using the group- wise shared dictionary for layers with large- rigee-imension features, e.g., Feed-forward Network (FFN). Last but not least, we show that training dictionary and coefficients are not trivial since indices used for looking up dictionary matrix and train it with 1 norm relaxation and convert this sparse matrix into dense coefficients and indices during inference.
not differentiable. We convert coefficients and indices into sparse
Extensive experiments demonstrate that our DictFormer provides significant improvements over existing transformers on three summarization, and (iii) language modeling. For machine translation, on IWSLT 2014 German- English, DictFormer attains transformer performance with 8.9x fewer parameters and 2x fewer Multi-Adds; on WMT 2014 Gerlan-English, DictFormer brings about 4.9x model compression and 1.9x computation reduction; on WMT 2014 English-French, DictFormer obtains consistent performance improvements: 4.9X model compression and 1.9x less computation with similar score. For abstractive summarization, DictFormer reduces the model size by more than 4.7x on CNN-DailyMail dataset. For language modeling, DictFormer matches the performance of trans- former with 3.6x to 5.7x fewer parameters than transformer on WikiText- 103 benchmark.
tasks, (i) machine translation, (ii) abstractive
2 RELATED WORK AND MOTIVATION
Related lightweight Transformers. Several methods have been proposed to design lightweight transformers. The first line of research is reduce the transformer computation complexities by
redesigning self-attention mechanism including (Katharopoulos et al., 2020 Zhou et al. [022]] Ra-
ganato et al., [2020] You et al. 2020 Correia et al. [2019] Kaiser et al. 2018)) These methods cannot reduce model size. The second line of research is model compression, e.g., quantization (Prato al., 20200] Lou et al. 2020b), pruning (Behnke & Heatield 2020), low-rank factorization (Ma et al. 2019). and knowledge distillation (Wang et 2020). These two directions of research can be combined with our Dictformer. The third line of research is efficient architecture design (So)
eet al. 2019; Wu et al. 2020: Mehta et al.) 2021) by improving the expressiveness of transformers.
The forth line of research weights sharing (Xia et al.] 2019: Ma et al., 2019 Dehghani et al.) 2019 Reid et al., 2021: Takase & Kiyono 2021) by reusing parameters across transformer blocks.
Weights sharing cannot reduce computations. Our Dictformer falls into the third and forth category. We show that our Dietformer with Dictionary sharing can reduce both model size and computations.
Transformers with Weights Sharing. Weight sharing is surprisingly effective to compress model size for discriminate NLP models based on Transformer encoders, e.g., BERT. For example, prior work ALBERT (Lan et al. 2020) shows that even sharing all parameters across layers does not introduce any accuracy reduction. However, for generative sequence-to-sequuncc models based on transformer's encoders and decoders, sharing all parameters will significantly decrease accuracy on multiple standard machine translation or language modelling tasks (Reid et al. 2021 Takase & Kiyono, (021). To match vanilla Transformer's 's accuracy, multiple works (Reid et al. 202 Takase & Kiyono 202T Xia et al. 2019 Ma et al., 2019) only share weights across partial layers instead of all layers. However, partial weights sharing remarkably brings down the model size compression effect of weights sharing. Also, how to decide which layers should be shared in partial weights sharing is difficult due to the large and dynamic search space that is dependent on the specific tasks.
Transformer with all- parameters sharing such as Universal Transformer (Dehghani et al., 2019) matches or improves transformer's performance at the cost of a wider or deeper transformer archi- tecture. A wider transformer with a larger embedding dimension enlarges the model size and brings larger computations (Mult-Adds). A deeper transformer with more encoder/decoder blocks does not only increase model size, but also introduces more computations. Importantly, weights sharing techniques cannot reduce Mult-Adds numbers and training/inference time. Figure a) in Appendix shows the comparisons of Transformers with weights sharing and our dictionary sharing. Weights sharing techniques cannot solve the deployment challenges of transformers on resource-limited de- vices for real-time NLP applications.
2

---

Published as a conference paper at ICLR 2022
29
Universal Transformer
28
DictFormer
27
size
DeLighT
Transformer
DictFormer-tin tiny
Subformer Tied Trasnformer
26
Smaller circle means fewer operations
Lite Transformer
model
25
Evolved Transformer
24
MMult-Adds: 500M
1500M
4500M
5 10
15
20
30 Params (in million)
25
35
40
45
50
Figure I: DictFormer achieves similar BLEU score with prior works using fewer parameters and Mult-Adds WMT 2014 De-En translation. DictFoormrr is flexible and scalable, e.g., for mo- bile devices, DictFormer-tiny can reduce 2.2x model size of lite transformer (Wu et al. (020) that achieves state-of the- performance under mobile settings. Results of existing works come from their own implementations and #Params does not include embedding parameters, egg., Trans- former Vaswani et al. 2017) has 44M #Params.
Motivation of DictFrrmer with Dictionary Sharing.
Figure shows that although transformers with weights sharing including Universal Trans- former (Dehghani et al,, 2019). Tied Transformer (Xia et al. 2019). and Subformer (Reid et al. 2021] significantly reduce the model size vanilla transformer and lightweight transformer De- LighT (Mehta et al. 2021). they cannot reduce the #Mult-Adds and even suffer from larger #Mult- Adds. Particularly, Tied Transformer (Xia al., 2019) and Subformer (Reid et al. 202T) compress transformer model by 1.7x ~ 2.6x but cannot reduce #Mult-Adds. Universal Transformer (De- BLEU score improvement with 1.4x less parameters, but the #Mult-Adds is increased by 4.3x, thereby significantly prolonging the NLP inference latency and restraining the deployment real-time NL.P applications on edge devices. To enable the de- ployment of transformer models mobile devices, recent work Evolved Transformer (So et al,, 2019) and Lite Transformer (Wu et al. 2020) try to design new lightweight transformer architecture to meet the defined mobile settings, c.g., IO million parameters, but their tiny architectures suffer from a large accuracy decrease. As Figure shoos, Evolved Transformer and Lite Transformer lose 2.9 and 2.4 BLEU score compared to base transformer, respectively.
hghani al., 2019) achieves
DICTFORMER
Overview. We propose DictFormer with dictionary sharing to enable fast, compact, and accurate transformer. When matching transformer's accuracy, DictFormer can reduce more than 3.6x param- eters and ~ 3x Mult-Adds shown in Figure] [ which outperforms prior transformers with higher BLEU score. Also, DietFormer is flexible to compress model size given an accuracy threshold. For instance, when DictFormer matches the accuracy of Lite Transformer, it could further compress the model of lite transformer by ~ 2.2x. Given a --layer transformer model, we can easily transform it into DictFormer by converting all the weights in N blocks into shared dictionary and few unshared look-up coefficients, shown Figure 2 For example, N N-layer weights attention W.A. and FEN W.F. where E [0, N - 1]. in Transformer (Figure a)) are represented by smaller dictio- naries DA, DF that are shared by blocks, and N-layer unshared coefficients C.A and C. for DictFormer (Figure 2b)). Meanwhile, the attention and FFN operations in vanilla Transformer are replaced with shared-dictionary attention and group-wise shared-dictionary FFN whose details are introduced in our following contents. Dictformer reduces Transformer's 's #Params from O(d" N) to (d(mn + tN)). This O(dN/(m + tN)) model size reduction since dictionary size m d, coefficient size t << d, where the embedding size is d. Dietformer also reduces #Mult-Adds from Odd Nn) to OdN + t)). where n is input sequence length. The details on how to define and calculate #Params and #Mult- Adds shown in Appendix A.1
3

---

Published as a conference paper at ICLR 2022
Embed
Embed
Embed
Embed
W
Attention
Dicl-Attention
Attention
Dict-Attention
Attention
Dict-Attention
N
N
WF
FFN
4d
Dict-FFN
FFN
Dict-FFN
Encoder Block
Decoder Block
Encoder Block
Decoder Block
#Params: O(d"N)
#Mult-Adds: O(d"Nn)
#Params: O(d(m+tN)) #Mult-Adds: O(dNn((++))
(a) Transformer: Weights w/ w/ shared
(b) DictFormer: Dictionaries (D* D²) shared by blocks
Figure 2: Our DictFormer replaces N-layer unshared, large attention and FFN weights W,A. WF E N - 1]) in transformer with smaller, shared dictionaries D D and coefficients C:², C.P The Mult-Adds operations between weights and inputs Transformer are also reduced by our dictionary look-ups and few linear projections with coefficients.
Shared-dictionas Attention. Given a N-layer transformer model, we define that Qi, Kü, and V, are the i-th layer query, key, and values. Same to Transformer (Vaswani et al. 2017), we also use equation D to calculate the attention scores once we have query, key, and values.
Attention(Q.. (Q,, K,, V.) softman Qi KT Vd
(1) Our DictFormer utilizes equation instead of A/ MultHHHadd Qu, K, V/,) MH; W used in Transformer /assaani et al. 2017) compute multi-head values, where M HH; is derived according to equation 3 and Da CT. and T are used to linearly project MH, instead of WTO The attention value head, for each head j in layer is computed by equation iintead of head' Attention(Q w&: K₁ wik, V4 W/) used in existing transformers.
V
MultitHead( "DDM,,, .P))
(2)
MH; Concal(teeud
(3)
SD(K,, head.- = Allention(SD(Q.. ( DA, "D c.
(4)
The reason why the shared attention dictionary D indices I, and coefficients C₁ to replace the larger and unshared weights W.O. W w.,, and w.O; used in previous transformers that the linear projections lookup of D with coefficients C,² can have similar representation abilities as W where X represents inputs type, e.g., X means input is query. To be specific, DA, It, and C, have size of x m t d, and & x d, respectively. A and t are the dictionary size and coefficient size in attention. As equation 5 shows, the linear projections in transformer's 's attention, e.g. W.O. Q3. are replaced by our lightweight shared dictionary projection function (SD), e.g., SD(Q,, DA C., ..(2), derived by equation and equation 7 SD(Q,, C. replaces 2 Q, since w.O: can be replaced by looking up D and few linear projections with coefficients 3(a) introduce the details of
che without accuracy decrease. The following paragraphs and Figure
look-up and scaling using equation 6 and equation 7
Q -w + SD(Q,, DA DA,
4

---

Published as a conference paper at ICLR 2022
In equation 6 the lookup of D by indices 1 is defined by [:, 19,, [t, dall that fetch the I collumn vector from DA; Unshared linear projection that is used to enlarge the represen-
tation abilitt of shared dictionary depicted by W.O: EE G it, dal @ [ 1 [t, iull where @ represents scaling fetched vector from a dictionary with scalar in coefficients. Therefore, lin- early projecting 2j with W; according W&,,, Q, Q2 is replaced by W.O) Qj in equation when we find proper DA, 1 C., to meet that W and W; have similar representation abilities, e.g., they have matched accuracy. Directly generating W 4 and multiplying with Q3 potentially increases the computations. To tackle this problem. we compute the multiplications between Q: and dictionary D as Og first according to equation 7 and reuse O₂ for the following look-ups and scaling with C.
SD,Q,, D ,,,,,,,,, ,1,1) = Q
Σ
c
[1, id]
D
t=1
Q, D [: 1, iall 0. it
(6)
Σ 0,1: 1 It اaهl o I
ia), E 10, d
O b] Q D lt. b). E [1,m*]]
We use Figure
3 ata) to show an example of equation 5 about how to lookup dictionary with in-
Index IF
Coaffi.
Coeffi. CF
Index 1
Coefficients CP
7. 271
10.3,
0.1,
[0.1. 0.6, 0.3]
3
11.33 (3)
10.2, 03,
0.3
0.1
0.8
13
0.2
0.3
0.5
W/+
0.6
0.3
W
O
d N-layer Weight W
d N-layer Weight Wf
Shared Dictionary D
Group-2 Dictionary D
(a) Replacing weight with shared-dictionary. index, coefficients
(b) Replacing weight with group-wise dictionary and coefficients
Figure 3: (a) An example of looking up dictionary with indices and scaling it with coefficients. This lookup and scaling able to reconstruct N l-layer weight. (b) An example of looking up group-wise dictionary and its scaling for farge-dimension representation projection.
dices and scaling it with coefficients. This process can create a N N-layer weight QQ., For example, given the shared dictionary DA, to generate the first column of weight W.?. the first column of index matrix r. is taken out, e.g., [1, 13, 35), and is used as indices to fetch the the corresponding columns from DA, e.g., [+ A [:][13]. D [35).. Then the first column of coefficients cº, e.g.. [0.2, 0.3, 0.5), are multiplied with D [.] [1]33] [+ respectively, and the sum of multiplication works as the first column of 8 In this way, weights W in attention with 4d2 N pa- rameters are compressed into m 1 and CO2 with size dm ANV. For example, DictFormer can reduce 8.7 x model size when d = 512, m = 256, t = 24, N 6. More calculation details can be found at appendix A.
Group-wise Shared-
FFN. The FFN in prior transformer includes two-layer compu- tations: (i) Fi = maz(0, X; W.
Fi + b1), (ii) Fy Fi W. + bg Instead of the regular linear projections X; W. + 6, and F1 w. + b2. DictFormer uses new lightweight pro- jection called group-wise shared dictionary projection (GSD), i.e., GSD(X, D, of 1.5 G) and
5

---

Published as a conference paper at ICLR 2022
Shared O
11. 13, 36
Index
10.2, 0.3, 6.5)
Coefficients C
13
35
0.2
0.3
0.5
m
Attention Results S
[0.2, 0.3, 0.5)
m
Sparse Coefficients Z
Since index / in DictFormer is not differentiable, we train sparse coefficients Z instead of jointly training I and C. After training, sparse Z is converted to and C' for deployment.
Figure 4:
GSD D. C. I,.h. G). in equation and equation 9 to compute the FFN. Here GSD projection is same as SD in shared-dictionary attention when G 1. In SD projection, it works well to replace a N weights of d x d with a x m dictionary where each column of dictionary multiplied with scalar in coefficients. However, when the column of dictionary large, e.g., 4 in the second layer of FFN, SD projection is difficult to achieve an accurate model since an vector of dictionary with 4 d, e.g. 2048, multiplied by same scalar is not flexible enough. To increase the flexibility of SD projection and improve performance, GSD firstly divides each column of dictionary into G groups equally and assign different scalars to multiply the numbers in each group.
mar(0, GSD D. C. 1/ (G).G)+ bb)
(8)
F. = G, D.C. (2) ()) +
(9)
Equation 10 and equation 11 show the computations of GSD. Dictionary DF is divided equally into G groups and the g-th group is defined as D [II[Inde :], where Inder, = [[g a 4d and E G1, Also, the multiplication result between g-th group dictionary and input is defined as [) shown in equation
11 where b E [1, m. Different groups of dictionary share the same look-up indices 1, The F queried vectors in each group will be scaled by its corresponding coefficients C. F2 and the scaled results are then accumulated to derive the g-th GSD result shown in equation 10. In addition, G group-wise GSD are further summed to compute the final GSD result.
G P2 0 1. id] @ of (1
GSD(X, 3
ΣΣ g=
(10)
4d
4d
O0 = X[: Inder,]] [Inder bl E Gl.. [1, m1, "1, Indezz 1(9
(11)
T
Figure 3bb) shows an example of group-2 shared dictionary for the the second layer of FFN. The dictionary Fa equally split into two parts that share the same indices I,"h but each group has unique coefficients, e.g., C. pagmi and - FFa= The first group and the second group use the corre- sponding coefficients, e.g., |0.3, 0.1, 0.6] and [0.1, 0.6, 0.3), scale the queried dictionary vectors, respectively. The accumulation of scaled vectors can be used to represent the N N-layer weight F
Training DictFormer via Constraints and Relaxation. DictFormer replaces transformer weights with linear projections of dictionary, thereby having three-step computations including small pro- jections, lookup and scale. DictFormer computes small multiplication between input and dictio- nary, generating intermediate variable O, looking up O and scaling it with coefficients. To train the DictFormer, one should jointly optimize dictionary, index / and coefficients C. Directly training the Dictformer leads to a combinatorial optimization problem since index / is non-continuous. Al- though one could use AutoML, such as evolutionary method and Reinforcement learning to jointly leam dictionary. I and C, these methods suffer from a large training time with low performance. To workaround the training of index I, we reformulate the method to a regular linear projection with sparse constraints, which can efficiently train the DictFormer.

---

Published as a conference paper at ICLR 2022
We convert the index I; and coefficients C₁ into sparse coefficients Z. so that the training of index I, and coefficients C2 is replaced with training sparse coefficients Z. During the deployment phase,
the sparse coefficients Z are reversely transformed into index I, and coefficients C, shown in Figure
4 In particular, the shape of coefficients C' and index / is 1 x d x N. The shape of sparse Z two steps derive Z: Initializing all elements 0 and copying C
m x d x N. There
elements to Z according to the index values in 4. For example, since the first column of 1 and Ci [1, 13, 35] and [0.2, 0.3, 0.5). all entries of the first column in 2 are zeros except the 1st entry, 13th entry, and the 35th entry are 0.2, 0.3, and 0.5. Therefore, the lookup and scale of O, can be converted into the matrix multiplication between O and Z, shown in equation [2]] The new coefficient C' has a sparsity constraint that requires that the non-zero elements (o norm) of each column is LA << m A shown in equation
13
Now we can train Z with || [:, ddllloo t instead of training and C.
O, itl O (t = O. 2
(12)
1 1:, fal Ilio = ta
(13)
The lo norm sparsity constraint in equation is non-differentiable and we cannot directly train Z. To achieve the training with sparsity, we inspired by existing sparse training methods Tsuruoka eet al. (2009); Li et al. (2016) to firstly loose this constraint to 1 norm constraint shown in equa- tion| 14 to penalize the non-zero parameters and clip the unimportant values that are near to zeros using equation [6] To be specific, the gradients of coefficients are calculated by equation [3] where parameter À is used to control the trade-off between the gradient of loss L, 5L and 11 norm sparsity constraint.
a7:
||
Σ
1 J. fallls
(14)
Ed
8(L. + 111211b))
8L
+ XZ)
(15)
5Z
52
Equation i6 is a threshold function during training. During the backpropagation, the derivative of threshold function is 1 if |a| > value(p), O otherwise. During the forward phase, Equation 16 can be used to globally clip the near-zero values to zero given a ratio p. where value(p) derives the value at ratio p in the ascending order. One also can force that at most / of the elements in each column of Z are non-zero while the rest are set to 0 by letting value(p) is the 1-th largest value of each column.
I. if |z| > value(p). otherwise.
m(x))
$0000
(16)
EXPERIMENTAL METHODOLOGY
Machine Translation Dataset. Three machine translation benchmarks are tested: IWSLT' 14 -nnlashE (De-En), WMT' 14 English to German (En-De), and WMT' English to France (Cn-Frr. For IWSLT' 14 De-En, we adopt the same settings in Wu et al. (020)) with 160K training sentence pairs and 10K joint byte pair encoding (BPE) vocabulary in lower case. For WMT 14 En-De, our models are trained with 4.5M sentence pairs, validated on newstest2013, and tested on newstest2014. Also, a 32K joint source and target BPE is used in the vocabulary. For WMT" 14 En- Fr, we follow the setup in /w et al. 2020) by training models on 36M training sentence pairs from WMT' 14, validating size of 40K and is based on a joint source and target BPE factorization. The evaluation setting used in Vaswani et al. (017)) is utilized with a beam size of 4 and length penalty of 0.6. We replicate the same BLEU calculations in Wu et al. (2020) with case- sensitive tokenization. The last 10 model checkpoints are averaged for testing and the lowest perplet-erylexity model is picked for validation.
newstest20 22 and 2013, and testing newstest2014. The vocabulary has
Abstractive Summarization Dataset. We evaluate DictFormer on CNN-DailyMail dataset (Chen et al. 2016) that has 280K news articles with multi-sentence summaries. We follow the settings
7

---

Published as a conference paper at ICLR 2022
in (Wu al. 2020). truncate the articles to 3000 tokens, 30K BPE vocabulary and F1- Fl-Rouge as the metric.
Language Modeling Dataset. We also evaluate DictFormer WIKITEXT-103 (Merity et al. 2016) that has 260K BPE vocabulary and contains 103M/217K/445K tokens for training, valida- tion, and testing. Language modeling performance about perplexity (ppl) is reported.
Architecture and Evaluation. For machine translation tasks, we follow the sequence-to- to-nneeeet--suueeeee transformer (Vaswani et al., 2017) with encoder and decoder to develop dataset, the general settings are same as (So et al. 2019). For WMT dataset, the setting is based on (Wu et al. 2020). Also, the same model for machine translation is used for summarization task. For language modelling, we use the transformer- based decoder architecture of (Baevski & Auli 2019) but with 512 model dimensions and 12 blocks that same (Wu al.) 2020). Fairseq's transformer implementation (Ott et al., 2019) is used the backbone for the baseline model.
model. For IWSLT
In our DictFormer architecture, encoder and decoder have three dictionaries, respectively, i.e., en- coder attention dictionary enee encoder F1 dictionary D encoder F/2 dictionary D.2 and enee
decoder attention dictionary dee: decoder F1 dictionary D.: decoder F2 dictionary D Each of them is shared by all blocks in encoder or decoder. For example, D is shared by all attention blocks in encoder. We study the effects of DictFormer with various dictionary size from 80 to 240 and different coefficient size from 12 to 84 shown in Figure g One can pick up dictionary size and coefficient size to control the trade-off between accuracy and model size according to the require- ments. For example, one could get the results in our tables when dictionary size is set m 2.3 m 2.5 F2 2 and t d F
10*
15
In this paper, #Params means parameter numbers of transformer without including embedding lay- ers; #TParams shows parameter numbers of model and embedding layers. The Mult-Adds caleu- lation is tested when model is used to translating two sentences with the length of 30 in default (same to previous work Wu et al. (2020).
Table 1: Comparison with state-of-the-art transformers on machine translation tasks. For a fair com- parison with existing tiny transformer, We follow Lite Transformer (Wu et al. 2020) to scaling down the hidden size to meet the deployment requirements on mobile settings. For example, Transformer uses a 128 hidden size and 2.8M #Params; #Ops represents Mult-Adds and Entries with + shows the average across three independent runs.
IWSLT' De-En BLEU
WMT'I4 En-De
WMT En-Fi
#Parann Ralios
ABLEU HOps #Params Ratios
BLEU
ABLEU #Ops
BLEU
ABLEU
Transformer (Vaswani
87M
Universal Trins (Dehghom
2019
2.1
1.5x
30.3
+2.5
152M
26M
x
22.4
+1.1
193M
35.8
+2.2
Tensorized. Trans Mia
2019
14M
2x
28.0
0.2
1.7M
1.6%
21.2
-0.1
33.4
.0.2
Lite Transformer (D
2
1.2x
29.610.4
+1.8
46M
1.8
21.810.3
+0.6
47M
35.320.4
+1.7
DeLighT MMhha al.
2021
1.1
2,5x
28.910.2
+1.1
49M
1.6M
18x
21.810.1
+0.5
64 M
34.2:0.22
+0.6
Subformer
33.5-0..4
DictFormmr
0.3
8.9% 30.0+0.3
+2.2
32M
0.6 M
49x
22.0+0.2
+0.7
46M
35.6=0.3
+2.0
5 RESULTS
Machine Translation. We first report the machine translation results and compare them with prior works. Our baseline transformer model settings are line with Lite Trans. (Wu al., 2020) which provides the best results under mobile settings. Our DictFormer generally outperforms the state-of- the-art transformers on IWSLT'1 De-En, WMT' En-De, and WMT' 14 En-Fr. Table represents the quantitative results of our DictFormer. Compared to Transformer (Vaswani al. 2017]. Dict- Former compresses 8.9x model size, reduces 1.9x #Mult-Adds, and improves 2.2 BLEU score on IWSLT' 14 De-En, which achieves better trade-off between performance and model size than pre- vious works. Similar to IWSLT' 14 De-En, our experiments on larger corpora including WMT' 14 En-De and En-Fr also show DictFormer obtains Transformers. For instance, only our DictFormer can compress more than 6x model size over Transformer without decreasing any model performance under mobile settings.
compact and accurate model than existing
8

---

Published as a conference paper at ICLR 2022
Table Comparison with state-of-the-ari transformers on abstractive summarization and Language modeling tasks. #Ops is calculated by longer sequence with the input length of 100. Entries with + represents the average across three independent runs.
CNN-DuilyMail for abstractive summarizztion #Params
WIKITEXTI03 for language modeling
#Ops
Ratios
Valid ppl.
Test ppl.
#Ops
#Params
Trans. /Adaptive
41.4
18.9
38.3
3.60
41.4M
1.0
23.2
24.0
50.3G
37.8M
10x
Transformer
21.4
DelighT
23.5
24.1
~13
L1X
Subformer
41.6
19.2
41M
1.01 x
20.8
21.1
~50
~18M
2.1x
DictFormer
41.32:0.2
18.9::0.1
38.3400.1
0.9G
8.9M
4.7%
21.340.2
22.24:0.2
20.3G
10.0M
388
Abstractive Summarization and Language Modeling. We then report the abstractive summariza- tion and language modeling results and compare them with prior works. Abstractive summarization model follows the transformer model, but language modeling model follows the adaptive inputs model (Baevski Auli 2019). Our DictFormer consistently outperforms prior transformers on both summarization and language modeling tasks. Table 2 represents the quantitative results of our DictFormer. For abstractive summarization on CNN-DailyMail, DictFormer achieves similar FI- Rouge score with Transformer but requires 4.7x less model size and 4x less computations. For language modeling on WIKITEXTI03, compared to adaptive inputs model (Baevski & Auli 2019), DictFormer compresses 3.8x model size with less computations and matched testing perplexity.
Sensitive Study and Ablation Study. We study the DictFormer's 's performance under different model size and hyper-parameters Appendix. We also use table 3 Appendix to show the ablation study of DictFormer's 's techniques. The performance of DictFormer improves with an increase in the number of model parameters, across different corpora. DictFormer achieves similar BLEU score with Transformer using fewer model parameters. Although DictFormer's performance improves with an increase in the size of dictionary and coefficients, the model size is also enlarged. So one should pick up proper dictionary coefficient size according to accuracy requirements.
CONCLUSION
We propose DictFormer with dictionary sharing to enable fast, compact, and accurate transformer. DictFormer significantly reduces both model size and computation parameters of prior transformers with similar performance. Also, DietFormer consistently improve the performance multiple tasks, such as machine translation, abstractive summarization, and language modeling with similar model size. DictFormer code is available at htthtss://ggtt
com/ SamNLP ct

---

Published as a conference paper at ICLR 2022
REFERENCES
Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. In International Conference on Learning Representations, 2019. hotLhttsss///peeeeee net/
Maximiliana Behnke and Kenneth Heafield. Losing heads in the lottery: Pruning transformer atten- tion in neural machine translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2664-2674, 2020.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess. Jack Clark, Christopher Bemnr,, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. CoRR, abs/2005. 14165, 2020. URL https: hattss://ariiv. org
www.//gg..b../.oo.///....t...e/e abs 20t.
Danqi Chen, Jason Bolton, and Christopher D. Manning. A thorough examination the cnn/daily mail reading comprehension task. CoRR, fbs//6606002888 2016. URL http:://axxv..co
abs/ 82888
Gonçalo M Correia, Vlad Niculae, and André FT Martins. Adaptively sparse transformers. arXiv preprint 1000001909.00015,, 2019.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. In International Conference on Learning Representations, 2019. URL https:
//openzeview www.:nne//eeeeee Hiy.::/
Feng, Qian Lou, Lei Jiang, and Geoffrey Fox. CRYPTOGRU: Low latency privacy-preservic text analysis with GRU. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pp. 2052-2057, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 00.18653/11/021..com www.leeninlnnnmmem 1566
Lukasz Kaiser, Samy Bengio, Aurko Roy, Ashish Vaswani, Niki Parmar, Jakob Uszkoreit, and Noam Shazeer. Fast decoding in sequence models using discrete latent variables. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2390 2399. PMLR, 10-15 Jul www.kaaiieelllaammammmmml html
2018. htttttt///poooeeeeee .mm.mlr
mlr prwpppessssseee
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers mns: Fast autoregressive transformers with linear attention. In International Conference on Ma- chine Learning, pp. 5156-5165. PMLR, 2020.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations, 2020. httphtttss//opppeevieee net www.:eeetforrun ffoumm? id=HleA7ARte
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint 16Xiv:1608.088700 2016.
Qian Lou and Lei Jiang. She: A fast and accurate deep neural network for encrypted data. In Advances in Neural Information Processing Systems, pp. 10035-10043, 2019.
Qian Lou, Bo Feng, Geoffrey C. Fox, and Lei Jiang. Glyph: Fast and accurately training deep neural networks on enerypted data. CoRR, abss/111..7110 2019.
Qian Lou, Song Bian, and Lei Jiang. Autoprivacy: Automated wise parameter selection for secure neural network inference. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.). Advances in Neural Information Processing Systems, volume 33, pp. 8638-8647. Curran Associates, Inc., 2020a.
10

---

Published as a conference paper at ICLR 2022
Qian Lou, Feng Guo, Minje Kim, Lantao Liu, and Lei Jiang. Autoq: Automated kernel-wist neural network quantization. In International Conference on Learning Representations, 2020b
Qian Lou, Yilin Shen, Hongxia Jin, and Lei Jiang. Safenet: A secure, accurate and fast neural network inference. In International Conference on Learning Representations, 2021.
Xindian Ma, Peng Zhang, Shuai Zhang, Nan Duan, Yuexian Hou, Ming Zhou, and Dawei Song. A tensorized transformer for language modeling. H. Wallach, Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, E. Fox, and R. Gareett (eds.), Advances in Neural Information Processing Sys- tems, volume 32. Curran Associates, Inc., 2019. URL https: pttptttttttttpeeiiiiiiiii
http://ooceedeeeeoooeooo ngs. neurips.
Paper .ddfrrpdff
Sachin Mehta, Marjan Ghazvininejad, Srinivasan Iyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Delight: Deep and light- transformer. In International Conference on Learning Represen- tations, 2021. URL http:ttttts////eeeeeeeie hwtp:/ewwwwwwweeeeeeeeeeeeeeeeee www.eoeorridduuuuuueeuuuu
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. CoRR, htss/1609.08443 2016. URL.
aww.aa/......ee..//////../.....e 07843
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing lstm lan- guage models. arXiv preprint 1708.v1708.000 02082, 2017.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. arXiv preprint arXiv:1904.01038, 2019.
Gabriele Prato, Ella Charlaix, and Mehdi Rezagholizadeh. Fully quantized transformer for machine translation. In Findings of the Association for Compulational Linguistics: EMNLP 2020, pp. 1- 14, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020 findings-emnlp.1. URL hattpss:/cclanthologg.ooo wttpl/oyyyorg//00...... findings--mmm1ppp1
2020
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text
transformer. CoRR, abs/1910.10683,, 2019. URL http///axvvvv/v org abs wwwpaass....... 1t1//1990110
Alessandro Raganato, Yves Scherrer, and Jörg Tiedemann. Fixed encoder self-attention patterns in transformer-based machine translation. CoRR, abs/2002.10260. 2020. URL httpss////xxiii...
org abs/ /b////bs20022. 2002. 10260
Machel Reid, Edison Marrese- Marrese-Taylor, and Yutaka Matsuo. Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformers. In Findings of the Association for Computa- fional Linguistics: EMNLP 2021, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
David R. So, Chen Liang, and Quoc V. Le. The evolved transformer. CORR, abs//abss..../ 1911.111111111 2019. URL htt.hhttp//axxv....o httpg//abs19111.11111
Sho Takase and Shun Kiyono. Lessons parameter sharing across layers in transformers. CoRR, 06022
abs/2104.06022, 2021. URL /ttp:///////xxxxor/////////
Yoshimasa Tsuruoka, Jun' Tsujii, and Sophia Ananiadou. Stochastic gradient descent training for Ll-rgullarized log-linear models with cumulative penalty. In Proceedings of the Joint Confer- ence of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pp. 477-485, Suntec, Singapore, August 2009, Association for Computational Linguisties.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In L. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gameet (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL /ttp:tttppp//pppcceiiisss.. http://sss..eerrs.oooooo//////// 3f.5ee2434476e (bdo::eeeeee1000111111111111c Paw.asaaa-aapr.ppp. pdf

---

Published as a conference paper at ICLR 2022
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers. arXiv preprint arXiv:2002.10997, 2020.
Zhanghao Wu, Zhijian Liu, Ji Lin, Yujun Lin, and Song Han. Lite transformer with long-short range attention. In International Conference on Learning Representations, 2020. URL https:
//openzeview.
HKPH
Yingce Xia, Tianyu He, Xu Tan, Fei Tian, Di He, and Tao Qin. Tied transformers: Neural machine translation with shared encoder and decoder. In The Thirty-Third AAAI Conference Artifi- cial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium Educational Advances in Artificial In- telligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 February 1, 2019, pp. 5473. AAAI Press, 2019. ISBN 978-1-57755-809-1. URL https:/ org ndex. http//AA///ttiille//veee// 44477
Weiqiu You, Simeng Sun, and Mohit lyyer. Hard-coded gaussian attention for neural machine translation. arXiv preprint arXiv:2005 00742, 2020.
Changsheng Zhao, Ting Hua, Yilin Shen, Qian Lou, and Hongxia Jin. Automatic mixed-precision quantization search of bert. Proceedings of the Thirtieth International Joint Conference on Artifi- cial Intelligence, Aug 2021. doi: 10.24963ficcai222214472 URL http::/dxx doi. org..ol/o..... 24963/1 2021///7722222222
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting, In Proceedings of AAAI, 2021.
12

---

Published as a conference paper at ICLR 2022
A
APPENDIX
A.1 DICTFORMER BENEFITS WITH DICTIONARY SHARING
The block comparison of Transformer and our DictFormer.
d
Multi-head Attention
W
W
SD (Eq.
D
SS (Eg.
SD Attention
Paras #:
8Ndt
Attention
Attention
Paras #: 4Nd? Ops: 4Nnd++dn
Concat
Ops: Nnd(m²+t")
Concat
d]
W
Add
d
Add
Fi
Paras #:
d
Paras #: 8Nd?
GSD FFN
3dm" +32Ndt
m
Lookup SD (Eq. 5)
W
Ops: 8Nnd
FFN
6: 4d
Ops:
W
Nnd +32f dn3
Lookup (Eq. 9)
d
Add
Add
(a) Transformer block
(b) Our DictFormer block
Figure 5: The block comparison of (a) transformer and (b) our DictFormer.
Figure 5 further depicts the comparison of transformer encoder block and DictFormer encoder block. The i-th Transformer block including attention and FFN has four weights W,A and W.". W,A has four weights including query W.O., key W:K,, value W.V. and output W thereby having df parameters and ndd Mult-Adds given a sequence of size n. W.F has two parts: WFFi and W.F: N-layer FFN blocks have d2 parameters and 8N Mult-Adds operations. contrast, Dict- Former significantly reduces the parameters and Mult-Adds of Transformer. Specifically, each Dict- Former block replaces the attention and FFN with shared dictionary (SD) attention and group group-wise shared dictionary (GSD) FFN. Linear projection is performed by looking up dictionary D using 5q.5. SD Attention has 8N + Ad parameters and Nnd(mn + (4) operations. GSD FFN has 3dm + 32 parameters and Nnd + 321F) operations, where t, coefficient and dictionary. To simplify the calculations of #params and #Mult-Adds, we can let ta = tf, m mf. Dietformer reduces Transformer's 's #Params from Odd O(d(m + tN)). This is a OddN/(m + tN))x model size reduction since dictionary size m < d, coefficient size t << d, where the embedding size is d. Dictformer also reduces #Mult-Adds from Odd Nn) to OdNN((m + !)).
the size of
A.2 THE COMPARISON BETWEEN WEIGHT SHARING AND DICTIONARY SHARING
Weight sharing is surprisingly effective to compress model size for discriminate NLP models based on Transformer encoders, e.g., BERT. For example, prior work ALBERT (Lan et 2020) shows that even sharing all parameters across layers does not introduce any accuracy reduction. However, for generative peuuence-to-seqeeee
models based on transformer's encoders and decoders, sharing all parameters will significantly decrease accuracy on multiple standard machine translation or lan- guage modelling tasks (Reid et al. [021 Takase & Kiyono 2021). To match vanilla Transformer's accuracy, multiple works (Reid al. 2021 Takase & Kiyono 2021 Xia et al. 2019 Ma et al.) 2019) only share weights across partial layers instead of all layers. However, partial weights sharing remarkably brings down the model size compression effect of weights sharing. Also, how to decide which layers should be shared partial weights sharing is difficult due to the large and dynamic search space that is dependent the specific tasks.
13

---

Published as a conference paper at ICLR 2022
Embed
Embed
Embed
Embed
W
Attention
Dict-Attention
Attention
Dict-Attention
Attention
Dict-Attention
N
X
N
N
D
Dicl-FFN
4d FFN
Dict-FFN
Encoder Block
Decoder Block
Encoder Block
Decoder Block
Wider: d" large model size Mult-Addstt
m < d & tiny blocks + Tiny model Mult-Addsft
Deeper: N" > N > Mull-Addsif
(a) Transformer with weight sharing: wider deeper (inefficient)
(b) DictFormer: efficient transformer with dictionary sharing
Figure 6: (a) Transformer with weights sharing. It contains three parts including embedding, N encoder blocks and N decoder blocks. Each encoder/decoder block contains attention and Feed- Forward Network (FFN). The embedding size is d and FFN feature size is 4 x d", Weights in the i-th Attention and FFN are denoted as W:A and W. To match or improve accuracy of transformer w/o weights sharing, transformer w/ wights sharing should be wider (d* > d) deeper (N* > N), where d and N are the embedding size and blocks number of transformer w/o wight sharing. (b) Our DietFormer with dictionary sharing. W.A. and W are represented by smaller dictionaries D DF, and coefficients C²¹ and C. where dictionary size m < d. coefficient size << d.
Transformer with all- pllaaarmeterss sharing such Universal Transformer (Dehghani et al. 2019) matches or improves transformer's performance at the cost of a wider or deeper transformer archi- tecture. A wider transformer with larger embedding dimension enlarges the model size and brings larger computations (Mult-Adds). A deeper transformer with more encoder/decoder blocks does not only increase model size, but also introduces ing techniques cannot reduce Mult-Adds numbers and training/inference time. Figure (((a) shows the comparisons of Transformers with weights sharing and our DictFormer with dictionary sharing. Weights sharing techniques cannot solve the deployment challenges of transformers on resource- limited devices and obtain real-time NLP applications. To reduce both model size and computations of Transformers, we introduce DictFormer with dictionary sharing instead of previous weights shar- ing. particular, DietFormer shares dictionary across all layers so there no need to decide which layers should be shared. Also, DictFormer with few unshared look-up coefficients does not require a wider embedding size or deeper encoder/decoder to improve accuracy. Rather than compressing model size by parameters sharing, DictFormer with dictionary sharing can also enable computations sharing to reduce running latency. Therefore, our DietFormer provides a compact, fast, and accurate transformer model for sequence-to-see se-quence NLP tasks.
computations. Importantly, weights shar-
A.3 TRAINING SETTINGS
To fairly compare with our baselines, all of our training settings are in line with (Wu et al., 2020). For machine translation tasks, dropout of 0.3 used and the dropout ratio is linearly scaled down when we shrink the dimension of the embeddings for WMT datasets. The learning rate linearly warms up from 10-7 to 10-3 followed by a cosine annealing with single cycle and Adam optimizer same as (So et al. 2019) and (Wu et al. 2020). Inverse linear square root learning rate scheduling used for IWSLT En-En. The summarization training settings are the same as machine translation. To train models on language modeling task, we follow the training setting in (Baevski & Auli, 2019), but decrease the dropout rate by half in FFN layers. The training experiments WMT. summarization, and language modeling conducted 8 NVIDIA Tesla V100 GPUs. IWSLT De-En is trained on two GPUs. Machine translations tasks are trained for 50K steps, but language modelling tasks trained for 286K steps. We further describe deployment settings in Appendix.
14

---

Published as a conference paper at ICLR 2022
35
35
34
Transformer
32
33
38
Test PPL
30
DictFormer
31
36
30
25
29
Transformer
Transformer
28
DictFormer
DictFormer
27
30
20
0 2
10 14 16 Parameters (in million)
10
20
40
30 40 50 60 70 Parameters (in million)
Parameters (in million)
(a) IWSLT'14 De-En
(b) WMT'14 En-Fr
(c) WIKITEXT-103
Figure 7: The performance of DictFormer improves with an increase in the number of model pa- rameters, across different corpora.
A.4 DEPLOYMENT SETTINGS.
Deploying deep learning models including transformer powerful cloud-based servers ser- vice suffers from data privacy issues when users' data private or confidential (Lou Jiang 2019 Lou et al. 2019). In addition, the inference cost of privacy-sensitive applications that performed over encry encrypted data using fully homomorphic encryption is prohibitively expensive (Feng et al. Lou et al. 2021 2020a). (i) Mobile settings. Scaling up transformer models leads to higher machine translation performance, but those large architectures are not suitable for real-world appli- cations, especially when we should deploy them on edge devices where computation and memory size are highly constrained. Multiple prior works like (Wu et al. 2020) formalize the deployment of lightweight transformers on mobile devices by defining the mobile settings regarding the amount of computation and parameter numbers. Particularly, given the ARM Cortex-A72 mobile CPU with computation performance of 48G FLOPS, the computation constraint for machine translation should be under 500M Mult-Adds and the parameter number should be around 10M with sequence of 30 tokens that are the general length machine translation. (ii) System-independent settings. Other than deployed mobile devices, DictFormer also supports system- inneeendddeenneent deployment. In this setting, we directly choose prior transformer architectures like DelighT (Mehta et al. (202I) and Subformer (Reid et al., 2021) as baselines and compare DietFormer with them under the the same performance constraint.
2021
A.5 SENSITIVITY STUDY
We study the DictFormer's performance under different model size as Figure 7 shows. This is important since inneeeiiddeeedeet settings require scalable DictFormer. The performance of Dict- Former improves with an increase in the number of model parameters, across different corpora. DictFormer achieves similar BLEU score with Transformer using 3x to 10x less model parame- ters. In Figure 8 we also report the hyper-parameter's effects on DictFormer's performance with WMT' 14 En-Fr dataset with embedding size d 512. Figure X a) shows the dictionary size study when the coefficient size is fixed to 60. All dictionaries share the same size, and all coefficients also share same size. The Figure 8 (b) shows the coefficient size study when the dictionary size is fixed 240. Although DictFormer's s performance improves with an increase in the size of dictionary and coefficients, the model size is also enlarged. So one should pick up proper dictionary coefficient size according to accuracy requirements.
A.6 ABLATION STUDY
We also use table 3 to show the ablation study of DictFormer's techniques. Transformer Vaswani et al. (2017) adopts the base architecture with hidden size 512. Transformer-Flatten flattens the ferd-forward hidden size transformer from 2048 to 512, thus reducing the parameters from 44 based on Transformer-
millions to 25 millions without hurting the accuracy. Only-Dictionary Flatten and uses unshared dictionary architecture. This method achieves 33.2 BLEU with 9.7 mil- lion model parameters. The #TParams is 14.3 million and 27.4 million when adding embedding parameters on IWSLT' 14 De-En and WMT' 14 En-Fr, respectively. When sharing dictionaries in at- tention, Shared-A ttention achieves similar BLEU score with 4.2 million model parameters. Mean-
15

---

Published as a conference paper at ICLR 2022
Parameters (in millionn
Model Size
BLEU
Parameters million!
Model Size
BLEU
6
40
4
3
m
2
100 140
180
200 220
240
12
24
36
48
60
72
84
Dictionary size (m)
Coefficient size (t)
(a) Dictionary study
(b) Coefficient study
Figure 8: Increasing the size of dictionary (a) and coefficients (b) improves DictFormer's perfor-
mance. (a) and (b) are tested on 1M En-Fr dataset for machine translation task.
while, Shared-FFN only shares FFN using group-wise shared dictionaries (G is group number). DictFormer that shares both FFN and attention with G=2, denoted by Shared-both, requires 2.6M model parameters and matches the accuracy of unshared DictFormer. Improved-Ene Hoproved-Embed represents that the embedding layers Shared-both further compressed by using existing method Mehta et al. (2021) so that the #TParams is reduced to 5.1M from 7.IM.
Table 3: Ablation study of DictFormer techniques on IWLS' 14 De-En and WMT' En-Fr. Entries with + represents the average three independent runs.
IWSLT 14 De-En
WMT' En-Fr
#Params
#TParams
BLEU
#Params
#TParams
BLEU
Transformer Vaswani et al.
2017
44M
48.6M
34.4
44M
40.7
Transformer-
25M
29.6M
34.4
25M
45M
40.7
Only Dictionary
9.7M
14.3M
33.5 =0.3
10.6M
27.4M
39.5 20.3
Shared-Attentinn
4.2M
=0.4
10.4
Shared-FFN (G=l)
7.8M
12.2M
33.2 =0.5
8.6M
25.4M
39.2 10.5
Shared-FFN (G=2)
8.1M
12.5M
33.2 =033
8.9M
25.7M
39.6 +004
Shared FFN (G=3)
8.4M
12.8M
33.3
9.1M
25.9M
39.5
Shared-both (G=2)
2.6M
7.1M
33.2 =0.4
2.5M
19.3M
39.6 +:0.4
Impeoved Embed
16


Reviews:
[{"review_id": "dlkVOVZ0gNc", "reviewer": "Reviewer_jV7A", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n- The proposed modification to the Transformer architecture reduces the number of model parameters and computational operations while sustaining competitive performance on various downstream tasks.\n- To the best of my knowledge, the idea of replacing layers of the Transformer with shared dictionaries is novel.\n\n### Room for Improvement\n\n*Shared-dictionary Attention*\n- I might be missing something but why is it stated that the unshared linear projection $\\tilde{W_{i}^{Q_{j}}}$ is approximately equal to $W_{i}^{Q_{j}}$? My understanding is that this is not directly optimized for in the model.\n\n*Group-wise Shared Dictionary FFN*\n- The motivation behind dividing columns of the dictionary into groups is a bit unclear. What is meant by “high-quality performance” of the shared dictionary projection? Also, have the authors considered using a larger number of dictionary elements $m$ to increase the “flexibility” of the model?  \n- How is the number of groups $G$ determined?\n\n*Training the DictFormer*\n- Since the sparse matrix Z is initialized using values in $C$, how are coefficients $C$ initialized? \n\n*Results tables*\n- Missing confidence intervals. Were the experiments run with multiple seeds? \n\n*Suggested related work*\n- How is this work related to work on Sparse Transformers (e.g. [1], [2]) or fixed attention such as [3], [4]? \n\n[1] Child R, Gray S, Radford A, Sutskever I. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509. 2019 Apr 23.\n\n[2] Correia, G.M., Niculae, V. and Martins, A.F., 2019. Adaptively sparse transformers. arXiv preprint arXiv:1909.00015.\n\n[3] You, W., Sun, S. and Iyyer, M., 2020. Hard-coded gaussian attention for neural machine translation. arXiv preprint arXiv:2005.00742.\n\n[4] Raganato, A., Scherrer, Y. and Tiedemann, J., 2020. Fixed encoder self-attention patterns in transformer-based machine translation. arXiv preprint arXiv:2002.10260. \n\n*Additional questions*\n- Is it necessary to have the dictionary size less than the embedding size, namely $m < d$? Would it not be feasible to have a large dictionary ($m > d$) but keep the number of selected components $t$ small (i.e. $t < d$) through a sparsity constraint? \n- Have the authors tracked whether all columns of the dictionaries are used in practice?\n- Have the authors tracked what percentage of the $t$ coefficients are non-zero on average? \n\n*Nitpicks*\n\nTypos:\n- p. 2, first line: “few unshared linear projection*s*”\n- p. 3, “Overview” paragraph: “given a*n* accuracy threshold”\n- p. 4, paragraph starting with “The reason why...”: “C_{i}^{x}” - should not $x$ be capitalized? \n- p. 5, “Group-wise Shared-dictionary FFN” paragraph: “a $N d \\times d$ weights” -> “$N$ weights of size $d \\times d$”   \n- p. 6, Figure 4: “training sparse coefficients” -> “we train sparse coefficients”\n- p. 6, first sentence of “Training DictFormer via Constraints and Relaxation” paragraph: “linear projections of *a* dictionary” \n- p. 7, last paragraph of “Architecture and Evaluation” paragraph: switch first sentence to present tense; “total #Params *i*n...”\n- p. 8, “Machine Translation” paragraph: “DictFormer obtain*s* more compact” \n- p. 8, “Sensitive Study” paragraph: rename to “Sensitivity Study”\n- p. 9 , first paragraph: “coefficient size is fixed *to* 60”\n- p. 9 , “Ablation” paragraph, first sentence: missing space after period\n- p. 9, “We will release code and data...”: Is there data to be released? \n\n", "summary_of_the_paper": "This work proposes a modification of the original Transformer architecture by replacing attention layers and layers in its Feed-Forward Networks across all of its blocks with learned shared dictionaries. The proposed model, called DictFormer, has a smaller number of parameters and uses a smaller amount of computational operations when compared to the original Transformer and some of its variations. When evaluated against these models on popular machine translation, summarization, and language modeling benchmarks, DictFormer achieves comparable or better performance.", "summary_of_the_review": "The proposed modification to the Transformer architecture is novel and I believe would be interesting for the community but the methodology and motivation could be explained more clearly and provided with more context, including more details on the hyperparameter selection and on how the DictFormer is trained. The experimental results would be even more convincing if confidence intervals are provided.\n\n#### Updates during paper discussion\nBased on the author's responses to the reviewers' questions and updates to the manuscript (including clarifying some of their methodology and statements and including confidence intervals in the results section), I've decided to increase my score. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "HZ45cf3Ecs", "reviewer": "Reviewer_aKni", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of the proposed method is interesting, but there are a few concerns in terms of the presentation.\nTherefore, it is hard to judge whether this paper has enough contribution for publishing as the conference paper.\n\nThe following are my concerns in the current version.\n\n### 1, Technical novelty\n* The idea of the proposed method is interesting and might be effective.\nHowever, the idea itself of sharing the parameter is not very innovative.\nI think that sharing parameters for compressing DNNs is a standard technique nowadays.\nTherefore, the authors need to clarify the contributions of the proposed method, such as the unique properties that previous similar compression methods cannot achieve.\nCurrently,  I do not find any strong properties in the proposed method.\n\n* If my understanding is correct, the proposed method is a reconstruction method. \nTherefore, we need a trained model for applying the proposed method.\nThis means the proposed method requires additional computation.\nI do not fully understand why this paper compares the computational cost with the standard Transformer.\n\n### 2, Notation and equation\n* The notations are incredibly messy and hard to understand.\nThe authors need to make notations much simpler for better understanding to readers.\n\n### 3, L1 constraint\n* If my understanding is correct, the relaxed L1 constraint does not guarantee to find the solution that satisfies the threshold of non-zero factors. This paper seems not to explain the way if such a situation occurs in the solution. ​\n\n### 4, Typo or misconfiguration?\n* In Table 1, it says the results for WMT De-En and WMT Fr-En.\nHowever, at the beginning of Section 4, the experiments are conducted on WMT \"En-De\" and \"En-Fr,\" which are not \"De-En\" and \"Fr-En.\" \n\n\n### 5, Confirmation of model sizes \n* According to the original Transformer paper [1], the numbers of parameters of Transformer (base) and (large) are 64M and 213M, respectively.\nHowever, in the experiments, the model size of the baseline Transformer is 3.6M (as shown in Table 1) for WMT En-De. \nMoreover, I checked the previous paper, such as the \"Lite Transfomer\" paper (Wu et al., 2020) and the \"Pay less attention\" paper (Wu et al., 2019).\nHowever, I could not find the precise experimental settings used in this paper. \nI recommend clearly showing the model configurations and hyper-parameter settings for keeping reproducibility.\nOtherwise, the reproducibility of the proposed method may not be sufficient.\n\n[1] Vaswani et al., Attention Is All You Need, In Proc. of NIPS-2017.\n\n\n\n\n### 6, Inconsistent results in Table 1 and 3\n* I thought that the ablation study of Table 3 is based on the results (settings) of Table 1.\nHowever, the numbers of parameters shown in Tables 1 and 3 differ entirely, so I do not understand the meaning of the ablation study in Table 3.\nPlease confirm it and clarify the configuration difference between Tables 1 and 3. \nMoreover, explain the results of the baseline Transformer and the proposed method corresponding to the ablation results in Table 3.\n\n* Additionally, it seems that there is no description about what the \"Improved-Embedded\" is shown in Table 3.\nIf I miss the description, please let me know. If the paper lacks explanation, this can be an apparent problem for this paper in terms of completeness.\n\n\n\n\n\n\n\n\n\n\n\n\n", "summary_of_the_paper": "This paper proposes a compression method for Transformer-based encoder-decoder or language models.\nThe key idea of the proposed method is to decompose the standard parameters into a much smaller shared parameter matrix and independent parameters for each original matrix.\nThen, the method can approximately recover the original Transformer models by simple additions and multiplications.\n\nThe experiments are conducted on three MT tasks, one summarization task, and one language modeling task.\nExperimental results show that the proposed method seems to reduce model sizes and computations successfully while preventing considerable performance degradation (in some cases, the proposed method appears to improve the performance).", "summary_of_the_review": "The idea of the proposed method is interesting and might be effective.\nHowever, the idea itself of sharing the parameter is not very innovative and rather incremental.\nExperimental settings are ambiguous and seems to use very weak settings.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "G7wBchti2ze", "reviewer": "Reviewer_EmuN", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper contains a lot of substance, but it is very dense and hard to follow. The core “dictionary” technique isn’t really explained at a high level before the paper plunges into the details. It seems to be something like the approach in [1] but it’s difficult to be sure (I gave up on section 3 after a while). The results in section 5 are very impressive, but some intuition about why a compressed approach like this could beat a much larger baseline on large data settings really need to be provided.\n\n[1] Kaiser, Lukasz, et al. \"Fast decoding in sequence models using discrete latent variables.\" International Conference on Machine Learning. PMLR, 2018.\n\nDetails:\n- The “first line of research”: would be good to add a word or two saying how these papers reduce computational complexity.\n- Figure 1 is really great, but you should say where these stats come from.\n- Figures 2 and 3: captions crash into text.\n- This is hard to understand: “\tIn this paper, the #Params omit word embedding size that would highly dependent on the sentence length and would significantly differ for various tasks. The total #Params in this paper includes the model size of word embedding.”\n- It’s difficult to align table 3 with figure 5. You should include a line corresponding to the point in figure 5 with highest BLEU (higher than anything that appears in table 3).\n", "summary_of_the_paper": "This paper describes a technique for reducing the size and computation of a Transformer model by projecting and factoring weight matrices. Experiments on MT, summarization, and language modeling show improved results over competing techniques, and even over standard Transformers, despite using significantly fewer parameters and less computation.\n", "summary_of_the_review": "Potentially a great paper, but if so it deserves to be much better explained.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AKmwT-4aOF8", "reviewer": "Reviewer_uPqt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- Clear description of background knowledge.  \n- Clear exposition of the proposed model.\n- The authors perform a comprehensive comparison on different downstream tasks, such as, machine translation, summarization, and language modeling.\n- The findings show that the proposed transformer model outperforms related work on the machine translation and language modeling tasks.\n\nWeaknesses\n- It is not clear how the initialisation of hyper-parameters affects model performance.  \n\nQuestions to the Authors.\nPlease address the following questions during the rebuttal:\n\n- Does parameter initialization could affect model performance? A possible extra contribution is to perform multiple  random runs and report variance. However, how expensive could this exercise become? \n- Please speculate on how attention representations behave across layers. For example, in Abnar, and Zuidema Quantifying Attention Flow in Transformers or Voita, et al. The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives \n- By using other pre-training objectives in the langgue modelling task (e.g. next sentence) would it change any finding or results? \n", "summary_of_the_paper": "The authors proposed an efficient transformer layer based on a dictionary of shared parameters instead of standard self-attention.  The goal is to reduce redundant parameters in transformer models.  The main contributions are: a lite transformer model, modification of the self-attention parameters, and evaluation on language dowstream tasks. The proposed transformer model outperforms related work on the machine translation and language modelling tasks.", "summary_of_the_review": "I recommend acceptance given that the paper clearly describes related work, and proposed model. The authors proposed an efficient transformer model that can be trained with less resources. The authors perform an  evaluation of the proposed model with different language downstream tasks, and the model outperforms related work on machine translation and language modelling.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: DictFormer is a method to reduce the redundancy in transformers so they can deployed on edge devices. In the method, a shared dictionary across layers and unshared coefficients are used in place of weight multiplications. The author proposed a l1 relaxation to train the non-differentiable objective  to achieve both higher performance and lower parameter counts.

All reviewers ended up giving the paper a score of 6 after increasing their scores during discussions. While the results are strong (better performance at much lower parameter counts), the paper is not clearly written. Several reviewers noted that the paper is difficult to understand and has a few unresolved points. For example, the method also ended up performing better than the base transformer model that DictFormer is supposed to compress. There seems to be a lack of understanding about what part of the model delivered the improvements. One reviewer said that this is potentially a great paper that deserves to be better explained. The basic concept of sharing a dictionary across layers should be simple enough to explain well and deserve a better explanation than eq 5. 

The authors promise to release the code, which would be necessary for a full dissemination of this work. I recommend accept.
Decision: Accept (Poster)