R2D2: Reuse & Reduce via Dynamic Weight Diffusion for Training Efficient NLP Models
Authors: Yi Tay
Aston Zhang
Shuai Zhang
Alvin Chan
Luu Anh Tuan
Siu Cheung Hui
Venue:  (2020)
Keywords: Deep Learning
Natural Language Processing

Abstract: We propose R2D2 layers, a new neural block for training efficient NLP models. Our proposed method is characterized by a dynamic weight diffusion mechanism which learns to reuse and reduce parameters in the conventional transformation layer, commonly found in popular Transformer/LSTMs models. Our method is inspired by recent Quaternion methods which share parameters via the Hamilton product. This can be interpreted as a neural and learned approximation of the Hamilton product which imbues our method with increased flexibility and expressiveness, i.e., we are no longer restricted by the 4D nature of Quaternion weight sharing. We conduct extensive experiments in the NLP domain, showing that R2D2 (i) enables a parameter savings of up to 2 times to 16 times with minimal degradation of performance and (ii) outperforms other parameter savings alternative such as low-rank factorization and Quaternion methods.


Under review as a conference paper at ICLR 2020
R2D2: REUSE & REDUCE VIA DYNAMIC WEIGHT DIFFUSION FOR TRAINING EFFICIENT NLP MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
We propose R2D2 layers, a new neural block for training efficient NLP models. Our proposed method is characterized by dynamic weight diffusion mechanism which learns to reuse and reduce parameters in the conventional transformation layer, commonly found in popular Transformer/l.. SSMMs models. Our method is inspired by recent Quaternion methods which share parameters via the Hamilton product. This can be interpreted as a neural and learned approximation Hamilton product which imbues method with increased flexibility and expres- siveness, i.e. we are no longer restricted by the 4D nature of Quaternion weight sharing. We conduct extensive experiments in the NLP domain, showing that R2D2 (i) enables a parameter savings of up to 2x to 16x with minimal degrada- tion of performance and (ii) outperforms other parameter savings alternative such as low-rank factorization and Quaternion methods.
the
INTRODUCTION
The transformation layer is one of the most ubiquitous and dominant component in existing and current deep learning literature (Goodfellow et al., 2016). Its pervasiveness cannot be understated, given its centrality many core building blocks in neural network research. Given widespread adoptions of FC layers, e.g., within Transformer (Vaswani et al., 2017) models and recurrent (Hochreiter & Schmidhuber, 1997) networks, further reduction in parameter complexity and memory footprint could be extremely useful to many real world applications.
This paper proposes a new neural component for reusing and reducing parameter costs of the transformation layer (i.e., memory footprint). Concretely, our proposed R2D2 layer is characterized by a new dynamic weight diffusion mechanism. The central idea is that we start off with a core set of parameters (weights) and learn dynamic 'diffusion' of these weight partitions to construct the final transformation weight W. The key novelty lies in our method of incorporating soft weight reuse, i.e., taking inspiration from multiplication in hypercomplex spaces.
Recent work in Quaternion spaces and Hamilton products have demonstrated reasonable success (Parcollet al., 2018; 2019; Tay al., 2019). The Hamilton product, which multiples two Quaternions by fusing latent inter-component representations, enables four time parameter saving as compared to the real-valued adaptation. Unfortunately, Hamilton products operate on 4D hypercomplex numbers, which limits its expressiveness and utility. To this end, our proposed R2D2 layer can be viewed as neural approximation of the Hamilton product, learning the permutation of the inter-latent component interactions in soft differentiable fashion. Moreover, our method can operate on any arbitrary aside from only on 4D Quaternion representations, facilitating up to n times parameter savings.
To demonstrate applicability, we equip two well-established models (LSTMs and Transformers) with R2D2 layers. We conduct extensive experiments on flagship benchmarks, i.e., neural machine translation for Transformers and natural language inference for LSTMs. Additionally, we include further validation on text style transfer and subject-verb agreement tasks. All in all, we find that R2D2 generally enables up parameter savings up to approximately 1/16 1/2 with minimal degradation in performance. Moreover, our proposed method also enables slight speed-up in terms of inference (decoding speed).

---

Under review as a conference paper at ICLR 2020
2
OUR PROPOSED METHOD
This section introduces our proposed R2D2 components.
2.1 VANILLA FC LAYERS
Before we delve into our proposed method, the standard FC transform layer is defined as:
y = FC(x) Wx b.
(2.1)
The FC layer (equation 2. is fundamental to many modern and traditional neural network architec- tures. Owing to its ubiquity and widespread usage, an efficient reduction of parameters in the FC layer may result in massive parameter savings, especially for models that heavily use transformation equation 2.1.
Remark 2.1 The degree freedom for the weight parameters W in equation 2.1 is kd. Since W dominates parameterization, the parameter size of the FC layer equation 2.1 is O(kd).
In the following, we describe the proposed R2D2 layer and its relationships with matrix multiplication in real space and Hamilton product in hypercomplex space.
2.2 REUSE AND REDUCE WITH DYNAMIC WEIGHT DIFFUSION (R2D2)
Preserving the same notation from the FC layer equation 2.1, a R2D2 layer transforms an input IRd into output IR* with H R and b R*
y R2D2 (x) Hx
(2.2)
The distinct difference in R21>2 is that construct the parameter H in equation 2.2 with reduced degree of freedom in order to reduce the parameter size. This is achieved by our Dynamic Weight Diffusion method described follows:
2.2.1 DYNAMIC WEIGHT DIFFUSION
This section describes proposed Dynamic Weight Diffusion mechanism. The central idea is to operate on partitioned weight blocks and learn dynamic diffusion of weights. There are two key parameter blocks A and S which central approach. Intuitively, A maxxxx controls the weight diffusion process and learns the soft interaction between S partitions. Here, is user defined hyperparameter.
Suppose that both and are divisible by n E Σ For I,..., n and j = I,..., -. denote by each partitioned parameter block S; E _ A; e |= is the weight diffusion matrix assigned each of n blocks. The parameter H in equation 2.2 is constructed by column-wise concatenation (:):
H (AA) ...
(2.3)
where each segment s( s(A)) is also formed by column- concatenation:
s s(A,) |v) (AAAAA)): -
(2.4)
In equation 2.4, function th PX + RM4 where (X) flattens the matrix X E Rxx by concate- nating each row of X then transposes the concatenated row vector into column vector of dimension pg. is easy to see that, (A,S,) E RR* s(A,) E xx g thus H kxxd
Here, the partitioning (or splitting) of weights into different components is analogous to complexification, i.c., splitting real vectors into real and imaginary components.
2

---

Under review as a conference paper at ICLR 2020
A:
Columna of H.
Size of H.
A1
S1
A1
So
A2
8i
A:
Parameters for H:
S:
Parameter size of H:
S:
S.
Figure 1: Illustration of our proposed Dynamic Weight Diffusion. Reusing partitioned parameter
blocks A; (i 2) and S, (j = 1..... .....4) 4) leads to reduction in the parameter size (n 2, d 8, le 6). Best viewed in color.
As illustrated in Figure 1, the reuse of partitioned parameter blocks S s(A,) (i = 1....,,, n) in equation 2.3 and the reuse the partitioned parameter block A, in each individual (A;;) segment in equation 2.4 may reduce the degree of freedom for H.
Sy in the segments
Remark 2.2 It is partitioned parameter blocks A; (i I..... n.) and S; (j I...., 4) that determine the degree of freedom for H. which is n². Since H dominates parameterization, the parameter size of the R2D2 in equation 2.2 ((),), where kd 2 4 assumed: this condition is mild for real-world problems, such in experiments (e.g., d == 512, 1c 2048, 2, 4, 16). Thus. for the same imput and output sizes, the parameterization cost of R2D2 layer approximately of that of FC layer under mild assumptions.
2.3 RELATIONSHIP WITH MATRIX MULTIPLICATION IN REAL SPACE
The R2D2 layer subsumes matrix multiplication in real space. Thus, it is generalization of the FC layer via hyperparameter n. To explain, referring equation 2.2, when -1, H aW, where scalar is the single element of the 1 x matrix A and elements of W E R come from the concatenation of ......, S. E R Since learning a and W separately is equivalent to learning their multiplication jointly, scalar can be dropped, which is learning the single weight matrix in an FC layer. Therefore, R2D2 layer is degenerated to an FC layer when n = 1, where there is no parameter saving.
2.4 RELATIONSHIP WITH HAMILTON PRODUCT HYPERCOMPLEX SPACE
Next, we explore the relationship between the R2D2 layer and Hamilton product in hypercomplex space.
For background, in hypercomplex space, inputs are represented by multiple imaginary components. For example, Quarternion has real component and imaginary components, while Octonion has I real component and 7 imaginary components. For the sake of illustration, let us take Quarternion Q as an example, which is a 4-dimensional hypercomplex number
Q = Q, + Q-i Quj Q.k.
(2.5)
where Q,, Qz, Qy: Qz real numbers, ijkk
12 j k2 -1 k.jk i
j.ji k, -i, ik = -j. The Hamilton product, which represents the multiplication of two Quaternions S S, S,i Syj + S.k and Q equation 2.5, is defined as
3

---

Under review as a conference paper at ICLR 2020
S
-Sy
-
Q+-
Sş
S,
-5
Sy
Qz
(2.6)
Sy S
ST
52
Qy
S2 -Sy
Sz
S,
QQ=
where the 4 output elements are the real values for the Quaternion unit basis [1, j. k) that = [Sr, Sz, Sy Sz] equation 2.6 can be rewritten as
Denoting
07
0
0
07
0
0
-1
0
0
0
0
-17
QQr
0
0
0
0
-1
0
1
0
0
0
8:
0
0
0
1
8;
0
0
0
8;
0
-1
0
S
Q2
0
-1
0
0
1
0
1
0
LQ=
(2.7)
can be observed that, when n = 4, the R21>2 layer may perform Hamilton Product of Quarternions. Specifically, partitioned parameter blocks A i:.... A4 in equation 2.3 parameterize the 4 permutation blocks (matrices composed of 1, in equation 2.7) that reflect the rules of Hamilton product, while S₁ in equation 2.4 are s in equation 2.7, and the layer input x in equation 2.2 Q2. Qy Q=] in equation 2.7. Likewise, Hamilton product of Octonions or Sedenions in hypercomplex space may also be performed by the R2D2 layer when k. d are equally set to 8 or 16.
2.5 GENERALIZING HYPERCOMPLEX MULTIPLICATION WITH NEURAL APPROXIMATION
In fact, weight reuse by component wise partitioning in Quaternion space has demonstrated reasonable success (Parcollet et al., 2018; Zhu et al., 2018; Parcollet et al., 2019; Tay et al., 2019). However, one key problem is that Hypercomplex algebra cannot be generalized to arbitrary n values simply because algebraic axioms cannot hold when takes values apart from power of 2. At this point, it is easy to see that 5-dimensional complex numbers cannot exist due to a clear inability to multiply two 5- dimensional complex numbers. Within the context of hypercomplex space, specialized multiplication rules, such as Hamilton product, have to be devised and encoded in the network as a fixed inductive bias. In sharp contrast, the R2D2 layer learns such specialized multiplication rules from data, as manifested in partitioned parameter blocks A, (i = ......7 n)) in equation 2.3. Thus, R2D2 subsumes and is degenerated to hypercomplex multiplication when such partitioned parameter blocks are set to reflect those predefined multiplication rules in hypercomplex space. Moreover, the proposed R2D2 layer can be seen as a trainable and parameterized form of n-dimensional hypercomplex multiplication, where can be values other than power of 2.
To sum up, the R2D2 layer reduces parameterization cost by reusing partitioned parameter blocks. also offers way to bridging multiplication between real space and hypercomplex space.
EFFICIENT NEURAL NLP MODELS WITH R2D2 LAYERS
To demonstrate the applicability the R2D2 layers, we develop two popular neural network models, LSTM and Transformer, which are able to benefit from R2D2 layers.
3.1 R2D2-LSTMs
Recurrent neural networks such as LSTMs are gated recurrent networks where gating functions are parameterized by linear transformations. We introduce the our efficient LSTM (R2D2-LSTM), which replace such linear transformations in the LSTM with the R2D2 layers:
Y: R2D2 (x+) + R2D2 (h.-1) 1) +
G, it, O,, X = o(y:)
c, = o. (f() c- + t0,
h o, Oc
4

---

Under review as a conference paper at ICLR 2020
where os is the sigmoid activation function and is the tanh activation function, Φ I - R a four way split on the last dimension, C₁, hy are the cell state and hidden state of the R2D2 -LSTM unit at time step t.
3.2 R2D2 TRANSFORMERS
Transformers (Vaswani et al., 2017), the state-of-the-art model for sequence transduction task, is stacked neural network architecture that aggressively exploits linear transformations. Each self- attention layer comprises of three Q, K, V transformations, along with N₁, heads per self-attention layer. Each layer also comprises of a two layered FC layer with nonlinearities. A large majority the Transformer parameters stem from the FC layers. In R2D2 Transformers, we replace all FC layers with R2D2 layers. The single-header headed self-attention module rewritten as:
QK vd
Q, V +(R2D2(X)) A softmax
VV,
where dy is the key dimension, & R 1xd + 33x 5 is a three way split on the last dimension, is the input sequence, and A is the self-attentive representation. For multi-headed using R2D2 layers also enables weight sharing not only among Q, K. V but also among heads. attention, the transformation of multiple heads also projected with R2D2 layers:
X R2D2 (Hi HN..)),
where N₁ is the number of heads and (i) is the column- wise concatenation, Finally, the position-wise FC layer now defined as
Y = R2D2(RLL
which transforms X with R2D2 layers.
EXPERIMENTS
This section reports the experimental results of R2D2-LSTMs and R2D2 Transformers that equipped with R2D2 layers. Overall, we conduct 4 main experiments in which all experimental settings and hyperparameters (e.g., parameter initialization) remain consistent with the vanilla version. We evaluate R2D2 LSTMs on natural language inference (NLJ) and compare them with standard LSTMs. Next, evaluate R2D2 Transformers and compare them with standard Transformers neural machine translation (NMT), text style transfer, and subject verb agreement (SVA) tasks.
4.1 NATURAL LANGUAGE INFERENCE
The task natural language inference is to determine two sequences entail or contradict with each other (MacCartney, 2009). NLI is fundamental task pertaining to language understanding. To this end, they serve as a suitable benchmark for evaluating recurrent models.
Table 1: Experimental results on natural language inference. Saving cost considers only the cell unit (other parts of architecture remains constant). R2D2-LSTMM reduces the parameter costs of the standard LSTT model and improves performance on out of 5 datasets.
Model
#Params
MNLI
QNLI
SNLI
DNLI
SciTail
ISTM Quaternion LSTM
721K
71.8271.89
84.44
84.18
85.16
74.36
180K (-75.0%)
71.57 72.19
84.73
84.21
86.45
75.58
R2D2-LSTM (n
361K (-49.9%) 9%)
71,82/ 71.82/72.08
72.08
84.39
84.38
85.77
77.47
R2D2 LSTM in
(-79.79)
71.80/71.77
84.58
86.47
R2D2-LSTM (n 10)
81K (-88.7%)
71.59/71.55 71.59
84.25
84.40
86.21
77.84
Datasets and Setup We run experiments on NLI datasets, i.e., (1) SNLI (Bowman et al., 2015),
(2) MultiNLI (Williams et al., 2017), (3) Dialogue NLI (Welleck al., 2018), (4) QNLI (Quora) (Wang et al., 2017), and (5) Scitail (Science Entailment) (Khot et al., 2018). We implement 300- dimensional unidirectional encoders with shared parameters for both premise/hypothesis.. We take
5

---

Under review as a conference paper at ICLR 2020
the concatenation of max and mean pooled representations imput to two-layered dineennnsii MLP for prediction. Our model is trained with the Adam with a learning rate of 0.0004 with a batch size of 256. Word embeddings are initialized with Glo Ve (Pennington al., 2014) and are fixed. No cross sentence attention (Parikh et al., 2016) is used, mainly to observe the effectiveness of standalone encoders. For R2D2--ISTM, since word embeddings are 300-dimensional, we select multiples of 5 instead of 4 for simplicity and ease of divisibility.
{2, 5, 10} and report the results accordingly. In this task,
Experimental Results Table I reports the results on all the 5 natural language inference datasets. All in all, results are extremely encouraging, showing that the R2D2 layer can only reduce the parameterization cost but also improve performance (4 out of 5 datasets show reasonable improve- ment). On QNLI (the only exception). the performance drop is marginal (< 1%), which is decent considering the parameter savings. It is also noteworthy that on SNLI, DNLI, and SciTail, all the R2D2 -LSTM variants outperform the base LSTM model. Overall, we think that the parameter reuse properties, in addition to learning to share such reused parameter blocks amongst recurrent gating functions, may contribute to a regularizing effect.
4.2 NEURAL MACHINE TRANSLATION
Machine translation (MT) is concerned with translating between source-target language pairs. To this end, sequence transduction models are central to this problem domain. In this experiment, the key goal is to compare R2D2 Transformers against the standard Transformer model.
Table 2: Experimental results on neural machine translation (BLEU scores). represents up- samplinli with factor of 2. R2D2 Transformers do not lose much performance despite enjoying huge saving in parameterization cost. Re-scaling to equal parameters can lead to improvement in results. Savings
do not account for token embedding parameters.
Sctting
#Params
En-Vi
En-Id
De-En
Ro-En
En-Et
En-MK
Ea-Ro
Transformer
44M
28.43
47.40
36.68
34.60
14.17
13.96
22.79
Factorized
IIM (-75.0%)
27.21
29.72
22.93
20.23
6.70
9.36
15.55
Quatemion
(-75.0%)
28.00
42.22
32.83
30.53
13.10
13.67
18.50
R2D2
(-50.0%)
29.25
46.32
33.40
14.98
21.73
R2D2
11M (-75.0%)
29.13
44.13
35.53
32,74
14,11
13.01
21.19
R2D2 n =
5.5M (-87.5%%
29.34
40.81
34.16
31.88
13.08
12.95
21.66
R2D2
16
2.9M (-93.4%)
29.04
33,48
33.89
31.53
12.15
11.97
19.63
R2D2 n
44M
29.54
49.05
34.32
33.88
14.05
14.41
22.18
R2D2 n
22M (-50.0%)
29.17
46.24
34.86
33.80
14.43
13.78
21.91
R2D2"
(-75.0%)
29.47
43.49
34.71
32.59
13.75
13.78
21.43
Table 3: Experimental results on neural machine translation (BLEU scores) on 1M En-De.
Table 4: Training time (seconds per 100 steps) and Decoding Time (seconds to decode test set) with Beam size of and length penalty of 0.6 on IWSLT 2014 Berman-Engliste
Reported Models
Transformer (Vaswani et al., 2017) 27.30
DynamicComy (Wu et al., 2019)
29.70
Transformer (our run)
27.86
Model
Train
Decoding
Compressed Models
Vanilla
7.79
Quaternion (Tay al., 2019)
25. 14
Factorized
7.26
291
R2D2 Transformer (n 2)
26.43
Quaternion
8.31
297
R2D2 Transformer (n 4)
26.52
R2D2 (r 4)
8.09
303
R2D2 Transformer (n 8)
21.11
R2D2 (n 8)
7.89
287
R2D2 Transformer (n 16)
23.01
Datasets and Setup We run experiments on 8 NMT datasets. The datasets are (1) IWSLT'
English-Vietnamese Vietnamese SEn-Vi), (2) IWSLT' 17 Ianglish-Indoeesann (En-Id)i, (3) IWSLT' 14 German- English (En-De)t, (4) IWSLT' 14 Comanian-Englicn (Ro-En)f, (5) WMT' 18 English-Estonian Estonian (En- Et)", (6) Setimes Agglish-Maacdoonian (En-Mk)", (7) WMT" 18 English- English-Romanian (En-Ro)* and (8)
English-
WMT' 16 English-German (En-De).

---

Under review as a conference paper at ICLR 2020
Datasets with i are with 50K steps while datasets with are trained for 100K steps also single GPU. For all tasks, we specify that Transformers have layers, heads, and a hidden size 512. We use beam size of 5 and 0.6 (length penalty) for evaluating all the models. For all R2D2 models, we benchmark several settings for the hyperparameter n = {2. 4, 8, 16). Likewise, we also re-scale the parameters by increasing the dimensions match the original number of parameters. This is denoted by hyperparameter U. In the experiments, the upsampling factor is fixed at 2.
We benchmark against two baselines that are also concerned with reducing parameterization of the FC layer. The first, is the Factorized Transformer which approximates each FC layer with low-rank approximation. In this case, the number of latent factors is set d enable direct comparison the Quaternion Transformer. The second is the Hamilton product based Transformer proposed in (Tay al., 2019).
Experimental Results Table 3 reports results on machine translation. Across 5 out of 7 NMT benchmarks, R2D2 Transformer outperforms the standard Transformer model. Overall, the empirical results are promising. On one hand, we observe that increasing n all the way to 16 does not cause significant degradation in performance on datasets such as En- Vi. On the other hand, for most datasets, even with significant parameter reduction, we find that the decrease in the BLEU score is overall manageable (= 1-3 BLEU points). However, we also note a rare occurrence where n = 16 results in significant decrease in the BLEU score, such as on the En-ld dataset.
On several datasets, the R2D2 Transformer model improves the performance. For example, datasets such En- Vi and En-Et enjoy a performance boost of about BLEU point with n 2. Generally, there is only marginal performance degradation upsampling with hyperparameter U, we are able to improve the performance three datasets: En-Vi, En-Id, and En-Mk.
= 4 or 8 for most datasets. Finally, by
Additionally, we note that Factorized Transformer performs substantially worse than the vanilla Transformer or our Transfranffrmmrr. On the other hand, our proposed R2D2 also makes reasonable gains over the Quaternion Transformer (Tay et al., 2019), signifying that flexible approximation of Hamilton products effective.
Table 4 reports the decoding and training time for each Transformer variant. We observe that R2D2 with 8 has the fastest decoding speed amongst all variants. All in all, the training speed is also approximately similar. As such, this ascertains that our proposed dynamic weight diffusion method is not computationally expensive, despite its complexity.
4.3 TEXT STYLE TRANSFER
We experiment with sequence transduction for text style transfer. The goal of this task is to convert text of certain style to another style.
Datasets and Setup We use the Badeem--Shakespena corpus² in the experiments. The key goal here is to convert modern writing into Shakespeare writing. This dataset comprises of 18395 parallel sentences for training. 1218 parallel sentences for evaluation (dev set), and 1462 parallel sentences for testing. We still specify that Transformers have 4 layers, 8 heads, and hidden size 512. Similar to NMT, we experiment with {2, 4, 8, 163. We train all the models for 10K steps.
Experimental Results Table 5 reports the re- sults text style transfer. We observe that the best performance is achieved with R2D2 Transformer (n = 4). Notably, all except the n 16 variant outperforms the standard Transformer model. This ascertains the effectiveness of the proposed R2D2 layer. This not only enables parameter savings but improves the performance of Transformer.
Table 5: Experimental results on text style transfer with Transformer models.
Model
#arams
BLEU
Transfommr
11.65
R2D2 (n 2)
22M (-50.0%)
12.20
R2D2
IIM (-75 0%)
12.42
R2D2 (n
8)
5.5M (-87.5%)
11.66
16)
(-93.4%)
10.76
7

---

Under review as a conference paper at ICLR 2020
4.4 SUBJECT VERB AGREEMENT
We conduct additional experiments on subject-verb agreement task (Linzen et al., 2016). The task is binary classification problem. It predicts if the sentence, e.g., 'The keys to the cabiner ----. follows by a plural or singular.
Dataset and Setup This dataset can be found online (Linzen et al., 2016). We evaluate the R2D2 Transformer encoder against the standard Transformer encoder for the subject verb agreement task. In contrast to the previous experimental settings, we use a smaller Transformer architecture. Specifically, Transformers here have 2 layers, head, and a hidden size 128. Since the hidden size is smaller than those in the previous experimental settings, we experiment with n {2,4.8).
Experimental Results Table reports the re- sults on the SVA task. Results are promising, demonstrating that all variants with R2D2 layers outperform the standard Transformer. The best performance peaks at n 8, despite parame- ter saving up to 1 1/8. R2D2 also outperforms Quaternion Transformers.
Table 6: Experimental results on subject verb agreement (SVA) with Transformer models.
Model
#Params
Acc
Transformer
400K
94,80
Quaternion
100K
94.70
R2D2 (n 2)
200K (-50.0%)
95.
R2D2 (n 4
101K (-74.8%)
95.05
R2D2 8)
56K (-86.0%%
95.62
RELATED WORK
FC layers are ubiquitous in deep learning research (Goodfellow et al., 2016). Many, if not all neural building blocks incorporate this functionality, i.e., transforming input vectors to an output space using a parameterized linear transformation matrix. Recurrent neural networks (Hochreiter & Schmidhuber, 1997), multilayer perceptrons, attention (Bahdanau et al., 2014), and Transformer models ( (Vaswani et al., 2017) are all heavily grounded in these transformation operations. Notably, recent state-of-the- art models in language domains aggressively exploit transformation layers, achieving very exceptional results (Devlin et al., 2018; Radford et al., 2019).
Our work be interpreted as form of soft parameter sharing, albeit dynamic and learned. A recent work (Savarese & Maire, 2019) proposed soft-weight sharing across stacked convolution layers, effectively simulating recurrence, i.e., RNNs can be considered be weight sharing across time. Quaternion networks (Zhu et al., 2018; 8; Parcollet et al., 2018; 2019) are also known to inhibit weight sharing qualities and have demonstrated reasonable success despite having fewer parameters.
Low-rank approximations (Markovsky, 2012) and factorization methods are known techniques to reduce parameters and improve generalization. Majority of techniques aimed reducing parameters of neural networks belong to this class (Sainath et al., 2013; Chen et al., 2018; Si et al., 2017). Within the context language, GroupReduce (Chen al., 2018) applies low-rank approximations to compress neural language models. (Si et al., 2017) proposed memory efficient kernels using low-rank approximation. Block diagonal approaches (Xie et al., 2017). structured sparsity (Gale et al., 2019) are also highly notable alternatives for improving the FC layer. Methods such as knowledge distillation (Hinton et al., 2015; Freitag et al., 2017) are also capable of reducing model complexity and can be considered an orthogonal direction from ours. In a similar vein, sharing parameters by the means of a meta network (HyperNetworks Ha et al. (2016) also perform a form of weight sharing.
6 CONCLUSION
We propose R2D2 layers, which can reduce the parameterization cost of their FC counterparts without significantly compromising performance. The R2D2 layer also offers a way to bridging multiplication between real space and hypercomplex space by subsuming and generalizing both matrix multiplication in real space and Hamilton product in hypercomplex space. is highly modular and applicable to many dominant models such LSTMs and Transformers. We evaluate such models equipped by R2D2 layers on comprehensive tasks, demonstrating substantial parameter savings with minimal degradation improvement in performance.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Dzmitry Bahdanau. Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint wrXiv:1409.04 2014.
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large annotated corpus for learning natural language inference. arXiv preprint arXiv: 1508. 05326, 2015.
Patrick Chen, Si Si, Yang Li, Ciprian Chelba, and Cho-Jui Hsieh. Groupreduce: Block-wise low-rank approximation for neural language model shrinking. Advances Neural Information Processing Systems, pp. 11988-10998, 2018.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:/810.0480 2018.
Markus Freitag, Yaser Al-Onaizan, and Baskaran Sankaran. Ensemble distillation for neural machine translation. arXiv preprint 1w0.v:1702.011 802, 2017.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks, arXiv preprint arXiv:/902.095:: 2019.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http: //www. www.:/deeplerrnnngoooe..oooo org.
David Ha, Andrew Dai, and Quoe V Le. Hypernetworks. arXiv preprint 1rXiv:1609.09100 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0253) 2015.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
Tushar Khot, Ashish Sabharwal, and Peter Clark. Scitail: A textual entailment dataset from science question answering. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. Assessing the ability of Istms to learn syntax- sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521-535, 2016.
Bill MacCartney. Natural language inference. Citeseer, 2009.
Ivan Markovsky. Low rank approximation. Springer, 2012.
Titouan Parcollet, Mirco Ravanelli, Mohamed Morchid, Georges Linarès, Chiheb Trabelsi, Re- nato De Mori, and Yoshua Bengio. Quaternion recurrent neural networks. arXiv preprint arXiv:1806.044/11 2018.
Titouan Parcollet, Mohamed Morchid, and Georges Linarès. Quaternion convolutional neural net- works for heterogeneous image processing. In ICASSP 2019-2019 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 8514 8518. IEEE, 2019.
Ankur P Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. A decomposable attention model for natural language inference. arXiv preprint 1606.0660.00/99 1606009936.0.... 2016.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543 2014.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. 2019.
Tara N Sainath, Brian Kingsbury, Vikas Sindhwani, Ebru Arisoy, and Bhuvana Ramabhadran. Low- rank matrix factorization for deep neural network training with high- nigh-dimensiooal output targets. In 2013 IEEE international conference on acoustics, speech and signal processing, pp. 6655-6651 IEEE, 2013.

---

matching for natural Under review as a conference paper at ICLR 2020
Pedro Savarese and Michael Maire. Learning implicitly recurrent cnns through parameter sharing. arXiv preprint arXiv:1902.09700 2019.
Si Si, Cho-Jui Hsich, and Inderjit S Dhillon. Memory efficient kernel approximation. The Journal of Machine Learning Research, 18(1):682-713, 2017.
Yi Tay, Aston Zhang, Luu Anh Tuan, Jinfeng Rao, Shuai Zhang, Shuohang Wang, Jie Fu, and Siu Cheung Hui. Lightweight and efficient neural natural language processing with quaternion networks. arXiv preprint arXiv: /00.//000004 043604433, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Zhiguo Wang, Wael Hamza, and Radu Florian. Bilateral language sentences. arXiv preprint 1722v.170..08814 2017.
Sean Welleck, Jason Weston, Arthur Szlam, and Kyunghyun Cho. Dialogue natural language inference. arXiv preprint 1xXiv:8811.006) 2018.
Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426, 2017.
Felix Wu, Angela Fan, Alexei Baevski, Yann N Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. arXiv preprint arXiv:1901.10430, 2019.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1492-1500, 2017.
Xuanyu Zhu, Yi Xu, Hongteng Xu, and Changjian Chen. Quaternion convolutional neural networks. In Proceedings of the European Conference Computer Vision (ECCV), pp. 631-647, 2018
10


Reviews:
[{"review_id": "SJx3GtTwtH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose R2D2 layers, which are trained to reduce and re-use existing parameters of a neural network layer, and apply this to Transformer and LSTM architectures. The authors conduct experiments on various NLP tasks, including NLI and NMT. \n\nThe main benefit of the proposed R2D2 layer is that the number of parameters can be reduced, as the existing parameters can be reused. I find this motivation compelling, particularly as it is well known Transformer networks are largely overparameterized. \n\nComments:\n1. There is no analysis on the specific choices made for dynamic weight diffusion- the way the partitioning is done could have a large effect on the end result. There's also little comparison to other ways to share weights across a model besides the proposed weight diffusion method. \n\n2. Sharing parameters contributes a regularization effect - it is difficult to untie the contributions of increased regularization from the proposed method. This is particularly problematic as the majority of the datasets used are \"small\" by current standards. WMT en-de (authors do not include the sizes of the datasets, but this is 4.5 million sentences) is the only large scale dataset, and the BLEU drop is quite large on this dataset compared to the smaller ones such as IWSLT. \n\nTo tie my points #1 and #2 together, I feel the authors did experiments on a variety of different tasks, but these style transfer and subject verb agreement tasks are not particularly interesting or realistic - instead this space should be devoted to discussions of the advantages of their method and analysis on its performance, which is quite lightly covered.\n\n3. The authors claim that the R2D2 Transformer outperforms standard Transformer models on 5 out of 7 NMT tasks. This appears true if up-sampling with a factor of 2 is used to make the models larger again. The authors should compare to factorized/quaternion baselines which have a larger quantity of parameters as well. \n\n4. Table 3, where results are reported on the competitive WMT en-de benchmark, lacks comparison for number of parameters and decoding speed. This table would probably have the most compelling and impactful results for this paper as this is the most competitive task (aside from the pre-training regime on MNLI/QNLI as part of GLUE). Can the authors complete this table so readers can understand the parameter reduction and inference speed possible from this method on this benchmark?\n\n(As an aside, the technique should be applicable to the DynamicConv model, which is a Transformer variant?)\n\n5. The related work section is quite light on other approaches to reducing model size, such as knowledge distillation or quantization? While the approach taken in this paper leverages parameter sharing, the motivation is similar and I feel acknowledging this entire area of work would be relevant.\n\n6. I'm not clear on why we see inference time decoding speed improvements based on the description of the method. Can the authors clarify this point for me? ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BJer5FcCYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper Summary:\n\nThis paper proposes to train smaller models by decomposing the weights of fully connected networks as the product of smaller matrices, along with a reordering/transpose of the outcome. The experiments shows that models with less parameters yield comparable performance with their larger counterparts.\n\nReview Summary:\n\nThe method is technically sound and the paper reads well. Experiments demonstrate the efficacy of the method, although some ablations are missing (see below). The paper is however not clear on the ultimate objective of the method (speed/accuracy/generalisation?) and does not compare with alternatives.\n\nDetailed Review:\n\nThe introduction does not make clear if your motivation to make model smaller is training speed, inference speed, memory usage, generalization accuracy. Please clarify.\n\nThe explanation of the method, i.e. Section 2.2.1, is not clear, in particular for the mapping \\psi. I feel it would cleared if somewhere in the paper there was an equation with the element-wise correspondence, i.e. H_{?,?} = \\sum_k A_i,k S_k,j\nIn that section, you should introduce that n is a hyperparameter before using it as well.\nIn that section, you could also discuss parameter initialization, and whether this model can use weight decay over H or A/S. it is also not clear to me if you control the norm ratio between A and S given the weight magnitude is over parameterized.\n\nThe experimental section lack a validation/ablation study to help the reader understand the interplay between the number of blocks and the number of latent dimensions. It will also be good to show learning curves to compare training speed of different parameterization. \nAlso no training errors are reported, does your method can be seen as a regularizer, i.e. is training objective closer to valid objective when n grows? Did you have to change other regularization parameters like dropout.\n\nTo me the main weakness of the paper lies in the lack of comparison with alternatives. Replacing fully connected layers with alternative has a rich literature that the authors ignore.\nI feel it is necessary to compare the approach with\n(i) block diagonal approaches, popular since ResNext for convolutions but equally applicable to linear layers. https://arxiv.org/abs/1611.05431\n(ii) other form of structured sparsity. https://arxiv.org/abs/1902.09574 (survey). https://arxiv.org/abs/1812.08301 (squantizer) https://arxiv.org/abs/1802.08435 (block sparsity)...\n(iii) distillation of large models into smaller models.  https://arxiv.org/abs/1503.02531 https://arxiv.org/abs/1702.01802\n(iv) it might not be necessary to compare, but at least mentioning approaches which predict weights from a meta network would be good.  https://arxiv.org/abs/1609.09106\n\nAs a reviewer, I am a bit annoyed that made no effort to have a decent list of related work and that they delegate that work to the reviewers to do so.\n\nDetails:\n\"transformation layer\": this is not common terminology, please prefer linear layer or fully-connected layer.\nplease define all acronyms, e.g. FC.\nThe experimental section does not define \\alpha (end of page 6).\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1xr0FcpFS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new Reuse and Reduce with Dynamic weight Diffusion (R2D2) layer as an alternative to feed-forward layers in neural networks. The layer is inspired by the Hamilton Product in a hypercomplex space (where numbers have multiple imaginary components). The main idea is to have two smaller parameter blocks that are partitioned, multiplied, and concatenated together in order to form the full weight matrix of the feed-forward layer.\nIn extensive experiments on NLI, NMT, text style transfer and subject-verb agreement, feed-forward layers in LSTMs and Transformers are replaced with R2D2 layers. The modified models achieve similar performance to the originals, while being more than 50% smaller. \n\nOverall, the proposed method is presented clearly and the experiments are comprehensive and convincing. For these reasons, I am leaning towards accepting this paper.\n\nThe proposed method is well explained. In particular, Figure 1 is helpful to obtain a conceptual picture of the method. This is in contrast to some of the previous methods based on hypercomplex operations, which often seem harder to grasp and visualize. In addition, it is helpful that connections to other operations such as matrix multiplication and the Hamilton product are highlighted.\n\nThe proposed method is evaluated extensively. It is applied to different models (LSTMs and Transformers) and on different tasks. Results are mostly convincing, as performance numbers are competitive with the baselines, while the models are much smaller. In addition, it compares to previous work, which it outperforms. \n\nThe main thing that I'm missing is some analysis of the dynamics of the model, what it is learning (in comparison to using FC layers) or why a smaller number of parameters is still competitive with the standard FC layers. Are feed-forward layers over-parameterized and only a smaller number of their weights are actually used in practice, similar to lottery tickets (https://arxiv.org/abs/1803.03635)? How do the learned A and S blocks look like? Is the entire model learning a different function or do the R2D2 layers just find a way to approximate a feed-forward layer? \n\nOverall, as the method seems straightforward enough to implement and achieves promising results, it has the potential to have some practical impact.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a very interesting alternative to feed-forward network layers, based on Quaternion methods and Hamilton products, which has the benefit of reducing the number of parameters in the neural network (more than 50% smaller) without sacrificing performance. They conducted extensive experiments on language tasks (NMT and NLI, among others) using transformers and LSTMs. 

The paper appears to be clearly presented and have extensive results on a variety of tasks. However all reviewers pointed out that there is a lack of in-depth analysis and thus insight into why this approach works, as well as questions on the specific effects of regularization. These concerns were not addressed in the rebuttal period, instead leaving it to future work. My assessment is that, with further analysis, ablation studies, and comparison to alternative methods for reducing model size (quantization, etc), this paper has the potential to be quite impactful, and I look forward to future versions of this work. As it currently stands, however, I don’t believe it’s suitable for publication at ICLR.

Decision: Reject