Title: R2D2: REUSE & REDUCE VIA DYNAMIC WEIGHT DIFFUSION FOR TRAINING EFFICIENT NLP MODELS

Abstract: We propose R2D2 layers, a new neural block for training efficient NLP models. Our proposed method is characterized by a dynamic weight diffusion mechanism which learns to reuse and reduce parameters in the conventional transformation layer, commonly found in popular Transformer/LSTMs models. Our method is inspired by recent Quaternion methods which share parameters via the Hamilton product. This can be interpreted as a neural and learned approximation of the Hamilton product which imbues our method with increased flexibility and expressiveness, i.e., we are no longer restricted by the 4D nature of Quaternion weight sharing. We conduct extensive experiments in the NLP domain, showing that R2D2 (i) enables a parameter savings of up to 2× to 16× with minimal degradation of performance and (ii) outperforms other parameter savings alternative such as low-rank factorization and Quaternion methods.

1. INTRODUCTION:

The transformation layer is one of the most ubiquitous and dominant component in existing and current deep learning literature (Goodfellow et al., 2016) . Its pervasiveness cannot be understated, given its centrality to many core building blocks in neural network research. Given widespread adoptions of FC layers, e.g., within Transformer (Vaswani et al., 2017) models and recurrent (Hochreiter & Schmidhuber, 1997) networks, a further reduction in parameter complexity and memory footprint could be extremely useful to many real world applications. This paper proposes a new neural component for reusing and reducing parameter costs of the transformation layer (i.e., memory footprint). Concretely, our proposed R2D2 layer is characterized by a new dynamic weight diffusion mechanism. The central idea is that we start off with a core set of parameters (weights) and learn dynamic 'diffusion' of these weight partitions to construct the final transformation weight W . The key novelty lies in our method of incorporating soft weight reuse, i.e., taking inspiration from multiplication in hypercomplex spaces.

Recent work in Quaternion spaces and Hamilton products have demonstrated reasonable success (Parcollet et al., 2018; 2019; Tay et al., 2019) . The Hamilton product, which multiples two Quaternions by fusing latent inter-component representations, enables a four time parameter saving as compared to the real-valued adaptation. Unfortunately, Hamilton products operate on 4D hypercomplex numbers, which limits its expressiveness and utility. To this end, our proposed R2D2 layer can be viewed as a neural approximation of the Hamilton product, learning the permutation of the inter-latent component interactions in a soft differentiable fashion. Moreover, our method can operate on any arbitrary n, aside from only on 4D Quaternion representations, facilitating up to n times parameter savings.

To demonstrate applicability, we equip two well-established models (LSTMs and Transformers) with R2D2 layers. We conduct extensive experiments on flagship benchmarks, i.e., neural machine translation for Transformers and natural language inference for LSTMs. Additionally, we include further validation on text style transfer and subject-verb agreement tasks. All in all, we find that R2D2 generally enables up parameter savings up to approximately 1/16 ∼ 1/2 with minimal degradation in performance. Moreover, our proposed method also enables slight speed-up in terms of inference (decoding speed).

2. OUR PROPOSED METHOD:

This section introduces our proposed R2D2 components.

2.1. VANILLA FC LAYERS:

Before we delve into our proposed method, the standard FC transform layer is defined as:

y = FC(x) = Wx + b.

(2.1)

The FC layer (equation 2.1) is fundamental to many modern and traditional neural network architectures. Owing to its ubiquity and widespread usage, an efficient reduction of parameters in the FC layer may result in massive parameter savings, especially for models that heavily use transformation equation 2.1.

Remark 2.1 The degree of freedom for the weight parameters W in equation 2.1 is kd. Since W dominates parameterization, the parameter size of the FC layer equation 2.1 is O(kd).

In the following, we describe the proposed R2D2 layer and its relationships with matrix multiplication in real space and Hamilton product in hypercomplex space.

2.2. REUSE AND REDUCE WITH DYNAMIC WEIGHT DIFFUSION (R2D2):

Preserving the same notation from the FC layer equation 2.1, a R2D2 layer transforms an input

x ∈ R d into an output y ∈ R k with H ∈ R k×d and b ∈ R k : y = R2D2 (x) = Hx + b. (2.2)

The distinct difference in R2D2 is that we construct the parameter H in equation 2.2 with a reduced degree of freedom in order to reduce the parameter size. This is achieved by our Dynamic Weight Diffusion method described as follows:

2.2.1. DYNAMIC WEIGHT DIFFUSION:

This section describes our proposed Dynamic Weight Diffusion mechanism. The central idea is to operate on partitioned1 weight blocks and learn a dynamic diffusion of weights. There are two key parameter blocks A and S which are central to our approach. Intuitively, A ∈ R n×n×n controls the weight diffusion process and learns the soft interaction between S partitions. Here, n is a user defined hyperparameter.

Suppose that both d and k are divisible by n ∈ Z >0 . For i = 1, . . . , n and j = 1, . . . , d n , denote by each partitioned parameter block S j ∈ R n× k n . A i ∈ R n×n is the weight diffusion matrix assigned to each of n blocks. The parameter H in equation 2.2 is constructed by column-wise concatenation (;):

H = [s(A 1 ); s(A 2 ); . . . ; s(A n )],

(2.3) where each segment s(A i ) is also formed by column-wise concatenation:

s(A i ) = [ψ(A i S 1 ); ψ(A i S 2 ); . . . ; ψ(A i S d n )].

(2.4)

In equation 2.4, function ψ : R p×q → R pq , where ψ(X) flattens the matrix X ∈ R p×q by concatenating each row of X then transposes the concatenated row vector into a column vector of dimension pq. It is easy to see that,

ψ(A i S j ) ∈ R k , s(A i ) ∈ R k× d n , thus H ∈ R k×d . A1 S1 A1 S3 A1 S2 A1 S4 A2 S1 A2 S3 A2 S2 A2 S4

Columns of H:

A1 A2 S1 S3 S2 S4

Parameters for H: Parameter size of H:

Size of H: As illustrated in Figure 1 , the reuse of partitioned parameter blocks S 1 , . . . , S d n in the n segments s(A i ) (i = 1, . . . , n) in equation 2.3 and the reuse of the partitioned parameter block A i in each individual s(A i ) segment in equation 2.4 may reduce the degree of freedom for H.

Remark 2.2 It is partitioned parameter blocks A i (i = 1, . . . , n) and S j (j = 1, . . . , d n ) that determine the degree of freedom for H, which is kd n + n 3 . Since H dominates parameterization, the parameter size of the R2D2 in equation 2.2 is O( kd n ), where kd n 4 is assumed: this condition is mild for real-world problems, such as in our experiments (e.g., d = 512, k = 2048, n = 2, 4, 8, 16) . Thus, for the same input and output sizes, the parameterization cost of a R2D2 layer is approximately 1 n of that of an FC layer under mild assumptions.

2.3. RELATIONSHIP WITH MATRIX MULTIPLICATION IN REAL SPACE:

The R2D2 layer subsumes matrix multiplication in real space. Thus, it is a generalization of the FC layer via hyperparameter n. To explain, referring to equation 2.2, when n = 1, H = aW, where scalar a is the single element of the 1 × 1 matrix A 1 and elements of W ∈ R k×d come from the concatenation of S 1 , . . . , S d ∈ R 1×k . Since learning a and W separately is equivalent to learning their multiplication jointly, scalar a can be dropped, which is learning the single weight matrix in an FC layer. Therefore, a R2D2 layer is degenerated to an FC layer when n = 1, where there is no parameter saving.

2.4. RELATIONSHIP WITH HAMILTON PRODUCT IN HYPERCOMPLEX SPACE:

Next, we explore the relationship between the R2D2 layer and Hamilton product in hypercomplex space.

For background, in hypercomplex space, inputs are represented by multiple imaginary components. For example, a Quarternion has 1 real component and 3 imaginary components, while an Octonion has 1 real component and 7 imaginary components. For the sake of illustration, let us take a Quarternion Q as an example, which is a 4-dimensional hypercomplex number

Q = Q r + Q x i + Q y j + Q z k, (2.5) where Q r , Q x , Q y , Q z are real numbers, ijk = i 2 = j 2 = k 2 = -1, ij = k, jk = i, ki = j, ji = -k, kj = -i, ik = -j.

The Hamilton product, which represents the multiplication of two Quaternions S = S r + S x i + S y j + S z k and Q equation 2.5, is defined as

   S r -S x -S y -S z S x S r -S z S y S y S z S r -S x S z -S y S x S r       Q r Q x Q y Q z    , (2.6)

where the 4 output elements are the real values for the Quaternion unit basis [1, i, j, k] . Denoting that s = [S r , S x , S y , S z ] , equation 2.6 can be rewritten as

      1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1    s;    0 -1 0 0 1 0 0 0 0 0 0 1 0 0 -1 0    s;    0 0 -1 0 0 0 0 -1 1 0 0 0 0 1 0 0    s;    0 0 0 -1 0 0 1 0 0 -1 0 0 1 0 0 0    s       Q r Q x Q y Q z    .

(2.7)

It can be observed that, when n = k = d = 4, the R2D2 layer may perform Hamilton Product of Quarternions. Specifically, partitioned parameter blocks A 1 , . . . , A 4 in equation 2.3 parameterize the 4 permutation blocks (matrices composed of -1, 0, 1 in equation 2.7) that reflect the rules of Hamilton product, while S 1 in equation 2.4 are s in equation 2.7, and the layer input

x in equation 2.2 is [Q r , Q x , Q y , Q z ] in equation 2.7.

Likewise, Hamilton product of Octonions or Sedenions in hypercomplex space may also be performed by the R2D2 layer when n, k, d are equally set to 8 or 16.

2.5. GENERALIZING HYPERCOMPLEX MULTIPLICATION WITH NEURAL APPROXIMATION:

In fact, weight reuse by component-wise partitioning in Quaternion space has demonstrated reasonable success (Parcollet et al., 2018; Zhu et al., 2018; Parcollet et al., 2019; Tay et al., 2019) . However, one key problem is that Hypercomplex algebra cannot be generalized to arbitrary n values simply because algebraic axioms cannot hold when n takes values apart from power of 2. At this point, it is easy to see that 5-dimensional complex numbers cannot exist due to a clear inability to multiply two 5dimensional complex numbers. Within the context of hypercomplex space, specialized multiplication rules, such as Hamilton product, have to be devised and encoded in the network as a fixed inductive bias. In sharp contrast, the R2D2 layer learns such specialized multiplication rules from data, as manifested in partitioned parameter blocks A i (i = 1, . . . , n) in equation 2.3. Thus, R2D2 subsumes and is degenerated to hypercomplex multiplication when such partitioned parameter blocks are set to reflect those predefined multiplication rules in hypercomplex space. Moreover, the proposed R2D2 layer can be seen as a trainable and parameterized form of n-dimensional hypercomplex multiplication, where n can be values other than power of 2.

To sum up, the R2D2 layer reduces parameterization cost by reusing partitioned parameter blocks. It also offers a way to bridging multiplication between real space and hypercomplex space.

3. EFFICIENT NEURAL NLP MODELS WITH R2D2 LAYERS:

To demonstrate the applicability of the R2D2 layers, we develop two popular neural network models, LSTM and Transformer, which are able to benefit from R2D2 layers.

3.1. R2D2-LSTMS:

Recurrent neural networks such as LSTMs are gated recurrent networks where gating functions are parameterized by linear transformations. We introduce the our efficient LSTM (R2D2-LSTM), which replace such linear transformations in the LSTM with the R2D2 layers:

y t = R2D2 (x t ) + R2D2 (h t-1 ) + b f t , i t , o t , x t = φ(y t ) c t = σ s (f t ) c t-1 + σ s (i t ) σ t (x t ) h t = o t c t

where σ s is the sigmoid activation function and σ t is the tanh activation function, φ : R 1×d → R 4× d 4 is a four way split on the last dimension, c t , h t are the cell state and hidden state of the R2D2 -LSTM unit at time step t.

3.2. R2D2 TRANSFORMERS:

Transformers (Vaswani et al., 2017) , the state-of-the-art model for sequence transduction task, is a stacked neural network architecture that aggressively exploits linear transformations. Each selfattention layer comprises of three Q, K, V transformations, along with N h heads per self-attention layer. Each layer also comprises of a two layered FC layer with nonlinearities. A large majority of the Transformer parameters stem from the FC layers. In R2D2 Transformers, we replace all FC layers with R2D2 layers. The single-headed self-attention module is rewritten as:

Q, K, V = Φ(R2D2(X)) , A = softmax( QK √ d k )V,

where d k is the key dimension, Φ : R 1×d → R 3× d 3 is a three way split on the last dimension, X is the input sequence, and A is the self-attentive representation. For multi-headed, using R2D2 layers also enables weight sharing not only among Q, K, V but also among heads. attention, the transformation of multiple heads are also projected with R2D2 layers:

X = R2D2([H 1 ; . . . ; H N h ]),

where N h is the number of heads and (; ) is the column-wise concatenation. Finally, the position-wise FC layer is now defined as

Y = R2D2(ReLU(R2D2(X))),

which transforms X with 2 R2D2 layers.

4. EXPERIMENTS:

This section reports the experimental results of R2D2-LSTMs and R2D2 Transformers that are equipped with R2D2 layers. Overall, we conduct 4 main experiments in which all experimental settings and hyperparameters (e.g., parameter initialization) remain consistent with the vanilla version. We evaluate R2D2 LSTMs on natural language inference (NLI) and compare them with standard LSTMs. Next, we evaluate R2D2 Transformers and compare them with standard Transformers on neural machine translation (NMT), text style transfer, and subject verb agreement (SVA) tasks.

4.1. NATURAL LANGUAGE INFERENCE:

The task of natural language inference is to determine if two sequences entail or contradict with each other (MacCartney, 2009) . NLI is a fundamental task pertaining to language understanding. To this end, they serve as a suitable benchmark for evaluating recurrent models. 4) QNLI (Quora) (Wang et al., 2017) , and ( 5) Scitail (Science Entailment) (Khot et al., 2018) . We implement 300dimensional unidirectional encoders with shared parameters for both premise/hypothesis. We take the concatenation of max and mean pooled representations as input to a two-layered 300-dimensional MLP for prediction. Our model is trained with the Adam with a learning rate of 0.0004 with a batch size of 256. Word embeddings are initialized with GloVe (Pennington et al., 2014) and are fixed. No cross sentence attention (Parikh et al., 2016) is used, mainly to observe the effectiveness of standalone encoders. For R2D2-LSTM, we use n = {2, 5, 10} and report the results accordingly. In this task, since word embeddings are 300-dimensional, we select multiples of 5 instead of 4 for simplicity and ease of divisibility.

Experimental Results:

Table 1 reports the results on all the 5 natural language inference datasets. All in all, results are extremely encouraging, showing that the R2D2 layer can not only reduce the parameterization cost but also improve performance (4 out of 5 datasets show reasonable improvement). On QNLI (the only exception), the performance drop is marginal (< 1%), which is decent considering the parameter savings. It is also noteworthy that on SNLI, DNLI, and SciTail, all the R2D2 -LSTM variants outperform the base LSTM model. Overall, we think that the parameter reuse properties, in addition to learning to share such reused parameter blocks amongst recurrent gating functions, may contribute to a regularizing effect.

4.2. NEURAL MACHINE TRANSLATION:

Machine translation (MT) is concerned with translating between source-target language pairs. To this end, sequence transduction models are central to this problem domain. In this experiment, the key goal is to compare R2D2 Transformers against the standard Transformer model. 

Datasets and Setup:

We run experiments on 8 NMT datasets. The datasets are (1) IWSLT'15 English-Vietnamese (En-Vi) † , (2) IWSLT'17 English-Indonesian (En-Id) † , (3) IWSLT'14 German-English (En-De) † , (4) IWSLT'14 Romanian-English (Ro-En) † , (5) WMT'18 English-Estonian (En-Et) * , (6) Setimes English-Macedonian (En-Mk) * , (7) WMT'18 English-Romanian (En-Ro) * and (8) WMT'16 English-German (En-De).

Datasets with † are run with 50K steps while datasets with * are trained for 100K steps also on a single GPU. For all tasks, we specify that Transformers have 4 layers, 8 heads, and a hidden size 512. We use beam size of 5 and α = 0.6 (length penalty) for evaluating all the models. For all R2D2 models, we benchmark several settings for the hyperparameter n = {2, 4, 8, 16}. Likewise, we also re-scale the parameters by increasing the dimensions to match the original number of parameters. This is denoted by hyperparameter U . In the experiments, the upsampling factor is fixed at 2.

We benchmark against two baselines that are also concerned with reducing parameterization of the FC layer. The first, is the Factorized Transformer which approximates each FC layer with low-rank approximation. In this case, the number of latent factors is set to d 4 to enable direct comparison to the Quaternion Transformer. The second is the Hamilton product based Transformer proposed in (Tay et al., 2019) .

Experimental Results Table 3 reports our results on machine translation. Across 5 out of 7 NMT benchmarks, R2D2 Transformer outperforms the standard Transformer model. Overall, the empirical results are promising. On one hand, we observe that increasing n all the way to 16 does not cause significant degradation in performance on datasets such as En-Vi. On the other hand, for most datasets, even with significant parameter reduction, we find that the decrease in the BLEU score is overall manageable (≈ 1-3 BLEU points). However, we also note a rare occurrence where n = 16 results in significant decrease in the BLEU score, such as on the En-Id dataset.

On several datasets, the R2D2 Transformer model improves the performance. For example, datasets such as En-Vi and En-Et enjoy a performance boost of about 1 BLEU point with n = 2. Generally, there is only marginal performance degradation at n = 4 or n = 8 for most datasets. Finally, by upsampling with hyperparameter U , we are able to improve the performance of three datasets: En-Vi, En-Id, and En-Mk.

Additionally, we note that Factorized Transformer performs substantially worse than the vanilla Transformer or our R2D2-Transformer. On the other hand, our proposed R2D2 also makes reasonable gains over the Quaternion Transformer (Tay et al., 2019) , signifying that flexible approximation of Hamilton products are effective.

Table 4 reports the decoding and training time for each Transformer variant. We observe that R2D2 with n = 8 has the fastest decoding speed amongst all variants. All in all, the training speed is also approximately similar. As such, this ascertains that our proposed dynamic weight diffusion method is not computationally expensive, despite its complexity.

4.3. TEXT STYLE TRANSFER:

We experiment with sequence transduction for text style transfer. The goal of this task is to convert text of a certain style to another style.

We use the Modern→Shakespeare corpus2 in the experiments. The key goal here is to convert modern writing into Shakespeare writing. This dataset comprises of 18395 parallel sentences for training, 1218 parallel sentences for evaluation (dev set), and 1462 parallel sentences for testing. We still specify that Transformers have 4 layers, 8 heads, and a hidden size 512. Similar to NMT, we experiment with n = {2, 4, 8, 16}. We train all the models for 10K steps. Experimental Results Table 5 reports the results on text style transfer. We observe that the best performance is achieved with R2D2 Transformer (n = 4). Notably, all except the n = 16 variant outperforms the standard Transformer model. This ascertains the effectiveness of the proposed R2D2 layer. This not only enables parameter savings but improves the performance of Transformer.

4.4. SUBJECT VERB AGREEMENT:

We conduct additional experiments on subject-verb agreement task (Linzen et al., 2016) . The task is a binary classification problem. It predicts if the sentence, e.g., 'The keys to the cabinet .' follows by a plural or a singular.

Dataset and Setup This dataset can be found online (Linzen et al., 2016) . We evaluate the R2D2 Transformer encoder against the standard Transformer encoder for the subject verb agreement task. In contrast to the previous experimental settings, we use a smaller Transformer architecture. Specifically, Transformers here have 2 layers, 4 head, and a hidden size 128. Since the hidden size is smaller than those in the previous experimental settings, we experiment with n = {2, 4, 8}. 

5. RELATED WORK:

FC layers are ubiquitous in deep learning research (Goodfellow et al., 2016) . Many, if not all neural building blocks incorporate this functionality, i.e., transforming input vectors to an output space using a parameterized linear transformation matrix. Recurrent neural networks (Hochreiter & Schmidhuber, 1997) , multilayer perceptrons, attention (Bahdanau et al., 2014) , and Transformer models (Vaswani et al., 2017) are all heavily grounded in these transformation operations. Notably, recent state-of-theart models in language domains aggressively exploit transformation layers, achieving very exceptional results (Devlin et al., 2018; Radford et al., 2019) .

Our work can be interpreted as a form of soft parameter sharing, albeit dynamic and learned. A recent work (Savarese & Maire, 2019) proposed soft-weight sharing across stacked convolution layers, effectively simulating recurrence, i.e., RNNs can be considered to be weight sharing across time. Quaternion networks (Zhu et al., 2018; Parcollet et al., 2018; 2019) are also known to inhibit weight sharing qualities and have demonstrated reasonable success despite having fewer parameters.

Low-rank approximations (Markovsky, 2012) and factorization methods are known techniques to reduce parameters and improve generalization. Majority of techniques aimed at reducing parameters of neural networks belong to this class (Sainath et al., 2013; Chen et al., 2018; Si et al., 2017) . Within the context of language, GroupReduce (Chen et al., 2018) applies low-rank approximations to compress neural language models. (Si et al., 2017) proposed memory efficient kernels using low-rank approximation. Block diagonal approaches (Xie et al., 2017) , structured sparsity (Gale et al., 2019) are also highly notable alternatives for improving the FC layer. Methods such as knowledge distillation (Hinton et al., 2015; Freitag et al., 2017) are also capable of reducing model complexity and can be considered an orthogonal direction from ours. In a similar vein, sharing parameters by the means of a meta network (HyperNetworks Ha et al. (2016) also perform a form of weight sharing.

6. CONCLUSION:

We propose R2D2 layers, which can reduce the parameterization cost of their FC counterparts without significantly compromising performance. The R2D2 layer also offers a way to bridging multiplication between real space and hypercomplex space by subsuming and generalizing both matrix multiplication in real space and Hamilton product in hypercomplex space. It is highly modular and applicable to many dominant models such as LSTMs and Transformers. We evaluate such models equipped by R2D2 layers on comprehensive tasks, demonstrating substantial parameter savings with minimal degradation or improvement in performance.

:

Here, the partitioning (or splitting) of weights into different components is analogous to complexification, i.e., splitting real vectors into real and imaginary components.

https://github.com/tlatkowski/st

Figures and Tables:

Figure 1: Illustration of our proposed Dynamic Weight Diffusion. Reusing partitioned parameter blocks A i (i = 1, 2) and S j (j = 1, . . . , 4) leads to a reduction in the parameter size of H (n = 2, d = 8, k = 6). Best viewed in color.

Experimental results on natural language inference. Saving cost considers only the cell unit (other parts of architecture remains constant). R2D2-LSTM reduces the parameter costs of the standard LSTM model and improves performance on 4 out of 5 datasets.

Table content: <table><tr><td>Model</td><td>#Params</td><td>MNLI</td><td>QNLI</td><td>SNLI</td><td>DNLI</td><td>SciTail</td></tr><tr><td>LSTM</td><td>721K</td><td>71.82 / 71.89</td><td>84.44</td><td>84.18</td><td>85.16</td><td>74.36</td></tr><tr><td>Quaternion LSTM</td><td>180K (-75.0%)</td><td>71.57 / 72.19</td><td>84.73</td><td>84.21</td><td>86.45</td><td>75.58</td></tr><tr><td>R2D2-LSTM (n = 2)</td><td>361K (-49.9%)</td><td>71.82 / 72.08</td><td>84.39</td><td>84.38</td><td>85.77</td><td>77.47</td></tr><tr><td>R2D2 LSTM (n = 5)</td><td>146K (-79.7%)</td><td>71.80 / 71.77</td><td>83.87</td><td>84.58</td><td>86.47</td><td>74.64</td></tr><tr><td>R2D2-LSTM (n = 10)</td><td>81K (-88.7%)</td><td>71.59 / 71.59</td><td>84.25</td><td>84.40</td><td>86.21</td><td>77.84</td></tr></table>



Table content: <table/>

Experimental results on neural machine translation (BLEU scores) on WMT'16 En-De.

Table content: <table><tr><td/><td/><td colspan="3">Table 4: Training time (seconds per 100 steps)</td></tr><tr><td>Reported Models</td><td/><td colspan="3">and Decoding Time (seconds to decode test set)</td></tr><tr><td>Transformer (Vaswani et al., 2017) DynamicConv (Wu et al., 2019)</td><td>27.30 29.70</td><td colspan="3">with Beam size of 4 and length penalty of 0.6 on IWSLT 2014 German-English.</td></tr><tr><td>Transformer (our run)</td><td>27.86</td><td>Model</td><td>Train</td><td>Decoding</td></tr><tr><td>Compressed Models</td><td/><td>Vanilla</td><td>7.79</td><td>341</td></tr><tr><td>Quaternion (Tay et al., 2019)</td><td>25.14</td><td>Factorized</td><td>7.26</td><td>291</td></tr><tr><td>R2D2 Transformer (n = 2) R2D2 Transformer (n = 4) R2D2 Transformer (n = 8)</td><td>26.43 26.52 24.11</td><td>Quaternion R2D2 (n = 4) R2D2 (n = 8)</td><td>8.31 8.09 7.89</td><td>297 303 287</td></tr><tr><td>R2D2 Transformer (n = 16)</td><td>23.01</td><td/><td/><td/></tr></table>

Experimental results on text style transfer with Transformer models.

Table content: <table><tr><td>Model</td><td>#Params</td><td>BLEU</td></tr><tr><td>Transformer</td><td>44M</td><td>11.65</td></tr><tr><td>R2D2 (n = 2)</td><td>22M (-50.0%)</td><td>12.20</td></tr><tr><td>R2D2 (n = 4)</td><td>11M (-75.0%)</td><td>12.42</td></tr><tr><td>R2D2 (n = 8)</td><td>5.5M (-87.5%)</td><td>11.66</td></tr><tr><td>R2D2 (n = 16)</td><td>2.9M (-93.4%)</td><td>10.76</td></tr></table>

Experimental results on subject verb agreement (SVA) with Transformer models.

Table content: <table><tr><td/><td/><td/><td>Experimental Results Table 6 reports the re-</td></tr><tr><td/><td/><td/><td>sults on the SVA task. Results are promising,</td></tr><tr><td/><td/><td/><td>demonstrating that all variants with R2D2 layers</td></tr><tr><td>Model</td><td>#Params</td><td>Acc</td><td>outperform the standard Transformer. The best</td></tr><tr><td>Transformer Quaternion</td><td>400K 100K</td><td>94.80 94.70</td><td>performance peaks at n = 8, despite parame-ter saving to up to 1/8. R2D2 also outperforms</td></tr><tr><td>R2D2 (n = 2) R2D2 (n = 4)</td><td>200K (-50.0%) 101K (-74.8%)</td><td>95.14 95.05</td><td>Quaternion Transformers.</td></tr><tr><td>R2D2 (n = 8)</td><td>56K (-86.0%)</td><td>95.62</td><td/></tr></table>

Peer review (iclr2020):
The authors propose R2D2 layers, which are trained to reduce and re-use existing parameters of a neural network layer, and apply this to Transformer and LSTM architectures. The authors conduct experiments on various NLP tasks, including NLI and NMT. The main benefit of the proposed R2D2 layer is that the number of parameters can be reduced, as the existing parameters can be reused. I find this motivation compelling, particularly as it is well known Transformer networks are largely overparameterized. Comments: 1. There is no analysis on the specific choices made for dynamic weight diffusion- the way the partitioning is done could have a large effect on the end result. There*s also little comparison to other ways to share weights across a model besides the proposed weight diffusion method. 2. Sharing parameters contributes a regularization effect - it is difficult to untie the contributions of increased regularization from the proposed method. This is particularly problematic as the majority of the datasets used are *small* by current standards. WMT en-de (authors do not include the sizes of the datasets, but this is 4.5 million sentences) is the only large scale dataset, and the BLEU drop is quite large on this dataset compared to the smaller ones such as IWSLT. To tie my points #1 and #2 together, I feel the authors did experiments on a variety of different tasks, but these style transfer and subject verb agreement tasks are not particularly interesting or realistic - instead this space should be devoted to discussions of the advantages of their method and analysis on its performance, which is quite lightly covered. 3. The authors claim that the R2D2 Transformer outperforms standard Transformer models on 5 out of 7 NMT tasks. This appears true if up-sampling with a factor of 2 is used to make the models larger again. The authors should compare to factorized/quaternion baselines which have a larger quantity of parameters as well. 4. Table 3, where results are reported on the competitive WMT en-de benchmark, lacks comparison for number of parameters and decoding speed. This table would probably have the most compelling and impactful results for this paper as this is the most competitive task (aside from the pre-training regime on MNLI/QNLI as part of GLUE). Can the authors complete this table so readers can understand the parameter reduction and inference speed possible from this method on this benchmark? (As an aside, the technique should be applicable to the DynamicConv model, which is a Transformer variant?) 5. The related work section is quite light on other approaches to reducing model size, such as knowledge distillation or quantization? While the approach taken in this paper leverages parameter sharing, the motivation is similar and I feel acknowledging this entire area of work would be relevant. 6. I*m not clear on why we see inference time decoding speed improvements based on the description of the method. Can the authors clarify this point for me?