Balancing Cost and Benefit with Tied-Multi Transformers
Authors: Raj Dabre
Raphael Rubino
Atsushi Fujita
Venue:  (2020)
Keywords: tied models
encoder-decoder
multi-layer softmaxing
depth prediction
model compression

Abstract: This paper proposes a novel procedure for training multiple Transformers with tied parameters which compresses multiple models into one enabling the dynamic choice of the number of encoder and decoder layers during decoding. In sequence-to-sequence modeling, typically, the output of the last layer of the N-layer encoder is fed to the M-layer decoder, and the output of the last decoder layer is used to compute loss. Instead, our method computes a single loss consisting of NxM losses, where each loss is computed from the output of one of the M decoder layers connected to one of the N encoder layers. A single model trained by our method subsumes multiple models with different number of encoder and decoder layers, and can be used for decoding with fewer than the maximum number of encoder and decoder layers. We then propose a mechanism to choose a priori the number of encoder and decoder layers for faster decoding, and also explore recurrent stacking of layers and knowledge distillation to enable further parameter reduction. In a case study of neural machine translation, we present a cost-benefit analysis of the proposed approaches and empirically show that they greatly reduce decoding costs while preserving translation quality.

Under review as a conference paper at ICLR 2020
BALANCING COST AND BENEFIT WITH TIED-MULTI TRANSFORMERS
Anonymous authors Paper under double-blind review
ABSTRACT
This paper proposes novel procedure for training multiple Transformers with tied parameters which compresses multiple models into one enabling the dynamic choice of the number of encoder and decoder layers during decoding. In sequence- to-sequence modeling, typically, the output of the last layer of the N --layer encoder is fed to the M M-layer decoder, and the output of the last decoder layer is used to compute loss. Instead, our method computes a single loss consisting of N x M losses, where each loss is computed from the output of of the M decoder layers connected to one of the N encoder layers. single model trained by our method subsumes multiple models with different number of encoder and decoder layers, and can be used for decoding with fewer than the maximum number of encoder and decoder layers. We then propose mechanism choose priori the number of encoder and decoder layers for faster decoding, and also explore recurrent stacking of layers and knowledge distillation enable further parameter reduction. In a case study of neural machine translation, we present cost-benefit analysis of the proposed approaches and empirically show that they greatly reduce decoding costs while preserving translation quality.
INTRODUCTION
Neural networks for sestennceto-sequenne modeling typically consist of an encoder and decoder coupled via an attention mechanism. Whereas the very first deep models used stacked recurrent neural networks (RNN) (Sutskever et al.) 2014, Cho et al.] 2014 Bahdanau et al. 2015) in the encoder and decoder, the recent Transformer model (Vaswani al.) 2017) constitutes the current tate-oof-the-art approach, owing to its better context generation mechanism via multi-head self- and cross-attentions.
Given encoder-decoder architecture and its hyper-parameters, such as the number of encoder and decoder layers, vocabulary sizes (in the case of text based models) and hidden layers, the parameters of the model, i.c., matrices and biases for non-linear transformations, updating them so that the loss for the training data is minimized. The pypeer-prrmeterss can also be tuned, for instance, through maximizing the automatic evaluation score the development data. However, in general, it is highly unlikely (or impossible) that a single optimized model suffices di- verse cost-benefit demands at the same time. For instance, practical low-latency scenarios, one may accept some performance drop for speed. However, a model used with a subset of optimized parameters might perform badly. A single optimized model cannot also guarantee the best perfor- mance for each individual input. Although this is practically important, it has drown only little attention. An existing solution for this problem ble choice. However, this approach is not very practical, because it requires unreasonably large quantity of resources. Furthermore, there are no well-established methods for selecting appropriate models for each individual input.
optimized by iteratively
host multiple models simultaneously for flexi-
As a more effective solution, we consider training a single model that subsumes multiple models which can be used for decoding with different hyper-parameter settings depending the input or on the latency requirements. In this paper, we focus on the number of layers as an important hyper- parameter that impacts both speed and quality of decoding, and propose multi-layer softmaxing method, which trains multi-layer neural models referring to the output of all layers during training. Conceptually, illustrated in Figure 1. this method involves tying (sharing) the parameters of mul-

---

Under review as a conference paper at ICLR 2020
Output
Output
Output
softmax
Oulput
ax
Output
softmax
Layer
Output
softmax ax
Layer
Output
softmax
Layer 3
Layer
Output
softmax
Layer 3
softmax
Layer 2
Layer 2
Layer 2
axftmax
Layer 2
Layer
Layer
Layer
Layer
Layer
Input
Input
Input
Input
Input
(a) Multiple lay-r vanilla models.
(b) Collapsing tied layers into one.
The general concept of multi-layer softmaxing for training multi layer neural models with an example of 4-layer model. Figure [alis depiction of our idea in the form of multiple vanilla
Figure 1:
models whose layers are tied together. Figure Tb shows the result of collapsing all tied layers into a single layer. The red lines indicate the flow of gradients and hence the lowest layer in the stack receives the largest number of updates.
tiple models with different number of layers and is not specific to particular types of multi-layer neural models. On top of the above method, we consider exploit the model further. For saving de- coding time, we design and evaluate mechanisms to choose appropriate number of layers depending on the input. As for further model compression, we leverage other orthogonal types of parameter tying approaches, such as those reviewed in Section
Despite the generality of our proposed method, in this paper, we particularly focus on encoder- decoder models with N encoder and M decoder layers, and compress N x M model by updating the model with a total of N M losses computed by softmaxing the output of each the M decoder layers, where it attends to the output of each of the N encoder layers. The number of parameters of the resultant encoder-decoder model is equivalent to that of the most complex subsumed model with N encoder and M decoder layers. Yet, we can now perform faster decoding using fewer number of layers, given that shallower layers are better trained. To evaluate our proposed method, we take the case study of neural machine translation (NMT) (Cho al., 2014 Bahdanau et al. 2013). where we focus on the numbers of encoder and decoder layers of the Transformer model (Vaswani et al.) 2017), and demonstrate that single model with N encoder and M decoder layers trained by our method can be used for flexibly decoding with fewer than N and M layers without appreciable quality loss. We evaluate our proposed method on WMT18 geglish-to-Germaae translation task, and give cost-benefit analysis for translation quality vs. decoding speed.
The rest of the paper organized as follows. Section briefly reviews related work for compressing neural models. Section 3 covers our method that ties multiple models by softmaxing all encoder- decoder layer combinations. Section 4 describes our efforts towards designing and evaluating a mechanism for dynamically selecting encoder-decoder layer combinations prior to decoding. Sec- tion 5 describes two orthogonal extensions to our model aiming at further model compression and speeding- up of decoding. The paper ends with Section containing conclusion and future work.
RELATED WORK
There are studies that exploit multiple layers simultaneously. Wang al. (0088) fused hidden rep- resentations of multiple layers in order to improve the translation quality. Belinkov et al. (2017) and Dou et al. (0018) attempted to identify which layer can generate useful representations for different natural language processing tasks. Unlike them, we make all layers of the encoder and decoder us- able for decoding with any encoder-decoder layer combination. In practical scenarios, we can save significant amounts of time by choosing shallower encoder and decoder layers for inference.
Our method ties the parameters of multiple models, which orthogonal to the work that ties pa- rameters between layers (Dabre & Fujita 2019) and/or between the encoder and decoder within
Rather than casting the encoder-decodee model into single column model with (N M) layers.
2

---

Under review as a conference paper at ICLR 2020
Algorithm Training tied-multi model
1: enco = X
2: for in 1 to N do
3: enc. L.n (encc)))
4: for j in 1 M do
5:
dee; = - (deej-1,
6: Ý softmar(dee
7: lossl, = cross_entropp(
8: end for
9: end for
10: ovcrall_oss = aggregate(loss1,11 I0s8N,M)
11: Back-propagate using overallJoss
single model (Xia et al. 2019 Dabre & Fujita 2019). Parameter tying leads to compact models enabling faster decoding, but they usually suffer from drops inference quality. In this paper, we counter such drops with knowledge distillation (Hinton et al. 2015: Kim & Rush, 2016 Freitag al. 2017). The working principle of this approach to utilize smoothed data or smoothed train- ing signals instead of the actual training data. A model with large number of parameters and high performance provides smoothed distributions that are then used as labels for training small models instead of one-hot vectors.
As one of the aims in this work is model size reduction, related to a growing body of work that addresses the computational requirement reduction. Pruning of pre-trained models (See al. 2016) makes it possible to discard around 80% of the smallest weights model without deteri- oration in inference quality, given it is re-trained with appropriate
after pruning. Currently, most deep learning implementations use 32-bit floating point representations, but 16-bit floating point representations (Gupta et al.) 2015 Out et al. 2018) aggressive binarization (Cour- bariaux et al. 2017) can be alternatives. Compact models are usually faster to decode; studies on quantization (Lin al. 2016) and average attention networks (Xiong al. 2018) address this topic.
To the best of our knowledge, none of the above work has attempted to combine multi-model param- eter tying, knowledge distillation, and dynamic layer selection for obtaining and exploiting highly- compressed and flexible deep neural models.
MULTI-LAYER SOFTMAXING
3.1 PROPOSED METHOD
Figure D gives simple overview of the concept of multi-layer softmaxing, taking generic model as an example. The rightmost 4-layer model takes an input and passes it through 4 layer before a softmax layer to predict the output. Typically, one would apply softmax to the 4th layer only, com- pute loss, and then back- proppgat gradients in order to update parameters. Instead, we propose to apply softmax to each layer, aggregate the computed losses, and then back- pack-propagate. This enables us to choose any layer combination during decoding instead of only the topmost layer.
Extending this to multi-layer iccoder-decoder model is straightforward. In incoderddecodd mod- els, the encoder comprises an embedding layer for the input (source language for NMT) and N stacked transformation layers. The decoder consists of embedding layer and softmax layer for generating the output (target language for NMT) along with M stacked transformation layers. Let X be the input the -llyy encoder, Y the anticipated output of the M- Mlyeer decoder as well as the input to the decoder (for training), and Y the predicted output by the decoder. Algorithm shows the pseudo-code for our proposed method. The line 3 represents the process done by the i-th encoder layer, L and the line 5 does same for the j-th decoder layer, Lgec, In simple words, we compute loss using the output of each of the M decoder layers which in turn is computed using
We make assumptions about the nature of the layers.
3

---

Under review as a conference paper at ICLR 2020
the output of each of the N encoder layers. In line 10, the N M losses aggregated before back-pue prckk-rrpaggtio. Henceforth, we will refer to this as the Tied-Mult model.
For a comparison, the vanilla model is formulated as follows: decg. = I so fttaa and overall Joss cross_entropy((,
y
3.2 EXPERIMENTAL SETUP
We evaluated the following two types of models on both translation quality and decoding speed.
Vanilla models: 36 vanilla models with I to encoder and to 6 decoder layers, each trained referring only to the last layer for computing loss.
Tied-Multi model: A single tied-multi model with N 6 encoder and M 6 decoder layers, trained by our multi-layer softmaxing.
We experimented with the WMT18 English-to English-to-German (En->De) translation task. We used all the parallel corpora available for WMT18, except ParaCrawl corpus consisting of 5,58M sentence pairs as the training data and 2,998 sentences in newstest2018 test data. The English and Ger- man sentences were pre-processed using the tokenizer, perl and truecase per scripts The true-case models for English and German were trained on 10M sentences ran- domly extracted from the monolingual data made available for the WMT18 translation task, using the t train-tuuecsser....... perl script available in Moses.
in Moses {
Our multi-layer softmaxing method was implemented on top of an open toolkit of the Trans- former model (Vaswani et al., 2017) in the version 1.6 branch of tensor2tensor For training, we used the default model settings corresponding to trans former base.s www.e.ssii.e single nggle.gpu in the implementation, except what follows. We used a shared sub- subwword vocabulary of 32k determined us- ing the internal sub-word segmenter of tensor2tensor and trained the models for 300k iterations. We trained the vanilla models on GPU and our tied-multi model on 2 GPUs with batch size halved to ensure that both models see the same amount of training data. We averaged the last 10 checkpoints saved every after Ik updates, and decoded the test sentences, fixing a beam size of 4 and length penalty of 0.6. We evaluated our models using the BLEU metric (Papineni et al. 2002) implemented in BLC (Post) 2018) R Prior to evaluation, the decoded test set was post-processee using the detoken zer perl and det ruecase perl scripts in Moses. We also report the time (in seconds) consumed to translate the test set, which includes times for the model instantiation, loading the checkpoint, sub-word splitting and indexing, decoding, and sub-word de-indexing and merging, whereas times for detokenization are not taken into account.
Note that we did not use any development data for two reasons. First, we train all models for the same number of iterations Seconn,, we use checkpoint averaging before decoding, which does not require a development set unlike early stopping. We adopted this approach, because known to give the best results for NMT using the Transformer implementation we use (Vaswani et al. 2017).
We averaged multiple losses in experiment, but there are number of options, such as weighted aver- aging.
|http:/ / http://wttttttttt. wwwpaatttttttog///ieooeeeeo http:/tt/////eo.oo//////oore//.t http://tttt/tttttttatsasseassaee www.iionntttaaaaa...e.o.
We excluded ParaCrawl following the instruction on the WMT18 website: "BLEU score dropped by 1.0" for this task.
"One can realize faster decoding by narrowing down the heam width. This approach is orthogonal ours and in this paper we do not insist which is superior to the other.
https:: / thub. wttpbb...o///josssssa...o
M..FUUcase..mxed--eee whixed-lang.en-de-mmiee
+ #test.wmtl weessswmtt8+tkk. 1tt.l1.......aaarttmre 334verrionn 1.3..7
9This enables fair comparison, because it ensures that each model sees roughly the same number of training examples.
4

---

Under review as a conference paper at ICLR 2020
36 vamilla models
BLEU
Single tied multi model
Dacoding time (see)
6
2
26.3
30.3
31.9
32.2
324
32.9
23.2
28.6
30.3
30.8
31.2
31.3
91.3
112.4
130.9
180.2
218.2
254.2
2
28.6
32.5
33.1
33.3
33.5
33.2
26.5
31.5
33.0
33.6
33.8
34.0
92.8
112.7
148.5
178.7
224.5
255.5
32.5
151.1
240.6
29.8
33.6
34.3
34.7
34.4
34.3
28.3
33.0
34.3
34,8
34,9
34.9
92.4
114.4
151.5
193.7
231.9
262.4
30.7
339
34.6
35.5
34,4
35.0
28.6
33.1
34.5
34,8
35.0
35.1
94.4
113.4
161.5
194.7
241.5
261.5
340
35.0
33.1
34.6
34.
35.0
93.2
114.9
158.3
246.3
Table 1: BLEU scores of 36 separately trained vanilla models and our single tied-multi model used with n (1 I N) encoder and m < m < .M) decoder layers. One of decoding times is also shown given the fact that a vanilla model with n encoder and m decoder layers and our tied-multi model have identical shapes when used with encoder and m decoder layers for decoding.
3.3 RESULTS
Table summarizes the BLEU scores and the decoding times of all the models, showing the cost- benefit property of our tied-multi model in comparison with the results of the corresponding 36 vanilla models.
Even though the objective function for the miulmmutti model substantially more complex than the one for the vanilla model, when performing decoding with the 6 encoder and 6 decoder layers, it achieved a BLEU score of 35.0, which approaching to the best BLEU score of 35.7 given by the vanilla model with 6 encoder and 4 decoder layers 10 Note that when using single encoder layer and/or a single decoder layer, the vanilla models gave significantly higher BLEU score than the tied- multi model. However, when the number of layers is increased, there is no significant difference between the two types of models: less than 1.0 BLEU point differences.
Regarding the cost-benefit property of our tied-multi model, two points must be noted:
BLEU score and decoding time increase only slightly, when we use more encoder layers.
The bulk of the decoding time is consumed by the decoder, since works in an auto- regressive manner. We can substantially cut down decoding time by using fewer decoder layers which does lead to sub-optimal translation quality.
One may argue that training single vanilla model with optimal number of encoder and decoder layers is enough. However, as discussed in Section I is impossible to know priori which com- bination is the best. More importantly, single vanilla model cannot suffice diverse cost-benefit demands and cannot guarantee the best translation for any input (see Section 4.1). Recall that we aim at flexible model and that all the results in Table T have been obtained using single tied-multi model, albeit using different number of encoder and decoder layers for decoding.
3.4 ANALYSIS AND DISCUSSION
We conducted analysis from the perspective of training times and model sizes, in comparison with vanilla models.
Training Time:
Given that all our models were trained for the same number of iterations, we compared the training times between vanilla and tied-multi models. As a reference, we use the vanilla model with 6 encoder and 6 decoder layers. The total training time for all the 36 vanilla models was 25.5 times M that of the reference model. In contrast, the training time for our tied- multi model was about 9.5 times that of the same reference model. Unexpectedly, training a tied- multi model was much more computationally efficient than independently training all the 36 vanilla models with different number of layers.
Model Size:
The number of parameters of our tied-multi model is exactly the same as the vanilla model with N encoder and M decoder layers. If we train a set of vanilla models with different
We consider this as an oracle, since we have no prior information about the best layer combination. We measured the collapsed time for a fair comparison, assuming that all vanilla models were trained on single GPU one after another, even though one may be able to use multiple GPUs to train the 36 vanilla models in parallel.
5

---

Under review as a conference paper at ICLR 2020
Player of dacoder for decoding
PLayer decoder used for cecoding
2
9
ALeyer el errodde usvd for decoding
ALayer of eroode used for deecoding
4.1%
1.7%
2.0%
2.0%
2.0%
2.1%
6.0%
5.0%
5.0%
3.%%
2.5%
2.8%
2,8%
2,5%
2.2%
2.0%
2.5%
2.5%
4.4%
5.4%
4.0%
3.1%
2,8%
2.0%
1.8%
2.4%
3.4%
2.7%
2.4%
2.2%
3.4%
3.0%
3.5%
3.4%
2.5%
3.1%
3.2%
2.4%
2.7%
244%
3.0%
3.7%
1.7%
2.5%
2.1%
3.4%
3.7%
0.4%
0.7%
0.0%
(a) 36 vanilla models.
(b) Single tied-multi model.
Figure 2: Distribution of oracle translations determined by chrF scores between reference and each of the hypotheses derived from the 36 combinations of encoder and decoder layers (2,998 sentences).
numbers of encoder and decoder layers, we end up with significantly more parameters. For instance, in case of N = .M- in our experiment, we have 25.2 times more parameters: total of 4,607M for the 36 vanilla models against 183M for our tied-multi model. We discuss the possibility of further model compression in Section
DYNAMIC LAYER SELECTION
To better understand the nature of our proposed method, analyzed the distribution of oracle trans- lations within 36 translations generated by each of the vanilla and our tied-multi models. Having confirmed that single encoder-decoder layer combination cannot guarantee the best translation for any input, we tackled an advanced problem: designing a mechanism for dynamically selecting one layer combination prior to decoding.
4.1 DECODING BEНAVIIR TIED-MULTI MODEL
Let (n.m) be an encoder-decoder layer combination of a given model with encoder and m de- coder layers. The oracle layer combination for an input sentence was determined by measuring the quality of the translation derived from each layer combination. We used reference-based metric, chrF (Popović) (016)), since has been particularity designed to sentence-level translation evalua- tion and shown to have relatively high correlation with human judgment of translation quality for the language pair involved in our experiment (Ma et al. (2018). In cases where multiple com- binations have the highest score, we chose the fastest one following the overall trend of decoding time (Table [). Formally, we considered a combination (n1, is faster than another combination (n2,, m2) if the following holds.
(n1, m2) < (71, m2) = m2 my V (m1 m2 A n1 < <22)
(I)
Figure presents the distribution of oracle layer combinations for the vanilla and our muut-multi models. comparison between the two distributions revealed that the shallower layer combinations in our tied-multi model often generates better translations than deeper ones unlike the vanilla models, thanks to the multi-layer softmaxing method, despite the lower corpus-level BLEU scores. This sharp bias towards shallower layer combinations suggests the potential reduction of decoding time by dynamically selecting the layer combination per input sentence prior to decoding, idcally without performance drop.
In what follows, we present our method for dynamic selection of layer combination and experiments.
4.2 METHOD
We formalize the encoder-decoder layer combination selection with supervised learning approach where the objective is to minimize the following loss function
1
arg min
C(((2':))
X'

---

Under review as a conference paper at ICLR 2020
where z' is the i-th input sentence (1 < i < [X), yf is the translation for x' derived from the k-th layer combination (1 S A I K = N x M) among all K possible combinations, f is the model with parameters 8, and L is loss function. Assuming that the independence of target labels (layer combinations) for a given input sentence allows for ties, the model is able to predict multiple layer combinations for the same input sentence.
The model f with parameters 0 implemented in our experiment is a multi-head self-attention neural network inspired by Vaswani et al. (2017). The number of layers and attention heads are optimized during a hyper-parameter search, while the feed-forward layer dimensionality is fixed to 2048. Input sequences of tokens are mapped to their corresponding embeddings, initialized by the embedding table of the beed-mutti NMT model. A specific token is appended to each input sequence before being fed the classifier. This token is finally fed during the forward pass to the output linear layer similarly to BERT (Devlin et al. 2019) for sentence classification. The output linear layer has K dimensions, allowing to output as many logits as there are layer combinations in the tied-multi NMT model. Finally, sigmoid function outputs probabilities for each layer combination among the K possible combinations.
The parameters 0 of the model are learned using mini-batch stochastic gradient descent with Nes- terov momentum (Sutskever et al. 2013) and the loss function L. implemented as a weighted binary cross-entropy (BCE) function detailed in and averaged over the batch.
LBCE. wi [ōk:% log (z) + (1 yk) log(1 (x))),
(3)
where 6x = (1 - p(yk) ) is the weight given to the k-th class based on class distribution prior knowledge and o the sigmoid function. During our experiment, we have found that the classifier tends to favor recall in detriment to precision. To tackle this issue, we introduce another loss using an approximation of the macro F's implemented following
CF2 = 11 --(((++8)) (1
P-R
(4)
(B2 P) + R.
with P=
1
R
1
and μ
Σ (vx yk),
Σx yic
Σnvk Uk
where yk is the output of (I)) for the k-th dimension. The final loss function is the linear interpo- lation of LECE averaged over the K classes and LF, with parameter X: À x LaCE (1 1--- A) x Lry- We tune a. 3, and Ả during the classifier hyper-parameter search based on the validation loss.
4.3 EXPERIMENTS
The layer combination classifier was trained on a subset of the tied-multi NMT model training data presented in Section 3.2 containing 5.00M sentences, whereas the remaining sentences compose a validation and a test set containing approximately 200k sentences each. The two latter subsets were used for hyper-parameter search and evaluation of the classifier, respectively. To allow for comparison and reproducibility, the final evaluation of the proposed approach in terms of translation quality and decoding speed were conducted on the official WMT development (newstest2017, 3,004 sentences) and test (newstest2018, 2,998 sentences) sets; the latter is the one also used in Section
The training, development, and test sets were translated with each layer combination of the tied- multi NMT model. Each source sentence was thus aligned with 36 translations whose quality were measured by the chrF metric. Because several combinations can lead the best score, the obtained dataset was labeled with multiple classes (36 layer combinations) and multiple labels (ties with re- gard the metric). During inference, the ties were broken by selecting the layer combination with the highest value given by the sigmoid function, or sacking-off to the deepest layer combination (6, 6) if no output value reaches 0.5. This tie breaking method differs from the oracle layer selection presented in Equation (1) and in Figure 2 which prioritizes shallowest layer combinations. In this experiment, decoding time was measured by processing one sentence at a time instead of batch de- coding, the former being slower compared to the latter, but leading to precise results. The decoding times were 954s and 2.773s when using a (1,1) and a (6,6) layer combination, respectively. By selecting the fastest encoder-decoded layer combinations according to oracle. the decoding times went down to 1,918s and 1,812s for the individual and tied-mut multi models, respectively. However, our objective is to be faster than default setting, that is, where would choose a (6,6) combination.
7

---

Under review as a conference paper at ICLR 2020
Classilier
Fine-tuning
Speed
BLEU
Baseline (tied (6,6))
2,773
35.0
Oracão (tied)
1.812
42.1
laycrs, beads
X
2.736
35.0
layers, heads
2,686
34.8
layers, heads
layers, heads
2,563
34.3
Table 2: Dynamic layer combination selection results in decoding speed (in seconds, batch size of 1) and BLEU, including the baseline and oracle for the WMT newstes22018 corpus using the tied-multi model architecture.
Several classifiers were trained before being evaluated on the WMT test set, with or without fine- tuning the WMT development decoding speed are presented in Table 2 Some classifiers maintain the translation quality (top rows), whereas others show quality degradation but further gain decoding speed (bottom rows).
The results obtained in terms of corpus-level BLEU and
4.4 ANALYSIS
The classification results show that gains in decoding speed are possible with a-priori decision for which encoder-decoder combination to select, based on the information contained in the source sentence only. However, no BLEU gains are observed in the results presented in Table trade- off between decoding speed and translation quality is present. Our best configuration for decoding speed reaches 210s decrease but leads to 0.7 point BLEU degradation. In addition, only 37s are gained when preserving the translation quality compared to the baseline configuration. The layer combination oracle indicates substantial gains both terms of BLEU (7 points) and decoding speed (961s). These oracle results motivate possible future work in layer combination prediction for the tied-multi NMT model.
FURTHER MODEL COMPRESSION
We examined the combination of our multi-layer softmaxing approach with another parameter-tying method in neural networks, called recurrent stacking (RS) (Dabre & Fujital 2019), complemented by sequence-leve knowledge distillation (Kim & Rush 2016), specific type of knowledge distillation (Hinton et al.) 2015). We demonstrate that these existing techniques help reduce the number of parameters in model even further.
5.1 DISTILLATION INTO RECURRENTLY STACKED MODEL
In Section 2 have discussed several model compression methods orthogonal multi-layer soft- maxing. Having already compressed N x M models with our approach, we consider further com- pressing it using RS. However, models that use RS layers tend suffer from performance drops due to the large reduction in the number of parameters. As way of compensating the performance drop, we apply sequence-level knowledge distillation.
First, we decode all source sentences in the training data to obtain pheudo-paralle corpus contain- ing distillation target sequences. By model which makes learning easier and hence is able to mimic the behavior of the parent model. Then, an RS child model is trained with multi-layer softmaxing on the generated pseudo-parallel corpus. Among variety of distillation techniques, we chose the simplest one to show the impact that distillation can have in our setting, leaving an extensive exploration of more complex methods for the future.
the data, we create soft-targets for the child
5.2 EXPERIMENT
We conducted experiment to show that RS and sequence distillation can lead to extremely compressed mied-multi model which no longer suffers from performance drops. We compared the following four variations of our tied-multi model trained with multi-layer softmaxing.
8

---

Under review as a conference paper at ICLR 2020
Tied-multi model
Tied-multi RS
mm
6
3
13.2
28.6
30.3
30.8
31.2
31.3
25.7
29.8
30.6
30.3
30.7
30.9
26.5
31.5
33.0
33.6
34.0
28.5
32.3
32.9
33.0
33.1
33.2
without
27.8
32.5
33.9
34.6
34.7
34.7
29.2
32.9
33.6
33.8
33.6
33.5
distillation
28.3
33.0
34.3
3488
34.9
34.9
29.3
33.2
33.7
33.9
33.6
33.7
28.6
33.1
34.5
34.8
35.0
35.1
29.4
13.2
33.7
33.9
33.9
34.0
28.7
33.1
34.6
34.7
34.9
35.0
29.2
33.2
33.7
33.9
34.0
13.8
30.1
34.0
35.1
35.3
35.6
35.7
312
13.5
34.1
34,2
34.3
34,3
33.4
35.8
36.6
36.8
17.1
37.3
33.7
35.5
35.7
35.7
35.8
35.8
with distillation
34.7
36.5
37.0
37.4
37.4
37,5
34.1
15.8
36.1
36.1
36.2
36.2
35,2
36.8
37.2
37.4
37.5
37.5
34.3
36.0
36.2
36.2
36.3
36.3
35.5
37.1
37.4
37.6
36.2
36.3
36.3
35.5
37.0
37.2
37,5
37.6
37.6
34.6
36.1
36.2
36.2
36.3
36.2
Table 3: BLEU scores of total of four NMT models: the tied-multi model with (left block) and without (center and right blocks) RS layers, each trained with (top block) and without (bottom block) sequence distillation.
Tied-multi model: A model that does not share the parameters across layers, trained on the original parallel corpus.
Distilled tied-multi model: The same model as above but trained on the pseudo-parre parallel corpus.
Tied-multi RS model: A tied-multi model that uses RS layers, trained with the original parallel corpus.
Distilled tied-mull multi RS model: The same model as above but trained on the pseudo-paras cor- pus.
Note that we defrayed much higher cost for training the distilled models than training models directly on the original parallel corpus. First, we trained vanilla models with encoder and decoder layers, because the performance of distilled models is affected by the quality of parent models, and NMT models vary vastly in performance (around 2.0 BLEU) depending on parameter initialization. We then decode the entire training data (5.58M sentences) with the ond with the highest BLEU score on the newstest2017 (used in Section 4.3) in order to generate pseudo- parallepaalllll corpus for sequence distillation. Nevertheless, we consider that we can fairly compare the performance of the above four models, since they were trained only once with random parameter initialization, without seeing the test set.
Table Eliives the BLEU scores for all models. Comparing top-left and top-right blocks of the table, can see that the BLEU scores for RS models are higher than their non-RS counterparts when using fewer than 3 decoder layers. This shows the benefit of RS layers despite the large parameter reduction. However, the reduction in parameters negatively affects (up to 1.3 BLEU points) when decoding with more decoder layers, confirming the limitation of RS as expected.
Comparing the scores of the top and bottom halves of the table, we can see that distillation dramat- ically boosts the performance of the shallower encoder and decoder layers. For instance, without distillation, the tied-multi model gave a BLEU of 23.2 when decoding with 1 encoder and decoder layers, the same layer combination reaches 30.1 BLEU through distillation. Given that RS further improves performance using lower layers, the BLEU score increases to 31.2. As such, distillation enables decoding using fewer layers without substantial drops in performance. Furthermore, the BLEU scores did not vary significantly when the layers deeper than were used, meaning that we might well train shallower models using distillation. The performance of our final model, i.e., the distilled tied-multi RS model (bottom-rrihht, was significantly lower (up to 1.5 BLEU points) similarly its non-distilled version. However, given that it outperforms our original tied-multi model (top-left) in all the encoder-decoder layer combinations, we conclude that we can obtain a substantially compressed model with better performance.
Ensemble of multiple models is commonly used for distillation, but we used a single model to save decod-
ing time.

---

Under review as a conference paper at ICLR 2020
Modelís)
Parameters
Relative
36 vanilia models
4,608M
26.16
Single tied-mulli model
183M
1.00
36 RS models
2,623M
14.35
Single tied-multi RS model
73M
0.40
Table 4: Model sizes for different encoder-decoder layer combinations. The relative size caleu- lated regarding the tied-multi model as a standard. Similarly to "36 vanilla models," "36 RS models" represents the total number of parameters of all RS models.
5.3 ANALYSIS
We now analyze model size and decoding speed resulted by applying RS and knowledge distillation. Note that RS has effect on training time because the computational complexity is the same with or without it.
Table 4 gives the sizes of various models that we have trained and their ratio with respect to the tied-multi model. Training vanilla and RS models with 36 different encoder-decoder layer combinations required 25.2 and 14.3 times the number of parameters of a single tied-multi model, respectively. Although RS led some parameter reduction, combining RS with our tied- multi model resulted further compressed single model: 0.40 times that of the single miudimulti model without RS. This model has 63.2 times and 36.0 times fewer parameters than all the individual vanilla and RS models, respectively. Given that knowledge distillation can reduce the performance drops due to RS (see Table 3), we believe that combining with this approach is an effective way to compress a large number of models into one.
Model Size:
Decoding Speed: Although we do not give scores due to lack of space, we observed that greedy decoding faster than beam decoding but suffers from significantly reduced scores of around 2.0 BLEU. By using our distilled models, however, greedy decoding reduced the scores only by 0.5 BLEU compared beam decoding. This happens because we have used translations generated by beam decoding as target sentences for knowledge distillation, which has the ability to loosely distill beam search behavior into greedy decoding behavior. For instance, greedy decoding with the dis- tilled tied-multi RS model with 2 encoder and 2 decoder layers resulted in BLEU score of 35.0 in Is. In comparison, beam decoding with the tied-multi model without RS and distillation with encoder and decoder layers led BLEU score of 35.1 in 261.5s (Table D.. showing that comparable translation quality is obtained with a factor of 4 decoding time when using RS and distillation. Even though intended to minimize performance drops besides the model compression, ob- tained an unexpected benefit in terms of faster decoding through greedy search without a significant loss in translation quality.
CONCLUSION
In this paper, we have proposed novel procedure for training encoder-decoder models, where the softmax function is applied to the output of each of the M decoder layers derived using the output of each of the N encoder layers. This compresses N M models into a single model that can be used for decoding with variable number of encoder (1 S n. < N) and decoder (1 < m < M) layers. This model can be used in different latency scenarios and hence is highly versatile. We have experimented with NMT as case study of encoder-decoder models and given a cost-benefit analysis of our method. We have proposed and evaluated two orthogonal extensions and show that we can (a) dynamically choose layer combinations for slightly faster decoding and (b) further compress models using recurrent stacking with knowledge distillation.
For further speed up in decoding well model compression, plan to combine approach with other techniques, such as those mentioned in Section[] Although we have only tested our idea for NMT, it should be applicable other tasks based on deep neural networks.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. In Proc. of ICLR, San Diego, USA, May 2015. URL https:
047004433.000 pdf
Yonatan Belinkov, Lluís Mårquez, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. Evaluating layers of representation in neural machine translation part-of-speech and semantic tagging tasks. Pruc of IJCNLP, pp. 1-10, Taipei, Taiwan, 2017. URL http://aclweb: Oxc/aatthology/II-1110
Kyunghyun Cho, Bart van Merriënboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN for statistical machine translation. In Proc. of EMNLP, pp. 1724-1734, Doha, Qatar, 2014. URL
http:://cc org/ anttooooy//D44--- 1179
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to +1
or CoRR, abs/1 10d/0200000200, 2017. URL pttt:://arxiv.oor//bbbs..
02830
Raj Dabre and Atsushi Fujita. Recurrent stacking of layers for compact neural machine translation models. In Proc. of AAAI, pp. 6292-6299, Honolulu, USA, 2019. htt.hhttps/aaat .rg/ ajs/index. AAAT wtt.://tttt.eeeeeeeeee/eeee//tt/
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional Transformers for language understanding, In Proc. NAACI, pp 4171-4186 Min- neapolis, USA, June 2019. URL UtU/:/ppp//aaalee...rr///tttttt/
Zi-Yi Dou. Zhaopeng Tu, Xing Wang, Shuming Shi, and Tong Zhang. Exploiting deep represen- tations for neural machine translation. In Proc. of EMNLP, 4253-4262, Brussels, Belgium, 2018. URL/tttp://aclwel
owg/oogg.o.......... wwtaalagoooyyaoaaoaoooooo BDllooyyyyyy-_/// DD1/ 1457
Markus Freitag, Yaser Al-Onaizan, and Baskaran Sankaran. Ensemble distillation for neural ma- chine translation. CoRR, abs/1702.01802, 2017. URL
01402
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishhan, and Pritish Narayanan. Deep learning with limited numerical precision. In Proc. of ICML, pp. 1737-1746, Lille, France, 2015. URLhttp:
CoRR, //proceedings. mlr ptress//33/quuee hhtml
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. 2015. Uttt::t:://arxi http://vvv.gggbb///saaaa// 02531
Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. In Proc. of EMNLP, pp. 1317-1327, Austin, USA, November 2016.
D16 016-1139
Darryl D. Lin, Sachin S. Talathi, and V. Sreekanth Annapureddy, Fixed point quantization of deep convolutional networks. In Proc. of ICML, pp. 2858, New York, USA, 2016. URLhttp:
wwtp//ottiiiiiiinnoooooo ctt.////ccc.oooo/caaaaaoooooo.o ctt.ttatioon.cmmmmmm0330555555 000.2222d--33330.33455555 3045690,
Qingsong Ma. Ondřej Bojar, and Yvette Graham. Results of the WMT18 metries shared task.
In Proc. of WMT, pp. 682-701, Brussels, Belgium, 2018 URL http://aclweb..oo
anthology/ Www.wW166666 6450
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation. In Proc. of WMT, pp. 1-9, Brussels, Belgium, October 2018. URL htllhttss://ccceer org/ anthology W18 6301
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: A method for automatic evaluation of machine translation. In Proc. ACL, pp. 311-318, Philadelphia, USA, 2002. URL http::/dx..oo doi wtt./0..311//07300000 1073333

---

Under review as a conference paper at ICLR 2020
Maja Popović. chrF deconstructed: parameters and n-gram weights. In Proc. of WMT, pp. 499- 504, Berlin, Germany, 2016. URL httpt://///:tbbb..... org/
Matt Post. A call for clarity in reporting BLEU scores. In Proc. of WMT, pp. 186-191, Brussels, Belgium, 2018. URL httLtttss://ww.accttt
www//ooog/aaeeeeeoo. anwataooooooo/oaaeoo /W18 6319
Abigail See, Minh-Thang Luong, and Christopher D. Manning. Compression of neural machine translation models via pruning. In Proc. of CoNLL, pp. 291-301, Berlin, Germany, 2016. URL
https::/aclweb.. http:///owee...r/r///toooooooo//
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance initial- ization and momentum in deep learning. In Proc. of ICML, pp. 1139- Atlanta, USA, 2013.
URL http:tttpppprroceeiiime
mir: htt.p/ppeee//////tsttseeeeeeeeee
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks.
In Proc. of NIPS. pp. 3104-3112, Montréal, Canada, 2014. URL URL/http://paperss nips. cc/ sequence www.eieaannnn wwiinngwwttti new.alllaallleeeeeeeee ntt
paper seuee//eee-eeeeeeeeee-e---e- sequence to -eseuecee----e-eeee------ pdf
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. of NIPS, pp. 5998-6008, Long Beach, USA, 2017. URL htttp://papera nips ctipps.cc/aper/ 7181-attention--le -otpttiiooo--sll--o-----e-ee-ee
pdf
Qiang Wang, Fuxue Li, Tong Xiao, Yanyang Li, Yinqiao Li, and Jingho Zhu. Multi-layer repre- sentation fusion for neural machine translation. In Proc. of COLING, pp. 3015-3026, Santa Fe, USA, 2018. http:t::: //tttt:://ccwbbb..../ http://www.....aaoooooooo///////
1255
Yingce Xia, Tianyu He, Xu Tan, Fei Tian, Di He, and Tao Qin. Tied Transformers: Neural machine translation with shared encoder and decoder. In Proc. of AAAI, pp. 14466-573, Honolulu, USA, 2019. URL https::ss/// http://aaaa..o. org //wndeee..eeex..e..ee//e//i/rei. index. whtp/AAAT/articieeeeeeeee
4487
Deyi Xiong. Biao Zhang, and Jinsong Su. Accelerating neural Transformer via an average attention network. In Proc. of ACI, pp. 1789-1798, Melbourne, Australia, 2018. URL http: //acl/ web.
org/ hrg/antho/l oogll
12


Reviews:
[{"review_id": "Syl23DhcFr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "** Summary **\nIn this paper, the authors propose a new variant of Transformer called Tied-multi Transformer. Given such a model with an N-layer encoder and an M-layer decoder, it is trained with M*N loss functions, where each combination of the nth-layer of the encoder and the mth-layer of the decoder is used to train an NMT model. The authors propose a way to dynamically select which layers to be used when a specific sentence comes.  At last, the authors also try recurrent stack and knowledge to further compress the models.\n\n** Details **\n1.\tThe first question is “why this work”:\na.\tIn terms of performance improvement, in Table 2, we can see that dynamic layer selection does not bring any improvement compared to baseline (Tied(6,6)). When compared Tied(6,6) to standard Transformer, as shown in Table 1, there is no improvement. Both are 35.0.\nb.\tIn terms of inference speed, in Table 2, the method can achieve at most (2773-2563)/2998 = 0.07s improvement per sentence, which is very limited.\nc.\tIn terms of training speed, compared to standard Transformer, the proposed method takes 9.5 time of the standard Transformer (see section 3.4, training time).\nTherefore, I think that compared to standard Transformer, there is not a significant difference.\n2.\t The authors only work on a single dataset, which is not convincing.\n3.\tIn Section 5, what is the baseline of standard RS + knowledge distillation?\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "H1gErESpKr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes a way to reduce the latencies incurred in inference for neural machine translation. Basic idea is to train a model with softmax attached to each output of decoder layers, and computes a loss by aggregating the cross entropy losses over the softmaxes. During inference, it could either use one of the softmax or train an additional model which dynamically selects softmaxes given an input string. Experimental results show that it is possible to reduce latencies by trading off the translation qualities measured by BLEU. Dynamically selection did not show any gains in latencies, though, this work empirically shows potential gains in oracle studies. This work further shows that the model could be compressed further by knowledge distillation.\n\nI have several concerns to this work and I'd recommend rejecting this submission.\n\n- One of the problems of this paper is presentation. This work basically combines three work together as a single paper, i.e., section 3 for the basic model, section 4 for dynamic selection and section 5 for distillation, with each section describing a separate experiment. I'd strongly suggest the author to focus on the main point, e.g., dynamic selection, and present the basic model and dynamic selection. Experiments should be presented in a single section for brevity.\n\n- Similarly, this work should have been submitted when meaningful gains were observed in the dynamic selection method, given that the proposal is somewhat new. Otherwise, I don't find any merits to see this accepted in ICLR, given the rather negative results in section 4.\n\n- The description in section 4.2 is totally messed up. x^i and y^i_k are strings since they are an input sentence and an output sentence, respectively,. However, they are treated as scalars in Equation 3 by multiplied with \\delta_k, subtracted from 1 and taking sigmoid through \\sigma. I strongly suggest authors to carefully check variables used in the equations and the description in the section.\n\n- The authors claim that the use of knowledge distillation is novel. However, it is already widely known in the research community and I don't think it is worthy to keep it as a single section. It could have been described as a yet another experiment in a single experimental section.\n\nOther comment:\n\n- Although this paper claims that attaching a softmax for each output layer is new, there is a similar work in language modeling, though the motivation is totally different.\n\n  Direct Output Connection for a High-Rank Language Model, Sho Takase, Jun Suzuki and Masaaki Nagata, EMNLP 2019.\n\n- In section 3.4, this paper claims that the training of all 36 models took 25.5 more time, but took 9.5 more time for a tied-model when compared with a basic 6-layer Transformer. It is not clear to me whether this comparison is meaningful given that it might be possible to employ multiple machines to train 36 models.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1liTs2TYS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper proposes a novel procedure for training multiple Transformers with tied parameters which compresses multiple models into one enabling the dynamic choice of the number of encoder and decoder layers during decoding. The idea is simple and reasonable and the results are promising.\n\nI have several questions about the paper:\n\t1. \"This enables a fair comparison, because it ensures that each model sees roughly the same number of training examples.\" This is not a fair comparison. Note that those models are of very different size, and thus they may need different numbers of samples for training. For example, a 1-1 model should need much less data for training than a 6-6 model. If the number of training samples is ok for the 1-1 model, it might be insufficient for the 6-6 model. Therefore, I think development set is necessary for a fair comparison.\n\n\t2. I don't understand Eq. (3). What do x and y_k mean in this equation? Are they corresponding to x^I and y^i_k in Eq (2)? However, y^i_k in Eq. (2) is a translation, i.e., a text sentence, while y_k in Eq. (3) looks like a number in [0,1].\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed a method for training multiple transformers with tied parameters and enabling dynamic choice of the number of encoder and decoder layers. The method is evaluated in neural machine translation and shown to reduce decoding costs without compromising translation quality. The reviewers generally agreed that the proposed method is interesting, but raised issues regarding the significance of the claimed benefits and the quality of overall presentation of the paper. Based on a consensus reached in a post rebuttal discussion with the reviewers, I am recommending rejecting this paper.
Decision: Reject