Year: 2019

Abstract:
This paper proposes a novel procedure for training multiple Transformers with tied parameters which compresses multiple models into one enabling the dynamic choice of the number of encoder and decoder layers during decoding. In sequenceto-sequence modeling, typically, the output of the last layer of theN -layer encoder is fed to the M -layer decoder, and the output of the last decoder layer is used to compute loss. Instead, our method computes a single loss consisting of N ×M losses, where each loss is computed from the output of one of the M decoder layers connected to one of the N encoder layers. A single model trained by our method subsumes multiple models with different number of encoder and decoder layers, and can be used for decoding with fewer than the maximum number of encoder and decoder layers. We then propose a mechanism to choose a priori the number of encoder and decoder layers for faster decoding, and also explore recurrent stacking of layers and knowledge distillation to enable further parameter reduction. In a case study of neural machine translation, we present a cost-benefit analysis of the proposed approaches and empirically show that they greatly reduce decoding costs while preserving translation quality.

heading: 1 INTRODUCTION
text: Neural networks for sequence-to-sequence modeling typically consist of an encoder and a decoder coupled via an attention mechanism. Whereas the very first deep models used stacked recurrent neural networks (RNN) (Sutskever et al., 2014; Cho et al., 2014; Bahdanau et al., 2015) in the encoder and decoder, the recent Transformer model (Vaswani et al., 2017) constitutes the current state-of-the-art approach, owing to its better context generation mechanism via multi-head self- and cross-attentions.
Given an encoder-decoder architecture and its hyper-parameters, such as the number of encoder and decoder layers, vocabulary sizes (in the case of text based models) and hidden layers, the parameters of the model, i.e., matrices and biases for non-linear transformations, are optimized by iteratively updating them so that the loss for the training data is minimized. The hyper-parameters can also be tuned, for instance, through maximizing the automatic evaluation score on the development data. However, in general, it is highly unlikely (or impossible) that a single optimized model suffices diverse cost-benefit demands at the same time. For instance, in practical low-latency scenarios, one may accept some performance drop for speed. However, a model used with a subset of optimized parameters might perform badly. A single optimized model cannot also guarantee the best performance for each individual input. Although this is practically important, it has drown only a little attention. An existing solution for this problem is to host multiple models simultaneously for flexible choice. However, this approach is not very practical, because it requires an unreasonably large quantity of resources. Furthermore, there are no well-established methods for selecting appropriate models for each individual input.
As a more effective solution, we consider training a single model that subsumes multiple models which can be used for decoding with different hyper-parameter settings depending on the input or on the latency requirements. In this paper, we focus on the number of layers as an important hyperparameter that impacts both speed and quality of decoding, and propose a multi-layer softmaxing method, which trains multi-layer neural models referring to the output of all layers during training. Conceptually, as illustrated in Figure 1, this method involves tying (sharing) the parameters of mul-
tiple models with different number of layers and is not specific to particular types of multi-layer neural models. On top of the above method, we consider to exploit the model further. For saving decoding time, we design and evaluate mechanisms to choose appropriate number of layers depending on the input. As for further model compression, we leverage other orthogonal types of parameter tying approaches, such as those reviewed in Section 2.
Despite the generality of our proposed method, in this paper, we particularly focus on encoderdecoder models with N encoder and M decoder layers, and compress N ×M models1 by updating the model with a total ofN×M losses computed by softmaxing the output of each of theM decoder layers, where it attends to the output of each of the N encoder layers. The number of parameters of the resultant encoder-decoder model is equivalent to that of the most complex subsumed model with N encoder and M decoder layers. Yet, we can now perform faster decoding using a fewer number of layers, given that shallower layers are better trained. To evaluate our proposed method, we take the case study of neural machine translation (NMT) (Cho et al., 2014; Bahdanau et al., 2015), where we focus on the numbers of encoder and decoder layers of the Transformer model (Vaswani et al., 2017), and demonstrate that a single model with N encoder and M decoder layers trained by our method can be used for flexibly decoding with fewer than N and M layers without appreciable quality loss. We evaluate our proposed method on WMT18 English-to-German translation task, and give a cost-benefit analysis for translation quality vs. decoding speed.
The rest of the paper is organized as follows. Section 2 briefly reviews related work for compressing neural models. Section 3 covers our method that ties multiple models by softmaxing all encoderdecoder layer combinations. Section 4 describes our efforts towards designing and evaluating a mechanism for dynamically selecting encoder-decoder layer combinations prior to decoding. Section 5 describes two orthogonal extensions to our model aiming at further model compression and speeding-up of decoding. The paper ends with Section 6 containing conclusion and future work.

heading: 2 RELATED WORK
text: There are studies that exploit multiple layers simultaneously. Wang et al. (2018) fused hidden representations of multiple layers in order to improve the translation quality. Belinkov et al. (2017) and Dou et al. (2018) attempted to identify which layer can generate useful representations for different natural language processing tasks. Unlike them, we make all layers of the encoder and decoder usable for decoding with any encoder-decoder layer combination. In practical scenarios, we can save significant amounts of time by choosing shallower encoder and decoder layers for inference.
Our method ties the parameters of multiple models, which is orthogonal to the work that ties parameters between layers (Dabre & Fujita, 2019) and/or between the encoder and decoder within a
1Rather than casting the encoder-decoder model into a single column model with (N +M ) layers.
Algorithm 1 Training a tied-multi model 1: enc0 = X 2: for i in 1 to N do 3: enci = Lenci (enci−1) 4: for j in 1 to M do 5: decj = Ldecj (decj−1, enci) 6: Ŷ = softmax(decj) 7: lossi,j = cross entropy(Ŷ , Y ) 8: end for 9: end for 10: overall loss = aggregate(loss1,1, . . . , lossN,M ) 11: Back-propagate using overall loss
single model (Xia et al., 2019; Dabre & Fujita, 2019). Parameter tying leads to compact models enabling faster decoding, but they usually suffer from drops in inference quality. In this paper, we counter such drops with knowledge distillation (Hinton et al., 2015; Kim & Rush, 2016; Freitag et al., 2017). The working principle of this approach is to utilize smoothed data or smoothed training signals instead of the actual training data. A model with a large number of parameters and high performance provides smoothed distributions that are then used as labels for training small models instead of one-hot vectors.
As one of the aims in this work is model size reduction, it is related to a growing body of work that addresses the computational requirement reduction. Pruning of pre-trained models (See et al., 2016) makes it possible to discard around 80% of the smallest weights of a model without deterioration in inference quality, given it is re-trained with appropriate hyper-parameters after pruning. Currently, most deep learning implementations use 32-bit floating point representations, but 16-bit floating point representations (Gupta et al., 2015; Ott et al., 2018) or aggressive binarization (Courbariaux et al., 2017) can be alternatives. Compact models are usually faster to decode; studies on quantization (Lin et al., 2016) and average attention networks (Xiong et al., 2018) address this topic.
To the best of our knowledge, none of the above work has attempted to combine multi-model parameter tying, knowledge distillation, and dynamic layer selection for obtaining and exploiting highlycompressed and flexible deep neural models.

heading: 3 MULTI-LAYER SOFTMAXING

heading: 3.1 PROPOSED METHOD
text: Figure 1 gives a simple overview of the concept of multi-layer softmaxing, taking a generic model as an example. The rightmost 4-layer model takes an input and passes it through 4 layers2 before a softmax layer to predict the output. Typically, one would apply softmax to the 4th layer only, compute loss, and then back-propagate gradients in order to update parameters. Instead, we propose to apply softmax to each layer, aggregate the computed losses, and then back-propagate. This enables us to choose any layer combination during decoding instead of only the topmost layer.
Extending this to a multi-layer encoder-decoder model is straightforward. In encoder-decoder models, the encoder comprises an embedding layer for the input (source language for NMT) and N stacked transformation layers. The decoder consists of an embedding layer and a softmax layer for generating the output (target language for NMT) along with M stacked transformation layers. Let X be the input to the N -layer encoder, Y the anticipated output of the M -layer decoder as well as the input to the decoder (for training), and Ŷ the predicted output by the decoder. Algorithm 1 shows the pseudo-code for our proposed method. The line 3 represents the process done by the i-th encoder layer, Lenci , and the line 5 does the same for the j-th decoder layer, L dec j . In simple words, we compute a loss using the output of each of theM decoder layers which in turn is computed using
2We make no assumptions about the nature of the layers.
the output of each of the N encoder layers. In line 10, the N ×M losses are aggregated3 before back-propagation. Henceforth, we will refer to this as the Tied-Multi model.
For a comparison, the vanilla model is formulated as follows: decj = Ldecj (decj−1, encN ), Ŷ = softmax(decM ), and overall loss = cross entropy(Ŷ , Y ).

heading: 3.2 EXPERIMENTAL SETUP
text: We evaluated the following two types of models on both translation quality and decoding speed.
Vanilla models: 36 vanilla models with 1 to 6 encoder and 1 to 6 decoder layers, each trained referring only to the last layer for computing loss.
Tied-Multi model: A single tied-multi model with N = 6 encoder and M = 6 decoder layers, trained by our multi-layer softmaxing.
We experimented with the WMT18 English-to-German (En→De) translation task. We used all the parallel corpora available for WMT18, except ParaCrawl corpus,4 consisting of 5.58M sentence pairs as the training data and 2,998 sentences in newstest2018 as test data. The English and German sentences were pre-processed using the tokenizer.perl and truecase.perl scripts in Moses.5 The true-case models for English and German were trained on 10M sentences randomly extracted from the monolingual data made available for the WMT18 translation task, using the train-truecaser.perl script available in Moses.
Our multi-layer softmaxing method was implemented on top of an open-source toolkit of the Transformer model (Vaswani et al., 2017) in the version 1.6 branch of tensor2tensor.6 For training, we used the default model settings corresponding to transformer base single gpu in the implementation, except what follows. We used a shared sub-word vocabulary of 32k determined using the internal sub-word segmenter of tensor2tensor and trained the models for 300k iterations. We trained the vanilla models on 1 GPU and our tied-multi model on 2 GPUs with batch size halved to ensure that both models see the same amount of training data. We averaged the last 10 checkpoints saved every after 1k updates, and decoded the test sentences, fixing a beam size7 of 4 and length penalty of 0.6. We evaluated our models using the BLEU metric (Papineni et al., 2002) implemented in sacreBLEU (Post, 2018).8 Prior to evaluation, the decoded test set was post-processed using the detokenizer.perl and detruecase.perl scripts in Moses. We also report the time (in seconds) consumed to translate the test set, which includes times for the model instantiation, loading the checkpoint, sub-word splitting and indexing, decoding, and sub-word de-indexing and merging, whereas times for detokenization are not taken into account.
Note that we did not use any development data for two reasons. First, we train all models for the same number of iterations.9 Second, we use checkpoint averaging before decoding, which does not require a development set unlike early stopping. We adopted this approach, because it is known to give the best results for NMT using the Transformer implementation we use (Vaswani et al., 2017).
3We averaged multiple losses in our experiment, but there are a number of options, such as weighted averaging.
4http://www.statmt.org/wmt18/translation-task.html We excluded ParaCrawl following the instruction on the WMT18 website: “BLEU score dropped by 1.0” for this task.
5https://github.com/moses-smt/mosesdecoder 6https://github.com/tensorflow/tensor2tensor 7One can realize faster decoding by narrowing down the beam width. This approach is orthogonal to ours
and in this paper we do not insist which is superior to the other. 8https://github.com/mjpost/sacreBLEU BLEU+case.mixed+lang.en-de+numrefs.1+smooth.exp +test.wmt18+tok.13a+version.1.3.7 9This enables a fair comparison, because it ensures that each model sees roughly the same number of training examples.

heading: 3.3 RESULTS
text: Table 1 summarizes the BLEU scores and the decoding times of all the models, showing the costbenefit property of our tied-multi model in comparison with the results of the corresponding 36 vanilla models.
Even though the objective function for the tied-multi model is substantially more complex than the one for the vanilla model, when performing decoding with the 6 encoder and 6 decoder layers, it achieved a BLEU score of 35.0, which is approaching to the best BLEU score of 35.7 given by the vanilla model with 6 encoder and 4 decoder layers.10 Note that when using a single encoder layer and/or a single decoder layer, the vanilla models gave significantly higher BLEU score than the tiedmulti model. However, when the number of layers is increased, there is no significant difference between the two types of models: less than 1.0 BLEU point differences.
Regarding the cost-benefit property of our tied-multi model, two points must be noted:
• BLEU score and decoding time increase only slightly, when we use more encoder layers. • The bulk of the decoding time is consumed by the decoder, since it works in an auto-
regressive manner. We can substantially cut down decoding time by using fewer decoder layers which does lead to sub-optimal translation quality.
One may argue that training a single vanilla model with optimal number of encoder and decoder layers is enough. However, as discussed in Section 1, it is impossible to know a priori which combination is the best. More importantly, a single vanilla model cannot suffice diverse cost-benefit demands and cannot guarantee the best translation for any input (see Section 4.1). Recall that we aim at a flexible model and that all the results in Table 1 have been obtained using a single tied-multi model, albeit using different number of encoder and decoder layers for decoding.

heading: 3.4 ANALYSIS AND DISCUSSION
text: We conducted an analysis from the perspective of training times and model sizes, in comparison with vanilla models.
Training Time: Given that all our models were trained for the same number of iterations, we compared the training times between vanilla and tied-multi models. As a reference, we use the vanilla model with 6 encoder and 6 decoder layers. The total training time for all the 36 vanilla models was 25.5 times11 that of the reference model. In contrast, the training time for our tiedmulti model was about 9.5 times that of the same reference model. Unexpectedly, training a tiedmulti model was much more computationally efficient than independently training all the 36 vanilla models with different number of layers.
Model Size: The number of parameters of our tied-multi model is exactly the same as the vanilla model with N encoder and M decoder layers. If we train a set of vanilla models with different
10We consider this as an oracle, since we have no prior information about the best layer combination. 11We measured the collapsed time for a fair comparison, assuming that all vanilla models were trained on a single GPU one after another, even though one may be able to use multiple GPUs to train the 36 vanilla models in parallel.
numbers of encoder and decoder layers, we end up with significantly more parameters. For instance, in case of N = M = 6 in our experiment, we have 25.2 times more parameters: a total of 4,607M for the 36 vanilla models against 183M for our tied-multi model. We discuss the possibility of further model compression in Section 5.

heading: 4 DYNAMIC LAYER SELECTION
text: To better understand the nature of our proposed method, we analyzed the distribution of oracle translations within 36 translations generated by each of the vanilla and our tied-multi models. Having confirmed that a single encoder-decoder layer combination cannot guarantee the best translation for any input, we tackled an advanced problem: designing a mechanism for dynamically selecting one layer combination prior to decoding.

heading: 4.1 DECODING BEHAVIOR OF TIED-MULTI MODEL
text: Let (n,m) be an encoder-decoder layer combination of a given model with n encoder and m decoder layers. The oracle layer combination for an input sentence was determined by measuring the quality of the translation derived from each layer combination. We used a reference-based metric, chrF (Popović, 2016), since it has been particularity designed to sentence-level translation evaluation and was shown to have relatively high correlation with human judgment of translation quality for the language pair involved in our experiment (Ma et al., 2018). In cases where multiple combinations have the highest score, we chose the fastest one following the overall trend of decoding time (Table 1). Formally, we considered a combination (n1, n2) is faster than another combination (n2,m2) if the following holds.
(n1,m1) < (n1,m2) ≡ m1 < m2 ∨ (m1 = m2 ∧ n1 < n2) (1)
Figure 2 presents the distribution of oracle layer combinations for the vanilla and our tied-multi models. A comparison between the two distributions revealed that the shallower layer combinations in our tied-multi model often generates better translations than deeper ones unlike the vanilla models, thanks to the multi-layer softmaxing method, despite the lower corpus-level BLEU scores. This sharp bias towards shallower layer combinations suggests the potential reduction of decoding time by dynamically selecting the layer combination per input sentence prior to decoding, ideally without performance drop.
In what follows, we present our method for dynamic selection of layer combination and experiments.

heading: 4.2 METHOD
text: We formalize the encoder-decoder layer combination selection with a supervised learning approach where the objective is to minimize the following loss function (2).
argmin θ
1 |X| ∑ xi∈X L(f(xi; θ), yik), (2)
where xi is the i-th input sentence (1 ≤ i ≤ |X|), yik is the translation for xi derived from the k-th layer combination (1 ≤ k ≤ K = N ×M ) among all K possible combinations, f is the model with parameters θ, and L is a loss function. Assuming that the independence of target labels (layer combinations) for a given input sentence allows for ties, the model is able to predict multiple layer combinations for the same input sentence.
The model f with parameters θ implemented in our experiment is a multi-head self-attention neural network inspired by Vaswani et al. (2017). The number of layers and attention heads are optimized during a hyper-parameter search, while the feed-forward layer dimensionality is fixed to 2048. Input sequences of tokens are mapped to their corresponding embeddings, initialized by the embedding table of the tied-multi NMT model. A specific token is appended to each input sequence before being fed to the classifier. This token is finally fed during the forward pass to the output linear layer similarly to BERT (Devlin et al., 2019) for sentence classification. The output linear layer has K dimensions, allowing to output as many logits as there are layer combinations in the tied-multi NMT model. Finally, a sigmoid function outputs probabilities for each layer combination among the K possible combinations.
The parameters θ of the model are learned using mini-batch stochastic gradient descent with Nesterov momentum (Sutskever et al., 2013) and the loss function L, implemented as a weighted binary cross-entropy (BCE) function detailed in (3) and averaged over the batch.
LBCEk = −wk [δkyk · log σ(x) + (1− yk) · log(1− σ(x))] , (3)
where δk = (1 − p(yk))α is the weight given to the k-th class based on class distribution prior knowledge and σ is the sigmoid function. During our experiment, we have found that the classifier tends to favor recall in detriment to precision. To tackle this issue, we introduce another loss using an approximation of the macro Fβ implemented following (4).
LFβ = 1− [ (1 + β2) · P ·R
(β2 · P ) +R
] (4)
with P = µ∑ k ŷk , R = µ∑ k yk and µ = ∑ k (ŷk · yk),
where ŷk is the output of σ(x) for the k-th dimension. The final loss function is the linear interpolation of LBCE averaged over the K classes and LFβ with parameter λ: λ×LBCE + (1− λ)×LFβ . We tune α, β, and λ during the classifier hyper-parameter search based on the validation loss.

heading: 4.3 EXPERIMENTS
text: The layer combination classifier was trained on a subset of the tied-multi NMT model training data presented in Section 3.2 containing 5.00M sentences, whereas the remaining sentences compose a validation and a test set containing approximately 200k sentences each. The two latter subsets were used for hyper-parameter search and evaluation of the classifier, respectively. To allow for comparison and reproducibility, the final evaluation of the proposed approach in terms of translation quality and decoding speed were conducted on the official WMT development (newstest2017, 3,004 sentences) and test (newstest2018, 2,998 sentences) sets; the latter is the one also used in Section 3.2.
The training, development, and test sets were translated with each layer combination of the tiedmulti NMT model. Each source sentence was thus aligned with 36 translations whose quality were measured by the chrF metric. Because several combinations can lead to the best score, the obtained dataset was labeled with multiple classes (36 layer combinations) and multiple labels (ties with regard to the metric). During inference, the ties were broken by selecting the layer combination with the highest value given by the sigmoid function, or backing-off to the deepest layer combination (6, 6) if no output value reaches 0.5. This tie breaking method differs from the oracle layer selection presented in Equation (1) and in Figure 2 which prioritizes shallowest layer combinations. In this experiment, decoding time was measured by processing one sentence at a time instead of batch decoding, the former being slower compared to the latter, but leading to precise results. The decoding times were 954s and 2,773s when using a (1,1) and a (6,6) layer combination, respectively. By selecting the fastest encoder-decoder layer combinations according to an oracle, the decoding times went down to 1,918s and 1,812s for the individual and tied-multi models, respectively. However, our objective is to be faster than default setting, that is, where one would choose a (6,6) combination.
Several classifiers were trained before being evaluated on the WMT test set, with or without finetuning on the WMT development set. The results obtained in terms of corpus-level BLEU and decoding speed are presented in Table 2. Some classifiers maintain the translation quality (top rows), whereas others show quality degradation but further gain in decoding speed (bottom rows).

heading: 4.4 ANALYSIS
text: The classification results show that gains in decoding speed are possible with an a-priori decision for which encoder-decoder combination to select, based on the information contained in the source sentence only. However, no BLEU gains are observed in the results presented in Table 2 and a tradeoff between decoding speed and translation quality is present. Our best configuration for decoding speed reaches a 210s decrease but leads to a 0.7 point BLEU degradation. In addition, only 37s are gained when preserving the translation quality compared to the baseline configuration. The layer combination oracle indicates substantial gains both in terms of BLEU (7 points) and decoding speed (961s). These oracle results motivate possible future work in layer combination prediction for the tied-multi NMT model.

heading: 5 FURTHER MODEL COMPRESSION
text: We examined the combination of our multi-layer softmaxing approach with another parameter-tying method in neural networks, called recurrent stacking (RS) (Dabre & Fujita, 2019), complemented by sequence-level knowledge distillation (Kim & Rush, 2016), a specific type of knowledge distillation (Hinton et al., 2015). We demonstrate that these existing techniques help reduce the number of parameters in our model even further.

heading: 5.1 DISTILLATION INTO A RECURRENTLY STACKED MODEL
text: In Section 2, we have discussed several model compression methods orthogonal to multi-layer softmaxing. Having already compressed N ×M models with our approach, we consider further compressing it using RS. However, models that use RS layers tend to suffer from performance drops due to the large reduction in the number of parameters. As a way of compensating the performance drop, we apply sequence-level knowledge distillation.
First, we decode all source sentences in the training data to obtain a pseudo-parallel corpus containing distillation target sequences. By forward-translating the data, we create soft-targets for the child model which makes learning easier and hence is able to mimic the behavior of the parent model. Then, an RS child model is trained with multi-layer softmaxing on the generated pseudo-parallel corpus. Among a variety of distillation techniques, we chose the simplest one to show the impact that distillation can have in our setting, leaving an extensive exploration of more complex methods for the future.

heading: 5.2 EXPERIMENT
text: We conducted an experiment to show that RS and sequence distillation can lead to an extremely compressed tied-multi model which no longer suffers from performance drops. We compared the following four variations of our tied-multi model trained with multi-layer softmaxing.
Tied-multi model: A model that does not share the parameters across layers, trained on the original parallel corpus.
Distilled tied-multi model: The same model as above but trained on the pseudo-parallel corpus.
Tied-multi RS model: A tied-multi model that uses RS layers, trained with the original parallel corpus.
Distilled tied-multi RS model: The same model as above but trained on the pseudo-parallel corpus.
Note that we defrayed much higher cost for training the distilled models than training models directly on the original parallel corpus. First, we trained 5 vanilla models with 6 encoder and 6 decoder layers, because the performance of distilled models is affected by the quality of parent models, and NMT models vary vastly in performance (around 2.0 BLEU) depending on parameter initialization. We then decode the entire training data (5.58M sentences) with the one12 with the highest BLEU score on the newstest2017 (used in Section 4.3) in order to generate pseudo-parallel corpus for sequence distillation. Nevertheless, we consider that we can fairly compare the performance of the above four models, since they were trained only once with a random parameter initialization, without seeing the test set.
Table 3 gives the BLEU scores for all models. Comparing top-left and top-right blocks of the table, we can see that the BLEU scores for RS models are higher than their non-RS counterparts when using fewer than 3 decoder layers. This shows the benefit of RS layers despite the large parameter reduction. However, the reduction in parameters negatively affects (up to 1.3 BLEU points) when decoding with more decoder layers, confirming the limitation of RS as expected.
Comparing the scores of the top and bottom halves of the table, we can see that distillation dramatically boosts the performance of the shallower encoder and decoder layers. For instance, without distillation, the tied-multi model gave a BLEU of 23.2 when decoding with 1 encoder and 1 decoder layers, the same layer combination reaches 30.1 BLEU through distillation. Given that RS further improves performance using lower layers, the BLEU score increases to 31.2. As such, distillation enables decoding using fewer layers without substantial drops in performance. Furthermore, the BLEU scores did not vary significantly when the layers deeper than 3 were used, meaning that we might as well train shallower models using distillation. The performance of our final model, i.e., the distilled tied-multi RS model (bottom-right), was significantly lower (up to 1.5 BLEU points) similarly to its non-distilled version. However, given that it outperforms our original tied-multi model (top-left) in all the encoder-decoder layer combinations, we conclude that we can obtain a substantially compressed model with better performance.
12Ensemble of multiple models is commonly used for distillation, but we used a single model to save decoding time.

heading: 5.3 ANALYSIS
text: We now analyze model size and decoding speed resulted by applying RS and knowledge distillation. Note that RS has no effect on training time because the computational complexity is the same with or without it.
Model Size: Table 4 gives the sizes of various models that we have trained and their ratio with respect to the tied-multi model. Training vanilla and RS models with 36 different encoder-decoder layer combinations required 25.2 and 14.3 times the number of parameters of a single tied-multi model, respectively. Although RS led to some parameter reduction, combining RS with our tiedmulti model resulted a further compressed single model: 0.40 times that of the single tied-multi model without RS. This model has 63.2 times and 36.0 times fewer parameters than all the individual vanilla and RS models, respectively. Given that knowledge distillation can reduce the performance drops due to RS (see Table 3), we believe that combining it with this approach is an effective way to compress a large number of models into one.
Decoding Speed: Although we do not give scores due to lack of space, we observed that greedy decoding is faster than beam decoding but suffers from significantly reduced scores of around 2.0 BLEU. By using our distilled models, however, greedy decoding reduced the scores only by 0.5 BLEU compared to beam decoding. This happens because we have used translations generated by beam decoding as target sentences for knowledge distillation, which has the ability to loosely distill beam search behavior into greedy decoding behavior. For instance, greedy decoding with the distilled tied-multi RS model with 2 encoder and 2 decoder layers resulted in a BLEU score of 35.0 in 66.1s. In comparison, beam decoding with the tied-multi model without RS and distillation with 5 encoder and 6 decoder layers led a BLEU score of 35.1 in 261.5s (Table 1), showing that comparable translation quality is obtained with a factor of 4 in decoding time when using RS and distillation. Even though we intended to minimize performance drops besides the model compression, we obtained an unexpected benefit in terms of faster decoding through greedy search without a significant loss in translation quality.

heading: 6 CONCLUSION
text: In this paper, we have proposed a novel procedure for training encoder-decoder models, where the softmax function is applied to the output of each of the M decoder layers derived using the output of each of the N encoder layers. This compresses N ×M models into a single model that can be used for decoding with a variable number of encoder (1 ≤ n ≤ N ) and decoder (1 ≤ m ≤ M ) layers. This model can be used in different latency scenarios and hence is highly versatile. We have experimented with NMT as a case study of encoder-decoder models and given a cost-benefit analysis of our method. We have proposed and evaluated two orthogonal extensions and show that we can (a) dynamically choose layer combinations for slightly faster decoding and (b) further compress models using recurrent stacking with knowledge distillation.
For further speed up in decoding as well as model compression, we plan to combine our approach with other techniques, such as those mentioned in Section 2. Although we have only tested our idea for NMT, it should be applicable to other tasks based on deep neural networks.

Review summaries:
In this paper, the authors propose a new variant of Transformer called Tied-multi Transformer. Given such a model with an N-layer encoder and an M-layer decoder, it is trained with M*N loss functions, where each combination of the nth-layer of the encoder and the mth-layer of the decoder is used to train an NMT model. The authors propose a way to dynamically select which layers to be used when a specific sentence comes.  At last, the authors also try recurrent stack and knowledge to further compress the models.

This work proposes a way to reduce the latencies incurred in inference for neural machine translation. Basic idea is to train a model with softmax attached to each output of decoder layers, and computes a loss by aggregating the cross entropy losses over the softmaxes. During inference, it could either use one of the softmax or train an additional model which dynamically selects softmaxes given an input string. Experimental results show that it is possible to reduce latencies by trading off the translation qualities measured by BLEU. Dynamically selection did not show any gains in latencies, though, this work empirically shows potential gains in oracle studies. This work further shows that the model could be compressed further by knowledge distillation.