Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: 
This paper proposes a novel procedure for training multiple Transformers with tied parameters which compresses multiple models into one enabling the dynamic choice of the number of encoder and decoder layers during decoding. The idea is simple and reasonable and the results are promising.

I have several questions about the paper:
	1. "This enables a fair comparison, because it ensures that each model sees roughly the same number of training examples." This is not a fair comparison. Note that those models are of very different size, and thus they may need different numbers of samples for training. For example, a 1-1 model should need much less data for training than a 6-6 model. If the number of training samples is ok for the 1-1 model, it might be insufficient for the 6-6 model. Therefore, I think development set is necessary for a fair comparison.

	2. I don't understand Eq. (3). What do x and y_k mean in this equation? Are they corresponding to x^I and y^i_k in Eq (2)? However, y^i_k in Eq. (2) is a translation, i.e., a text sentence, while y_k in Eq. (3) looks like a number in [0,1].

review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: This work proposes a way to reduce the latencies incurred in inference for neural machine translation. Basic idea is to train a model with softmax attached to each output of decoder layers, and computes a loss by aggregating the cross entropy losses over the softmaxes. During inference, it could either use one of the softmax or train an additional model which dynamically selects softmaxes given an input string. Experimental results show that it is possible to reduce latencies by trading off the translation qualities measured by BLEU. Dynamically selection did not show any gains in latencies, though, this work empirically shows potential gains in oracle studies. This work further shows that the model could be compressed further by knowledge distillation.

I have several concerns to this work and I'd recommend rejecting this submission.

- One of the problems of this paper is presentation. This work basically combines three work together as a single paper, i.e., section 3 for the basic model, section 4 for dynamic selection and section 5 for distillation, with each section describing a separate experiment. I'd strongly suggest the author to focus on the main point, e.g., dynamic selection, and present the basic model and dynamic selection. Experiments should be presented in a single section for brevity.

- Similarly, this work should have been submitted when meaningful gains were observed in the dynamic selection method, given that the proposal is somewhat new. Otherwise, I don't find any merits to see this accepted in ICLR, given the rather negative results in section 4.

- The description in section 4.2 is totally messed up. x^i and y^i_k are strings since they are an input sentence and an output sentence, respectively,. However, they are treated as scalars in Equation 3 by multiplied with \delta_k, subtracted from 1 and taking sigmoid through \sigma. I strongly suggest authors to carefully check variables used in the equations and the description in the section.

- The authors claim that the use of knowledge distillation is novel. However, it is already widely known in the research community and I don't think it is worthy to keep it as a single section. It could have been described as a yet another experiment in a single experimental section.

Other comment:

- Although this paper claims that attaching a softmax for each output layer is new, there is a similar work in language modeling, though the motivation is totally different.

  Direct Output Connection for a High-Rank Language Model, Sho Takase, Jun Suzuki and Masaaki Nagata, EMNLP 2019.

- In section 3.4, this paper claims that the training of all 36 models took 25.5 more time, but took 9.5 more time for a tied-model when compared with a basic 6-layer Transformer. It is not clear to me whether this comparison is meaningful given that it might be possible to employ multiple machines to train 36 models.
review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: ** Summary **
In this paper, the authors propose a new variant of Transformer called Tied-multi Transformer. Given such a model with an N-layer encoder and an M-layer decoder, it is trained with M*N loss functions, where each combination of the nth-layer of the encoder and the mth-layer of the decoder is used to train an NMT model. The authors propose a way to dynamically select which layers to be used when a specific sentence comes.  At last, the authors also try recurrent stack and knowledge to further compress the models.

** Details **
1.	The first question is “why this work”:
a.	In terms of performance improvement, in Table 2, we can see that dynamic layer selection does not bring any improvement compared to baseline (Tied(6,6)). When compared Tied(6,6) to standard Transformer, as shown in Table 1, there is no improvement. Both are 35.0.
b.	In terms of inference speed, in Table 2, the method can achieve at most (2773-2563)/2998 = 0.07s improvement per sentence, which is very limited.
c.	In terms of training speed, compared to standard Transformer, the proposed method takes 9.5 time of the standard Transformer (see section 3.4, training time).
Therefore, I think that compared to standard Transformer, there is not a significant difference.
2.	 The authors only work on a single dataset, which is not convincing.
3.	In Section 5, what is the baseline of standard RS + knowledge distillation?

review_title: Official Blind Review #1

Meta-review: The paper proposed a method for training multiple transformers with tied parameters and enabling dynamic choice of the number of encoder and decoder layers. The method is evaluated in neural machine translation and shown to reduce decoding costs without compromising translation quality. The reviewers generally agreed that the proposed method is interesting, but raised issues regarding the significance of the claimed benefits and the quality of overall presentation of the paper. Based on a consensus reached in a post rebuttal discussion with the reviewers, I am recommending rejecting this paper.
Decision: Reject