Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper dives deeper into understand reward augmented maximum likelihood training. Overall, I feel that the paper is hard to understand and that it would benefit from more clarity, e.g., section 3.3 states that decoding from the softmax q-distribution is similar to the Bayes decision rule. Please elaborate on this.

Did you compare to minimum bayes risk decoding which chooses the output with the lowest expected risk amongst a set of candidates?

Section 4.2.2 says that Ranzato et al. and Bahdanau et al. require sampling from the model distribution. However, the methods analyzed in this paper also require sampling (cf. Appendix D.2.4 where you mention a sample size of 10). Please explain the difference.
review_title: Official review

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper interprets reward augmented maximum likelihood followed by decoding with the most likely output as an approximation to the Bayes decision rule.

I have a few questions on the motivation and the results.
- In the section "Open Problems in RAML", both (i) and (ii) are based on the statement that the globally optimal solution of RAML is the exponential payoff distribution q. This is not true. The globally optimal solution is related to both the underlying data distribution P and q, and not the same as q. It is given by q'(y | x, \tau) = \sum_{y'} P(y' | x) q(y | y', \tau).
- Both Theorem 1 and Theorem 2 do not directly justify that RAML has similar reward as the Bayes decision rule. Can anything be said about this? Are the KL divergence small enough to guarantee similar predictive rewards?
- In Theorem 2, when does the exponential tail bound assumption hold?
- In Table 1, the differences between RAML and SQDML do not seem to support the claim that SQDML is better than RAML. Are the differences actually significant? Are the differences between SQDML/RAML and ML significant? In addition, how should \tau be chosen in these experiments?

review_title: some interesting ideas, but the paper need to be strengthened

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors claim three contributions in this paper. (1) They introduce the framework of softmax Q-distribution estimation, through which they are able to interpret the role the payoff distribution plays in RAML. Specifically, the softmax Q-distribution serves as a smooth approximation to the Bayes decision boundary. The RAML approximately estimates the softmax Q-distribution, and thus approximates the Bayes decision rule. (2) Algorithmically, they further propose softmax Q-distribution maximum likelihood (SQDML) which improves RAML by achieving the exact Bayes decision boundary asymptotically. (3) Through one experiment using synthetic data on multi-class classiﬁcation and one using real data on image captioning, they show that SQDML is consistently as good or better than RAML on the task-speciﬁc metrics that is desired to optimize. 

I found the first contribution is sound, and it reasonably explains why RAML achieves better performance when measured by a specific metric. Given a reward function, one can define the Bayes decision rule. The softmax Q-distribution (Eqn. 12) is defined to be the softmax approximation of the deterministic Bayes rule. The authors show that the RAML can be explained by moving the expectation out of the nonlinear function and replacing it with empirical expectation (Eqn. 17). Of course, the moving-out is biased but the replacing is unbiased. 

The second contribution is partially valid, although I doubt how much improvement one can get from SQDML. The authors define the empirical Q-distribution by replacing the expectation in Eqn. 12 with empirical expectation (Eqn. 15). In fact, this step can result in biased estimation because the replacement is inside the nonlinear function. When x is repeated sufficiently in the data, this bias is small and improvement can be observed, like in the synthetic data example. However, when x is not repeated frequently, both RAML and SQDML are biased. Experiment in section 4.1.2 do not validate significant improvement, either.

The numerical results are relatively weak. The synthetic experiment verifies the reward-maximizing property of RAML and SQDML. However, from Figure 2, we can see that the result is quite sensitive to the temperature \tau. Is there any guidelines to choose \tau? For experiments in Section 4.2, all of them are to show the effectiveness of RAML, which are not very relevant to this paper. These experiment results show very small improvement compared to the ML baselines (see Table 2,3 and 5).  These results are also lower than the state of the art performance. 

A few questions:
(1). The author may want to check whether (8) can be called a Bayes decision rule. This is a direct result from definition of conditional probability. No Bayesian elements, like prior or likelihood appears here.
(2). In the implementation of SQDML, one can sample from (15) without exactly computing the summation in the denominator. Compared with the n-gram replacement used in the paper, which one is better?
(3). The authors may want to write Eqn. 17 in the same conditional form of Eqn. 12 and Eqn. 14. This will make the comparison much more clear.
(4). What is Theorem 2 trying to convey? Although \tau goes to 0, there is still a gap between Q and Q'. This seems to suggest that for small \tau, Q' is not a good approximation of Q. Are the assumptions in Theorem 2 reasonable? There are several typos in the proof of Theorem 2. 
(5). In section 4.2.2, the authors write "the rewards we directly optimized in training (token-level accuracy for NER and UAS for dependency parsing) are more stable w.r.t. τ than the evaluation metrics (F1 in NER), illustrating that in practice, choosing a training reward that correlates well with the evaluation metric is important". Could you explain it in more details?


review_title: good theoretical interpretation of RAML, weak experiment results

Meta-review: There are some interesting ideas discussed in the paper, but the reviewers expressed difficulty understanding the motivation and the theoretical results. The experiments do not seem convincing in showing that SQDML achieves significant gains. Overall, the the paper needs either stronger and clearer theoretical results, or more convincing experiments for publication at ICLR.
Decision: Reject