Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This work introduces a new algorithm to improve decoding in discrete autoregressive models (ARMs). Because exact decoding of ARMs is computationally difficult, the contribution of this paper is to introduce a new approximate solution. The authors show experimentally that this new framework is very effective and outperforms multiple baselines such as greedy and bean search and another text filing algorithm called TIGC on a text-filling benchmark. This work is not in my domain of expertise so I am not able to have a very accurate evaluation. However, based on the references cited in this paper, there are other approximate solution for ARMs and I believe the authors need to use those as baselines to show that the proposed approximate solution is useful.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The paper proposes a decoding algorithm for auto-regressive models (ARMs) to improve the decoding accuracy. The key idea is to introduce auxiliary continuous states variables, and alternatively optimize the discrete output variables (i.e., the tokens) and the continuous states. Given the hidden states, the decoding can be efficiently done by veterbi-like algorithms. Then the constraint of the auxiliary variables can be imposed by penalty based continuous based optimization. The paper also extends the idea to the ensemble of ARMs.  On two NLP tasks, the paper shows improvement upon the existing greedy or beach search based approaches. 

Overall, the proposed method can be very useful in RNN decoding and structure prediction. The idea, however, is quite straightforward to people who are familiar with probabilistic graphic models and inference. I am surprised (if the authors' claim is correct) the RNN community still relies on greedy/beam search for decoding. In graphical model language, the hidden states (h_i or g_i in the paper) are typically viewed as continuous latent random variables. The function f that links consecutive hidden states are factors or potential functions. For inference/prediction, you can jointly optimize the hidden states and discrete outputs, where the alternative updates are a natural choice. Similar ideas were used long time ago when (hierarhical) conditional random fields were popular. Honestly, I don't see anything new here.  Here are a few comments:

1. Don't say ARMs are non-Markov model and/or with unbounded Markov order. This is very misleading and over bragging. RNNs are just a nonlinear version of HMM/hidden Kalman filter. The only difference is that the states are continuous (Kalman filter uses continuous states as well), and the state transition kernel is nonlinear, constructed in a very black-box way. People like to make some analogy with brains --- unfortunately, these explanations are at most an analogy.  Given hidden states, RNNs are just first order Markov-chains, nothing special. If you integrate out hidden states, of course, every output is dependent on all the previous outputs. But the same argument applies to all the hidden markov models/dynamic systems. This is not something unique to RNNs, and shouldn't be hyped everywhere.

2. Is there a way to show the standard deviations/error bars in the test results, say, Table 1 & 2 and the figure? One single number is usually not representative for the performance, unless you have a large test set. 
review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: I think this is a good study, unless I miss something. It proposes a new solution to one of the fundamental problems in the field, which significantly improves previous solutions in terms of accuracy. I will recommend it for acceptance unless I miss something. (I’m not an expert in the field and the problem seems to be so fundamental, making me cautious about judging the novelty of the proposed solution.)

There seems to be some unclarity about the optimization algorithm. In short, I suspect that the proposed optimization has some difficulty with its convergence. 
-	In Sec.4, the authors suggest “a two-step block coordinate descent algorithm to alternate between (i) optimizing y’s while g’s are fixed, and (ii) optimizing g’s while y’s are fixed”. They further suggest `a variant of the Viterbi algorithm’ for (i) and gradient-based algorithms for (ii). These seem to make sense to me so far. 
-	Then, in Sec.6, they state that “We use the Nesterov optimizer with a learning rate of 0.1. All \mu’s are initialised with 0.5 and are multiplied by 1.2 after 5 epochs”. I think this needs some explanation. 
-	For instance, what is the thought behind the choice of the optimizer? Why was Nesterov’s acceleration necessary instead of plain GD? Is it essential to use the scheduled adjustment of \mu?
-	It states “All \mu’s” but there seems only a single \mu in Eq.(5). Am I missing something?
-	 Some explanation on what “Nesterov optimizer” and \mu would also be necessary for a wide range of readers. 
-	There is a statement on the initialization of the optimization in p.7: “h corresponding to the beam search solution to initialize the g variables”. How sensitive is the optimization to the initial values? For instance, will the results change if g’s are initialized by the solution of the greedy method. 
-	There is a plot without a figure caption in p.8, which shows how the objective cost decreases with parameter updates. Why are their values at ‘epoch’=0 lower than those at the subsequent epochs? Does this mean the initial values give more optimal parameters?

Additional comments:
-	It would be more friendly to the readers to show the definition of the notations like y^n_1. 
-	There is a statement like “Decoding was run for 10 epochs.” I suppose epochs here mean iteration count of the alternated parameter updates in the proposed algorithm. Why do the authors call it `epoch’? 
-	No figure caption for the plot in p.8. It also uses `epochs’ for the horizontal axis title. No axis title for the vertical axis. 
review_title: Official Blind Review #3

Meta-review: This paper proposes an approximate inference approach for decoding in autoregressive models, based on the method of auxiliary coordinates,  which uses iterative factor graph approximations of the model.  The approach leads to nice improvements in performance on a text infilling task.  The reviewers were generally positive about this paper, though there was a concern that more baselines are needed and discussion was very limited following the author responses.  I tend to agree with the authors that their results are convincing on the infilling task.  The impact of the paper is a bit limited by the lack of experiments on more standard decoding tasks, which, as the authors point out, would be challenging as their approach is computationally demanding.  Overall I believe this would be an interesting contribution to the ICLR community.
Decision: Accept (Poster)