Venue: ICLR 2022 Track Blog

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The blog post is about this paper https://arxiv.org/pdf/2110.15797.pdf which provides a new way of doing speech synthesis.
The blog post is very detailed and targets a broad audience, especially with section 2, which provides a lot of context, and with the recall on Sinkorn networks (section 3).

Minor:
- in thm 2 I would replace "moreover" by "then".
review_title: Very detailled and complete blogpost

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The blogpost focuses on Li et al., 2021 (https://openreview.net/pdf?id=jP1vTH3inC), which presents a novel approach for generating natural languages in a non-monotonic manner. Specifically, the blogpost details how Li et al. utilize a domain-independent unsupervised non-autoregressive learner based on Sinkhorn Networks to generate a good set of non-monotonic, adaptive orderings (represented as permutation matrices), which guides a final autoregressive token generation model to generate natural languages.

Overall, the blogpost is very well-written. The detailed background section and the emphasis that the permutation generator (encoder) $\phi$ is only an assistive network are very helpful for understanding the methodologies in Li et al.

Some comments:

- An updated version of paper has been posted at https://arxiv.org/pdf/2110.15797.pdf , so I would suggest to also include the updated pdf link in the blog post. (the updated paper exceeds the ICLR final version page limit, so it was posted on Arxiv)

- It might be better to add a link to the section "How are NMAO discovered?" at the beginning of the background section. The background section is quite long, so people might skip the background and read the main methodologies first. If they have something that they don't understand, they can refer to the background section.

-  In "How are NMAO discovered - Formulation", it would be better to add a note that in practice, we do not first create a fixed-length sequence of blanks and then replace the blanks with actual values. Instead, we adopt an insertion-based approach, and dynamically insert a new value at a position relative to the previous values (using similar approaches as Pointer Networks). Therefore, in practice, we use position $\mathbf{r}$ relative to the previous generated values instead of $\mathbf{z}$. Since there is a bijection between $\mathbf{z}$ and $\mathbf{r}$, for simplicity, we still use $\mathbf{z}$ throughout the blogpost / paper.

- When introducing the Bethe Permanent, it would be better to also link to the implementations (Appendix C of http://proceedings.mlr.press/v118/mena20a/mena20a.pdf , and https://github.com/xuanlinli17/autoregressive_inference/blob/master/voi/nn/base/bethe.py)

- The figure in "How are NMAO discovered - Variational Order Inference" has typos: (1) $\log p_{\theta}(y_{z_i}|y_{<z_i},z_i,\mathbf{x})$ should be $\log p_{\theta}(y_{z_i}|y_{z_{<i}},z_{<i},\mathbf{x})$; (2) $\log p_{\theta}(z_i | y_{\le z_i}, z_{<i}m, \mathbf{x})$ should be $\log p_{\theta}(z_i | y_{z_{\le i}}, z_{<i}, \mathbf{x})$ (note that $y_{<z_i}$ and $y_{z_{<i}}$ have different meanings, and the latter is correct)

- It would be better to emphasize in "Gradients for the Decoder Network (θ)" that each $\mathbf{z_i}$ refers to an entire permutation due to the bold text, which is not an element within a single permutation (it's easy for people to miss the bold text detail here).

- For "MDP Formulation" section, it would be helpful to emphasize that the baseline $b$ is not taken with respect to the whole dataset (and is state-space dependent). Empirically Li et al. find that this significantly stabalizes training.

- The figure 2 of https://arxiv.org/pdf/2110.15797.pdf features specific architectures implemented for sequence modeling tasks (in particular, a Transformer with non-causal attention that outputs permutations in one forward pass). I would suggest to add this figure in "VOI in practice" section.

minor: 

- In "Permutation Sequences and Permutation Matrix", "A set of all such permutations is denoted as" -> "A set of all such permutations on $[1\dots n]$ is denoted as $S_n$"

- In "Decoding Strategies - Adaptive Non-Monotonic Ordering", it would be better to clarify that $\mathbf{z}=(z_1,z_2,\dots,z_n) \in S_n$ is a one-dimensional representation of permutation.

- combinatorical -> combinatorial

- In "How are NMAO Discovered - Formulation", "Note that $y_{z_i}$ denotes that the generated token needs to be put in position $z_i$ of the output sequence" -> "Note that $y_{z_i}$ denotes that the ith generated token needs to be put in position $z_i$ of the output sequence"

- The softmax in "Discrete Approximations for Vectors" is missing right parenthesis

- The $x$ in $X = \phi(\mathbf{y}, {x}) \in \mathbb{R}^{n \times n}$ in "Optimization Procedure - Stage 2.a" should be boldface

- In "Optimization Procedure - Inference", it would be better to state that the source value is $\mathbf{x}$; output from autoregressive decoder is $\mathbf{y}$ through ordering $\mathbf{z}$; in practice $\mathbf{y}$ is generated through the insertion, and the insertion order is parametrized by $\mathbf{z}$ (by first converting $\mathbf{z}$ to $\mathbf{r}$).

- In "What's Next", $y$, $z$ -> $\mathbf{y}$, $\mathbf{z}$

- Minor parenthesis mistakes in "MDP Formulation" section

review_title: Review

Decision: Accept