Venue: ICLR 2022 Track Blog

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The blog post is on the ICLR 2021 paper called Text Generation by Learning from Demonstration.

Pros
- The blog introduces basic concepts well, so even readers who don’t have extensive RL or text generation experience will be able to understand the blog post and benefit from it. 
- The flow of the blog is very nice. It’s easy to follow along. 

More comments

In the second paragraph of the background section, the authors might be slightly confusing training and decoding a bit. In MLE training, no decoding algorithm is involved. The core issue is that MLE maximizes E_{y \sim p_{human}} \sum_t \log p_\theta(y_t | y_{<t}, x), but the evaluation objective maximizes E_{y \im p_\theta} R(y) where R is some reward function; to achieve good evaluation score, we don’t necessarily need to do MLE to fit the human/dataset distribution. 

In the first equation of the policy gradient section, J is an objective, and J(\theta) should not include the max symbol. Per-step reward uses r in the first equation, but R in the next paragraph–the notations are inconsistent.

In the “how GOLD uses importance sampling” paragraph, the writing is good. I do wonder if the original paper’s approximation of importance weights is too liberal. I wonder what the authors of the blog post think. 

The concluding remark is interesting, and I agree that GOLD might not work if we need diversity in generated text (also depending on the definition of “diversity”). More commentary on the paper (instead of describing the paper) will be nice, in my opinion. 

Better phrasing
- In the paragraph “exposure bias: GOLD trains the model to choose the next token given a context and tokens generated by the model”, it’s worthwhile to emphasize that the tokens are not explicitly generated by the model. Instead, GOLD are treating the reference translations *as if* they were generated by the model (but they are not actually generated). 

Potentially more background
- I would appreciate it if the authors use a paragraph or two to introduce value-based and model-based approaches for text generation. Or expand “off-policy vs. on-policy learning” a bit more. This is not necessary but probably very helpful. (Recent examples for Q learning: https://arxiv.org/pdf/2106.07704.pdf, https://arxiv.org/pdf/2112.08670.pdf)

Potentially more illustration toward the end of the blog post: A figure might help in the last few sections. In general, imo it's a good idea to make the blog post more lively (e.g., illustrating the concepts through examples).

Minor issues
- In the third paragraph of the “background” section, the state should include the text generated so far (prefix/history) as well as source. 

Minor style issues
- Usually commas follow “i.e.”?
- “This” should usually be followed by a noun. For example, in the sentence “this is valid when R is discontinuous…” it’s hard to know what “this” refers to.
- In the background section, there’s a punctuation typo in the first paragraph. 
- In the importance sampling section, “x has a data distribution p using samples another distribution q” is not fluent

review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: The blog is very informative, it includes all needed background and covers what is needed to understand the paper.

It could have been more lively, easy going or easy to read. It often feels like a broad paper, where not a lot of the cognitive load needed to read a paper is lifted, and if that is so, why read a blog?]

The table of notations is a nice idea.

In a blog it is much better to reduce some of the details to gain 
minor:
typo:
publicaión
review_title: review

Decision: Accept