Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper draws a connection between teacher-forcing and inverse reinforcement learning. The authors derives theoretically from the teacher-forcing objective to an equivalent reward function (i.e., a model trained with teacher-forcing recovers a reward function). With this insight, the authors leverage this learned reward function to train on non-parallel data. The reward can be applied step-wise, therefore mitigates the common instability issues when training seq2seq models.
strengths_and_weaknesses: Strengths
- The idea that links teacher-forcing to its inherently learned reward function is, to my knowledge, novel and the theoretical connection has not been explicitly drawn before
- The paper is easy to follow and the methods are well motivated
- The proposed method is simple and easy for the community to extend and build research upon
- The experimental results and the analysis are illuminative

Weaknesses phrased as questions below (if addressed or answered, I'm happy to change the score).


questions: - In equation 7, the second term requires computing a $\max_{a' \in \mathcal{A}} f_w(s + [a], a')$. Does this mean that at each generation step the model needs to run a forward path for every action, i.e., word in the vocabulary? If so it might render the method slow and hard to scale.
- The Periodically Synchronized Behavior Policy is described as a contribution but to my knowledge similar issues (discrepancy between the behavior policy and the model policy) are long-standing and has been discussed as a simple fix such as in PPO (Schulman et al., 2017) or other off-line RL methods (update after K steps). Is there any main difference that I missed here?
limitations: The authors appropriately addressed potential limitations in a standalone section.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents an approach for training a text-generation model by transforming an existing conditioned language model into a reward function. The paper suggests that the pre-softmax logits of a pre-trained sequence-to-sequence language model can be used to compute a per-token reward for new input sequence data. The paper also proposes an algorithm for training a model with this reward function, including off-policy exploration. The experiments compare the proposed approach against several other methods for training with additional non-parallel data (e.g., self-training, and R-regression, which uses a reward function derived from BLEU score. Compared to existing approaches which use single-sentence reward (e.g., BLEU score), the proposed approach provides a denser reward which results in moderate improvement over the compared systems.
strengths_and_weaknesses: Strengths:

To the best of my knowledge, I haven't seen any other work that trains a model with RL using rewards derived from another language model. It is a clever approach. Experiments were done on two different text generation datasets.

---

Weaknesses:

The improvements in performance reported in Table 1 seem quite small, except for BLEU2. Are they significant?

There are some experiments I would have liked to see: 
* The ability to compute the proposed reward function is general, but it still requires a domain-specific f_w to compute. Would have been nice to see experiments on deriving reward functions from other f_w besides in-domain, e.g., rewards from a general pre-trained language model.
* How much does the quality of f_w matter? It would have been interesting to see performance relative to the number of training examples used to train f_w. How little parallel data do you need?
* It would be nice to see learning curves / reward curves comparing the proposed approach with other methods. How many samples and gradient updates do you need for the proposed approach vs. self-training, R-regression, etc?

And some evaluation I would have liked to see:
* Some deeper analysis of model outputs after different training methods. E.g., more than a one or two pieces of generated text per approach.
* Evaluation with a more recent text generation metric, e.g. BERTScore or similar

Some claims / phrasing are a bit confusing:
* Teacher-forcing isn't necessarily the only way to train a text generation model; masked training objectives are also effective and popular these days. Also, you don't need parallel data to use teacher-forcing based training objectives.
* I'd also be interested to see non-conditional text generation with this proposed method. It doesn't seem like anything in the proposed method requires the task to be a sequence-to-sequence task (maybe this is what the authors are suggesting for future work with the reference to GANs in L153?).

Some missing related work:
* Kreutzer et al. 2016 / 2017 / 2018 use BLEU score for training text generation

--- 

Some minor suggestions (e.g. readability):

* In 2.1, I'd suggest moving lines 61 and 64 to closer to the top[ (e.g., after L56). I also am not sure what Eq 2 adds, besides it being standard in RL papers.
* What does it mean for a reward function to be "naturally defined" in the context of text-generation; why would the suggested reward function be "natural"? (L65) 
* L100 -- policy q should be \pi_q, right?
* The discussion in the beginning of 2.3 is a bit confusing; the term "behavior policy" is unfamiliar to me ("inference policy" seems more natural)
* The claim that any behavior policy can generate any trajectory seems broad to me. What if \pi_b = \pi_\phi?
* Nitpick on L130 -- the policy starts to generate fixed trajectories because the entropy is collapsing, not because it's becoming more optimal. Also in L254, this seems to be the case as well.
* I was relatively confused about what was being plotted in Fig 1 -- what's the x-axis?
questions: * What is the intuition behind the derived reward function in Eq 7? It looks kind of like an advantage function.
* What happens if r(s, a) < 0, if this ever happens (it seems like it should)? Does this cause problems with training, as this means Alg 1 is directly minimizing log \pi?
* Would training with teacher forcing basically be \pi_b \prop f_w, with k = \inf?
* How is \pi_\phi initiated? With f_w or just the original T5 base parameters?
* Is the training for self-training using teacher forcing, just with sequences derived via inference from unpaired data?
* How do you choose to stop training?
* Eq 7 -- f_w returns unbounded logits, right? Does this cause issues for training?
* Did you experiment with a discounting factor in Algorithm 1 that attenuates return backwards in time? If f_w is usually positive, this seems like it would give much higher rewards to actions at the beginning of a sequence rather than at the end.
* Did you experiment with an entropy term to avoid collapse (similar to what you are seeing with k=1)?
limitations: I'd be interested in hearing a bit about computational efficiency for training and how the different methods compare. Self-training seems more efficient as it doesn't require sampling/inference at training time, except for a fixed number of examples. I believe R-regression requires running inference on a separate model, and obviously the proposed approach does as well, but it seems twice as many times, as in Eq 7 f_w is computed both for s and s + [a].
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work proposes a novel reward function for text generation models, which is derived from a pre-trained supervised model. Authors also suggest using the new reward function to utilize monolingual data for fine-tuning. This training framework aims to resolve two main weaknesses of text generation models, namely exposure bias and lack of parallel data. Empirical results on two different generation tasks show improvements upon the baselines.
strengths_and_weaknesses: Strengths:
- Building an interesting connection between training with teacher-forcing objective and RL objective
- Significant improvements over the baseline (however, see weaknesses)
 
Weaknesses:
- The cited previous results in Table 1 need verification 
- Sloppy citations
- Paper is hard to understand and needs clarification
- Thus: the argumentation in the paper is not convincing

**UPD**: Weaknesses are addressed by authors during discussion; Score is updated
questions: Comments and questions:
- line 12: You say "common training method" and give one citation from 2014. It does not sound convincing. At least you can cite Transformer model/Bert/BART
- While you speak about text generation training, do you mean the structure of the dataset or the input/output of the model by parallel dataset? Because the text generation model can be trained on monolingual data without any need for labels. 
- You claim that large models intensify the problem of creating parallel datasets. But these large models are usually trained on **monolingual data**, and can be fine-tuned on the smaller task-specific dataset
- line 34: what objective do you use for training seq2seq?
- line154: you do not say what seq2seq model: is it Transformer? or T5?
- Table 1: Do I understand the training process correctly: T5 base pretrained --> train it on task-parallel data with teacher forcing --> to add additional parallel/non-parallel data using RL? Which of that does line 3 Table 1(a) (T5-Base) report?
- You cite results from "DialogBERT: Discourse-Aware Response Generation via Learning to Recover and Rank Utterances". Why can the numbers from your paper not be found in the source? Same for AdaLabel
- According "Exploring Diverse Expressions for Paraphrase Generation", the quora paraphrase test set contains 30k, you report 20k on test set. Why such difference?
- "Exploring Diverse Expressions for Paraphrase Generation" does not report iBLEU4, but you cite the number. What is this number?
- "Learning to Selectively Learn for Weakly-supervised Paraphrase Generation" does not report SBLEU, how did you get the number?

**UPD**: Questions are mostly answered by authors; Score is updated
limitations: Limitations are addressed in Section 5
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper investigated the connection between supervised teacher forcing and inverse reinforcement learning. The authors showed that under the same parametrization and a deterministic transition function, the loss function of teacher forcing and maximum entropy IRL are equivalent. Then the authors proposed to use the teacher forcing model to replace the q function in the bellman optimality equation to derive a reward function. Empirical results show that the proposed methods outperform semi-supervised learning baselines on text generation tasks.


-------
POST REBUTTAL:

The authors addressed my concerns. After reading the other reviews and the discussion, I increased the score to 6.
strengths_and_weaknesses: Strengths:

The proposed algorithm is interesting. It avoids estimating the partition function (e.g. using GAN or sampling methods) in maximum entropy IRL, but instead using the teacher forcing model to derive a reward function. The training in this case will be more stable.

Weaknesses:

I am not entirely convinced that the proposed reward function in equation (7) is valid.
(1) Theorem 1 in the paper shows that, given the same parameterization, MaxEnt IRL and teacher forcing both maximize the likelihood of the trajectories in the data and hence are equivalent. However, the q function in MaxEnt IRL (in equation (3)) has to additionally satisfy bellman equation and cannot be optimized freely. Therefore, the q function q_w cannot be simply replaced by the logit function f_w, and f_w(s, a) does not equal to q_w(s, a) for a given (s, a) pair. Therefore, I am not entirely convinced that equation (7), which simply substitute q_w with f_w, is valid.

(2) Even if we can replace q_w with f_w, f_w is learned from data and represents the behavior policy, while q_w in equation (7) should be the q function of the optimal policy.


questions: In terms of experiments, since this paper is about IRL, why not include IRL baselines such as GAN-based IRL algorithms?

limitations: N/A
review_title: null

Meta-review: metareview: This paper proposes a method to design a reward function from a pre-trained language model, and uses it to train a text generation model using reinforcement learning. The approach is novel and the paper presents both theoretical derivations (drawing connections between maxent IRL and the supervised teacher forcing loss) and empirical results to demonstrate the effectiveness of the proposed algorithm, offering new insights for a challenging problem. 
confidence: Certain
Decision: Accept