Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper considers the problem of learning text generation models using reinforcement learning. The problem is challenging in that RL algorithm becomes inefficient or unstable when dealing with large action space and the sparse reward situations in text generation. To address these problems, this paper adapts the path consistency learning approach to the text generation setting. It allows to train the model with both on- and off-policy samples, bridge the sparse reward signal to directly supervise the Q-function learning, and makes efficient updates to Q-values by considering all candidate actions together. Experiments result show that the proposed approach are effective in solving a wide range of applications where MLE are not applicable, where it achieves better performance than previous RL-based methods as well as task-specific methods. In addition, on standard MLE-based tasks, the proposed approach could also achieve competitive performance when training the models from scratch.
main_review: Strength: This paper establishes a connection between the text generation problem and path consistent learning. It leads to a principled and elegant RL-based text generation method that naturally inherits many desirable advantages from PCL algorithm, such as being capable of using both on- and off-policy samples. It is widely applicable to text generation tasks where MLE training is not directly applicable, and demonstrate superior performance on 3 such tasks.

Weakness: The work is almost a direct application of PCL to text generation, and there is no newly developed RL algorithm for text generation. This may not be a serious weakness given that this paper mainly focuses on RL-based text generation problem, and the good performance could be encouraging for research on RL-based text generation methods. The experiments on standard MLE-based tasks are relatively small scale. Demonstrating competitive performance of SQL on standard machine translation tasks would be more convincing.

Comments: 
- It would be helpful to explain the intuition of the single-step PCL loss (9) and the multi-step PCL loss (10) in the text generation context. In particular, compare its intuition to the MLE loss could make it better received in NLP community.

- It seems that the perplexity of the proposed method (SQL full) is not as good as the others (Figure 3 Middle and also Table 1). The authors may need to discuss it more thoroughly.

- Compared to the original PCL learning, the SBEED algorithm [Dai et al 2018] provides a provably-stable algorithm for path consistency learning when there is nonlinear function approximation (e.g., by using deep neural networks to parameterize the Q-function, the state-value function and the policy here).

[Dai et al 2018]: “SBEED: Convergent reinforcement learning with nonlinear function approximation”, Proc. ICML 2018.

summary_of_the_review: This paper establishes a connection between text generation and path consistent learning, which leads to an elegant RL-based text generation algorithm that inherits many advantages from PCL algorithm. The performance of the algorithm is encouraging for RL-based text generation, although having more large-scale experiments would be more convincing.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Authors proposed to use soft Q-learning (SQL) to formulate the RL problem in sequence generation. The stability can be increased when techniques of path consistency learning is corporated.
main_review: 1. I think when mentioning text generation, people will recall language modeling or machine translation. This paper only did experiments on some tasks which I don't think it's central to the topic. I think it's a misleading title and content. Since authors used this title, I think at least one of these 2 central tasks should be added in experiments.

2. I think overall all the technical details are not new (correct me if I am wrong, and in this case kindly pointed out what's the contribution). It reads to me that the major contribution is trying some existing technical methods in these sort of problems. If that's the case, I feel it's more appropriate to submit to EMNLP I guess. 

3. On technical issues, I feel like the claim is not well justified. In many places, authors claims SQL is the important thing in overcoming the existing problems. However, in SQL the major concept to me is the entropy term. If that's the case, that means we want the term to be large and it translates to we want an evenly distributed actions space and in this thinking, normal SARSA or Q-learning or with larger epsilon greedy search should also work. I think if SQL is the reason, this should also be verified.

4. Somehow it reads to me that SQL itself is not even enough to overcome the challenges so path consistency learning is the most important. So I think 1) there should be an ablation analysis on SQL only, and 2) I feel path consistency learning can be applied to other RL problems as well, which should also be included. And if it cannot be combined, please discuss the reason.

summary_of_the_review: Overall, I have the experience in using RL for machine translation so I agree with the authors that those problems mentioned in the paper are indeed challenging. But I am not familiar with SQL but from the derivation provided in the paper I am skeptical on the effectiveness of SQL. Since most if not all technical stuff are existing work, I think the novelty is limited and thus justification of each component in the method is important to me.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new text generation framework based on the existing soft Q-learning of reinforcement learning. The experiments demonstrate that the proposed text generation method achieves superior performance to baselines. 
main_review: This paper models text generation as a reinforcement learning problem, and uses the existing reinforcement-learning techniques (soft Q-learning and path consistency learning) to relieve the sparsity reward and large action space problems in the existing reinforcement learning based text generation methods. 

Strong Points:
- The paper is well-written.
- The illustration of this paper is clear and good.
- The proposed methods achieve moderate improvement over previous methods.


Weak Points:
- The novelty of this paper seems limited as the techniques used in this paper are directly from the reinforcement learning area (soft Q-learning and path consistency learning). Considering that the reinforcement learning methods had been widely used in the text generation methods, e.g, MIXER [1] SeqGAN [2], the idea of using reinforcement learning methods for the text generation seems less novel.

- One of the motivations of this paper is to relieve the sparsity problem in text generation. However, this problem had been clearly pointed by the previous text generation methods, e.g. LeakGAN [3] and IRLGAN [4]. Specifically, IRLGAN tried to relieve this problem using the inverse reinforcement learning method, which is highly related to the submission. However, the paper does not compare with them and even does not mention them. This is limiting.

- How significant is the sparsity problem in the chosen three text generation tasks? If the authors emphasise the sparsity reward problem is significant in the text generation, the related experiments, e.g. generating different lengths of texts, should be performed. However, the related analysis is absent in the paper.

- One of the baselines in this paper is MLE+PG. There are many MLE+PG text generation methods, but authors do not clearly state which one they use in the paper, which leads to confusion.

- The paper shows the generation results of the prompt generation which is an interesting application of text generation. However, there are many meaningless and less fluent generated prompts in the shown examples, e.g. macintoshintoshintoshintosh in Table 7. What causes this phenomenon?

- In addition to the prompts results, It would be better to show generated examples of other tasks.

[1] Ranzato, M. A., Chopra, S., Auli, M., & Zaremba, W. (2015). Sequence level training with recurrent neural networks. arXiv preprint arXiv:1511.06732.

[2] Yu, L., Zhang, W., Wang, J., & Yu, Y. (2017, February). Seqgan: Sequence generative adversarial nets with policy gradient. In Proceedings of the AAAI conference on artificial intelligence (Vol. 31, No. 1).

[3] Guo, J., Lu, S., Cai, H., Zhang, W., Yu, Y., & Wang, J. (2018, April). Long text generation via adversarial training with leaked information. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 32, No. 1).

[4] Shi, Z., Chen, X., Qiu, X., & Huang, X. (2018, July). Toward diverse text generation with inverse reinforcement learning. In Proceedings of the 27th International Joint Conference on Artificial Intelligence (pp. 4361-4367).
summary_of_the_review: In summary, this paper is well-written but it lacks relevant analysis and experimental evidence for the problem they want to solve. Further, the method they proposed is directly from the reinforcement learning area, which significantly reduces the contribution of this paper. Last, the paper lacks the experimental comparison with the related work aiming to solve the same problem.
review_title: null

Meta-review: This work proposes an approach to improve non-ML based methods of text generation. It reformulates the problem with the soft Q-learning approach from RL instead of standard hard RL formulations from previous text generation work. By doing this, the work allows application of path consistency learning. This is an elegant formulation. However, this reformulation into soft Q-learning appears quite straightforward and so the application of path consistency learning does not require much change to be used for text generation. This limits the novelty of the work. The experiments are also relatively small-scale and consists of some non-standard tasks such as prompt generation (which is typically evaluated indirectly, the response to the prompts rather than the prompt itself). As the reviewers mention, evaluating on more large-scale standard tasks such as summarisation or dialog would be more convincing. Finally the work lacks references to recent works in the field, such as LeakGAN.
Decision: Reject