Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper elaborates on the idea of the DPG (Parshakova et al., 2019b) and GDC (Khalifa et al., 2021) paper, explicitly states that "KL-control", a specific reward maximization method, is also a distribution matching method with energy-based models. The paper also clearly explains the similarity and difference between distributional gradient descent and standard gradient descent. An advantage-based GDC algorithm is introduced in the paper as a standard technique in RL to reduce the variance of gradient estimates. Numerical experiments show the effectiveness of the proposed method.
strengths_and_weaknesses: DPG framework could be a powerful tool for fine-tuning LMs to maximize some non-differentiable objectives using RL/EBM, but also prevent the catastrophic forgetting problem. The idea is neat with mostly straightforward derivation. The paper clarifies the key concepts and connections to existing training paradigms. Readers that are perplexed about fine-tuning LMs with reinforcement learning would greatly benefit from this paper.
questions: I don't have any critical questions or confusions about the paper.
limitations: Limitations and future directions are well discussed at the end of the paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper focuses on fine-tuning language models for controllable language generation tasks ( constraining topic, sentiment, gender distributions, etc). The authors derive the connection between distribution matching approaches and approaches that use KL-divergence penalty to prevent distribution shifts. By deriving a parametric reward baseline for Distributional Policy Gradients (DPG) and use it to extend the work by Khalifa et al., a distribution matching approach for nudging autoregressive generation distributions, this work achieves variance reduction and superior performance. 
strengths_and_weaknesses: Strengths:
* The mathematical derivations for connecting policy gradients and distributional policy gradients (DPG),  connecting KL-control and distributional matching, DPG baseline are easy to follow. I appreciate the effort. These derivations provide good perspectives and should be valuable to the community.
* The main claim of introducing a baseline to reduce variance of DPG is well-justified and supported by evidence (Section 4).

Weaknesses:
* I think the contribution on making connection between policy gradients and distributional policy gradients is unnecessarily exaggerated, such as line 77: "So far, the connections between these two seemingly distinct paradigms have not been thoroughly explored". I think even the similarity in names already suggest that they are closely related, and they indeed aren't that different from each others. I think this claim actually makes the paper's logic difficult to follow for me. I would suggest focusing more on the contribution of introducing the parametric baseline, which should be the main product of this paper in my opinion.
* I'd like to see a consolidated definition of the control language generation task. Right now, they are scattered in intro, Sec. 4.1, Sec. 4.2. What's the context and desired outcomes of such tasks? How does DPG achieve the desired outcomes (now it's sort of described in Sec. 4.2)? Can the authors give some examples?
questions: Overall, I think the technical contributions are solid and novel, but I have concerns about how these contributions are framed and positioned. Can the authors address or clarify the concerns I raised above?
limitations: Limitations and potential negative societal impact are not discussed. Please add relevant pieces to the conclusion section.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies controllable text generation with an emphasis on sequence level rewards and distribution matching. It first establishes a connection between reward maximization (RM) with KL control and distribution matching (DM) in a general sense. They show that there is an equivalent formulation of RM in DM terms and also DM can be connected to RM by thinking the importance sampling (IS) term as a reward. Using these connections, the authors highlight that a recent DM method, distributional policy gradient (DPG), suffers from the same high variance of REINFORCE estimators as the latter appears in the loss function of DPG. As is typical in RL, the authors propose using advantage estimations to reduce the variance of the DPG estimator and show that the intuitive expected reward (estimated as mean IS weights) gives unbiased estimates. The authors evaluate the proposed method, GDC++, on 10 controllable generation tasks including pointwise and distributional constraints using moment matching, KL-divergence, BLEU score metrics. When compared to other baselines, GDC++ shows lower divergence from the prior language model while also having competitive constraint matching. Using an advantage estimation is also shown to perform better and more stable results.
strengths_and_weaknesses: The main strength of the paper is the connection between RM and DM to motivate using a baseline in DPG. There are also several weaknesses that needs clarification. I detail these below.

## Strength
1. The paper is well written and easy to follow. 
2. The connection between RM and DM to motivate using a baseline in DPG and showing that the particular expected reward baseline is unbiased are interesting.
3. Experimental results show improvement compared to GDC and Ziegler et al.

## Weaknesses
1. While DM is not exactly the same as RM, I think there are previous work that incorporates KL-divergence between an online distribution and an offline distribution as a reward signal to train a policy using RL. Some of the more recent work are SAC$^1$, SMM$^2$ that induce a reward using a distribution matching and an older work *. While these don't necessarily utilize a KL-control, they study KL-divergence objective with energy based models or reward shaping.
2. The KL-divergence appears in different forms at different places. You define DPG in Line 105 and Algorithm 1 using $D_{KL}(p || \pi_\theta)$ while in Line 135 it is reverse.
3. Ziegler uses a reward shaping with a KL-control and the model is optimized via PPO. As PPO uses GAE, it also subtracts a baseline from this new reward. I think the relationship between this and the proposed baseline needs to be addressed.
4. It is also not clear how the $\beta$ is tuned in Ziegler baseline. By varying this hyperparameter, Ziegler can do better on $D_{KL}(\pi_\theta, a)$ while doing worse on $E_{\pi_theta} \phi(x)$, performing very similar to GDC++.
5. There is no human evaluation study in the paper and it is not clear how to judge the metrics proposed. For example, does higher KL from the language prior always means worse generation?

$^1$ Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor.
$^2$ Efficient Exploration via State Marginal Matching.
\* Reinforcement Learning by Probability Matching.
questions: I have several questions regarding my concerns above.

1. Could you clarify how your derivations are related to prior work on distribution matching and max entropy RL ? 
2. Can you clarify the form of KL divergence used in the paper?
3. What is the relationship between the baseline used in the paper and that of Ziegler?
4. Could you clarify how Ziegler is trained, especially $\beta$ hyperparameter? Can you get better tradeoff by tuning this hyperparameter?
5. Can you discuss how these metrics translate into human satisfaction?
limitations: The authors addressed the limitations.
review_title: null

Meta-review: metareview: All reviewers consistently agree that this paper provides valuable contribution in identifying the connection between different RL training paradigms for language models with clear derivations. In addition, it also develops an improved method by incorporating a baseline into DPG.
confidence: Certain
Decision: Accept