Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper shows a connection between the reward maximization (RM) approach and the distribution matching (DM) approach for fine-tuning language models. The authors further suggest borrowing the baseline idea from reinforcement learning and applying it to DM to reduce variance. Experiments show that adding a baseline not only makes training more stable, but also converges to a better solution.
main_review: While I think it's nice to analyze the connection between RM and DM, the math provided in the paper is simple, and the main contribution is just to add a baseline to the algorithm of Khalifa et al. I encourage the authors to try out ideas they mentioned as future work to have a substantial contribution for a conference publication.

Typos: In Sec 5.2 Tasks (f), "$\phi_2(x)=1$" should be "$\bar \mu_2=1$". In Sec 5.5 Gradient Variance, $\pi$ is missing in "$G_\theta(x)=A(x)\nabla_\theta \log_{\theta}(x)$"
summary_of_the_review: I think this paper is below the acceptance threshold because it is just a simple addition to the method of Khalifa et al.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper untangles the connections between the RM and DM paradigm, and exploit these connections to propose a baseline for the DPG method. Experiments on a set of controllable language generation experiments show that the propose baseline can lead to better performance, stability and sample efficiency.

main_review: * Strength
    * This paper proposes an interesting idea and interpretation to connect RM and DM paradigm
    * This paper proposes a variance reduction method for DPG which demonstrates its improvement on performance, stability and sample efficiency
* Weakness
    * From my understanding, the baseline mostly comes from the observation in 3.3, which has limited technical novelty. It will be great if authors can justify more on the technical novelty.
    * It will be cool to show some results on BLEU improvement on machine translation, or user studies on language generation tasks, to demonstrate its practical impact, as the quantitative metrics right now are still kind of artificial.

summary_of_the_review: This paper addresses an interesting direction to connect RM and DM paradigms and shows promising results for better controlled language generation with their proposed variance reduction technique. Though I'm not sure if the technical novelty is sufficient and the empirical results are strong or practical enough given limited technical novelty.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors explore the connections between reward maximimization (RM) with REINFORCE and distribution matching (DM) using distributional policy gradients (DPG), which uses importance sampling from a proposal distribution $q$ to minimize  $L = E_p[log\pi]$, where $p(x) = P(x)/Z$ is a distribution (energy-based model) that is difficult to sample from (but incorporates generation constraints, such as gender percentage, etc.). The corresponding DM sequence-level reward in REINFORCE is p/q. The authors derive and utilize $E_q[p/q] = \pi/q*Z$ as a baseline, which lowers variance, and leads to improvements in loss L and sampling efficiency.

main_review: Strengths:

- presents novel baselines for DPG, which improve optimization performance.

Weaknesses:

- The connection between importance sampling based distillation and REINFORCE was introduced in in the DPG paper, and from this perspective, adding a baseline is a straightforward exercise, and lower in novelty.

- While baselines improve the DPG objective, constraint satisfaction, as shown by the second subfigure in figure 3, is NOT improved, contrary to claims in the abstract and main body of the paper.
- With lower objective and equal constraint satisfaction, text generated by $\pi$ may nevertheless be better. Are the improvements evident? No test results, quantitative or qualitative, are given on an application task to demonstrate this (the Khalifa 2021 paper, which utilizes the same technique less baselines, shows improved constraint satisfaction by using DPG-based DM, which is the point of utilizing DPG since EBMs are more difficult to sample)
- More generally, it is not clear to me that the DPG approach (baseline or not) is highly effective. The constraints in the EBM are NOT satisfied in either case, it would be great to investigate this in detail and figure out why.
- Related, the importance sampling step in DPG assumes that q(x)>0 where p(x)>0 to make the equality in (5) true, although this in the current context feels like a minor limitation.
- Based on Algorithm 1, it seems that the DPG importance weights are not normalized, which is the most common way to reduce variance (at the expense of bias) in IS. Is this the case?
summary_of_the_review: Baselines for DPG methods are introduced, which improve optimization performance. However constraint satisfaction, the ultimate goal of the process, does not appear to improve, and neither qualitative nor qualitative evidence of improved test performance has been included. This, and the fact that DPG baselines are quite straightforward to work out, make the contribution of the paper seem quite low.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper builds upon previous work on fine-tuning pre-train language models to conform to pre-specified "distributional constraints" (such as: "the model should generate the same proportion of sentences with female or male subjects", etc...). Specifically, the authors draw parallel between a previous approach (distribution matching, DM) and RL-based approaches a la REINFORCE. Based on this comparison, they propose to adapt variance reduction techniques from RL (in particular the use of a baseline) to DM in order to facilitate training. Experimental results show that the resulting improved approach boasts better sample-efficiency, improved stability and overall competitive performance.
main_review: The idea to use a baseline to stabilize training in DM is rather simple but quite efficient as demonstrated by the experiments, and I think this is an overall useful, if somewhat incremental contribution. That being said is not quite clear to me that there isn't a simpler, more straightforward way to reduce variance in DM (see my full review for details).

Overall I think that the paper would be interesting to the community and I recommend acceptance, however I set my score as "borderline" until the concerns I raised in my review are addressed by the authors.

Strengths:
- Fine-tuning pre-trained language models with distributional preferences is an important research topic
- Simple, yet clever use of baselines.
- Convincing experimental results

Weaknesses:
- Some of the results of the paper (esp. section 3.2 on interpreting REINFORCE w/ KL control) are not particularly insightful as they don't really inform any modeling decision (as opposed to the "parametric reward" idea which suggests using variance reduction techniques from RL to apply to DM). What's more, I think that saying "KL-control developed in the RM paradigm can also be construed as belonging to DM" is slightly misleading because DM's core advantage is not so much in that it minimizes a KL but presumably that it minimizes the KL *in a specific direction*.
- Missing baseline for DM? It seems to me like optimizing the DM objective by actually sampling from the target distribution $p(x)$ (by eg. sampling from a large unlabeled dataset to get samples from $a$ and reweighting/subsampling according to $P(x)$) would be a more natural approach than first sampling from the model, and then reweighting with the likelihood ratio. The reason I think this is worth using as a baseline is that the variance issue the paper tries to address arises precisely because the DM objective is estimated by sampling from the model rather than sampling from the data. If we need RL variance reduction techniques solely because we are formulating a straightforward log-likelihood/cross-entropy objective as an RL objective, this begets the question: why formulate DM as RL in the first place when it doesn't need to be? I hope the authors can clarify this point.
- Results are reported exclusively as curves. I think that at least the core results should be reported as a table with a single number (with standard deviation) for each method (and each metric). It is also not entirely clear to me if the curves are training curves, or whether they represent quantities computed on a test/validation set.
summary_of_the_review: The idea to use a baseline to stabilize training in DM is rather simple but quite efficient as demonstrated by the experiments, and I think this is an overall useful, if somewhat incremental contribution. That being said is not quite clear to me that there isn't a simpler, more straightforward way to reduce variance in DM (see my full review for details).

Overall I think that the paper would be interesting to the community and I recommend acceptance, however I set my score as "borderline" until the concerns I raised in my review are addressed by the authors.
review_title: null

Meta-review: This paper explores the connections between reward maximization (RM) with REINFORCE and distribution matching (DM) with distributional policy gradients (DPG) for fine-tuning language models. Based on this, the paper proposes to apply a baseline (an idea in reinforcement learning) in DM to reduce variance and improve sample efficiency. Reviewers have concerns on the technical novelty as claimed in the paper, since the application of baseline is a straightforward practice and the resulting method is a simple addition to the existing method. More analysis (such as on the tradeoff between prior and constraint satisfaction, etc) was also suggested.
Decision: Reject