Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary
The paper considers the problem of attribute-based sequence generation, particularly in language models. Authors propose a framework “GeDi” which learns a generative classifier for controlling generation from a large language model. With experiments on publicly available datasets and models, and including human-evaluation, the authors empirically demonstrate that the algorithm is computationally efficient and is competitive against strong baseline algorithms like CTRL, Plug&Play language models (PPLM).

Reason for the score
I vote for rejecting the current version of the paper (marginally below acceptance threshold). While the premise of the problem is well motivated, I think several sections of the paper are difficult to follow. I would strongly encourage the authors to include a pseudo code of the algorithm to improve the presentation of the central idea. The paper includes several experiments, though I think some critical ablations are missing.

Strengths
+ The problem is practically well motivated and is very relevant to the language learning community. 
+ The proposed algorithm of using generative classifiers is computationally efficient compared to strong baselines like CTRL and PPLM. The experimental results suggest that the algorithm also allows for better control over generation from a LM while maintaining linguistic quality.

Weaknesses
- Several sections of the paper are hard to follow. To improve the presentation of the idea, I would encourage the authors to distill the central idea into a pseudo code which goes along with Section 3.
- The experiments on detoxification are critical to the thesis of the paper, however it seems that experiments in Section 5.2 consider only GPT-2 baselines? I think a strong baseline based on prior-work, like a CTRL generator conditioned on the positive label (as mentioned in Introduction), would help evaluating the gap between proposed approach and current algorithms.


review_title: Proposes algorithm for controllable sequence generation

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposed a method —- GeDi — to generate guided and controlled text from a large language model (LM). The method utilizes smaller LMs as generative discriminators to guide generation from large LMs to make them safer and more controllable. By safer and controllable they emphasis on the toxicity, hate, bias, and negativity contains in the training of the large LM. The proposed method guides generation at each time step by computing classification probabilities for all possible next tokens via Bayes rule by normalizing over two class-conditional distributions (i.e. contrastive discrimination); one conditioned on the desired attribute, or control code, and another conditioned on the undesired attribute (i.e. contrastive attribute), or anti-control code. 

The paper explores ways to increase generation speed and claimed that with the proposed techniques the generation speeds more than 30 times faster compared to PPLM model. The paper explores different heuristics to impose the guided generation including bias parameter, weighted decoding and filtering heuristics. The findings are that GeDi gives stronger controllability than the state of the art method (i.e.PPLM, CC-LM, CTRL).  

Experiments show that, training GeDi on four topics (i.e. Business, Science/Tech, Sports, World ) allows the controlled generation of new topics zero-shot from just a keyword. They also demonstrate that GeDi can make GPT-2 (1.5B parameters) significantly less toxic without sacrificing linguistic quality.


Re: “so long as the LM and GeDi share the same tokenization”: can you please elaborate the constraint on ‘same tokenization’?

Re: “If the GeDi was trained on movie reviews for sentiment control, its direct class-conditional predictions will be biased towards predicting movie review words (illustrated by next word prediction of “cinematic”). However, by contrasting the predictions of opposing control codes via Bayes rule, the bias towards movie reviews can be cancelled out.”: The word cinematic can reveal a neutral/negative sentiment, is there any possibility that pushing the sentiment towards positive might degrade the accuracy of the overall generation?

Re: GeDi training (λ < 1 in Equation (10)) and standard generative training(λ = 1 in Equation (10)). : How the value for λ = 0.6 was chosen? What is the impact of other values for this hyper-parameter? 
  
Re: “In order to have prompts that are more likely to trigger aggressive generations but less likely to be explicitly toxic, we pass candidate prompts through a RoBERTa (Liu et al., 2019) model trained to classify toxicity, and only kept prompts where RoBERTa was less confident about the toxicity label.“: how did you measure model confidence about the toxicity label?
review_title: The paper is written well

final_score: rating: 4: Ok but not good enough - rejection
review_content: #### Summary

The authors propose a method for controlling attributes of generated text (sentiment, topic, toxicity, etc) by reweighting a base language model's token-level distributions with auxiliary conditional language models, bayes rule, and additional heuristics.

The core idea is simple - which is a strength in my view - and does not require retraining the base language model, which could be important as language models become more expensive to train. However, the clarity and experiments in this paper fall short: the experimental setup has issues (detailed below), the effect on perplexity is quite large but relegated to the Appendix, several claims are speculative and lacking corresponding experimental evidence, and it is unclear how the additional heuristics affect performance.

The method seems promising, but with the current experiments it is difficult to draw conclusions about how the method affects performance and which parts of it are necessary; given that this is an empirical paper, I would therefore not recommend acceptance in its current form.

#### Experimental setup

- **Human evaluation**. There is little information given about how the human evaluation prompts are written (and why they are written that way), how many evaluators are used, and there are no significance tests (e.g. "We run human evaluation to measure toxicity" does not given enough details). This is concerning since all of the results in the main text use human evaluation, sometimes with small differences between methods.

- **Automatic eval/perplexity**. The authors only measure perplexity in one set of experiments in the appendix (and it gets worse by introducing GeDI training). It would be good to have perplexity and automatic metrics to compare against the human evaluation (e.g. see Table 4 in the PPLM paper).

- **Decoding algorithms**. The authors only show results for greedy decoding with a repetition penalty (with no ablation on the choice of penalty parameter). Results with a sampling method (e.g. nucleus) are needed for this open-ended setting, or an argument for why these aren't considered.

#### Effect of the method
- **Effect on perplexity**. The perplexity gets much worse as the gedi training is introduced (i.e. $\lambda$ decreases), e.g. going from 25 to 45 on IMDb. This result is in the Appendix, and perplexity is never evaluated/reported in the other experiments.

- **Gedi training**. It's unclear whether the gedi training (i.e. the $\mathcal{L}_d$ loss) is beneficial: in some experiments $\lambda=1.0$ performs similarly, and on the IMDb/MNLI/QNLI experiments decreasing $\lambda$ either hurts, has no effect, or improves performance (i.e. no consistent trend).

- **Detoxifying**. It's unclear how significant the results in Table 3 are, and there are no baselines; in general it it difficult to draw conclusions from these results.

#### Speculative claims or conclusions
- **Domain transfer**. Figure 1 gives an intuition for why domain transfer might be possible, but only an anecdote (first paragraph of 5.1, "we noticed that") and a single experiment is done, where the method performs similarly to PPLM. Crucially, the GEDI training does not appear to help over just re-weighting with the conditional LM ($\lambda=1.0$ vs. $\lambda=0.6$). Could the authors comment on this result? How well does domain transfer work for less similar domains? How is perplexity affected for the models reported in Table 2?

- **Zero-shot control codes**. The authors only provide anecdotes for evaluating the Zero-shot control codes. Based on the evaluation it's quite speculative to say "GeDi’s ability to generalize to new control codes zero-shot gives the ability to generate text corresponding to many topics and subtopics.".

- **Smaller language models guiding larger language models**. To be fair, the authors use GPT-2 medium as the conditional language model, and GPT-2 XL as the base language model, which is larger, but there was no investigation of this aspect of size difference. How small can the conditional LM be? Why was medium used instead of small? What if large was used? Does the conditional LM need to be a large-scale pretrained model (it would be nice to see a baseline of a simpler conditional LM)?

#### Heuristics
- Several heuristics are used: $\alpha/T_i$ weighting, $\omega$ weighting, nucleus filtering, keeping tokens over a threshold, repetition penalty, and rescaling the logits to positive (used in only one experiment).
- How does each of these affect performance? There are no ablations, and given the small differences in some of the experiments it is unclear whether performance would actually be worse if we changed one of the heuristics. One outcome may be that the method only works for a careful balance of hyperparameters, which could be fine, but we don't have a sense of the variation.
review_title: Review: GeDi: Generative Discriminator Guided Sequence Generation

final_score: rating: 5: Marginally below acceptance threshold
review_content: ##########################################################################
Summary:
This paper proposed using small-sized LM as a generative discriminator to guide large-sized LM for better controllability and decoding efficiency. 
 
##########################################################################
Reasons for score: 
 
My score is marginally below the acceptance threshold. 
 
Pros:

1. The most exciting part of this paper is to factor out the opposite labels of each token (e.g., positive and negative, or toxic or non-toxic) using Bayes rule in generative models. 

 
Cons: 

1. The contribution of controllability and producing safe output should be separated out. Safer LMs seem to be the outcome of the controllability of the LM by canceling out the opposite part of the target label using the Bayes rule. Highlighting this point might be helpful for showing out the novelty and value of this work. The current frame of the work seems quite distributed between applications and architectural contributions. 
 
2. The main concern of this work is the lack of focused contributions and their validations. The authors claim that this model is good at almost everything; efficiency, controllability while maintaining linguistic quality, reducing the toxicity of GPT2, zero-shot topical generation, etc. However, in fact, most of the experiments in Section 5 are very shallow, uncontrolled, and lack statistical significance. I appreciate the general effectiveness of the model and I don’t doubt it. However, as a conference paper with limited pages, it would be better to make one or two points among them and providing more in-depth with valid setups of experiments. I put additional notes about the experiments below. 
 
3. As mentioned above, the novelty of this comes from using the Bayes rule to make positive and negative labels far from each other. The authors also mention that this is a sort of contrastive learning, and they used contrastive generation. However, contrastive learning is often used to refer to learning by separating two different instances out far each other. In this work, there is no such auxiliary optimization during training time, but the posteriors are re-weighted using the Bayes rule. I would recommend using a more exact term to describe this rather than contrastive generation. 
 
4. In Section 3.1.1, various heuristics are used. I expected to see the effect or ablation of each heuristic and how important each of them is in terms of generation quality. Also, the baseline models such as CTRL, CC-LM, and PPLM seem to be not using the same heuristics, which seems to be not fair. 
 
5. The output in Table 6 makes me doubt how the experiments are badly controlled. The outputs from positive and negative sentiment are totally different and almost random text, meaning that the content of the generators is not controlled properly. In preparation for prompts for GeDis (5.2) or for measurement of label fidelity (5.3), authors used the pre-trained BERT or RoBERTa on the target attribute like toxicity and topics. As far as I know, these automatic classifiers are not correlated with human judgment, in fact, leading to huge wrongly-predicted labels. I wonder why human annotations are not used here. 

6. In Table 2, I don’t see any significant improvements of GeDi against PPLM in its attribution score (i.e., positivity) and transferability to the target domain. Similar to the comment above, none of the experiments are controlled in content. Measuring how the text is similar to the domain (e.g., book-like) sounds interesting but there are no further details of how the human evaluation is studied, what kinds of guidelines are provided to annotators, how the output looks like, etc. 

7. In Table 3 and 4, have you performed the same experiments with PPLM and CTRL? 
 
##########################################################################

review_title: A novel approach but limited experiments with a lack of valid experimental setups. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: [Summary]
In this paper, the authors propose an efficient method for controllable language generation of large pre-trained LMs (e.g., GPT2). The main idea is to use a smaller, compared to the LM to control, language model trained with control code (Keskar et al., 2019) to generate a per-token score $P(c|x_{1:t}$) to steer the original Language Model distribution. The author proposes two ways, contrastive and discriminative, to approximate $P(c|x_{1:t})$ using bayesian rules. Experiments on open-ended language generation have been shown for positive/negative, detoxification, and topic-control, including a zero-shot topic-control. 

[Pros]
- the proposed methodology is novel for the task and it is effective in controlling the desired attributes. 
- the proposed method is more efficient than WD (Ghazvininejad et al., 2017) since it does not require a forward to the discriminator for each to token in the vocabulary, and computationally more efficient then PPLM (Dathathri et al. 2020) which requires several updates per token. 
- the paper is well-written and easy to follow, except for some minor (later for more info). To the best of my knowledge, the paper is technically correct and reproducible.

[Cons/Question for the authors]
- I have read through the paper, but I could not find any significant test (e.g., annotator agreement, t-test etc.), are the reported human evaluation results significant? could you provide p-values for the results? 
- Both detoxification and topic control has no baselines to compare with. For instance, CC-LM-non-tox, CTRL, PPLM could have been used to detoxify the generation. For detox, why not using Universal Triggers (Eric et.al. 2019) for making the model generate toxic text, instead of using prompt from the dev set of the same dataset used for training GEDI? 
- Zero-Shot Topics: why PPLM and CTRL cannot do zero-shot on a topic (from the conclusion)? PPLM can use a bag-of-word discriminator, so no training required and generate any kind of topics, and CTRL can use different link/prompt to generate unseen topics?
- Greedy decoding: why using greedy decoding for a language generation task? it is well known that top-p and top-k greatly improve the model generation, are there performance drop if using top-p? how GEDI compare to CTRL, PPLM  in this setting? 

[Reason to accept]
The proposed method is a simple and effective way to control the generation of large language models. This is an important and timely problem, especially for language detoxification.

[Reason to reject]
The experiments are a bit unclear, looking forward to the author response

[Suggestions and some more questions]
- With reference to the sentence: " In addition to class-conditional generation, CC-LMs can be used as generative classifiers by applying Bayes rule to compute $P(c|x_{1:T})$, as is done by Keskar et al. (2019) for source attribution." Could you please add the inline formula, $p_θ(c|x) \approx p_θ(x|c)p(c)$, it saves one jump to the paper and makes the paper more readable :)
- Could you please elaborate on why GEDI would be 10k fold less computation as compared with a unidirectional classifier? Could you include a more detailed computational cost analysis?


review_title: Review

Meta-review: Reviewer #2 has written a nice summary of the paper which I quote below.

“The core idea is simple - which is a strength in my view - and does not require retraining the base language model, which could be important as language models become more expensive to train. However, the clarity and experiments in this paper fall short: the experimental setup has issues, the effect on perplexity is quite large but relegated to the Appendix, several claims are speculative and lacking corresponding experimental evidence, and it is unclear how the additional heuristics affect performance.

The method seems promising, but with the current experiments it is difficult to draw conclusions about how the method affects performance and which parts of it are necessary; given that this is an empirical paper, I would therefore not recommend acceptance in its current form.”

Key Strengths
+ Well-motivated problem of considerable interest 
+ A relatively straightforward Bayesian solution
+ Proposed solution is computationally efficient compared to other competing approaches.

The paper has been thoroughly reviewed by the reviewers and as a result numerous questions has surfaced. While the authors addressed most of the questions adequately, there are still many unanswered questions. They include:
- Readability issues highlighted by Reviewer #1
- Reviewer #1: “"how did you measure model confidence about the toxicity label"
- Reviewer #4: The perplexity gets much worse as the gedi training is introduced (i.e. Λ decreases), e.g. going from 25 to 45 on IMDb. This result is in the Appendix, and perplexity is never evaluated/reported in the other experiments.
- Reviewer #4: Crucially, the GEDI training does not appear to help over just re-weighting with the conditional LM ( vs. ). Could the authors comment on this result? How well does domain transfer work for less similar domains? How is perplexity affected for the models reported in Table 2?
- Reviewer #4: How small can the conditional LM be? Why was medium used instead of small? What if large was used? Does the conditional LM need to be a large-scale pretrained model (it would be nice to see a baseline of a simpler conditional LM)?
Several heuristics are used:  weighting, nucleus filtering, keeping tokens over a threshold, repetition penalty, and rescaling the logits to positive (used in only one experiment).
- How does each of these affect performance? There are no ablations, and given the small differences in some of the experiments it is unclear whether performance would actually be worse if we changed one of the heuristics. One outcome may be that the method only works for a careful balance of hyperparameters, which could be fine, but we don't have a sense of the variation.
- Reviewer #2: The output in Table 6 makes me doubt how the experiments are badly controlled. The outputs from positive and negative sentiment are totally different and almost random text, meaning that the content of the generators is not controlled properly.

Decision: Reject