Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on classifier guided controllable generation of text with autoregressive models. Following an observation that the stepwise classifiers tend to always be peaked, this work argues that it is an undesirable characteristic because of the loss of contrast between classifier's predictions on partial completions and fuller completions which might cause stepwise generation to be miscalibrated. The solution proposed for this problem is TOTY, which instead of training just 1 classifier at each step, trains a teacher classifier at each step and additionally, simultaneously trains a student classifier to match the teacher's logits which causes encourages teacher's predictions to be less peaked. The approach is tested on topic and sentiment control using lightweight GRU and GPT-2 based classifiers as students. It is empirically compared against other ways of smoothing/calibrating peaked distributions.
strength_and_weaknesses: Strengths:

-- The observed problem is important to tackle.

-- The proposed regularizer is reasonable and simple to implement.

-- The proposed approach exhibits a greater amount of smoothing than "label smoothing" and "focal loss" baselines, especially with GPT-2 based students.

-- The generated text seems to be on par or better than the vanilla approach with GPT-2 student classifier.

Weaknesses:

-- In the evaluation setup for topic control, only 4 topics instead of 7 topics as explored in the prior work are considered. Justification for omission of these topics and the potential impact on the empirical comparison should be provided.

-- This paper lacks human evaluation of the generated sentences which is important because the automatic metrics, while helpful and informative, are imperfect.

-- The paper shows that GPT-2 based student is crucial for the success of the approach. However, more analysis on why this is the case would make the paper stronger.

-- Some examples of generated samples would enhance the understanding the capability of the proposed approach.
clarity,_quality,_novelty_and_reproducibility: Please see the review above
summary_of_the_review: Overall, this paper identifies an important problem for classifier-guided autoregressive controlled generation and proposes a reasonable solution which is shown to beneficial empirically.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a simple regularization method named TOTY to address the biased probability distribution problem. The proposed method introduces a teacher and student model for the classifier and employs a regularization loss to improve the classifier. The proposed method achieves better performance on IMDB and AG News.
strength_and_weaknesses: The proposed method is simple and achieves better performance with the help of the regularization loss.
clarity,_quality,_novelty_and_reproducibility: 1. Although the paper shows that the proposed method smooths probability distributions of classifiers and achieves better performance. I still feel confused about the motivation. Why smoothing the probability distributions is helpful? Could you provide more insight into this?
2. It is also not clear to me why the standard GRU classifier has the biased probability distribution problem? It seems that a much heavier classifier GPT-2 also has the same problem. 
summary_of_the_review: The proposed method is simple and effective. But I am confused about the motivation, which improves the generation via smoothing the probability distributions.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose a simple regularization method named Teaching Others is Teaching Yourself (TOTY) to address the BPD problem. The proposed method is similar to self-distillation. Different from knowledge distillation in which knowledge only flows from the teacher to the student, in TOTY, the authors align the teacher and the student together, such that they can learn from each other. Experiments show that TOTY works well on smoothing the probability distribution of classifiers, and significantly improves the performance of classifier-guided language models.
strength_and_weaknesses: Strength
1. The proposed method can significantly improves the performance of classifier-guided language models, and get better performance than simple label smoothing and focal loss.
2. The paper is well-written and easy to read.

Weakness
1. The baselines are a little bit weak. It would be better to compare with PPLM (Dathathri et al., 2019) and FUDGE (Yang & Klein, 2021) discussed in the related works.
2. The proposed method is actually self-distillation or may be called co-training. It's a well-known method. Not surprised to see the smoothing ability on classifier. I think the title is talking about self-distillation loss as the regularization. There may be no need to name it "TEACHING OTHERS IS TEACHING YOURSELF REGULARIZATION".
clarity,_quality,_novelty_and_reproducibility: Why is the classifier still based on GRU, but not pre-trained language models, such as RoBERTa? 
summary_of_the_review: Overall, I think the baselines are not strong enough and the proposed method is very similar to self-distillation which has been widely used in many areas.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose a method called Teaching Others is Teaching Yourself (TOTY) regularization that smoothes the probability distribution that attribute discriminators produce when guiding generation. They show that augmenting classifier-guided language models with TOTY helps for sentiment control and topic control.
strength_and_weaknesses: Strengths:
- The method is relatively simple, although has a lot of potential confounders and moving parts (choice of architecture of the student/teacher models, hyperparameters with their training/fine-tuning etc.)

Weaknesses:
- The body of prior work on classifier-guided generation with language models in both a supervised and unsupervised perspective is vast and not compared with or discussed. For example: Plug and Play Autoencoders for Conditional Text Generation (Mai et al. 2020), DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts (Liu et al. 2021), and Extracting Latent Steering Vectors from Pretrained Language Models (Subramani et al. 2022) all work on sentiment control in either a supervised or unsupervised context. Quark (Lu et al. 2022) has really strong results on supervised sentiment control. There are numerous others, but these are just a few.
- It's really unclear whether the smoothing is significantly helpful. The results here seem mixed and for relatively weak generators.
- It'd be nice to see other attributes be controlled such as toxicity.


clarity,_quality,_novelty_and_reproducibility: Clarity: I think the paper is written decently and can be followed.

Quality/Novelty: I think the method and idea for calibration is important, but one that's been studied in many contexts and perhaps this one. I'm not an expert in the calibration literature to comment more, but this seems like a relevant and important problem to work on for many reasons. I'd say its reasonably novel.

Reproducibility: I'm not sure I could reproduce this exactly from the descriptions in the paper.
summary_of_the_review: I think the work is a good step towards a comprehensive study of whether the smoothing is helpful and how it affects different tiers of generators. The experimentation is quite limited and not contextualized with the broader literature/models on steering generation/controllable text generation/etc. Without a much stronger set of experiments, I cannot recommend acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This work aims to solve the difficult problem of controlled generation. The work points out that existing classifier-guided language models have issues in that their predicted probabilities are extreme (close to 0/1) and that this can make it hard to guide the generation. The work proposes the 'Teaching Others is Teaching Yourself' TOTY regularisation method, which trains both a teacher classifier and a student classifier which prevents the teacher's predictions from being so peaked.

Strengths:
* The work tackles an important problem.
* The approach is simple to implement and easy to understand.
* The approach performs better than label smoothing and focal loss baselines.
* The paper is well-written.

Weaknesses:
* The paper doesn't have a through discussion of related approaches and the baselines are relatively weak.
* The motivation for the approach is unclear such as why or whether the smoothing is significantly helpful.
* The approach is similar to existing self-distillation methods so there may be no need for a new name.
justification_for_why_not_higher_score: This work proposes a simple approach to fix an important problem. However, the motivation for the approach is unclear, particularly why smooth distributions help. The experiments are also quite limited without enough comparison or discussion of existing similar approaches for controlled text generation.
Decision: Reject