Teaching Others is Teaching Yourself Regularization For Controllable Language Models
Authors: Han Liu
Bingning Wang
Ting Yao
Haijin Liang
Jianjin Xu
Xiaolin Hu
Venue: Submitted to ICLR 2023 (2023)

Abstract: Large-scale pre-trained language models have achieved great success on natural language generation tasks. However, it is difficult to control the pre-trained language models to generate sentences with the expected attribute such as topic and sentiment. Recent efforts on controllable language generation employ an additional attribute classifier, which guides the generation of large-scale pre-trained language models, have been shown to be efficient in controllable language generation. These methods are named ''classifier-guided language models'' (CGLMs).
However, we find that the probabilities predicted by the attribute classifiers usually approaches 0 or 1, which make it hard to distinguish sentences with different matching degrees to the expected attribute. The problem is named \textit{the biased probability distribution} (BPD) problem. 
To address the problem, we investigate different methods for adjusting probability distribution and propose a ''Teaching Others is Teaching Yourself'' (TOTY) regularization method to smooth the probability distribution.
Experiments on sentiment control and topic control tasks show that CGLMs can get better performance with guiding classifiers trained with TOTY.

Under review as a conference paper at ICLR 2023
TEACHING OTHERS IS TEACHING YOURSELF REGU- LARIZATION FOR CONTROLLABLE LANGUAGE MOD- ELS
Anonymous authors Paper under double-bling review
ABSTRACT
Large-scale pre-trained language models have achieved great success natural language generation tasks. However, is difficult to control the pre-trained lan- guage models to generate sentences with the expected attribute such topic and sentiment. Recent efforts (Yang & Klein 2021 Krause et al., 2021 Dathathri let al., 2019) controllable language generation employ additional attribute classifier, which guides the generation of large-scale pre-trained language models, have been shown be efficient in controllable language generation. These methods are named maassifier-uuiee language models (CGLMs). However, we find that the probabilities predicted by the attribute classifiers usually approaches which makes it hard to distinguish sentences with different matching degrees to the expected attribute. The problem named the biased probability distribution (BPD) problem. To address the problem, we investigate different methods for ad- justing probability distribution and propose a Teaching Others Teaching Yourself (TOTY) regularization method to smooth the probability distribution. Experiments on sentiment control and topic control tasks show that CGLMs can get better performance with guiding classifiers trained with TOTY.
1,
INTRODUCTION
Recently, with the advances in large-scale pre-trained language model (PLM) (Radford al. 2017
2018; 2019 Brown et al.) 2020), great progress has been made natural language generation tasks. With billions or even trillions of parameters, and abundant unlabeled training data, PLMs can generate diverse and realistic sentences. Formally, autoregressive PLM models the probability distribution of text = {21, B2 ..., Fr} with the chain rule:
T
p(X)= = II 21 D2 24-1).
(I)
However, those models usually trained general purpose corpus and the sentences generated by those PLMs are usually inconsistent with task requirements. Therefore, Controllable Language Generation (CLG), which aims generate sentences that meet the requirements, has become more important in natural language generation. Controllable language generation attempts to model p(X) where desired attribute (e.g. topic, length and sentiment):
(X II [ (1=1,,))
To simplify the expression, use X1s denote the sequence {21,
has been found that using an attribute classifier to guide the generation of PLMs was an efficient approach to control the PLMs to generate sentences with expected attributes (Dathathri al. 2019 Krause et al.] 2021 Yang & Klein 2021). These methods are called classifier- classifier-guided language models (CGLMs). In CGLMs, the conditional probability each generation step is calculated by the Bayes Rule:
P(z:(X1s-1,a)) a) ((:

---

Under review as a conference paper at ICLR 2023
0.6
0.3
6.4
probabiity
0.4
probability
0.4
TOTY
0.2
Regularization
0.3
0.2
0.2
6.1
6.1
cn
(0:02)
19.... (0102)
19.88.9
(0.5..)
luaan (2.70.)
n.
1000011
1242
19655
10.5,0 104001
1es Lay
(a) GRU classifier
6.8
63-
0.1
6.4-
probabilty
TOTY Regularization
probabiltt
0.3
02-
6.1-
c.
10.011
192...1
99.38.4
(0.00.))
12) a.e)
20 sa
198011
24.4)
(04.5))
10.500
1960
1...0
18 a.a)
10
(b) GPT2 classifier
Figure I: The probability distribution of positive sentiment predicted by different classifiers trained on the IMDB dataset. (a) GRU classifiers trained without/with TOTY regularization. (b) GPT2 classifiers trained without/with TOTY regularization.
In the formula above, (1==1)) the unconditional probability of generating x; at step i, which is usually instantiated by a large-scale language model such as GPT. p(a)X1:)) is the attribute probability that the generation result contains the attribute a when starting with the X14- CGLMs usually use the output an attribute classifier, known as guiding classifier, to model p(a[X1:a) However, in experiments, we found that the probability distribution of p(a) X1.) predicted by guiding classifiers was usually very biased. To be specific, for most of sentences X1.1. the probability predicted by guiding classifiers either approaches 0 or approaches 1. We call this phenomenon the biased probability distribution (BPD) problem.
In classification tasks, the BPD problem has little influence since these tasks only need to pick out the class with the highest probability. In other words, the concrete value of probability does not have a direct impact on classification accuracy. However, in CGLMs, the attribute of many sentences could be ambiguous. Especially for autoregressive models, the tokens are generated one after another, so the classifiers need to predict the attribute probability of many incomplete sentences. Obviously to these sentences unreasonable. For example, for the following sentences, the probabilities of positive sentiment predicted by the GRU classifier trained on the IMDB dataset (Maas et al., 2011) for a), b), and c) was 89.5%, 98.5% 99.9%, respectively. However, only c) has a clear positive sentiment, while a) and b) do not have clear sentiment.
assigning a probability approaching
a) This tale takes place in
b) This tale takes place in the Namib Desert of Africa.
c) This impressive tale takes place in the Namib Desert of Africa.
A good guiding classifier for CGLMs should distinguish sentences with different matching degrees to the expected attribute, meaning that we should smooth the probability distribution predicted by the classifier. The existing methods (Wang et al., 2021 1) Gupta & Ramdas, 2021 Platt 2000 Weil
et al. 2022 Zadrozny & Elkan 2001 Szegedy et al. [016]] Müller et al. 2019) for adjusting the
probability distribution predicted by classifiers are usually used to address the mismatch between
model's 's confidence and its correctness. It will be shown that these methods does not significantly smooth the probability distribution.
2

---

Under review as a conference paper at ICLR 2023
In this work, we propose simple regularization method named Teaching Others is Teaching Yourself (TOTY) to address the BPD problem. In TOTY, we have two classifiers for the same classification task, one is named the "teacher" and the other is the "student". The teacher classifier learns from the ground truth and teaches the student classifier. Different from knowledge distillation in which knowledge only flows from the teacher to the student, in TOTY, we align the teacher and the student logether, such that they can learn from each other. Figure demonstrates the probability distribution of positive sentiment predicted by classifiers trained the IMDB dataset (Maas et al. 2011) without TOTY regularization and with TOTY regularization.
Experiments show that TOTY works well on smoothing the probability distribution of classifiers, and significantly improves the performance of CGLMs. Moreover, TOTY is an easily applicable method since it does not require complicated designing of model structure or training scheme.
2
RELATED WORK
2.1 MODELS FOR CONTROLLABLE LANGUAGE GENERATION
Controllable language models can be classified into two categories: class-conditional language models (CCLMs) and classifier-guided language models (CGLMs).
Given an expected attribute a, CCLM is the most straightforward approach controllable language generation which directly models the probability p( X [a) for generating sentence X. There are various methods to implement CCLM, such as training conditional generative models by concatenating the expected attribute to inputs (Fieler & Goldberg 2017 Kikuchi et al., 2016), training generative adversarial networks (Yu et al.) 2017), and training variational auto-encoders (Hu et al. 2017 7).
In recent years, with the development of large-scale pre-trained language models (Radford et al.
2017; 2018: 2019 Brown et al. 2020) and prompt methods (Jiang et al. 2020 Li & Liang 202 I Shin al.] 2020), great progress has been made in CCLMs. CTRL (Keskar al. 2019) builds large controllable language model trained on a large-scale corpus with 55 different control codes.
Following CTRL, COCON (Chan et al. 2020) proposes three self-supervised learning loss to enhance controllability. Moreover, ACB (Yu et al. (021]) tries to disentangle the irrelevant attributes in corpus and introduces wrefi--tuning in CCLM to avoid tuning a large number of parameters.
Since the training data for specific attributes is limited, CCLMs usually lead to corpus overfitting, meaning that the generated sentences of CCLMs usually highly resemble sentences the training corpus. For example, sentences sampled from CCLMs with sentiment control trained on IMDB (Mass et al. 201T). dataset of movie reviews for sentiment classification, usually mention words in movie reviews (e.g. movie, film, and character). However, the attribute we wish CCLMs to learn is the sentiment in IMDB rather than the context relevance of movie reviews.
CGLMs attempt to find the generation path that matches the expected attribute by using external attribute classifier. Instead of modeling p(Xla) directly, CGLMs sample X according to the Bayes Rule: p(X]a) α p(a]X] X X)))). PPLM (Dathathri et al., 2019) proposes iterative method that updates the hidden states of a language model by backpropagating gradients of an attribute model to maximize the likelihood of the expected attribute. Since PPLM requires iterations of forward and backward processes in inference, it is computationally intensive. GeDi (Krause et al.. 202T) uses smaller language models model p(a| FUDGE ( (Yang Klein (021) trains discriminator to distinguish whether the desired attribute will be true in the future and use the discriminator to model the performance of CGLMs is heavily influenced by the
p(a) X). However, as discussed in Section [ probability distribution of the external classifiers.
2.2 METHODS FOR ADJUSTING PROBABILITY DISTRIBUTION PREDICTED BY CLASSIFIERS
Most of previous methods for adjusting probability distribution predicted by classifiers are proposed to address the miscalibration problem- the mismatch between a mismatch between a model's 's confidence and its correctness. Miscalibration makes the predictions of deep neural networks hard to rely on and leads to the overconfidence problem. Some classical works address the problem by post-hoc methods, such as Platt Scaling (Platt) 2000) and Histogram Binning (Zadrozny & Elkan| 2001). Guo jet al.) (2017) systematically analysed the previous post-hoc methods. Following these works, many post-hoc methods were proposed (Kull et al. 2019 Rahimi et al. 2020 Gupta et al. 2020). Another
3

---

Under review as a conference paper at ICLR 2023
Ground Truth
Ground Truth
classification
classification loss
xa)
Pr (a(a))
TOTY regularization loss
fx
(x()
classifier
teacher classifier
student classifier
(b)
Figure 2: (a) The sketch of standard classifier training process. (h) The sketch of classifiers trained with TOTY regularization.
series of methods propose to adjusting the probability distribution by modifying the training process. Such label smoothing (Szegedy al. 2016 Müller al. 2019), focal loss (Lin et al. 2017
Mukhoti et al. 2020), and LogitNorm (Wei et al. 2022).
3 APPROACH
3.1 TOTY REGULARIZATION
In this Section introduce the TOTY regularization method. Since TOTY is not restricted train the classifiers of CGLMs, we introduce the TOTY regularization in a general classífication task. Different from classical training methods, TOTY contains separate classifiers: the "teacher" and the "student". Figure 2 TOTY respectively. The two classifiers has same input and output in training.
(a) and (b) demonstrates the training sketch of classical training method and
Let be the input of the classification task, and be the label of u. The TOTY regularization is defined as the euclidean distance between the logits output of the teacher and the student:
LTOTY (u, Or, 0g) = IIʃe (u, 0+) (u, 0s)11?
(4)
where (u, 0+) and (u, 0a) denote the logit output the teacher and the student respectively, and 0_ and 0g the parameters of the teacher and the student respectively. The student classifier only learns from LroTy:
Lstudent LTOTY (u, 0c,0s)
For the teacher classifier, besides the LTOTY. it also learns from the normal loss Lnozma (u, 0.) for classification in classical training methods. The forms of the loss for classification in different models are various. We do not modify their own losses. Overall, for the teacher,
Leeacher Linomnal (u, y/ 0c) Lrozx 0,, 0z)
(6)
3.2 THEORETICAL ANALYSIS TOTY REGULARIZATION
In this section, analyse how TOTY regularization smooths the probability distribution. To simplify the problem, we treat the classification task as a binary classification task. In this case, the logit output fo (u, O.) and (u, 0g) would be scalars, and Pa (vlu) is the sigmoid output of fu, 6.). And we use the cross entropy loss as the normal loss:
Tinormal In P- (ylu)
= = In
I exp (-(f(uu,0.)
4

---

Under review as a conference paper at ICLR 2023
For classifier trained with only the normal loss, ideally the training converges when
Onorra
a (11 - (=(l)))) 0. (u, 0t)
(8)
a
Since the partial derivative of (u, 0.) with respect to 0_ usually not zero, obviously Pe (yluu) approaches when it converges.
For the teacher classifier, the gradient of the TOTY loss can formulated as the follows:
aLreacer
@Lnormal
aLIOTY
00 +
00
00.
(9)
=(ff 0.) (u, - (1)-((1)p(() 05 (u, 0.)
(y(l)))
a
Ideally, with TOTY regularization, when the training of the teacher classifier converges, the gradient of Lteacher would approach zero, such that
(u, 0.)
(2ft(u, 0.) (u. -LL-P(y)lu))
(10)
00,
When the training converges, we get
Pt (ylyu) (w, fu,
(11)
Since the alignment of the teacher classifier and the student classifier would have fitting deviation = (u, 0.) fa (u, 0s), pr (ylu) would converge to a value less than 1 when 0. So, after convergence, with the TOTY regularization the value range of Pc. (y/u) becomes wider, which smooths probability distribution.
3.3 APPLYING TOTY TO CGLMS
In training classifiers for guiding CGLMs, the input is complete sentence X1.7, and the label is the attribute a of XI.r Since in inference the classifiers need to classify the incomplete sentence X1a, apply the TOTY regularization for every incomplete sentence X1:: where ranges from to T. So the TOTY regularization for classifiers of CGLMs is
LATATY = T Σ (Ife(X1a,,.) 0c) - Sa(X1s, 0,)112
(12)
As for the normal classification loss, we adopt the FUDGE Yang & Klein 2021) loss:
Iaoimal
Σ Inp (1)
(13)
4 EXPERIMENTS
To evaluate our approach, we trained TOTY with various pairs of classifiers as the teacher and the student, then tested the performance of CGLMs with these classifiers as the guiding classifier. We experimented on two controllable generation tasks: sentiment control and topic control.
4.1 DATASETS
IMDB: We used the IMDB dataset (Maas et al.] 2011) for sentiment control. It contains 50k samples of movie reviews labeled with sentiment tag (25k for positive sentiment and 25k for negative sentiment). We randomly chose Ik positive samples and IK negative samples as the test set of sentiment classification. We trained model on both positive attribute and negative attribute. In evaluation, we used the same 15 prefixes (See Section A) of sentiment control from the prior work (Dathathri al. 2019) and generated samples started with these prefixes.
AG News: We used the AG News dataset (Zhang et al. 2015) for topic control. The AG News dataset contains 120k samples of news articles. We randomly chose 500 samples from each class as the test

---

Under review as a conference paper at ICLR 2023
set of sentiment classification. The samples classified into one of the classes: worlds, sports, business, and science. We experimented on all of the classes. In evaluation, we used the same 20 prefixes (See Section[A] of topic control from the prior work (Dathathri et al. 2019) and generated samples from these prefixes.
4.2 GUIDING CLASSIFIERS
We adopted three different guiding classifiers in our experiments. GRU classifier is a randomly initialized GRU (Cho al. 2014) model trained on the appropriate task. GPT2 classifier a GPT2 model with a one-layer classification head on top. The GPT2 was initialized with the GPTg-large implemented in Huggingface Transformers" Since applying the GPT2 classifier as the guiding classifier of CGLMs would have a huge computational cost in inference, this classifier is only used as the student in our experiments. We will have more discussion on the computational cost of CGLMs guided by the GPT2 classifier in Section C GPT2-token classifier is the model introduced in Section C
4.3 IMPLEMENTATION DETAILS
For CGLMs, a pre-trained language model is needed for computing the unconditional generation probability p(r. X1x-1). We used the GPT2-large (Wolf et al. 2020) implemented in Huggingface Transformers as the pre-trained language model in all experiments.
In training, the AdamW optimizer (Loshchilov & Hutter, 2017) was adopted in all experiments. The learning rate of AdamW was set to 5 x 10-0. We used a batch size 64 and trained the models for 100 epochs. On average, on a Nvidia A100 Tensor Core GPU machine, each epoch took 7 minutes and minutes for sentiment control and topic control respectively.
In inference, we adopted an attribute-driven nucleus sampling as the decoding strategy in experi- ments. The attribute-driven nucleus sampling will be introduced in Section B
4.4 EVALUATION METRICS
In controllable language generation, need to evaluate both the attribute relevance and the linguistic quality of the generated sentences. In evaluation, for each model, we generated 50 samples per prefix and report the average score over the samples.
Attribute relevance (AR): For both tasks, we used classifiers based on BERT (Devlin et al. 2019) measure the consistency between the generated sentences and the desired attribute. The BERT was initialized with the BERT-large implemented in Huggingface Transformer: and fine-tuned on the task-specific corpus. Perplexity (PPL): Since we used GPT2-large as the language model of CGLMs, for the sake of fairness, we measured the linguistic quality of generated sentences by the average perplexity of a GPT2-XI
4.5 PROBABILITY DISTRIBUTION ANALYSIS
To further analyse the effect of TOTY, we plot the probability distribution predicted by GRU classifiers with different regularization methods trained on the IMDB dataset and the AG News dataset in Figures and
To simplify the expression, in the following we use TOTY(z) to denote the TOTY regularization with model æ as the student. For the probability of sentences in IMDB dataset, as can be seen from (a). the vanilla GRU classifier has very high peaks at interval [0.0,0.1) and [0.9.1]. According to Figure 3 (b), with label smoothing, the peaks of distribution shifted, but it did not smooth the
Figure (a),
distribution. The distance that the peaks shifted similar to the hyper-paramett a in label smoothing.
Figure 3 (c) shows that the focal loss did not significantly alter the probability distribution, since
https://uggingffac..o///g/ http:/accccccp//laaaaeeeee
"nttss//hugggnngface..o wttp:aaeecooppt-eaaaeeeeeee
https:/htgggngffccccooooeee wtwp:iinnnnnncnnnnttaaiieeee cw/been--arge-ucccee

---

Under review as a conference paper at ICLR 2023
03
pobeoiin
probusato
meam anse
Mivnna
smeering
probet4
Figure 3: The probability distributions predicted by GRU classifiers with different regularization methods trained the IMDB dataset. (a) Vanilla. (b) Label smoothing. Focal loss. (d) TOTY (student: GRU classifier). (e) TOTY (student: GPT2 classifier). TOTY (student: GPT2-token classifier).
PILO
Probodtty
8.20
8.20
037
810
085
0.30
8.30
probotiityy
10 1OTEYPO takers)
The probability distributions predicted by GRU classifiers with different regularization methods trained on the News dataset. The y-axis is truncated 0.3 to detailed display the distribution around secondary peak. (a) Vanilla. (b) Label smoothing. (c) Focal loss. (d) TOTY (student: GRU classifter). (e) TOTY (student: GPT2 classifier), (f) TOTY (student: GPT2-token classifier).
Figure 4:
the probability distribution predicted by models with focal loss still had very high peaks (0.0,0.1) and [0.9.1]. Figure 3 (d), (c), and (f) show the probability distributions of classifiers trained with TOTY by using different models as the student. is shown that with TOTY( the probability distribution predicted by the teacher GRU classifier was not well smoothed, while with TOTY(GPT2) and GPTT2G9T-))))))), the probability distribution got much smoother.
think this phenomenon can be explained with the theoretical analysis in Section 3.2 As we discussed in Section 3.2 the probability distribution will get smoother when the fitting deviation e gets larger. Compared with aligning models with different structures. the absolute value of is small when the teacher and the student had the same structure, such that the value range probability didn't get wider and the probability could not be well smoothed.
7

---

Under review as a conference paper at ICLR 2023
Table I:
Results on sentiment control topic control task. and Focal denotes the label smoothing and tocal loss method respectively. TOTY()) means that the student model is æ.
Negative Sentiment AR(
Positive Sentiment AR(%))
Topic
Classifier
Regularization
PPL.
PPL.
AR(%))
PPL.
vanilla
90.3
33.6
98.8
72.1
97.7
33.0
LS (a 0.1)
89.7
32.0
98.7
37.0
96.9
33.0
LS 0.2)
84.4
28.1
96.8
34.2
96.2
31.9
Focal (7 1.0)
86.7
34.6
98.3
57.0
96.6
31.5
GRU
Focal (? = 5.0)
86.1
35.1
96.5
67.6
91.4
32.3
TOTY (GRU)
88.0
32.0
98.3
47.0
97.8
31.0
TOTY (GPT2)
93.7
35.4
99.2
26.6
97.8
27.2
TOTY (GPT2-token)
93.9
26.5
99.3
36.7
98.5
29.1
vanilla
92.5
28.7
98.4
25.9
98.3
22.9
LS 0.1)
58.4
61.6
91.7
39.9
98.5
22.7
I.S = 0.2)
54.4
88.1
40.7
97.4
22.2
Focal (? 1.0)
60.1
36.2
90.4
27.2
25.5
26.8
GPT2-token
Focal (T = 5.0)
65.6
34.7
91.6
30.7
90.9
43.0
TOTY (GRU)
94.4
24.3
99.4
26.7
99.0
22.2
TOTY (GPT2)
24.0
98.8
18.4
98.5
16.9
TOTY (GPT2-token)
92.0
25.1
99.5
21.1
99.3
19.9
For the probability of sentences in the AG News dataset, since the dataset 4-classes classification task and we only compute the probability distribution of one of the classes, it's a normal phenomenon that there is high peak of distribution interval [0.0.0.1). So, in Figure 4 we only about the secondary peak in the distribution. From Figure we can draw similar conclusion with the conclusion of Figure 3 such as the label smoothing only shifted the secondary peak and the focal loss did not significantly alter the probability distribution. From Table [] control, the AR of CGLM with GRU classifier trained by TOTY(GPT2-token) is better than trained by TOTY (GRU) and of these models since compared to the Figure (d) and (e) the distribution in Figure ([) has lower secondary peak.
we find that on the topic
This result can be explained from the probability distribution
4.6 EXPERIMENTAL RESULTS
In our experiments, we used the vanilla classifier trained with the classification loss in equation 13 without any regularization method as the baseline model. To verify the effect of TOTY on CGLMs, we compared classifiers trained with TOTY with two previous regularization methods for adjusting
probability distribution: label smoothing (Szegedy et al., 2016: Müller al. 2019) and focal
loss (Lin et al. 2017). In experiments of label smoothing, we tested the syper-parameter a 0.1
and a 0.2 separately. And in experiments of focal loss, we tested the 5.0 separately.
y 1.0 and
Table shows the results on sentiment control and topic control. most of experiments, compared to the vanilla model, label smoothing and focal loss could not improve the performance AR or PPL, and in some cases (such as the GRU classifier with label smoothing a = 0.2 on negative sentiment control, and the GPT2-token classifier with focal loss 7 1.0 topic control) these regularization methods even seriously harmed the performance.
And compared with the bascline models, in most of experiments, TOTY significantly improved the attribute relevance with negative sentiment, positive sentiment and expected topic, meanwhile reduced the perplexity metric.
However, we also found the GRU classifier for sentiment control when the student was a GRU classifier had no improvement. Comparing the results with the probability distribution in Figure 3
we conclude that whether TOTY works well on smoothing the probability distribution determines the performance of CGLMs with the classifiers the guiding classifier.
8

---

Under review as a conference paper at ICLR 2023
Table 2:
The stepwise probabilities assessed by different classifiers. data each step stands for probability of the desired attribute for the sentence from the beginning to the current step.
Positive Sentiment
The
food
is
awful
but
the
service
is
nice
GRU-vanilla
0.50
0.43
0.41
0.02
0.05
0.08
0.09
0.10
0,02
GRU-TOTY
0.54
0.47
0.49
0.01
0.01
0.01
0.01
0.01
0.01
GRU-T)
0.51
0.48
0.49
0.22
0.33
0.34
0.36
0.46
GRU-TOTY(GPT2--okn))
0.51
0.50
0.52
0.19
0.29
0.35
0.34
0.37
0.44
Sports Topic
The
Italian
athlete
attended
the
World
War
in
1940
GRU-vanilla
0.22
0.22
0.94
0,97
0.99
1.00
1.00
1.00
GGU-
0.24
0.30
0.80
0.88
0.89
0.89
0.86
0.87
GRU-TOTY(GPT)
0.24
0.24
0.63
0.64
0.67
0.68
0.58
0.61
0.54
GIRU-TOTY(GPT2-tooe))
0.23
0.29
0.78
0.72
0.73
0.77
0.62
0.55
0.49
4.7 STEPWISE ANALYSIS OF CLASSIFIERS WITH TOTY
Table 2 demonstrates the stepwise probabilities of positive attributes and sports topics. From the stepwise probabilities, we found that the stepwise probabilities assessed by GRU classifiers with TOTY(GPT2) and TOTY in the sentences. For "The food is awful but the service is nice". the sentiment in the sentence converts from negative positive, and the stepwise probabilities of positive attribute decreased firstly and then increased gradually. For "The Italian athlete attended the World War 1940", the stepwise probabilities decreased after the word "War", which is reasonable. However, the vanilla classifier and the GRU classifier with TOTY(GRU) performed poorly on both the sentences. They classified "The food is awful but the service nice" a very negative sentence, and classified "The Italian athlete attended the World War in 1940" as a sentence strongly related to the sports topic. To conclude, the stepwise probability models with biased probability distribution usually could not adapt the attribute conversion of the sentences, while the models with smoothed probability distribution performs well on attribute conversion.
flowed along with the conversion of the sentiment and topic
5 CONCLUSION
In this work, we identify the BPD problem which harms the performance of CGLMs. To address the BPD problem, we introduce TOTY, an easily applicable and efficient regularization method for smoothing the probability distribution of classifiers. By aligning the teacher classifier and the student classifier together, TOTY smooths the probability distribution of classifiers. Experiments on sentiment control task and topic control task show that TOTY can significantly improve the performance of CGLMs. We also think that TOTY might be useful regularization method for other tasks.
REFERENCES
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Ben- jamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, F. Balcan, and Lin (eds.), Advances Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL ht https:::pproeeeiing....t. cc/ [2//a//8//2220//L111144466666662 http://c......a....22.2.aaaa2223
Alvin Chan, Yew-Soon Ong. Bill Pung, Aston Zhang, and Jie Fu. Cocon: A self-supervised approach for controlled text generation. arXiv preprint arXiv:2006.03535, 2020.

---

Under review as a conference paper at ICLR 2023
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using mnn encoder-decoder for statistical machine translation. arXiv preprint MeXiv:14400 1078, 2014.
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. arXiv preprint arXiv: 1912.02164 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pp. 4171- 4186. Association for Computational Linguistics, 2019. doi: 10.18653/v1/n19 1/1991133/ URL
https::/abi.org// htt.:/a..................///111.
Jessica Ficler and Yoav Goldberg. Controlling linguistic style aspects in neural language genera- tion. In Proceedings of the Workshop on Stylistic Variation, pp. 94-104, Copenhagen. Denmark, September 2017. Association for Computational Linguistics. doi: 10.18633/1/117-49122. URL
https::/aclanthol.oo w W17 0rg//17-4912
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference machine learning, pp. 1321-1330. PMLR, 2017.
Chirag Gupta and Aaditya K Ramdas. Top-label calibration and satticlass-t-bbinngy reductions.
arXiv preprint arXiv:2107.08355 2021.
Kartik Gupta, Amir Rahimi, Thalaiyasingam Ajanthan, Thomas Mensink, Cristian Sminchisescu, and Richard Hartley, Calibration of neural networks using splines. arXiv preprint arXiv:2006.12800, 2020.
Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. The curious case of neural text degenera- tion. CoRR,
2019. URL /uRL:ttt:://aariv.o////aa//////
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric Xing. Toward controlled generation of text. In Proceedings of the 34th International Conference Machine Learning Volume 70, ICML'17, 17, pp. 1587-1596. JMLR.org, 2017. 2017.
Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How can know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438, 2020.
Nitish Shirish Keskar, Bryan McCann, Lav Varshney, Caiming Xiong, and Richard Socher. CTRL A Conditional Transformer Language Model for Controllable Generation. arXiv preprint 2019.
Yuta Kikuchi, Graham Neubig, Ryohei Sasano, Hiroya Takamura, and Manabu Okumura. Con- trolling output length in neural encoder-decoders. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1328-1333 Austin, Texas, Novem- ber 2016. Association for Computational Linguistics. doi: 10. 10.18653/v1/D11-11444 URL
Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. GeDi: Generative discriminator guided sequence generation.
In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 4952, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 18.18653/v 1/2021. findings-emnlp.424. URL https::///// hittps:/accaanthology. orgp://
Findings emn 1p. 424
Meelis Kull. Miquel Perello Nieto. Markus Kängsepp, Telmo Silva Filho, Hao Song, and Peter Flach. Beyond temperature scaling: Obtaining call-calibrated multi-class probabilities with dirichlet calibration. Advances in neural information processing systems, 32, 2019.
10

---

Under review as a conference paper at ICLR 2023
Xiang Lisa Li and Percy Liang. Prefix- Prefix-tuning: Optimizing continuous prompts for generation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers), abs/2101. 100100, 2021.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection, In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017.
Hya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 111iv1711.05100,, 2017.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150. Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL https 7/aclantho ogy. www.yoogg/P1--1-
Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, and Puneet Dokania. Calibrating deep neural networks using focal loss. Advances in Neural Information Processing Systems, 13:15288-15299 2020.
Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? Advances in neural information processing systems, 32, 2019.
John Platt. Probabilistic outputs for support vector machines and comparisons to regularized likeli- hood methods. Adv. Large Margin Classif., 10, 06 2000.
Alec Radford, Rafal Jozefowicz, and Ilya Sutskever. Learning to generate reviews and discovering sentiment. arXiv preprint arXiv: 704.0 11700001444. 01444, 2017.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. 2018
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188):9 2019.
Amir Rahimi, Amirreza Shaban, Ching-An Cheng, Richard Hartley, and Byron Boots. Intra order-preserving functions for calibration of multi-class neural networks. Advances in Neural Information Processing Systems, 33:13456-13467, 2020.
Taylor Shin, Yasaman Razeghi, Robert IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. pp. 422244222444 4235, 01 2020. doi: 10. 1/20200cmmmppmmiin.3....
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe. Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2818-2822 2016. doi: 10. 10.1100//2P...016....
Deng-Bao Wang, Lei Feng, and Min-Ling Zhang. Rethinking calibration of deep neural networks: Do not be afraid of overconfidence. Advances in Neural Information Processing Systems, 34: 11809-11220, 2021.
Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng, Bo An, and Yixuan Li. Mitigating neural network overconfidence with logit normalization. arXiv preprint arXiv:2205.092: 093110 2022.
Thomas Wolf, Julien Chaumond, Lysandre Debut, Victor Sanh, Clement Delangue, Anthony Moi, Pierric Cistac, Morgan Funtowicz, Joe Davison, Sam Shleifer, et Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45, 2020.
Kevin Yang and Dan Klein. Fudge: Controlled text generation with future discriminators. arXiv preprint arXiv:2104.05218. 2021.
11

---

Under review as a conference paper at ICLR 2023
Dian Yu, Zhou Yu, and Kenji Sagae. Attribute alignment: Controlling text generation from pre-trainne trained language models. arXiv preprint arXiv:2103. 11000, 2021.
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seggan: Sequence generative adversarial nets with policy gradient. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, A77I'17, pp. 2852-2858, AAAI Press, 2017.
Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In Ieml, volume I. pp. 609-616. Citeseer, 2001.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas- sification. Proceedings of the 28th International Conference on Neural Information Processing Systems Volume 1, NIPS 15, pp. 649-66999999 657, Cambridge, MA, USA, 2015. MIT Press.
A PREFIXES USED IN EXPERIMENTS FOR SENTENCE GENERATION
We used the same 15 prefixes for sentiment control and 20 prefixes for topic control from PPLM.
"Once upon a time", "The book". "The chicken", "The city", "The country", "The horse", "The lake", "The last time", "The movic", "The painting". "The pizza", "The potato", "The president of the country", "The road", "The year is 1910.".
Prefixes for sentiment control:
Prefixes for topic control: "In summary,", "This essay discusses", "Views on", "The connection", "Foundational to this is", "To review,", "In brief,", "An illustration of", "Furthermore,", "The central theme", "To conclude,", "The key aspect", "Prior to this", "Emphhsisee are", "To summarise,", "The relationship". "More importantly"", "It has been shown", "The issue focused on", "In this essay".
B ATTRIBUTE-DRIVE NUCLEUS SAMPLING
As we discussed Section T CGLMs use the Bayes Rule in equation to calculate the conditional probability. However, in inference, is to be generated at step in inference. To get the probability distribution 2,, we need to calculate (xx1)) for all token w in the vocabulary, such that we need to calculate p(a ( w) for all token w in the vocabulary.
Following previous CGLM works (Dathathri et al. 2019; Krause et al. 2021), we applies weighted decoding in model by introducing non-negative hyper-parameteed X. The conditioned probability with weighted decoding is
(14) a apper-parameter to balance controllability and fluency of generation. When À is 0. the conditioned probability degrades into the unconditioned probability. In all of our experiments, À was set to 5.
p( a) x P(a[X1.-1, "))pp)(pp (14-1)- p(
Moreover, inspired by the decoding strategy in the original paper of GeDi (Krause et al.) 202 we design an attribute-driven nucleus sampling in inference to improve the controllability and the linguistic quality. The attribute-driven nucleus sampling contains two filters. The first filter is standard nucleus sampling filter (Holtzman et al., 2019) acting on the unconditional probability distribution p(w) [[w X1:-1) to maintain the linguistic quality of generation. We define Vi8 as the set of k tokens w with the highest (w)Xi=-1).. With filter probability pl. k((1)) defined as k((p)) = argmin(
Σ X(a pz).
(15)
&
WEV,
With the first filter, we get the k(p1) tokens with the highest p(w) (I)-)) which form a set Vi(py Inspired by the decoding strategy of GeDi (Krause et al.) 2021), the second filter, named "attribute- driven filter", acts on the conditional probability distribution p(w) (| a), aiming at making the generated sentences well conditioned on the attribute a. We define Um as the set of m tokens w in Vi())) with the highest p(a) X 1:i-1. w). With filter probability P2. mmp2)) is defined as
(p))) argmin( Σ
P2).
(16)
WEU
Finally we sample from the set Un(p2)) Experimentally, set P =0.9, P2 0.3 in all experiments.
12

---

Under review as a conference paper at ICLR 2023
C GPT2-TOKEN CLASSIFIER
As we discussed Section B we need to calculate p(a| 1 w) for all token to the vocabulary. For the GPT2 classifier, we need extract the features of the sentence { w) for all token w in the vocabulary. Usually the vocabulary size of GPT2 is more than 50k, and on a single NVIDIA A100 Tensor Core GPU machine, extracting the feature of sentence by GPT2-large takes about 0.03s. So it is huge cost for extracting the features of sentence {X1::-1. w) for all token w in the vocabulary. To reduce the computational cost, we introduce the GPT2-token classifier.
Let hi...., hr denote the last hidden state of GPT2 model with X1. as the input. For each the dimension of is dn-
Let Ewr denote the embedding of token w. The dimension of ew is de. The GPT2-token classifier M receives h and en as the imput. Different from the GPT2 classifier, GPT2-token classifier do not need to extract the feature of (X1xi w) by GPT2 for all token w in vocabulary, which significantly reduces the computational cost. The structure of the GPT2-token classifier is gated neural network:
-i-1 W bg.
= o cW + W2 + by),
2; (Wizew + W22fii-1 1 bz);
(17)
T4 = Benh(Winew + bnn + ri (W + b2n)),
0, ReLu((1 2,)m4 + 2/91-1);
M(h,-1,ew) = Woos:
where W, is d2 x d, matrix, a WirrWi..Wi,, are d, x de matrices, W..,W.,WW. are d, x dn matrices, by by. by. bin and 62n are d,, dimensional vectors, Wo is dy dimensional row vector and Tanh is the hyperbolic tangent function.
13


Reviews:
[{"review_id": "dWC9j-76H-f", "reviewer": "Reviewer_V4K8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on classifier guided controllable generation of text with autoregressive models. Following an observation that the stepwise classifiers tend to always be peaked, this work argues that it is an undesirable characteristic because of the loss of contrast between classifier's predictions on partial completions and fuller completions which might cause stepwise generation to be miscalibrated. The solution proposed for this problem is TOTY, which instead of training just 1 classifier at each step, trains a teacher classifier at each step and additionally, simultaneously trains a student classifier to match the teacher's logits which causes encourages teacher's predictions to be less peaked. The approach is tested on topic and sentiment control using lightweight GRU and GPT-2 based classifiers as students. It is empirically compared against other ways of smoothing/calibrating peaked distributions.", "summary_of_the_review": "Overall, this paper identifies an important problem for classifier-guided autoregressive controlled generation and proposes a reasonable solution which is shown to beneficial empirically.", "strength_and_weaknesses": "Strengths:\n\n-- The observed problem is important to tackle.\n\n-- The proposed regularizer is reasonable and simple to implement.\n\n-- The proposed approach exhibits a greater amount of smoothing than \"label smoothing\" and \"focal loss\" baselines, especially with GPT-2 based students.\n\n-- The generated text seems to be on par or better than the vanilla approach with GPT-2 student classifier.\n\nWeaknesses:\n\n-- In the evaluation setup for topic control, only 4 topics instead of 7 topics as explored in the prior work are considered. Justification for omission of these topics and the potential impact on the empirical comparison should be provided.\n\n-- This paper lacks human evaluation of the generated sentences which is important because the automatic metrics, while helpful and informative, are imperfect.\n\n-- The paper shows that GPT-2 based student is crucial for the success of the approach. However, more analysis on why this is the case would make the paper stronger.\n\n-- Some examples of generated samples would enhance the understanding the capability of the proposed approach.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NQC1fAgOOo6", "reviewer": "Reviewer_absA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a simple regularization method named TOTY to address the biased probability distribution problem. The proposed method introduces a teacher and student model for the classifier and employs a regularization loss to improve the classifier. The proposed method achieves better performance on IMDB and AG News.", "summary_of_the_review": "The proposed method is simple and effective. But I am confused about the motivation, which improves the generation via smoothing the probability distributions.", "strength_and_weaknesses": "The proposed method is simple and achieves better performance with the help of the regularization loss.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "G6zPxk2HrG0", "reviewer": "Reviewer_eggk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a simple regularization method named Teaching Others is Teaching Yourself (TOTY) to address the BPD problem. The proposed method is similar to self-distillation. Different from knowledge distillation in which knowledge only flows from the teacher to the student, in TOTY, the authors align the teacher and the student together, such that they can learn from each other. Experiments show that TOTY works well on smoothing the probability distribution of classifiers, and significantly improves the performance of classifier-guided language models.", "summary_of_the_review": "Overall, I think the baselines are not strong enough and the proposed method is very similar to self-distillation which has been widely used in many areas.", "strength_and_weaknesses": "Strength\n1. The proposed method can significantly improves the performance of classifier-guided language models, and get better performance than simple label smoothing and focal loss.\n2. The paper is well-written and easy to read.\n\nWeakness\n1. The baselines are a little bit weak. It would be better to compare with PPLM (Dathathri et al., 2019) and FUDGE (Yang & Klein, 2021) discussed in the related works.\n2. The proposed method is actually self-distillation or may be called co-training. It's a well-known method. Not surprised to see the smoothing ability on classifier. I think the title is talking about self-distillation loss as the regularization. There may be no need to name it \"TEACHING OTHERS IS TEACHING YOURSELF REGULARIZATION\".", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "EgsllAcDmzb", "reviewer": "Reviewer_jJwL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a method called Teaching Others is Teaching Yourself (TOTY) regularization that smoothes the probability distribution that attribute discriminators produce when guiding generation. They show that augmenting classifier-guided language models with TOTY helps for sentiment control and topic control.", "summary_of_the_review": "I think the work is a good step towards a comprehensive study of whether the smoothing is helpful and how it affects different tiers of generators. The experimentation is quite limited and not contextualized with the broader literature/models on steering generation/controllable text generation/etc. Without a much stronger set of experiments, I cannot recommend acceptance.", "strength_and_weaknesses": "Strengths:\n- The method is relatively simple, although has a lot of potential confounders and moving parts (choice of architecture of the student/teacher models, hyperparameters with their training/fine-tuning etc.)\n\nWeaknesses:\n- The body of prior work on classifier-guided generation with language models in both a supervised and unsupervised perspective is vast and not compared with or discussed. For example: Plug and Play Autoencoders for Conditional Text Generation (Mai et al. 2020), DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts (Liu et al. 2021), and Extracting Latent Steering Vectors from Pretrained Language Models (Subramani et al. 2022) all work on sentiment control in either a supervised or unsupervised context. Quark (Lu et al. 2022) has really strong results on supervised sentiment control. There are numerous others, but these are just a few.\n- It's really unclear whether the smoothing is significantly helpful. The results here seem mixed and for relatively weak generators.\n- It'd be nice to see other attributes be controlled such as toxicity.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject