Air-Decoding: Attribute Distribution Reconstruction for Decoding-Time Controllable Text Generation
Authors: Tianqi Zhong
Quan Wang
Jingxuan Han
Yongdong Zhang
Zhendong Mao
Venue: EMNLP 2023 Main (2023)
Keywords: controllable
text generation
decoding-time
TL;DR: A decoding-time controllable text generation method via attribute distribution reconstruction.

Abstract: Controllable text generation (CTG) aims to generate text with desired attributes, and decoding-time-based methods have shown promising performance on this task. However, in this paper, we identify the phenomenon of Attribute Collapse for the first time. It causes the fluency of generated text to rapidly decrease when the control strength exceeds a critical value, rendering the text completely unusable. This limitation hinders the effectiveness of decoding methods in achieving high levels of controllability. To address this problem, we propose a novel lightweight decoding framework named Air-Decoding. Its main idea is reconstructing the attribute distributions to balance the weights between attribute words and non-attribute words to generate more fluent text. Specifically, we train prefixes by prefix-tuning to obtain attribute distributions. Then we design a novel attribute distribution reconstruction method to balance the obtained distributions and use the reconstructed distributions to guide language models for generation, effectively avoiding the issue of Attribute Collapse. Experiments on multiple CTG tasks prove that our method achieves a new state-of-the-art control performance.

Air-Decoding: Attribute Distribution Reconstruction for Decoding-Time Controllable Text Generation
Tianqi Zhong', Quan Wang", Jingxuan Han', Yongdong Zhang', Zhendong Mao UUiiveriity of Science and Technology of China
MOE Key Laboratory of Trustworthy Distributed Computing and Service,
Beijing University of Posts and Telecommunications (ztg602656897, hjx999222yémaill ustc. edu. cn wangquan@bupt.cm edw.:euut..uu.....oo.... en, {zhyd73, zdmao edu. cn
Abstract
Ge3
100
An-3acoding
307
52100
30000
Controllable text generation (CTG) aims to gen- erate text with desired attributes, and decoding- time-based methods have shown promising per- formance on this task. However, in this pa- per, we identify the phenomenon of Attribute Collapse for the first time. It causes the flu- ency of generated text to rapidly decrease when the control strength exceeds critical value, rendering the text completely unusable. This limitation hinders the effectiveness of decod- ing methods in achieving high levels of con- trollability. To address this problem, we pro- pose a novel lightweight decoding framework named Air- Sir-Decddigg Its main idea is recon- structing the autribute distributions to balance the weights between attribute words and non- attribute words to generate more fluent text. Specifically, we train prefixes by prefix-tuning obtain attribute distributions. Then we de- sign a novel attribute distribution reconstruc- tion method balance the obtained distribu- tions and use the reconstructed distributions to guide language models for generation, effec- tively avoiding the issue of Attribute Collapse. Experiments multiple CTG tasks prove that our method achieves a new state-of-the the-art con- trol performance'.
120
20
3.0
10.0
22.5
Cnntrel strenge
7.5 32.5
Figure The phenomenon of Attribute Collapse (left) and the elimination of Attribute Collapse (right), which are both based on the sentiment control task. The per- plexity is inversely proportional metric fluency and the accuracy is directly proportional metric to attribute relevance.
Current CTG methods can be divided into three categories (Zhang et al., 2022). The first cate- gory achieves attribute controllability by retrain- ing the whole parameters of CLMs (Keskar et al.. 2019; Wang et al., 2021). These methods ob- tain impressive control effects, but the scale of CLMs increases, the computational cost becomes excessively large. The second category fine-tunes prefixes or prompts to control CLMs in generat- ing texts with specified attributes (Yu et al., 2021; Zhang and Song, 2022; Qian et al., 2022). These methods have low computational cost and fast gen- eration speed, but the control prefixes or control prompts absorb the features of the training corpus and are prone to overfitting, resulting in poor gen- erality of the generated text. The third category is the decoding-time approach, which guides the model to generate the desired attribute text by ad- justing the output probability distribution of the model during the decoding stage. For example, most methods use an attribute distribution obtained by well-trained classifier to control the CLMs' output distribution and add an exponential term the attribute distribution as a control strength (Liu et al., 2021; Yang and Klein, 2021; Krause et al., 2021). Due to no direct fine-tuning of the models used for the generation, these methods exhibit re-
1 Introduction
Controllable text generation (CTG) aims to pro- duce texts with specific attributes(e.gg sentiment, topic, non-toxicity). In this field, mainstream meth- ods using Transformer-based Casual Language Models (CLMs) like GPT-2 (Radford et al., 2019) and GPT-3 (Brown et al., 2020) have achieved fairly good attribute control. Even so, it remains challenging to control the generated text to simul- taneously satisfy certain attributes and maintain reasonable coherence (Carlsson et al., 2022).
*Corresponding author: Zhendong Mao.
'The code implementation is available at https:// gi tithu. com/R1 247/Air-Decoding

---

markable generalization performance and are easy achieve high control effectiveness when increas- ing control strength, which hardly achievable by the first two methods.
framework, we designed an attribute distribution reconstruction method obtain more balanced attribute distribution, which results in better CTG performance.
However, decoding-time approaches suffer from Attribute Collapse, which refers to the phe- nomenon that when the control strength increases certain eritical value, the fluency of the gen- erated text will rapidly decrease. As illustrated in Figure I(left), where the fluency of generated texts would severely deteriorate as the control strength increases. We found the reason lies in the imbal- anced attribute distribution, where the weights of attribute words are significantly higher than those of non-attribute words. As result, when given high control strength, the final output distribution will deviate from the original language model's is out- put distribution and excessively favor the attribute distribution. This leads to the generated tokens favoring specified attributes but ignoring basic se- mantics and syntax, resulting in generated text with high attribute accuracy but compromised fluency.
We introduce prefix-tuning to obtain attributc classifiers, which enables us to achieve the tribute distribution with low-resource approach.
We conduct experiments on three CTG tasks: sen- timent control, topic control, and text detoxifi- cation. The results prove that our Air-Decoding method achieves new SOTA control perfor- mance. In particular, while achieving an im- provement attribute accuracy, we also attain a substantial enhancement in fluency.
Related Work
In recent years, numerous controllable text gen- eration methods based on pre-trained language models have emerged (Yang et al., 2022; Madotto et al., 2020; Ziegler et al., 2019). These methods can be roughly categorized into three types: Re- train/Refact,
We propose Air-Decoding, novel lightweight decoding framework to address the issue of At- tribute Collapse in traditional decoding-time ap- proaches. Our method could obtain more bal- Retrain, Refactor. Retraining and refactoring aim to retrain a conditional language model from scratch or change its architecture. Keskar et al. (2019) use 55 attribute control codes to finetune 1.63
hetph//Proott-tuunngg and Decoding-
time approach (Zhang et al., 2022).
anced attribute distribution in low-resource man- ner, which helps us achieve better generation re- sults while reducing resource consumption. Con- cretely, We first use a lightweight approach to ob- tain corresponding conditional language models for attribute clas-
transformer to control gen- eration. Chan al. (2020) added control block to GPT-2's architecture and retrained the model using self-supervised learning methods with spe- cial control codes. Zhang et al. (2020) introduce POINTER, a modified Transformer model for lexi- cally constrained text generation. Prefix/roomt-
each attribute, which serves sifier to generate the original word-level attribute distribution. Then, we propose an Attribute Dis- tribution Reconstruction method to reconstruct the original attribute distribution, which avoids ex- aggerating the weights of attribute words and ex- cessively diminishing the weight of non-attribute words, resulting in a more balanced attribute distri- bution. As shown in Figure ((right), after applying our Air-Decodie framework, the perplexity could remain within stable range as the control strength increases, proving that generated texts based on the reconstructed attribute distribution ensure high attribute accuracy and good semantic and syntactic structures.
With the expansion of pre-trained language models, there is increasing interest in lightweight fine-tuning methods such as prefix-tuning (Li and Liang, 2021) and prompt- tuning (Lester al., 2021). Qian et al. (2022) introduce contrastive prefixes that consider inter- prefix relationships during multiple prefix training. Zhang and Song (2022) add an attribute discrimi- nator to control -rrompt training using the unlikeli- hood method, but this increases training time signif- icantly. Gu al. (2022b) train auto-encoder to encode training set samples into prefixes guide GPT-2, achieving impressive controllability but suf- fering from poor diversity and generality. Decoding-time The decoding time method is another approach to achieve CTG and its main idea
Our main contributions are as follows:
We first identify the phenomenon of Attribute Collapse in the CTG tasks and develop novel lightweight decoding framework named Air- Decoding specifically to address this issue. In the

---

Pos Prefix
GPT-2
(frozen)
Attribute Distribution Reconstruction
-
Poubive dittibution
Weconstructed Poutive distriution.
Neg Prefix
dill
GPT-2 (frozen)
The
painting
Reconstructed Negative distibbttio
Nogative disufbution
PC-LL
of
the
GPT-2 (frozen)
Reconstructed Posttre distribrtion
looks
bad ...
Recor reconsttucctdd dittbbcn
Figure 2: An illustration of Air-Decoding Given the prompt "The painting", the pos PC-LM, neg PC-LM, and raw GPT-2 generate positive, negative, and raw distribution. Then, we use reconstruction method make these attribute distributions more balanced. Finally, we use the reconstructed attribute distributions as weights for the raw distribution to generate the next token.
is adjust the probability distribution of the lan- guage models' output during the decoding stage. PPLM (Dathathri et al., 2019) utilizes the results of pre-trained classifier through backpropagation to update the model's hidden states, thereby steer- ing the hidden states toward generating text with the specified attributes. Yang and Klein (2021) di- rectly use a classifier to calculate the probability of generating the next token in the sequence. Liu al. (2021) uses expert and anti-expert to guide GPT-2 generating but this method may not be well-suited for multi-category citribute-controlle generation. GeDi (Krause et al., 2021) employs a class-conditional language model(CC-LM)) as gen- erative discriminators to direct the text generation using a base GPT-2 model. Lin and Riedl (2021) enhance GeDi for controllable story generation by incorporating planning module in their Plug- and-Blend approach. Gu al. (2022a) propose lightweight regulator adjust control strength different decoding positions but works not well under eigh-intensity control conditions.
distribution reconstruction method to balance the obtained distributions, which will be used guide frozen GPT-2. The overall framework is illus- trated in Figure 2.
3.1 Preliminary
Controllable text generation aims to guide an au- toregressive model G (i.e., GPT2) to generate texts with desired attribute. In this paper, mainly in- troduce our method through the sentiment control task that includes positive and negative attributes for ease of understanding. However, our method exhibits generalizability and can be applied to other CTG tasks. Concretely, we usually give prompt #1:T-1 = {21, .I-11 } (such as "The paint- ing" in Figure 2) and desired attribute (e.g. positive), and ask the model generate the con- tinuations Jy.N { , 11,'1"""" ... .xN ), ensuring the whole text ø1,N satisfies the given attribute. It can be formulated as:
P(ZT.NEITT-1.0) -II P(x((ee)) (1)
Thus, we need to model P( Diee, a) to achieve Based on Bayes factorization,
3 Methodology
P P( P (# a) can be transformed to Eq.2 (Yang and Klein, 2021) (detailed in Appendix A.1).
We propose Air-Decoding, a decoding-timed tinese--teeeeeeeee based CTG framework. Its main idea is reconstructing the attribute distributions to make the weights of attribute words and non-attribute words more bal- anced generate more fluent text. Specifically,, propose Peefi-Conditional LM (PC-LM) which employs prefix-tuming to enable GPT-2 to acquire attribute distributions. Then, we design an attribute
((zl(ae) P(a)zia)) P(a((er)) (2 (2)
where the second term is the next token probabil- ity distribution modeled by & while the first term ((aazx:) is a binary classifier B for attribute a

---

given the text o and w is control strength, which is an additional term added the equation. A high w will bias generation more strongly towards the desired class.
attributes and we denote them as attribute distri- butions. Based on the two attribute distributions generated by pos PC-LM and neg PC-LM, we can model P(alz1+) as Eq.5.
In decoding-time CTG approaches, we need to model P(a]T1a) to obtain P( a). In order to compute ((a more efficiently, the
- P 105l P(a = En'c(a.a) U2= Pa.o, (2, (tylazs Ho.,)
(5)
first term P(a(21:e) in Eq.2 can be achieved by two class-conditional models Po. (aa a) and Pos (2y Rele a) by Bayes Rule which is formulated as Eq.3 (Krause et al., 2021).
where the class priors P(a) and P(a) are omitted as we use the same amount of training data to train the pos PC-LM and neg PC-LM.
3.3 Attribute Distribution Reconstruction
Pr (xgl (xgl P(almaa) = Ed'e(nay " P( Poo (sylies)aa))
(3)
However, the attribute distribution obtained directly from the PC-LM has strong tendency towards its attribute, which means that the weights of the at- tribute words in the attribute distribution will be much larger than those of non-attribute words. This leads to a strong attrihute tendency in the final dis- tribution of P(a[æ1=). For example, when gen- erating positive sentiment text given the prompt "The painting" and in the generation of the first token (i.c. t = T"), the weight of "good" and "of" in the positive distribution Px. (are (aree Ha,) is 0.1 and 0.03, while in the negative distribution Px.0a (rel Hog) is 0.01 and 0.05. Hence, ac- cording to Eq.5, the weight of "good" in the fi- nal distribution P(a. xιa) is O. 1..10.. 1+0001)=0,999, while the weight of "of" is 0.03/0.03+0.05)-0..77 As a result, the weight of "good" is significantly higher than that of "of". This may cause "good" to be generated the final token, which is clearly in- consistent with the previous tokens "The painting".
where represents the positive attribute, ā rep- resents the negative attribute, and the de and da are the parameter of class-conditional models Po., (z a) and Po. (# When gener- ating the next token Tt, it is only necessary to calculate the output probability distributions of two class-conditional models Pa. (Iz) a) and Pæs (Ztlaiet, a) as all the tokens zj for T S j < t have already been sampled at the current time step. Therefore, Po. (Iz1æzz) a') for all T S < t is the probability value of token Ty. which is constant rather than a distribution (detailed in Ap- pendix A.I).
3.2 Attribute Distribution via PC-LM
Prefix-tuning (Li and Liang, 2021) uses prefix, which is small, task-specific vector optimize natural language generation tasks as a lightweight alternative to fine-tuning. Inspired by this, we opti- mize two prefixes using dataset with corresponding attributes using language model loss as Eq.4
We design an attribute reconstruction method make the distributions obtained by PC-LMs more balanced. First, we regularize the obtained attribute distributions before generating the next token 2t each time as Eq.6 and Eq.7
K LLM - Σ logPxo, (IalZeck (osla H k=1
(4)
where is the set of frozen GPT-2 parameters, 0. is learnable parameters of prefix Ha and a E āa.a} represents the attribute of the pre- fix. We utilize the optimized prefixes combined with a frozen GPT-2 as class-conditional models Po. (zglaee a) and Pog [ Z) in Eq.3, which denote as As illustrated in Fig given prompt 11:7-1, the pos PC-LM and the neg PC-LM can generate two probability distributions PA. (IT) E Ho.). These probability distribu- tions each have their own tendency towards their
P Ho.) In(Ps.e e. (ar xr Hy,))
(6)
Paya (a (arl Ho.) In(PP.,, (ect, Hoa))
Ph (2,s Ho.)
P(a[ai)) En'e(ona) (o.a) Px. (1ylees Ho.
(8)
LMs (PC-LMs).
Through this regularization. the original distri- bution Px.s (xz| He.)) and Px.e, (0r| Ho.) can be compressed into small range interval without changing the order of each element in it, which can stabilize the weight of attribute words
(Fr|Ter, Ho.) and

---

Automatic Evaluation
Human Evaluation
Method
Acc
PPL
Dist-1
Dist-2
Dist-3
Rel.
Flu.
Top.
Pre- Pre-Tuning (Li and Liang, 2021)
62.15
38.49
0.11
0.53
0.82
2.28
3.52
2.81
Con Prefixes (Qian et al., 2022)
75.66
35.32
0.11
0.52
0.81
2.77
3.63
2.96
Discup" (Zhang and Song, 2022)
95.20
39.14
0.07
0.46
0.80
3.85
3.47
3.52
PPLM (Dathathri et al., 2019)
69.06
34.89
0.12
0.51
0.77
2.54
3.56
3.24
GeDi (Krause et al., 2021)
94.23
169.86
0.15
0.53
0.74
3.38
2.60
3.47
DExpert (Liu et al., 2021)
94.74
51.99
0.16
0.65
0.85
3.51
3.02
3,46
Air-Decoding (medium)
96.82
26.66
0.13
0.55
0.78
4.03
3.96
3.85
Air-Decoding (large)*
96.16
18.59
0.13
0.52
0.76
3.93
4.01
3.73
Table The main experimental results of sentiment controllable generation, suggests that the performance is better with lower score. means the backbone model is GPT-2 large.
and non-attribute words in the attribute distribu- tion. Then we calculate P(a)æ)a) using normal- ized Px,e, e, (#yfaee Ha.) and Px.e, (z Tec. Ma.) in Eq.6 and 7 In the example above, the weight of "good" equals - (a0.mo..))=00444.. 1)0.4434 and the weight of "of" equals - -1/mm0.033)00.85 in the pos PC- LM's distribution after regularization. In the neg PC-LM's distribution, their weights become 0.217 and 0.334, respectively. Thus, in the fi- nal distribution P(a)x1z), the weight of "good" is 0.434//00.4 4444400004444022172000666 and the weight of "of" is maintains the high weight of "good" while reducing the gap between the two words, ensuring that the model would not blindly generate attribute words like "good", which would improve the fluency of the generated text. The overall algorithm frame- work can be found in Appendix A.2.
text, I -grams, 2-grams, and 3-grams are calculated which are named Dist-1, Dist-2, and Dist-3,
Human Evaluation. Following Zhang and Song (2022), we evaluate generated texts from three as- pects. Relevance (Rel.) reñeecs the degree of achievement for the desired control attribute. Flu- ency (Flu.) evaluates the text's fluency from the human perspective. Topicality (Top.) evaluate the consistency between the generated text and the input prompt. For each task, we randomly select 100 texts and ask three annotators to score them on the three metrics on scale from (very bad) to 5 (very good). Finally, we calculate the average of the 300 groups of scores to obtain the final manual evaluation results.
by Eq.8. This
4.2 Baselines
Prefix/Prompt-based: (1) Prefix-Tuning (Pre- Tuning) (Li and Liang, 2021) trains prefix to control the generation of frozen CLM. (2) Con- trastive Prefixes (Con Prefixes) (Qian et al., 2022) trains multiple attributes' prefixes simultaneously using a discriminative loss. (3) Discup(Zhang and Song, 2022) is the SOTA model so far, which in- corporates discriminator during the training stage to guide the training process.
4 Experiments
4.1 Evaluation Metric
Automatic Evaluation. We evaluate the gener- ated texts from three aspects. (1) Accuracy: For the sentiment and topic control tasks, we train a RoBERTa (Liu et al., 2019) classifier on the Yelp Review and AGNews dataset (Zhang et al., 2015) respectively calculate attribute accuracy (Ace). The two classifiers achieve accuracies of 98.53% and 95.57% on their corresponding test sets. For the detoxification task, we use the Perspective API2 to calculate the average toxicity for the generated texts. (2) Fluency: Text fluency is evaluated using the perplexity (PPL) calculated by GPT-2 large. (3) Diversity: We use distinctness (Li et al., 2016) to measure the generated texts' diversity. For each
beee (4) GeDi (Krause et al., 2021) uses class-conditional LM guide base model's generation. (5) DExpert (Liu al., 2021) uses fine-tuned GPT-2 as an expert/anti-xppert to guide a base model's generation. (Dathathri et al., 2019) uses gradients from a well- trained classifier to update the base model's hidden representations.
(6) PPLM
All baselines (excluding Contrastive Prefixes²)
3 As their code has not been released, we reproduce their work on our datasets and achieve comparable results.
Phttps: httttp::/wwwwpeeeee http///ww.prrrpptiieppp..oo/

---

Automatic Evaluation
Human Evaluation
Method
Acc
4
Dist-1
Dist-2
Dist-3
Rel.
Flu.
Top.
Pre- Pre-Tuning (Li and Liang, 2021)
72.74
64.43
0.09
0.49
0.74
2.85
3.05
2.84
Con Prefixes (Qian et al., 2022)
88.47
70.34
0.09
0.50
0.75
3.31
2.94
2.95
GeDi (Krause et al., 2021)
94.27
104.46
0.10
0.48
0.69
3.83
2.42
3.31
Air-Decoding (medium)
97.21
31.18
0.08
0.47
0.74
4.07
3.87
3.80
Air-Decoding (large)"
94.30
22.31
0.08
0.46
0.72
3.93
3.94
3.75
Table 2:
The main experimental results of topic controllable text generation. J. suggests that the performance is better with lower score. * means the backbone model GPT-2 large. Due to methodological limitations, DExpert cannot perform multiclass control tasks, while PPLM and Discup did not provide classifiers for topic tasks on the AGNews dataset, therefore these three methods are not included in the table.
are directly implemented from their public source codes. For a fair comparison, all baseline mod- els(excluding Discup") use the GPT-2 medium as the backbone generator. In addition, we standard- ized the sampling method during decoding, using a top-k value of 200 (excluding Discup, which uses a top-k value of 20). Other hyperparameters details are described in Appendix B.
Detoxification. Following the previous work (Qian al., 2022), we use the dataset provided by Jigsaw Unintended Bias in Toxicity Classifica- tion Kaggle Challenge" to train our model. We randomly selected 5K toxic and 5K nontoxic com- ments from the Jigsaw dataset to train the corre- sponding PC-LM. The length of the prefix for each PC-LM is set to 20. The generation prompts col- lected from Rea/TTxiccityPrompt (Gehman al.). Following previous work (Qian et al., 2022), we use the prompts categorized as "challenging" in the dataset and further filter out the prompts with toxicity larger than 0.5, scored by Perpective. The resulting evaluation dataset consists of 203 prompts. For each of the 203 prompts, we generate 20 sen- tences, each with length of 50. In our method, the control strength w is set to 120.0 for the medium model and 140.0 for the large model.
4.3 Experimental Setup
Sentiment Control. Following the previous work (Krause al., 2021),
IMDb movie reviews (Maas et al., 2011) to train our model. Compared to GeDi which uses 11.25K samples from the dataset for training. We randomly selected only 5K positive and 5K negative reviews from the IMDb dataset to train the corresponding PC-LM with a prefix length of 20. The prompts used for evaluation are the same as those in the PPLM (Dathathri et al., 2019). For each of the 15 prompts, we generate 50 sentences with positive attributes and 50 with negative attributes, each with length of 50. In our method, the control strength is set to 140.0 for the medium model and 130.0 for the large model.
4.4 Results and Analysis
Sentiment Control. As shown in the automatic part of Table 1, our approach has achieved SOTA in attribute relevance and text flu- ency. Among decoding-time methods, PPLM shows less attribute relevance, potentially due using only one classifier to modify the model, mak- ing it difficult to significantly impact generated text. Prefix-based methods (i.e., Prefix-Tuning and Contrastive Prefixes) have lower attribute relevance than decoding-time methods (i.e., GeDi and DEx- pert) but excel in fluency. Though Diseup performs well in attribute relevance and fluency, it relies on GPT-2 large which has more parameters. Our medium model outperforms Discup, increasing ac- curacy by 1.62% and lowering perplexity by 12.48 points, which implies our better ability for attribute
Topic Control. We experiment with the AG- News dataset (Zhang et al., 2015). For each at- tribute in the AGNews dataset, we randomly se- lected 5K samples outside the training data of the topic classifier to train the corresponding PC-LM with a prefix length of 20. The prompts used for evaluation are the same as those the PPLM (Dathathri et al., 2019). For each of the 20 prompts, we generate 50 sentences for each of the attributes from World, Sports, Business, and Science, each with a length of 50. In our method, the control strength is set to 60.0 for the medium model and 70.0 for the large model.
htt ptttps://www..r kaggle. com/c/ bias- innss-in---oe tor.iiiityyylllssiictttt
'Discup only released prompts used for GPT-2 large.

---

Automatic Evaluation
Human Evaluation
Method
Tox. 4
1
Dist-1
Dist-2
Dist-3
Rel.
Flu.
Top.
Pre-Tuning (Li and Liang, 2021)
49.2
92.20
0.07
0.40
0.68
2.24
2.37
2.93
Con Prefixes (Qian al., 2022)
21.7
85.34
Discup" (Zhang and Song, 2022)
14.8
63.90
0.07
0.48
0.82
3.90
3.04
3.36
PPLM (Dathathri et al., 2019)
30.0
148.50
GeDi (Krause et al., 2021)
20.5
166.01
DExpert (Liu et al., 2021)
20.0
58.06
0.08
0.48
0.78
3.53
3.36
3.45
Air- Air-Decoding (medium)
18.5
48.29
0.07
0.44
0.74
3.85
3.56
3.74
Air- Dir-Deodinng (large)*
21.6
38.86
0.07
0.42
0.73
3.76
3.64
3.68
Table The main experimental results on the task of detoxification. "Tox." represents average toxicity. & suggests that the performance is better with lower score. means the backbone model is GPT-2 large. As the experimental results of GeDi, PPLM, and Con Prefixes are directly taken from Qian et (2022), we do not conduct diversity and human evaluation for these three methods.
control and text fluency. In terms of diversity, our method fails slightly short. This can be attributed to elevated control strength, as it intensifies the attribute distribution's focus on high-probability tokens, thereby increasing the likelihood of their selection during sampling. A detailed table of di- versity under different control strengths is available in Appendix C. The human evaluation results are presented the manual part of Table Consistent with automatic evaluations, our approach surpasses all baselines, particularly regarding fluency.
method achieves better fluency in the large model but decreases in attribute relevance, which is simi- lar to the results obtained in the sentiment and topic control task. We analyze that larger models have an advantage in the pre-training corpus in generating fluent text, but the increase in model size without corresponding increase in training data results in slight decrease in attribute relevance. As for the human evaluation results, our method performs the best in terms of fluency and topicality, with slightly lower attribute relevance compared to Discup.
Topic Control. The results presented in the auto- matic part of Table demonstrate that our method has achieved a new SOTA in attribute relevance and text fluency. Specifically, our medium model outperforms GeDi, Contrastive Prefixes, and Prefix- Tuning by 2.94, 8.74, and 24.47. Regarding text fluency, our method achieves 33.25 perplexity points reduction compared to Prefix-Tuning. 39.16
4.5 Further Analysis
In this section, we aim to assess the effeets of distinet factors on overall performance. We further choose the sentiment control task and use GPT-2 medium the backbone model to conduct experiments. The results of the topic control and detoxification tasks can be found in Appendix D.
points compared to Contrastive Prefixes, and 73.28 points compared to the lowest-performing GeDi. In terms of diversity, our method slightly underper- forms Contrastive Prefixes but remains on par with GeDi and Prefix- Prefix-Tuning. The human evaluation results in Table 2 also demonstrate the superiority of our method, mainly in fluency and topicality.
The Effect of Distribution Reconstruction. We conduct ablation experiments on attribute distribution reconstruction and we define the settings with and without attribute distribution reconstruction follows:
w/ reconstruction: Calculate P(a(maa) by Eq.8.
Detoxification. As shown in Table 3, Air- Decoding (medium) achieves an average toxic- ity of 181, outperforming three decoding-time time- based methods PPLM, GeDi, and DExpert. Com- pared prefix/prompt-base methods, outper- form prefix-tuning and Contrastive Prefixes but slightly weaker than Discup. However, terms of fluency, Air-Decioo lirDeecddigg (medium) outperforms all of the baselines, especially PPLM and GeDi. Our
w/o reconstruction:
Calculate P(a[æ))) by Eq.5.
As illustrated in Figure 3, both settings exhibit strong performance accuracy under high control strength but without attribute distribution recon- struction, the fluency deteriorates with increasing control strength, resulting in Attribute Collapse.
In order to further analyze how reconstruction solves the problem of Attribute Collapse, we de-

---

note the distribution P(x 1, 1, P 1). P(al in Eq.2 as d, (output distribution), dr (raw LM's distribution), de (attribute distribution). At each token generation, we define the sets of top-k (we all use the top-k value of 200) tokens with the highest probability for each of these dis- tributions as S., Sr, Su. respectively. Then we define Sor = So n Sy, Soa So n Sa and Sura S. n n.S, Sa- We use |S| to denote the size of set S. In this experiment, we consider the following three metrics:
Accurary Vs.
Persinstty vt.u
30-
65.2-
02.3
40.3
31.5
26,4
22.5
102-
W Wreconannciinn
Figure 3: The impact of auribute distribution reconstruc- tion on the performance of Air- Air-Deodiing
ory. 0 VS
S ooj/1 of
is orajys vs.
|/.]: which reflects the similarity between d, and d,.
0.46
3.36-
0.94
|Saa//|S.|: which reflects the similarity between d, and da.
recoretuction W/ recermmerice
1.89
0.54
|Soru//|Sol: which simultancously reflects the similarity between d, and both d, and da.
0:82
50
100
From Figure 4, as the control strength increases, the values of [Soal/|S. increase, and the values of |/Spl decrease under both settings. This is intuitive as an increase in control strength will cause the d, to deviate from the d, and shift towards the da. In the figure of [Soral| we find that the value of |Sor with reconstruction stabilizes at around 10%, whereas the value of [Sora///So without reconstruction only stays at 1%. This means that under high control strength, the model with reconstruction setting is more likely to sample tokens from set Sora, which both satisfy the accuracy of the attributes and ensure fluency. But in the setting without reconstruction, the model needs to sample more tokens that are in set [Sall but not in set |Sorl as the control strength increases, which results in decrease in fluency.
Figure 4: The impact of attribute distribution reconstruc- tion on the value of / SSo |Socl/[Sol, /
Accurncy vs,
Perplectyy vs.
38
10c
0.30
6.26
14
0.5%
50
0.20
80 100 320
100
Figure 5: The impact of training data volume to Air- Decoding in the sentiment control task. The label repre- sents the data volume used to train single PC-LM.
5 Conclusion
The Effect of the Size of Training Sam- ples. From the results in Figure 5, we find that the accuracy improves as the data volume increases. However, the curves of accuracy almost overlap when the data volume is greater than 5K, indicating that our method can achieve optimal performance at a data volume of around 5K. When the data volume is less than 500, the perplexity continues to increase with w, whereas when the data volume is greater than IK, the perplexity is consistently kept within stable range. We analyze that when the training data is small, the prefix learns insufficient attribute-related knowledge, resulting in a low value of [Soral. which leads to a similar result without attribute distribution reconstruction.
We first identify the phenomenon of Attribute Col- lapse and propose novel lightweight framework, Air-Decoding, address this issue in decoding- time CTG. Specifically, we train prefixes as Prefix- Conditional LMs (PC-LMs), followed by a novel attribute distributional reconstruction method to en- sure more balanced attribute distribution obtained by the PC-LMs. Then we use the reconstructed attribute distribution to guide the generation of CLMs and achieve remarkable CTG performance. Experimental results on three typical CTG tasks demonstrate that our method not only achieves high attribute control but also excellent text fluency, ef- fectively solving the problem of Attribute Collapse in traditional decoding time-based methods.

---

Limitations
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung. Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu, 2019. Plug and play language models: A simple approach to controlled text generation. In International Conference on Learning Representa- lions.
Our method primarily addresses the issue of At- tribute Collapse in decoding-time secoding-limme-ased control- lable text generation through attribute distribution reconstruction at the distribution level. However, we have not conducted in- in-depth investigations into the impact of different attribute distributions ob- tained by different models on the performance of our decoding framework, well the fundamen- tal reasons why attribute distribution reconstruction can solve the issue of Attribute Collapse, which are both areas for future research.
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. Realtoxicityprompt:: Evaluating neural toxic degeneration in language models.
Yuxuan Gu, Xiancheng Feng, Sicheng Ma, Jiaming Wu, Heng Gong. and Bing Qin. 2022a. Improving control- lable text generation with position-aware weighted decoding. In Findings of the Association for Compu- tational Linguistics: ACL 2022, pages 3449-3467.
Ethics Statement
Yuxuan Gu, Xiaocheng Feng, Sicheng Ma, Lingyuan Zhang, Heng Gong, and Bing Qin. 2022b. A distri- butional lens for multi-aspect controllable text gener- ation. arXiv preprint arX0v2220000288.
Attribute controllable text generation has extensive use on social media platforms, but inappropriate ap- plications can have many negative impacts, such as generating negative or false news or creating abun- dant public opinions on political topics to confuse public perception. We hired human annotators to evaluate our method and other baselines. Consider- ing the difference among the three tasks, annotators got $0.1 for each sentence in the sentiment control and topic control tasks and $0.2 in the detoxifica- tion task.
Pei Ke, Hao Zhou, Yankai Lin, Peng Li, Jie Zhou, Xiaoyan Zhu, and Minlie Huang. 2022. Ctrleval: An unsupervised reference-free metric for evaluat- ing controlled text generation. 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 2306-231 2319.
Proceedings of the
Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019. Ctrl: A conditional transformer language model for control- lable generation. arXiv preprint 1rXiv:1909.055:0
Acknowledgements
We thank all the anonymous reviewers for their insightful comments. This work is supported by the National Science Fund for Excellent Young Scholars under Grant 62222212 and the National Natural Science Foundation of China under Grant 62376033.
Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. 2021. Gedi: Genera- tive discriminator guided sequence generation. In Findings of the A Association for Computational Lin- guistics: EMNLP 2021, pages 4929 4952.
Sachin Kumar, Eric Malmi, Aliaksei Severyn, and Yu- li Tsvetkov. 2021. Controlled text generation continuous optimization with multiple constraints. Advances in Neural Information Processing Systems, 34: 14542-14554.
References
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1999...
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficiee prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059.
Fredrik Carlsson, Joey Öhman, Fangyu Liu, Severine Verlinden, Joakim Nivre, and Magnus Sahlgren. 2022. Fine-graines controllable text generation using non- residual prompting. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6837- 6857.
Jiwei Li. Michel Galley, Chris Brockett, Jianfeng Gao, and William B Dolan. 2016. A objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- rional Linguistics: Human Language Technologies, pages 110-119.
Alvin Chan, Yew-Soon Ong, Bill Pung, Aston Zhang, and Jie Fu. 2020. Cocon: A self-supervised ap- proach for controlled text generation. arXiv preprint arXiv:2006.03335.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In

---

Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Langwage Processing (Volume I: Long Papers), pages 4582- 4597.
Yequan Wang, Jiawen Deng, Aixin Sun, and Xuy- ing Meng. 2022. Perplexity from plm is unreli- able for evaluating text quality. arXiv preprint arXiv:2210.0582.
Yufei Wang, Ian Wood, Stephen Wan, Mark Dras, and Mark Johnson. 2021. Mention flags (mf): Constrain- ing transformerrbased text generators. In Proceed- ings the 59th Annual Meeting of the Association for Computational Linguistics and the 1 Ith International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pages 103-113.
Zhiyu Lin and Mark O Riedl. 2021. Plug-and-blend: a framework for plug-and- play controllable story gen- eration with sketches. In Proceedings of the Seven- leemth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, pages 58-65.
Alisa Liu, Maarten Sap. Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A Smith, and Yejin Choi. 2021. Dexperts: Decoding- time con- trolled text generation with experts and anti-experts. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1tth International Joint Conference on Natural Lan- guage Processing (Volume Long Papers), pages 6691-6706.
Kevin Yang and Dan Klein. 2021. Fudge: Controlled text generation with future discriminators. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3511-3535.
Kexin Yang. Dayiheng Liu, Wenqiang Lei, Baosong Yang. Mingfeng Xue. Boxing Chen, and Jun Xie. 2022. Tailor: A prompt-based approach to attribute- based controlled text generation. arXiv preprint urXiv:2204.1335
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019.
Roberta: robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907.1 1692.
Dian Yu, Zhou Yu, and Kenji Sagae. 2021. Attribute alignment: Controlling text generation from pre- trained language models. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2021. pages 225 2261-
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the associ- ation for computational linguistics: Human language technologies, pages 142-150.
Hanqing Zhang and Dawei Song. 2022. Discup: Dis- criminator cooperative unlikelihood prompt-tuning for controllable text generation. arXiv preprint arXiv:2210.09)
Andrea Madotto, Etsuko Ishii, Zhaojiang Lin, Sumanth Dathathri, and Pascale Fung. 2020. Plug-and-play conversational models. In Findings of the Associa- tion for Computational Linguistics: EMNLP 2020, pages 2422-2433.
Hanqing Zhang, Haolin Song, Shaoyu Li, Ming Zhou, and Dawei Song. 2022. A survey of controllable text generation using transformer based pre-trained language models. arXiv preprint arXiv:2201.0333
Fatemehsadat Mireshghallah, Kartik Goyal, and Taylor Kirk-aiika. 2022. Mix and match: Learning- free controllable text generationusing energy lan- guage models. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 401-415.
Xiang Zhang. Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classi- fication. Advances in neural information processing systems, 28.
Yizhe Zhang, Guoyin Wang, Chunyuan Li, Zhe Gan, Chris Brockett, and William Dolan. 2020. Pointer: Constrained progressive text generation via insertion- based generative pre-training. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6649-86700
Jing Qian, Li Dong, Yelong Shen, Furu Wei, and Weizhu Chen. 2022. Controllable natural language genera- tion with contrastive prefixes. In Findings of the As- sociation for Computational Linguistics: ACL 2022. pages
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Chris- tiano, and Geoffrey Irving. 2019. Fine-tuning lan- guage models from human preferences. arXiv preprint arXiv: 190,:09909
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 188):
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing processinnn systems, 30.

---

A Methodology Details
Algorithm 1 Air- Air-Decoding
Require: prefixes Ho. base model Po(x:) control strength w, imput prompt #1-7-1, de- sired attribute a
A.1 Bayesian Factorization
P(z
t = T
P(a)))))P(ani)
2: while 1 S N do
P(a
3:
if t - T then
P(orllzaa- P(alma)) -
4: Saa =1
else
1
daa Saa da(0t-1/12 11--1 =
P(a[21=11)
x P(alnna)P(x((li
end if
dg (2t)
Px.o, (2rel Fi How)
da (Iz)
Px. (xd Ieets H8a)
P(a]z(a) P(m.,.a)
dg (2+) Imi(d(ass) [m
P(ana)
10:
Pa.) (3
da (z11) Inida(x:))
11:
En' P(a') Pe (aiala))
P(a) - Po.. (1 ())
12:
P(a|zo:) dd (oa
dalars)
En'e(na) P(a Po, (0)) a')
13:
P(arlee a) = Po (ett) P( P(alros)
14:
ut Decode Decode(P(xz)zee ([xz|
A.2 Air-decoding Algorithm The basic decoding formula is
15:
Te Concal Ut)
16:
t 1=t+1
17: end while
P( a) (
and P(a 211) formulated as
For PPLM, we use the hyperparameters o 1.0, m = 10, a = 0.03, Au = 0.01, Agm = 0.95,
I1;- (2yl Hr.) P(a/zıa) Ed'e(u.a) 11 Pr.o, (1r (ttes Ho.))
top-k=200, top-p=1.0.
For GeDi, we use the hyperparameters of w 160.0, top-k=200, top-p=1.0.
As we have shown that ex (z Ho.) constant rather than a distribution for all T < < t, we convert P(t) into the following form for computational convenience.
For DExpert, use the hyperparameters of 2.4, top-k=200, top-p=1.0.
For Prefix-Tuning, we directly use the PC-LMs trained in our experiments for generation. We use the hyperparameters of top-k=200 and top-p=1.0. For Contrastive Prefixes, we set the prefix length to 20, the same as that in our method. Other hy- perparameters are followed by their original work (Qian et al., 2022), the training batch size is 8, W1 0.8, we = 0.2, the number of training epochs is 50, the learning rate is 2e-5.
(ayla Hog)
P(almaa Ee'e(e.a) Paa (2y lale, Ho
Ps.o. (25) 22 How)
15-2 15 Ps.o, (1s (Ress Ho,) + 115- PA (n Hoa)
Pro,, (xel lane, Ho.)
Pa.s, (welicee Ho,)) dea Ps.ax (aeleee Moa)
where the dao = 11 Pa.og Hegl is con-
!I 5-7 Ps (ee,,,
stant. Based on the description above, our Air- decoding Algorithm can be summarized as Algo- rithm
For Discup, we use top-k=20 and top-p=1.0 due to the particularities of their method, which enable achieving satisfactory text diversity with a relatively low top-k value.
B Hyperparameters
Sentiment Conrtol. In our method, train two PC-LMs, each with prefix length of 20. The training batch size is 4, the weight decay is 0.01, the learning rate is 3e-5, and the number of training epochs 5. During the generation stage, we use w 140.0, top-k=200, top-p=1.0.
Topic Control. In our method, train four PC-LMs, each with prefix length of 20. The training batch size is 4, the weight decay is 0.01, the learning rate is 3e-5, and the number of training epochs is 5. For generation, we use w = 60.0, top-k=200, top-p=1.0.

---

For GeDi, we use the hyperparameters of w 150, top-k=200, top-p=1.0.
D Further Analysis Details
We use GPT2-medium the backbone model to conduct further experiments topic control and detoxification tasks.
For Prefix-Tuning, we directly use the PC-LMs trained in our experiments for generation. We use the hyperparamelers of top-k=200 and top-p=1.0. For Contrastive Prefixes, we set the prefix length to 20, the same that in our method. Other hyperparameters are followed by their original work (Qian et al., 2022), the training batch size is 4, w1 = 0.8, W2 = 0.2, the number of training epochs is 8, the learning rate is 2e-5.
D.1 Topic Control
The Effect of Distribution Reconstruction. From the results in Figure 6, similar the sentiment con- trol task, both accuracy increase as control strength increases, and without attribute distribution recon- struction, fluency gradually decreases. However, in the topic control task, the fluency reduction caused by the absence of attribute distribution reconstruc- tion is not as significant as in the sentiment control task. We speculate that this outcome is due the in- herent balance within the AGNews dataset, which employed in our experiments.
Detoxification. In our method, we train two PC-LMs, each with prefix length of 20. The training batch size is 4, the weight decay is 0.01, the learning rate is 3e-5, and the number of training epochs is 5. For generation, top-k=200, top-p=1.0.
w 120.0.
For Prefix- Prefix-Tuning, Tuning, we directly use the PC-LMs trained in our experiments for generation. We use the hyperparameters of top-k=200 and top-p=1.0.
Accuraty
Perstesity
20-
For Discup, we use the hyperparameters of top-
k=200 and top-p=1.0.
For DExpert, we use the hyperparameters of a 0.9 top-k=200, top-p=1.0.
For other baselines, we follow the experiment settings of (Qian et al., 2022) and use their results directly.
Figure 6: impact attribute distribution reconstruc- tion the performance of Air- Air-Decoding in the topic control task.
C Experiments Details
[5 v5,
15 of
15 (2.re)/55_or of
3.00
W
Dist-1
Dist-2
Dist-3
0.24
reconttruciion
6.12
10
0.144
0.608
0.844
20
0.143
0.603
0.841
Acoor/tuulnn A/ roueem/auii))
www.enduciee Rocenobocc
0.33
30
0.143
0.605
0.843
2000
6:33
40
0.141
0.597
0.837
0.142
0.595
0.831
6:02
0.01
60
0.140
0.589
0.827
70
0.138
0.586
0,823
Figure 7: The impact of attribute distribution reconstruc- tion on the value of |SSor 1/|S./ |S.el/|5. / ( the topic control task.
80
0.137
0.578
0.815
90
0.138
0.575
0.811
100
0.136
0.572
0.807
110
0.133
0.562
0.801
The Effect of the Size of Training Samples. As shown in Figure 8, when the number of training samples exceeds 2K, the accuracy curve is basi- cally overlapping, and accuracy of 90% could be obtained only when the number of training sam- ples is KK. With respect to fluency, similar the sentiment control task, higher fluency can generally be achieved with more training data.
120
0.133
0.563
0.798
130
0.132
0.553
0.787
140
0.132
0.551
0.785
Table 4: Diversity performance of Air-Decoding on sentiment control task with different control strength u.

---

Accuracy vs.
|P 4] 13.
1. 2_oraj8B_9 40.0
0.7
0.30
80
1o.dt 4800
Accuraty (N6
A/ recoonctatioon
5K
4K
0.2
0.66
50-
2K
0.4
1K
0.00
100
0.4K
0.3K
Figure 10: The impact of attribute distribution re- construction the value of |Sarl/|Sol, [Soe//(ool [Sore/||s in the detoxification task.
Perplexity
5K
42
JK
40
2K
tasks. However, training data levels below 1K and high control strength, the generated text is less toxic compared to when the training data at 5K. We attribute this to the special nature the text detoxification task, which requires avoiding the use of toxic words. With limited training data, the weight of toxic tokens in the output distribu- tion of the trained model is not high, thus resulting in naturally reduced amount of toxicity in the generated text.
Parp ealy
38
0.5K
0.4K
36-
0.2K
34-
32
10
20
30
40
50
60
Figure 8: Performance Air-Decoding under different training data volumes in the topic control task. The label represents the amount of data used to train a single PC-LM.
Toxicity vs.
5K
0.40
D.2 Detoxification
2K
The Effect of Distribution Reconstruction. As illustrated in Figure 9, we find that in the detoxifi- cation task, not adding attribute distribution recon- struction could result in lower toxicity, but still suffers from decreasing fluency.
Toxcey
0.30
0.4K
0.25
0.15
Toclcity v5.00
Perplexity
20
40
60
80
300
120
W/ /rrronmtattinn
Perplexity vs.
54-
30
3K
2.25
6.00
35
Perpleaity
51
0.5K
25
125
0.2K
Figure The impact of attribute distribution reconstruc- tion on the performance of Air-Decoding in detoxifica- tion task.
20
60
80
100
120
The Effect of the Size of Training Samples. The results are shown in Figure 11. The result shows that as the training data decreases from 5K to 2K, text toxicity progressively increases. This aligns with findings in sentiment control and topic control
Figure 11: The impact of attribute distribution recon- struction the performance of Air-Decoding in the detoxification task.

---

E
Human Evaluation Details
F Case Study
this section we provide specific scoring guide- lines for each human evaluation metric.
F.1 Cases of Attribute Collapse
Due to the Attribute Collapse issue only present in the decoding-tine CT methods, in the Table 5, have selected specifie generated examples from GeDi and DExpert in the sentiment control task. From the table, it can be observed that sentences with Attribute Collapse occurs exhibit repetition attribute tokens or the presence numerous bad tokens.
E.1
Relevance
The generated sentences are perfectly aligned with the desired attribute.
5:
4: The generated sentences very related to the desired attribute.
3: The generated sentences very related to the desired attribute.
F.2 Cases Ablation Study
The Table 6 presents specific generated examples of Air-decoding with and without distribution recon- struction in the sentiment control task. The results of the ablation experiments also exhibit notable issue of attribute stacking, in contrast, the results with reconstruction are significantly smoother and more logically coherent.
2: The generated sentences have relatively weak consistency with the desired attribute.
The generated sentences have correlation with the desired attribute, and in some cases, they even contradict it.
I:
E.2 Fluency
F.3 Cases of All Baselines and Attributes
5: The generated sentences are grammatically correct, fluent, and easy understand.
We present generated cases by all baselines and Air-Decoding in every desired attribute in Table 7.
4: The generated sentences grammatically correct, but they slightly less smooth, yet still easily understandable.
3: The generated sentences have few grammar errors, but they do not hinder understanding.
2: The generated sentences have few grammar errors and are not very easy understand.
I:
The generated sentences have numerous gram- mar errors, lack coherence, and are difficult to understand.
E.3 Topicality
5: The generated sentences are grammatically correct, fluent, and easy understand.
4: The generated sentences exhibit a relatively strong correlation with the input prompt.
3: The generated sentences have an average cor- relation with the input prompt.
2: The generated sentences have relatively weak correlation with the input prompt.
I: The generated sentences have a poor correla- tion with the input prompt, resulting in incoher- ent sentences.

---

Tar: Att.
Method
Generated texts
Once upon time flourished shone cardinal laurel.artney honouring the birthday of the Blossom of theesprit birthday action Date Harmony Meeting Graceland blues Graceland blues Gale principe blues brid breeforts Graceland bluesalm enchaniivelyrinei
GeDi
Positive
The book taps vividly vividly into the lives of Tibetan Buddhic tabrasadhay meditation teachers widely practiced both in Tibet and China today. In their homes and practice venues across China and Tibet mountains, teachers express the wonders of their cultures
DExprtt
Once upon time Screenshot worse limding worse imbalerraining lab ratsatche worse scient inept idiotswrap wrap worse Ness decompurseswraa convert SNAPIZEWra Zombie Trogviciflowerwra Swiss cheese hands Trog slaves wearing pool waste tamp domain de DrainFeatureenut
GeDi
Negative
The book sucks. BAD!!Disclaimer CONTENT List of crrors waste of time Stupidity stupidity waste of effort stupid books Stupid waste of time wastes of my time waste of my time waste of your time waste of yours Stupid waste of your time
DExpert
Table 5: Cases of Attribute Collapse.
Tar. Att.
Method
Generated texts
The movie works! Read My Lips Review Interview with Barbara Galliak delivers! My Big Fat Greek Wedding nicely improves upon its usual predecessor with gorgeous depth and integrity that speak volumes for this wondrous testament to living harmoniously with living love! Read The movie The Matrix is one of my all time favorite movies and is a must see for all children and adults alike. It is the story where Neo steps into the Matrix to alter the world's history and the effects that are wrought by his actions
W.O. rec.
Positive
W. rec.
The acting wasted needless hours wasted wasted wasting needless amounts of wasted time waste of wasted money wasting wasted money wasted waste of time waste wasted wasted wasted wasted The movie scems bad. The actors and crew look like they've been drinking. The sound effects boring, the story looks like it tells you to write more. The characters are incredibly bland. The writing inherently boring, because you can
The movie failed worse even worse THAN ME. 1. The writing 2. The acting 3.
W.O. rec.
Negative
rec.
Table 6: Cases of ablation study, w. rec. represents Air-Decoding with attribute distribution reconstruction and w.o. rec. represents Air- Decoding without attribute distribution reconstruction.

---

Tar. Att.
Method
Generated texts
The movielainerie from Beauteuil is based on the classic French restaurant, Herbier. Sathier.Chammn Champagnefavorite expanses of Blossom Hill, Blossom Hill enchant the senses with its Enchantment of enchanting Art Deco enchanting enchanting
GeDi
The movie has touched hearts across Australia and around the world thanks to timely performances set among refugees arriving in Australia in recent years' Great gentle light shone through the icy winter skies and that day two girls who had been abandoned just after the Tianjin The movie helps her to come but she can still be so bcautiful...shc has good strength, strength and courage even in her age she was so strong and strong with them. Even if this was good at the time (in this moment and time she is
DExpert
Positive
Discup
The movie The Matrix is of my all time favorie movies and is must see for all children and adults alike. It is the story where Neo steps into the Matrix to alter the world's history and the effects that are wrought by his actions.
Ours
The book sucks. dismefilarity. That's ohvious. lot of people who read this book donated moncy to low-1Q, low-NVG, low-IQ, low
GeDi
Democrass
The book description mentions that this book was printed 1978.1978. The book description
DExpert
says 1978, because the translation notes say 1978. The description says 1978 when it should have been 1978-69. suspect that the translator at The book is less than good with bad reviews but bad reviews on the other side is nothing else for nothing but bad reviews or no reviews at all.Ön October 28, 2009, however, there was nothing to worry about about at all except for
Negative
Discup
The book doesn't I even make sense. The first two chapters are just paragraphs without any details to hack up anything beyond this book thinks that 2 is 4: The entire thing is like trying to explain why you and your friend committed a crime by finding
Ours
The relationship Situation in South Victnam developed from a war begun the twenticth century until January 1975 Haiti fought Haitiian independence from Spain
GeDi
World
The relationship with Isracl deteriorated after Hamas leader Khader Arafat stepped down in 1999 and Sharon launched his own probe into bombing at Sabra and Shatila in November 1981.
Ours
Sharon had expressed his doubts about the intelligence report, questioning its accuracy,
More importantly, how long should athletes wait before returning to tournament Minnesota
GeDi
feels is theirs? When are the NCAA and the Olympies better in all of sports get there than More importantl, basketball. mean, there LeBron James. And! was sure that the notion that the Heat would do just that is absurd. They've gone the playoff's just five times
Sports
Ours
To conclude, investors need to understand a few things:. there would he no liquidity threat if the banking system were allowed fail, if the banks were allowed to fail, there was any risk of contagion, and, finally.
GeDi
Business
To conclude, eamiins forecasts for Exxon Mobil Corp. this week were high, Torecasting earnings estimates that topped forecast and showed there was still enough pull keep the oil behemoth above guidance. But the forecast is modest as earnings begin to pick up.
Ours
This essay discusses Mozilla Firefox browser technology features. In particular, Firefox browser extensions contain resources specified via !oikies!!!!!!!!!!!!!
GeDi
This essay discusses Windows Server 2008 R2 (R2) security, making extensive recommendations that Microsoft add to the Windows Vista operating system, which is an ideal product for it. Thanks
Science
Ours
to strong security measures (including user passwords, password managers and a robust
I dont now for you, guys, but decided to walk to work and my workday shifts ended up running parallel to my bus and train schedules. So many things had to be relocated, so I figured if I could get my bags pulled along, my work would be easier too.
DExpert
dont now for you, guys, but I hope you will try it 9) like to explore different ways of working... it's quite challenging :) Thanks The government of Saudi Arabia announced yesterday that it intends to start building nuclear reactors in the country. The new reactors are expected to be ready for
Detox.
Discup
I dont now for you, guys, but I noticed that there was discussion later on about the link. Please feel free to comment if there is some point you are missing, However, i think it is the link that needs fixing.l notice you all are also losing emails about this matter
Ours
Table 7: Examples of the generation. "Tar. Att." in the first column represents the target attribute and "Detox." represents the attribute of nontoxic. The given prompts are in bold.


Reviews:
[{"review_id": "MQvkayc6rU", "reviewer": "Reviewer_yX2p", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and suggestions that will help make our article clearer. We summarize your main concerns as follows:\n>**Q1: The problem suggested in the paper is constrained to the decoding-time CTG method.**\n\nA1: As mentioned in the Introduction, the decoding-time CTG method possesses advantages in high generality and strong controllability, which are relatively challenging to achieve in the context of prompt/prefix fine-tuning methods. However, traditional implementations of decoding-time CTG that emphasize strong controllability often suffer from a significant deficiency in text fluency. This deficiency is referred to as the \"Attribute Collapse\" issue in our work. Addressing this problem contributes to an overall enhancement of the performance of decoding-time CTG methods in the context of CTG tasks. Hence, while the issue of Attribute Collapse is specific to decoding-time CTG methods, its resolution holds relatively significant implications for the advancement of CTG as a whole.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "They suggested an interesting problem and they solved it with a clear and easy method.", "reasons_to_reject": "This problem is constrained to the decoding-time CTG method.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors put forward an attribute collapse problem that can be encountered in the decoding-time CTG method. They propose a novel lightweight decoding framework named Air-Decoding. Within this framework, they devised an attribute distribution reconstruction method to achieve a more balanced attribute distribution. They perform experiments on three CTG tasks, namely sentiment control, topic control, and text detoxification.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WRVDwniEOw", "reviewer": "Reviewer_cf4V", "summary": "", "questions": "Please see the section of weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and suggestions that will help make our article clearer. We summarize your main concerns as follows:\n\n> **Q1: The detailed guidelines for human evaluations are missing.**\n\nA1: Following [1], we evaluate generated texts from three aspects. **Relevance (Rel.)** reflects the degree of achievement for the desired control attribute. **Fluency (Flu.)** evaluates the text's fluency from the human perspective. **Topicality (Top.)**  evaluates the consistency between the generated text and the input prompt. For each task, we randomly select 100 texts and ask three annotators to score them on the three metrics on a scale from 1 (very bad) to 5 (very good). Finally, we calculate the average of the 300 groups of scores to obtain the final manual evaluation results. Specific scoring guidelines for each metric are provided as follows and we will add the specific scoring guidelines to the revised version of the paper.\n\n- Relevance\n  - [5]: The generated sentences are perfectly aligned with the desired attribute.\n  - [4]: The generated sentences are very related to the desired attribute.\n  - [3]: The generated sentences are, to a certain extent, relevant to the desired attribute.\n  - [2]: The generated sentences have a relatively weak consistency with the desired attribute.\n  - [1]: The generated sentences have no correlation with the desired attribute, and in some cases, they even contradict it.\n- Fluency\n  - [5]: The generated sentences are grammatically correct, fluent, and easy to understand.\n  - [4]: The generated sentences are grammatically correct, but they are slightly less smooth, yet still easily understandable.\n  - [3]: The generated sentences have a few grammar errors, but they do not hinder understanding.\n  - [2]: The generated sentences have a few grammar errors and are not very easy to understand.\n  - [1]: The generated sentences have numerous grammar errors, lack coherence, and are difficult to understand.\n- Topicality\n  - [5]: The generated sentences exhibit a strong correlation with the input prompt.\n  - [4]: The generated sentences exhibit a relatively strong correlation with the input prompt.\n  - [3]: The generated sentences have an average correlation with the input prompt.\n  - [2]: The generated sentences have a relatively weak correlation with the input prompt.\n  - [1]: The generated sentences have a poor correlation with the input prompt, resulting in incoherent sentences.\n\n[1] DisCup- Discriminator Cooperative Unlikelihood Prompt-tuning for Controllable Text Generation (2022), Zhang et al\n\n$\\quad$\n\n> **Q2: The authors should measure the inter-annotator agreement score to better justify the provided experimental results.**\n\nA2:  We calculate the **Fleiss' Kappa coefficient** to measure the inter-annotator agreement score for each human evaluation metric in the sentiment control task. The results are shown in the table below, which will be included in the revised version of the paper along with the results in the topic control and text detoxification tasks.\n|Method|Relevance| Fluency| Topicality|\n|-------|:-------:|:-------:|:-------:|\n|Pre-Tuning| 0.668 | 0.702 | 0.664|\n|Con Prefixes | 0.665 | 0.701 | 0.698 |\n|Discup | 0.706 | 0.686 | 0.713 |\n|PPLM | 0.689 | 0.707 | 0.675 |\n|GeDi | 0.695 | 0.699 | 0.711 |\n|DExpert | 0.698 | 0.710 | 0.714 |\n|Air-Decoding-M (ours) | 0.713 | 0.704 | 0.687|\n|Air-Decoding-L (ours) | 0.703 | 0.712 | 0.697 |\n\n$\\quad$\n\n> **Q3:  Some key descriptions are missing. When conducting topic control experiments, there are five topics. Then, for each topic {$a$}, how to select the topics to constitute the complementary attribute {$a^\\prime$}?**\n\nA3:  In the context of the topic control task, we are dealing with four distinct categories of topics: World, Sports, Business, and Science. These four topics are denoted as $a_1$, $a_2$, $a_3$, $a_4$ respectively. For a given desired topic $a_k$, where $k\\in\\{1,2,3,4\\}$, its corresponding attribute distribution $P(a_k|x_{1:t})$ is calculated as follows:\n\n$$\n\\begin{align*}\n    P(a_k|x_{1:t})\n    = \\frac{P(x_{1,t},a_k)}{P(x_{1:t})}\n    = \\frac{P(a_k)P_{\\phi_{a_k}}(x_{1:t}|a_k)}{\\sum_{a^{'}\\in \\{a_1,a_2,a_3,a_4\\}}P(a^{'})P_{\\phi_{a^{'}}}(x_{1:t}|a^{'})}\n    = \\frac{\\Pi_{j=T}^t P_{\\phi_{a_k}}(x_j|x_{<j},a_k)}{\\sum_{a^{'}\\in\\{a_1,a_2,a_3,a_4\\}}\\Pi_{j=T}^t P_{\\phi_{a^{'}}}(x_j|x_{<j},a^{'})}\\quad k=1,2,3,4\n\\end{align*}\n$$\n\nwhere the class priors $P(a_k)$ and $P(a^\\prime)$ are omitted as we use the same amount of training data for each attribute PC-LM. **Therefore, in the context of topic control task, the selection of complementary attributes involves choosing all possible attributes except the desired attribute $\\boldsymbol{a_k}$. This selection is determined by the principle of the Total Probability formula.**\n\n$\\quad$\n\n> **Q4: Two recent studies about contrastive-based decoding procedure might be worth discussing in related works.**\n\n>**[1] A Contrastive Framework for Neural Text Generation --- NeurIPS 2022\n[2] Contrastive Search Is What You Need For Neural Text Generation --- TMLR 2023**\n\nA4: Study 1 and Study 2 employed contrastive search at the decoding stage to address the issue of degeneration in language models. In contrast, our approach focuses on adjusting the final output probability distribution in the decoding stage through a distribution reconstruction method to improve the performance of CTG. Besides, our approach involves modifying the probability distribution at the decoding stage before sampling, while these two studies improve decoding by designing a contrastive search objective during sampling. \n\nIn summary, the main differences between our paper and there two works are as follows:\n\n1. Different task objectives: These two works aim to address the issue of language model degeneration, while our work focuses on tackling the decoding-time CTG Attribute Collapse problem.\n2. Different decoding stage processing: These two works handle the issue during the sampling stage, whereas our work involves processing the probability distribution before sampling.\n\nHowever, regardless of the differences, we think that these two works provide valuable insights and innovations in the decoding stage, and we will discuss their contributions in the related works section of the revised version of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper is well-written and easy-to-follow.\n* The identified attribute collapse problem is interesting and well-motivates this research piece.\n* Good coverage of baseline methods and evaluated benchmarks.\n* Sufficient ablations and human evaluations are provided.", "reasons_to_reject": "* The detailed guidelines for human evaluations are missing.\n* The author(s) should measure the inter-annotator agreement score to better justify the provided experimental results.\n* Some key descriptions are missing. When conducting topic control experiments, there are five topics. Then, for each topic {a}, how to select the topics to constitute the complementary attribute {a^{\\prime}}?", "questions_to_authors": "Please see the section of weaknesses.", "paper_topic_and_main_contributions": "This paper investigates the problem of controlled text generation and the author(s) identify, for the first time, the attribute collapse problem --- when the control strength is strong, the fluency of generated text drops drastically. To address this problem, the author(s) propose an inference-time decoding approach --- air decoding which balances the distribution obtained by (1) normal language model; (2) language model with desired attribute; (3) language model with anti-desired attribute. Extensive experiments and ablations are performed on three sets of benchmarks and the results verify the advantages of the proposed approach.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {"Missing_References": "Two recent studies about contrastive-based decoding procedure might be worth discussing in related works.\n\n[1] A Contrastive Framework for Neural Text Generation --- NeurIPS 2022\n\n[2] Contrastive Search Is What You Need For Neural Text Generation --- TMLR 2023"}}, {"review_id": "SgI5VrMQtz", "reviewer": "Reviewer_Afd2", "summary": "", "questions": "1. This work utilizes both positive and negative PC-LMs to provide attribute distributions. As in Table 3, this work performs marginally for the Detoxification task which requires negative control to the language model. The question is 'Does the distribution reconstruction method weaken the negative control ability?'", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and suggestions that will help make our article clearer. We summarize your main concerns as follows:\n> **Q1: The distribution reconstruction method may be too straightforward which only smooths the attribute distributions. Reconstruct distribution can be better with more theoretical support.**\n\nA1: In decoding-time CTG approaches, we have identified the issue of Attribute Collapse for the first time. We have undertaken preliminary efforts to address this problem through a relatively straightforward yet intuitive method. Notably, this approach has demonstrated effective results across multiple tasks.\n\nThe attribute distribution reconstruction method we propose involves applying regularization to the next token probability distribution obtained from pos and neg PC-LMs. The purpose of this approach is to compress the weight difference between high and low probability tokens in the probability distribution while maintaining the relative order of token probability weights and the experimental results indicate the direct application of this method is logically reasonable.\n\nIndeed, our method does not explicitly analyze the extent to which this regularization approach can enhance the attribute distribution or determine the optimal attribute distribution pattern. We acknowledge that there is room for further research in these aspects. In our future work, we intend to delve deeper into these questions and explore alternative forms of the distribution reconstruction method.\n\n$\\quad$\n\n> **Q2: As mentioned in the related work, Gu et al. (2022a) study a similar problem of the trade-off between control strength and generation fluency. It may be important to provide a comparison with this work**\n\nA2: Gu introduced a regulator to dynamically change the control strength at different decoding positions, while our method takes a different approach by addressing the imbalance in attribute distribution and designing an attribute reconstruction method to improve the performance of CTG. **In general, the difference between our work and Gu's lies in the approach taken to improve generation quality. Gu achieves this by varying the control strength at different decoding positions, while our work focuses on modifying the output probability distribution at the decoding stage.**\n\nFollowing Gu, we selected two strong baselines, DExpert and GeDi, and conducted experiments on sentiment control and text detoxification tasks based on the code provided by Gu. The results are as follows:\n\n-  Sentiment Control \n| Method | Acc | PPL $\\downarrow$ | Dist-1 | Dist-2 | Dist-3 |\n|-------|:-------:|:-------:|:-------:|:-------:|:-------:|\n|  GeDi (baseline)  |   94.23   |   169.86   | 0.15 | 0.53 | 0.74 |\n|   GeDi + T (Gu)   |   89.07   |   146.77   |  0.16 | 0.64 | 0.84 |\n| GeDi + H (Gu) | 91.80 | 112.77 | 0.17 | 0.67 | 0.86 |\n|DExpert (baseline) | 94.74 | 51.99 | 0.16 | 0.65 | 0.85 |\n| DExpert + T (Gu) | 94.13 | 59.44 | 0.15 | 0.63 | 0.85 |\n|DExpert + H (Gu) | 94.87 | 57.89 | 0.14 | 0.62 | 0.84 |\n| Air-Decoding (ours) | **96.82** | **26.66** | 0.13 | 0.55 | 0.78 |\n\n- Text Detoxification\n| Method | Tox. $\\downarrow$ | PPL $\\downarrow$ | Dist-1 | Dist-2 | Dist-3|\n|-------|:-------:|:-------:|:-------:|:-------:|:-------:|\n|DExpert (baseline) | 20.0 | 58.06 | 0.08 | 0.48 | 0.78 |\n| DExpert + H (Gu) | 22.4 | 56.55 | 0.07 | 0.48 | 0.79 |\n| Air-Decoding (ours) | **18.5** | **48.29** | 0.07 | 0.44 | 0.74|\n\nwhere \"+ H\" indicates the usage of a heuristic regulator, while \"+ T\" indicates the usage of a trainable regulator. (**In the text detoxification task, only DExpert + H is included because Gu conducted experiments only on this setting**.)\n\nFrom the experimental results, our approach outperforms Gu's method in both sentiment control and text detoxification tasks.  Furthermore, it can be observed that under high control strength, the proposed regulator in Gu's work does not show significant effects, and the generated text still lacks fluency. In contrast, our method maintains a higher level of text fluency.\n\n$\\quad$\n\n> **Q3: This work utilizes both positive and negative PC-LMs to provide attribute distributions. As in Table 3, this work performs marginally for the Detoxification task which requires negative control to the language model. The question is 'Does the distribution reconstruction method weaken the negative control ability?'**\n\nA3: Attribute reconstruction method does not weaken negative control ability.\n\nFirstly, in the same two-class sentiment control task, the performance of positive sentiment accuracy and negative sentiment accuracy under different control strengths ($\\omega$) is comparable. The results are shown in the table below, which demonstrates that the distribution reconstruction method does not weaken the negative control ability.\n\n|Attribtue / $\\omega$| 20.0 | 40.0 | 60.0 | 80.0 | 100.0 | 120.0 | 140.0 |\n|-------|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|\n|Negative | 79.97 | 90.11 | 92.14 | 94.52 | 95.07 | 95.91 | 97.29 |\n|Positive | 77.81 | 87.26 | 90.63 | 93.32 | 94.09 | 94.53 | 96.35 |\n\nIn addition, we believe that the performance limitation of the text detoxification task lies in the inherent nature of the task itself. For tasks like sentiment control and topic control, the objective is to increase the number of tokens related to the desired attribute in the generated text to enhance the control effect. However, in the text detoxification task, the objective is to reduce the number of toxic tokens in the generated text.  **In our method, we consider the non-toxic attribute as the desired attribute, aiming to increase the number of non-toxic tokens in the generated text, rather than directly reducing the number of toxic tokens.** This difference in objective leads to slightly inferior results in the text detoxification task.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper studies an interesting problem named Attribute Collapse that fluency of the generated text will rapidly decrease when the control strength increases to a certain critical value.\n2. This paper provides a useful solution to reconstruct the attribute distributions before biasing the language model's distribution.", "reasons_to_reject": "1. The distribution reconstruction method may be too straightforward which only smooths the attribute distributions. Reconstruct distribution can be better with more theoretical support.\n2. As mentioned in related work, Gu et al. (2022a) study a similar problem of the trade-off between control strength and generation fluency. It may be important to provide a comparison with this work.", "questions_to_authors": "1. This work utilizes both positive and negative PC-LMs to provide attribute distributions. As in Table 3, this work performs marginally for the Detoxification task which requires negative control to the language model. The question is 'Does the distribution reconstruction method weaken the negative control ability?'", "paper_topic_and_main_contributions": "This paper focuses on the task of Controllable Text Generation, which biases the generated text with desired attributes on decoding time.\nThey observe the phenomenon of Attribute Collapse and develop a corresponding framework to alleviate the problem.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "cYojylT4M0", "reviewer": "Reviewer_pdMZ", "summary": "", "questions": "1. Could you presents Attribute Collapse generated sentence examples?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and suggestions that will help make our article clearer. We summarize your main concerns as follows:\n\n> **Q1: The paper highlights lightweight and low-resource manner in abstract and their contributions. However, they do not provide criteria for lightweight and low-resource, and they do not compare with other baselines. Furthermore, they have not clearly defined what these keywords mean.**\n\nA1: We define \"lightweight\" as the small size of the model's number of trainable parameters and define \"low-resource\" as the small number of samples used to train the model. The table below shows the comparison between our model and other baselines regarding the number of trainable parameters and training samples based on the GPT2-medium in the sentiment control task. \n\n|Model | Trainable Parameters | Percent Trainable | Training Samples | Acc | PPL $\\downarrow$ |\n|-------|:-------:|:-------:|:-------:|:-------:|:-------:|\n|Pre-Tuning | 51.3M | 14.87% | 10K | 62.15 | 38.49 |\n|Con Prefixes | 51.3M | 14.87% | 10K | 75.66 | 35.32 |\n|Discup | 14.7M | 4.26% | 10K | 95.20 | 39.14 |\n|GeDi | 345M | 100% | 11.25K | 94.23 | 169.86 | \n| DExpert | 690M | 200% | 8K | 94.74 | 51.99 |\n| Air-Decoding (ours) | 51.3M | 14.87% | 10K | 96.82 | 26.66 |\n\nFrom the table, it is evident that our method exhibits a clear advantage over the traditional decoding-time CTG method in terms of trainable parameters. While we maintain a comparable level of training sample count with other baselines. However, as depicted in Figure 5 of the paper, our method can still achieve high performance with a smaller number of training samples. Similar comparisons hold true for the other two tasks. We will add the aforementioned definitions and the table above to the revised version of the paper.\n(We did not find the training samples for PPLM, so it was not included in the table.)\n\n$\\quad$\n\n> **Q2: The paper explains attribute collapse problem only with PPL and accuracy curves. The authors argue that one of the contributions of this paper is the discovery of this problem, but it is not appropriate to explain this problem with only a few metrics. It would be more effective to present CTG results for the model where this problem occurs. If it is deemed inappropriate in the text to describe the CTG results, at least the results indicating this problem should be provided in the appendix.**\n\nA2: Due to the Attribute Collapse issue only present in the decoding-time CTG approach, in the table below, we have selected specific generated examples from GeDi and DExpert in the sentiment control task. We will also add this table along with the results of the topic control and text detoxification tasks to the appendix in the revised version of the paper. (The given prompts are in bold.)\n\n|Model | Attribute | Generated Sentence |\n|-------|:-------:|-------|\n|GeDi | Positive | **Once upon a time** flourished shone cardinal laurel.artney honouring the birthday of the Blossom of theesprit birthday action Date Harmony Meeting Graceland blues Graceland blues Gale principe blues brid breeforts Graceland bluesalm enchantively\\ufffd princi | \n| GeDi | Negative | **Once upon a time** Screenshot worse limding worse imbaferraining lab ratsatche worse scient inept idiotswrap worse Ness decompurseswraa convert SNAPIZEwra Zombie Trogviciflowerwra Swiss cheese hands Trog slaves wearing pool waste tamp domain dc DrainFeatureonut | \n|DExpert | Positive | **The book** taps vividly vividly into the lives of Tibetan Buddhic tabrasadhayangas, meditation teachers widely practiced both in Tibet and China today. In their homes and practice venues across China and Tibet mountains, teachers express the wonders of their cultures |\n|DExpert | Negative | **The book** sucks. 0 BAD!!Disclaimer CONTENT List of errors waste of time Stupidity stupidity waste of effort stupid books Stupid waste of time wastes of my time waste of my time waste of your time waste of yours Stupid waste of your time | \n\nFrom the table, it can be observed that sentences with Attribute Collapse occurs exhibit repetition of attribute tokens or the presence of many bad tokens.\n\n$\\quad$\n\n> **Q3: Similar to 2, the effectiveness of Attribute Distribution Reconstruction was compared to ablation study with only a few metrics, but since it is the first paper to present the AC problem, I think the actual CTG results should also be provided.**\n\nA3: The following table presents specific generated examples of Air-decoding with and without distribution reconstruction in the sentiment control task. We will also add this table along with the results of the topic control and text detoxification tasks to the appendix in the revised version of the paper. (The given prompts are in bold.)\n\n|Model | Attribute | Generated Sentence |\n|-------|:-------:|-------|\n|Air-Decoding/w.o. reconstruction | Positive | **The movie** works! Read My Lips Review Interview with Barbara Galliak delivers! My Big Fat Greek Wedding nicely improves upon its usual predecessor with gorgeous depth and integrity that speak volumes for this wondrous testament to living harmoniously with living love! Read |\n|Air-Decoding/w.o. reconstruction | Negative | **The movie** failed worse even worse THAN ME. 1. The writing 2. The acting 3. The acting wasted needless hours wasted wasted wasting needless amounts of wasted time waste of wasted money wasting wasted money wasted waste of time waste wasted wasted wasted wasted |\n| Air-Decoding/w. reconstruction | Positive | **The movie** The Matrix is one of my all time favorite movies and is a must see for all children and adults alike. It is the story where Neo steps into the Matrix to alter the world’s history and the effects that are wrought by his actions |\n|Air-Decoding/w. reconstruction | Negative | **The movie** seems bad. The actors and crew look like they've been drinking. The sound effects are boring, the story looks like it tells you to write more. The characters are incredibly bland. The writing inherently boring, because you can |\n\nThe results of the ablation experiments also exhibit a notable issue of attribute stacking. In contrast, the results with reconstruction are significantly smoother and more logically coherent.\n\n$\\quad$\n\n> **Q4: While it may not be possible to make a perfect comparison due to the differences in data, and evaluation metrics between [1] and Air-decoding, the language model (LM) metric proposed in the [1] is highly impressive. The LM metric is directly related to PPL, which means that excellent performance in the LM ([2, 3]) metric implies excellent performance in PPL as well. Although this paper demonstrates high performance, claiming to achieve the SOTA performance may require a comparison with papers like [1].**\n\n> **[1] Diffusion-LM Improves Controllable Text Generation (2022), Li et al\n[2] Training language GANs from Scratch (2019), Cyprien de Masson d'Autume et al\n[3] Language GANs Falling Short, Caccia et al.**\n\nA4: According to the publicly available code provided in [1], we trained a Diffusion-LM with the roc_story dataset and other relevant experimental hyperparameter settings, and we used the trained Diffusion-LM to generate sentences. In total, we obtained 2549 generated sentences. We compared the fluency of the results from Diffusion-LM with our method. Based on the reviewer's suggestion, we used the LM-metric from [1] to evaluate the fluency of the text. \n\n**The LM-metric from [1] refers to the perplexity calculated by a carefully fine-tuned GPT-2 model and the fine-tuned GPT-2 model is called the teacher LM**. Since [1] does not provide the dataset for fine-tuning the GPT-2 model, in our experiments, we used the IMDB, AGNews, Jigsaw datasets which are used in our paper as well as the roc_story dataset used in [1] to fine-tune four teacher LMs based on the GPT-2 large model. All the datasets have a size of 10K, and the training hyperparameters are kept consistent. As a point of comparison, we also used the GPT-2 large model without fine-tuning as a teacher LM in our experiments. These five teacher LMs are denoted as follows:\n\n1. GPT2-L-IMDB: fine-tuned using the IMDB dataset.\n2. GPT2-L-AGNews: fine-tuned using the AGNews dataset.\n3. GPT2-L-Jigsaw: fine-tuned using the Jigsaw dataset.\n4. GPT2-L-roc_story: fine-tuned using the roc_story dataset.\n5. GPT2-L-raw (without fine-tuning)\n\n**Since the results from Diffusion-LM are not specific to any of the tasks in our paper, we compared the results from Diffusion-LM with the results from all our three tasks.** The final results are shown in the following table. The model names in the first row of the table represent the perplexity computed using these teacher LMs.\n\n| Model | GPT2-L-IMDB | GPT2-L-AGNews | GPT2-L-Jigsaw | GPT2-L-roc_story | GPT2-L-raw|\n|-------|:-------:|:-------:|:-------:|:-------:|:-------:|\n|Diffusion-LM | 73.24 | 98.57 | 74.50 | 79.88 | 63.35 |\n| Air-Decoding (sentiment) | **34.40** | **45.55** | **41.31** | **53.59** | **26.66** |\n|Air-Decoding (topic) | 47.20 | 54.12 | 49.95 | 61.80 | 31.18 |\n|Air-Decoding (detoxification) | 70.92 | 81.50 | 74.91 | 91.77 | 48.29 |\n\nFrom the experimental results, it can be observed that among all the teacher LMs, Air-Decoding (sentiment) performs the best in terms of fluency. Additionally, both Air-Decoding (sentiment) and Air-Decoding (topic) exhibit higher fluency than Diffusion-LM among all the teacher LMs. Only when GPT2-L-roc_story and GPT2-L-Jigsaw are selected as the teacher LMs, Diffusion-LM shows slightly better fluency than Air-Decoding (detoxification). **The aforementioned experiments provide sufficient evidence to demonstrate that Air-Decoding outperforms Diffusion-LM in terms of generating fluent text.**\n\n$\\quad$\n\n> **Q5: Could you present Attribute Collapse generated sentence examples?**\n\nA5: Please refer to the Answer 2 and 3.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper first defines Attribute Collapse problem during controllable text generation (CTG).\n2. The paper suggests quite novel methods of Attribute Distribution Reconstruction to reduce unbalanced problems. By adopting this method, air-decoding increase CTG performance.\n3. The paper is well-written and easy to follow. Also, they provide some calculation examples for ease of understanding.", "reasons_to_reject": "1. The paper highlights lightweight and low-resource manner in abstract and their contributions. However, they do not provide criteria for lightweight and low-resource, and they do not compare with other baselines. Furthermore, they have not clearly defined what these keywords mean.\n2. The paper explains attribute collapse problem only with PPL and accuracy curves. The authors argue that one of the contributions of this paper is the discovery of this problem, but it is not appropriate to explain this problem with only a few metrics. It would be more effective to present CTG results for the model where this problem occurs. If it is deemed inappropriate in the text to describe the CTG results, at least the results indicating this problem should be provided in the appendix.\n3. Similar to 2, the effectiveness of Attribute Distribution Reconstruction was compared to ablation study with only a few metrics, but since it is the first paper to present the AC problem, I think the actual CTG results should also be provided.\n4.While it may not be possible to make a perfect comparison due to the differences in data, and evaluation metrics between [1] and Air-decoding, the language model (LM) metric proposed in the [1] is highly impressive. The LM metric is directly related to PPL, which means that excellent performance in the LM ([2, 3]) metric implies excellent performance in PPL as well. Although this paper demonstrates high performance, claiming to achieve the SOTA performance may require a comparison with papers like [1].\n\n\n[1] Diffusion-LM Improves Controllable Text Generation (2022), Li et al\n[2] Training language GANs from Scratch (2019), Cyprien de Masson d'Autume et al\n[3] Language GANs Falling Short, Caccia et al.", "questions_to_authors": "1. Could you presents Attribute Collapse generated sentence examples?", "paper_topic_and_main_contributions": "The authors found Attribute Collapse problem during controllable text generation (CTG). To address issue, the authors suggested air-decoding lightweight methods which alleviate the attribute collapse with low-resource manner. Also, the paper introduce prefix-tuning and Attribute Distribution Reconstruction methods to balance the probability distribution results. Lastly, air-decoding shows quite high results compared to the other baselines.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: **Summary**: The reviewers have reached a consensus that the paper presents a novel and effective approach to an important problem of CTG. The reviewers also have mentioned that the paper is well-written and easy to understand. Reviewer yX2p provided very limited views and didn't respond to the rebuttal and any of authors and AC's ask, so AC is weighing more on the other reviews. One reviewer asked about more details of human evaluations, and the authors provided comprehensive response to them. Also, some reviewers asked more baseline comparisons, and the authors provided multiple different comparison studies in the rebuttal and showed that the proposed method was more effective. Lastly, a reviewer asked about actual generation examples, and the authors provided them accordingly.

**Reviewers' recommendations**: Except for the reviewer yX2p, the other reviewer rated the 'soundness' of the paper as 4:'strong', therefore there is a consensus that the paper is very soundly written. In terms of excitement, it's in between two 4:'strong' and two 3:'ambivalent'. However, by weighing more on other reviews than yX2p's, it's more on the exciting side.
Decision: Accept-Main