An Extensible Plug-and-Play Method for Multi-Aspect Controllable Text Generation
Year: 2023

Abstract:
Recently, multi-aspect controllable text generation that controls the generated text in multiple aspects (e.g., sentiment, topic, and keywords) has attracted increasing attention. Although methods based on parameter efficient tuning like prefix-tuning could achieve multi-aspect controlling in a plug-and-play way, the mutual interference of multiple prefixes leads to significant degeneration of constraints and limits their extensibility to training-time unseen aspect combinations. In this work, we provide a theoretical lower bound for the interference and empirically found that the interference grows with the number of layers where prefixes are inserted. Based on these analyses, we propose using trainable gates to normalize the intervention of prefixes to restrain the growing interference. As a result, controlling training-time unseen combinations of aspects can be realized by simply concatenating corresponding plugins such that new constraints can be extended at a lower cost. In addition, we propose a unified way to process both categorical and free-form constraints. Experiments on text generation and machine translation demonstrate the superiority of our approach over baselines on constraint accuracy, text quality, and extensibility.1

heading: 1 Introduction
text: Multi-aspect controllable text generation (MCTG), which aims at generating fluent text while satisfying multiple aspects of constraints simultaneously, has attracted increasing attention in recent years (Chan et al., 2021; Qian et al., 2022; Gu et al., 2022). To effectively control diverse aspects such as sentiment, topic, and detoxification, extensive efforts have been devoted to the task, including methods based on conditional generative model (Keskar
† indicates equal contribution. * Corresponding authors: M.Sun (sms@tsinghua.edu .cn) and Y.Liu (liuyang2011@tsinghua.edu.cn) 1The source code is available at https://github.com/ THUNLP-MT/PromptGating4MCTG
et al., 2019), decoding-time regulation (Lin and Riedl, 2021; Kumar et al., 2021), and parameter efficient tuning (Qian et al., 2022; Gu et al., 2022).
Despite their effectiveness, existing methods still suffer from low extensibility. Ideally, suppose a multi-aspect controllable text generation system has learned how to control sentiment, topic and keywords separately, it should be extensible to any combinations of the three aspects, e.g., generating a sports-themed sentence with negative sentiment containing keywords “New York” (see Figure 1). Moreover, an extensible system should also be easily extended to control new aspects in a plug-andplay way. However, it is non-trivial for existing methods to achieve this goal. Specifically, the dedicated conditional generative models (Keskar et al., 2019) mostly need to be trained from scratch or finetuned when facing new aspect combinations. The decoding-time regulation based methods (Lin and Riedl, 2021; Kumar et al., 2021) intervene in the probabilities of sentences by light-weight attribute classifiers or language models during inference, which significantly impairs text fluency when multiple distributions are interpolated. The parameter efficient tuning based methods (Qian et al., 2022; Gu et al., 2022) control aspects by inserting trainable prompts or prefixes into the model, referred to as plugins. By leveraging one plugin for each aspect, these methods can naturally work in a plug-and-play way, showing better potential to achieve high extensibility.
However, existing studies show that directly combining multiple plugins results in significantly lower controllability of the corresponding aspects than before combining (i.e., attribute degeneration) (Qian et al., 2022; Gu et al., 2022). Gu et al. (2022) argue that mutual interference of the plugins is the major reason for attribute degeneration, which is further justified by our theoretical and empirical analyses. Previous works alleviate the problem by introducing connectors to connect mul-
15233
tiple plugins (Yang et al., 2022), latent variables to represent the unsupervised aspects (Qian et al., 2022), or objectives to narrow the discrepancy of aspects (Gu et al., 2022). However, these methods require joint training of plugins and are designed for pre-defined closed-set constraints. In consequence, their extensibility is limited.
In this paper, we propose an extensible plug-andplay method, PROMPT GATING, for multi-aspect controllable text generation. We derive a theoretical lower bound for the interference of plugins and reveal that it accumulates with the increasing number of layers where prefixes are inserted. Based on these findings, we propose attaching trainable gates to the plugins, which normalize the interventions of plugins. As a result, the mutual interference has been significantly reduced such that the control of arbitrary combinations of aspects can be realized by simply concatenating the corresponding plugins. Thus, our method is both extensible and plug-andplay. Moreover, we represent the constraints of the aspects in textual form, which makes our method applicable not only to categorical aspects (e.g., sentiment) but also to free-form aspects (e.g., lexical constraint).
Our contributions are three-fold:
• We propose an extensible plug-and-play method, PROMPT GATING, for multi-aspect controllable text generation, which is able to control training-time unseen aspect combinations by simply concatenating plugins.
• We provide a theoretical lower bound along with empirical analyses for the mutual interference problem, which we believe will facilitate future research.
• Experiments show that our approach has lower mutual interference, leading to superiority over strong baselines on text quality, constraint accuracy, and extensibility.

heading: 2 Background
text: In this section, we illustrate the widely-used prefixtuning-based method (Qian et al., 2022; Gu et al., 2022; Yang et al., 2022) for multi-aspect controllable text generation. Generally, prefix-tuning (Li and Liang, 2021) prepends light-weight continuous vectors to the multi-head attention sublayer of each Transformer layer (Vaswani et al., 2017):
H = Att ( Q, [ PK ;K ] , [ PV ;V ]) , (1)
where Att(·) is the attention function, Q are queries of inputs, K and V are activations of inputs, PK and PV are trainable prefixes, [·; ·] denotes the concatenation operation, H is the output of the attention sublayer. We use ϕ to denote the set of prefixes in all Transformer layers.
Specifically, for multi-aspect controllable text generation, we assume that there are N aspects of constraints. Due to the lack of multi-aspect labeled data, each set of prefixes, which usually represents a specific constraint (e.g., “positive” for the sentiment aspect), is trained on the corresponding single-aspect labeled data:
ϕ̂i = argmax ϕi
{ P (y|x;θ,ϕi) } , 1 ≤ i ≤ N, (2)
where θ are the fixed parameters of the pretrained model, y is the controlled target sentence, x is the
input sentence2, P (y|x;θ,ϕi) is the conditional probability of y, and ϕ̂i are the learned parameters of prefixes for the i-th aspect.
During inference, for a combination of multiple aspects, corresponding prefixes are selected and synthesized by either concatenating (Qian et al., 2022; Yang et al., 2022) or finding their intersection (Gu et al., 2022), and then the generation is conditioned on the synthesis. Without loss of generality, we take two aspects as an example. The conditioned probability can be represented as
P ( ŷ|x;θ, syn(ϕ̂1, ϕ̂2) ) , (3)
where syn(·) is a synthesize function, ŷ is the candidate sentence, ϕ̂1 and ϕ̂2 are two sets of prefixes corresponding to two aspects (e.g., “positive” for sentiment and “sports” for topic), respectively.
Although existing methods alleviate the mutual interference of prefixes by joint training (Qian et al., 2022; Gu et al., 2022; Yang et al., 2022), they are based on pre-defined closed-set constraints, which increases the overhead of extending a new constraint and thus limits extensibility. Thus, to maintain high extensibility, reducing mutual interference without joint training still remains a challenge.

heading: 3 Analyses on Mutual Interference
text: To alleviate mutual interference while maintaining extensibility, we conduct theoretical and empirical analyses. First, we provide a definition of mutual interference as follows.
Definition. Mutual interference (MI) is the interference between multiple plugins which are trained separately during training but are combined to guide the pretrained model simultaneously during inference (i.e., in the zero-shot setting). However, the exact interference is hard to analyze because of the complexity of deep neural networks. Intuitively, suppose multiple plugins are optimized simultaneously during training, which requires multi-aspect labeled data, their interference will be minimized because they have learned to work cooperatively under supervision (i.e., in the supervised setting). Therefore, we use the differences between the hidden states of the supervised and zero-shot settings
2Note that x is the source sentence or context for tasks like machine translation (Bahdanau et al., 2015) or summarization (Nallapati et al., 2016) and can be eliminated when it is not present.
to approximate the mutual interference of multiple plugins. Specifically, let ϕ̂i and ϕ̃i be the parameters of plugins learned from the single- and multi-aspect labeled data, respectively. Taking twoaspect controlling as an example, the output of a Transformer layer is given by H(x,ϕ1,ϕ2), where x is the layer input, then mutual interference can be defined as
MI ≈ ∥∥∥H(x, ϕ̂1, ϕ̂2)−H(x, ϕ̃1, ϕ̃2) ∥∥∥. (4)
Empirical Analysis. Then, as mutual interference has been defined as the norm of gap between hidden states in the zero-shot and supervised settings, we can empirically estimate it on the authentic dataset. By calculating the averaged norm on the Yelp dataset (Lample et al., 2019), we plot the variations of mutual interference with the number of Transformer layers for Prompt-tuning (Lester et al., 2021) and Prefix-tuning (Li and Liang, 2021) in Figure 2. We can find that the interference accumulates with insertions of trainable parameters. Moreover, the magnitude of mutual interference at the last Transformer layer (i.e., the 12-th layer in Figure 2) is consistent with the performance gap, which is the difference between the fulfillment of constraints in single- and multi-aspect settings (see Table 1). Meanwhile, too few trainable parameters cannot guide the pretrained model effectively. In summary, the key point for remaining effective in the zero-shot setting is restraining the growth of mutual interference (for a lower performance gap)
while providing sufficient trainable parameters (for better supervised performance).
Theoretical Analysis. Next, to find a way to alleviate mutual interference, we conducted a theoretical analysis.3 As a result, we found that the mutual interference, which is caused by the interactions in attention sublayers, has a theoretical lower bound4:
MI > α∥∆h1(x, ϕ̂1)∥+ β∥∆h2(x, ϕ̂2)∥, (5)
where 0 < α, β < 1, and ∥∆hi(x, ϕ̂i)∥ is a norm that is positively related to the magnitude of ϕ̂i. Moreover, the lower bound might accumulate with Transformer layers like in Figure 2. Intuitively, applying normalization (e.g., gates) to the parameters of the i-th plugin to reduce its magnitude will decrease the lower bound of mutual interference.

heading: 4 PROMPT GATING
text: We propose a novel approach that attaches trainable gates to the plugins, which alleviates the mutual interference of multiple plugins and makes the model highly extensible. Figure 3 shows the architecture of our approach. We first provide intuition in §4.1, then define our approach formally in §4.2.

heading: 4.1 Intuition
text: Although prefix-tuning provides sufficient interventions and avoids long-range dependencies by inserting continuous vectors into each attention sublayer, it suffers from the accumulation of mutual interference of these plugins (see §3). On the one hand, the vectors are inserted into the attention sublayer, where they interact with each other, which directly enhances mutual interference. On the other hand, the vectors are not normalized, which leads to a large lower bound of mutual interference (Eq. (5)). Intuitively, injecting the vectors in a position-wise manner will avoid direct interaction between them. Moreover, normalizing the vectors can limit the magnitude of the lower bound, which might decrease mutual interference. Therefore, we first propose attaching vectors outside the attention sublayer, which can be realized by appending trainable vectors to the output of the embedding layer and adding trainable vectors to the hidden states in each Transformer layer (see Figure 3). Then, trainable gates are applied to these hidden states to alleviate
3Please refer to Appendix A for more detail. 4For brevity, we show the lower bound in one head of attention (obtained by simplifying Eq. (13)), and a similar conclusion can be obtained on the multi-head attention (Eq. (14)).
mutual interference further. In this way, we expect our approach to restrain the growth of mutual interference while providing sufficient interventions.

heading: 4.2 Method
text: Prompting. We present our model in the order of forward propagation. To change how the trainable parameters are injected into the model, we first follow prompt-tuning (Lester et al., 2021) to append trainable prompts to the output of the embedding layer. Moreover, to make our model applicable not only to categorical aspects (e.g., sentiment) but also to free-form aspects (e.g., lexical constraint), we present the constraints of aspects in textual form and feed them to the model. When two aspects of constraints are required during inference, the model input is given by
H(0) = [ E(x); E(c1); E(c2);P (0) 1 ;P (0) 2 ] , (6)
where E(·) is the embedding function, and x is the source sentence for sequence-to-sequence generation like machine translation and can be eliminated
for text generation. c1 and c2 are textual form of constraints (e.g., “This is a positive review” for positive review generation, and “New York” for lexical constraint). P(0)1 ,P (0) 2 ∈ Rp×d are continuous prompts, where the right superscript (j) represents the j-th layer, p is the number of continuous vectors, and d is the dimension of hidden states. To avoid the discrepancy between training and inference in position, each textual sequence has its own position indexes starting from 1 and its own segment embedding (Devlin et al., 2019). Note that only one textual constraint and one set of trainable parameters are injected during training.
Gating. Then, the model input H(0) is fed to the encoder, where trainable gates are attached to the hidden states in a position-wise manner, which alleviates mutual interference as well as steers the model. Formally, A(j) = Self-Att(H(j−1)) is the output of the j-th attention sublayer, and it is normalized by the gates:
Ã(j) = [ A
(j) X ;σ
( G
(j) 1
) ⊙ ( A
(j) P1 +P (j) 1
) ;
σ ( G
(j) 2
) ⊙ ( A
(j) P2 +P (j) 2
)] , (7)
where A(j)X ∈ R(|x|+|c1|+|c2|)×d and A (j) Pi ∈ Rp×d are hidden states split from A(j), P(j)i ∈ Rp×d are trainable vectors that add to the hidden states, σ is the sigmoid(·) function and G(j)i ∈ Rp×d are trainable vectors. ⊙ denotes the Hadamard product and the normalized vectors σ(G(j)i ) serve as gates to selectively rescale the output of the attention sublayer in a position-wise manner and Ã(j) ∈ R(|x|+|c1|+|c2|+2p)×d is the result of the normalization. Next, the normalized output is fed to the feed-forward sublayer: H(j) = FFN(Ã(j)). Finally, the output of the last encoder layer is fed to a standard Transformer decoder to guide the generation.
Training & Inference. As shown in Figure 1, during training, each plugin (including prompts and gates) for a single aspect of constraints is attached to the pretrained generative model and optimized by corresponding single-aspect labeled data separately (refer to Eq. (2)). In contrast, during inference, the control of arbitrary combinations of aspects can be realized by simply concatenating the corresponding plugins (refer to Eq. (3)).
Moreover, our approach treats the training and inference processes for pre-existing and newly-
introduced constraints identically. The total training cost of N pre-existing aspects and M newlyadded aspects is O((N +M)C), where C denotes the cost of training on a single aspect. In this way, the cost of introducing new constraints is relatively low.

heading: 5 Experiments
text: We conducted experiments on two representative tasks in natural language generation, which are text generation and machine translation.

heading: 5.1 Multi-Aspect Controllable Text Generation
text: Dataset. Following previous work (Yang et al., 2022), we adopted the widely-used Yelp dataset (Lample et al., 2019), which contains restaurant reviews with the sentiment (positive and negative) and the topic (American, Mexican, and Asian) labels. To evaluate the extensibility of methods, we added two additional aspects of constraints: keywords (He, 2021) and tense (past and present) (Ficler and Goldberg, 2017), where their labels are automatically extracted from the reviews. Due to the page limit, please refer to Appendix B for more details about the experimental setup.
Evaluation. Following previous work, we adopted automatic and human evaluations for constraint accuracy and text quality (Lyu et al., 2021; Dathathri et al., 2019; Gu et al., 2022). Specifically, we finetuned two RoBERTa-based (Liu et al., 2019) classifiers for the evaluations of sentiment and topic. The tense accuracy was evaluated by the same tool adopted in the training set, and we used word-level Copy Success Rate (CSR) (Chen et al., 2020) to evaluate the lexical constraint. In addition, we used the perplexity (PPL) given by GPT-2medium (Radford et al., 2019) and averaged distinctness (Li et al., 2016) to evaluate the fluency and diversity of the generated text, respectively. For human evaluation, each sentence received a score of 1 to 5 on sentiment and topic relevance as well as fluency given by three evaluators. The final scores are averaged over three ratings.
Baselines. We compared our approach with several representative methods for multi-aspect controllable text generation:
• GEDI (Krause et al., 2021): a decoding-time regulation method that uses light-weight con-
ditional generative discriminator to guide pretrained models. The distributions given by multiple discriminators are normalized for controlling multiple aspects of target sentences.
• DIST. LENS (Gu et al., 2022): a prefix-tuningbased method that introduces an autoencoder and additional objectives to map several constraints of attributes to one latent space (i.e., joint training of prefixes). It finds the intersection of prefixes of multiple constraints during inference.
• PROMPT-TUNING (Lester et al., 2021): a pa-
rameter efficient method that appends continuous prompts to the model input. Multiple prompts are trained separately and are simply concatenated during inference.
• PREFIX-TUNING (Li and Liang, 2021): a parameter efficient method that appends continuous prefixes to the activations at attention sublayers. Multiple prefixes are trained separately and are simply concatenated during inference.
• TAILOR (Yang et al., 2022): a prompt-tuningbased method that further modifies the attention mask and position indexes during inference to narrow the gap between training and inference.
Results. Table 1 shows the automatic evaluation on double-aspect controllable text generation. We demonstrate the averaged accuracies to represent the overall performance on satisfying multiple aspects of constraints. Furthermore, we provide the performance gap between double- and single-
aspect settings to represent the ability to combine multiple plugins in a zero-shot manner. Although GEDI achieves the highest scores on sentiment accuracy and distinctness, its perplexity explodes, and its tense accuracy is significantly decreased, which can be attributed to the interpolation of multiple discriminators. As PROMPT-TUNING does not have sufficient trainable parameters, it performs poorly on constraint accuracies. However, it has a relatively minor performance gap due to only inserting vectors once. PREFIX-TUNING suffers from severe mutual interference because of the insertions in all Transformer layers, leading to poor performance on either constraint accuracies or perplexity. Compared with PREFIX-TUNING, DIST. LENS has better constraint accuracies and lower performance gaps because of the joint training of prefixes. We found that DIST. LENS is sensitive to constraint distributions in the training set because it attempts to find the intersection of multiple constraints. Our approach (PROMPT GATING) achieves the highest constraint accuracies, lowest perplexity and a relatively small performance gap while our plugins are trained separately.
Table 2 shows the extensibility of the methods. When extended from double-aspect to triple-aspect, DIST. LENS has to be retrained because of its joint training strategy. In contrast, our approach and PREFIX-TUNING only need to train one plugin, then combine plugins and plug them into the pretrained model. Unfortunately, when extended from triple-aspect to quadruple-aspect, as plugins of PREFIX-TUNING badly interfere with each other, its ability to control multiple aspects significantly degenerates. However, our approach has a slight performance gap with a relatively small training cost, revealing its high extensibility.
The human evaluation results are illustrated in
Table 3 with an inter-annotator agreement of 0.31 (Fleiss’ κ). Experiments indicate that our approach significantly outperforms both baselines with p < 0.01 on all three aspects, determined by paired bootstrap and t-test using a popular opensource tool (Dror et al., 2018)5. Unlike automatic evaluations, GEDI performs the worst in sentiment relevance. It can probably be attributed to the fact that GEDI often generates ambivalent-sentiment and non-fluent sentences, and human annotators tend to give low ratings to them. The other results are in line with automatically evaluated results.

heading: 5.2 Multi-Aspect Controllable Machine Translation
text: Dataset. To thoroughly compare our approach with baselines, we also adopted a sequence-tosequence generation task (i.e., machine translation (Bahdanau et al., 2015)). Experiments are conducted on the WMT14 German → English benchmark. We adopted three aspects of constraints in machine translation, and the labels are all automatically obtained from target sentences. We use keywords (Post and Vilar, 2018) and tense (Ficler and Goldberg, 2017) like in the text generation task to control translations. Specifically, we adopt French sentences with the same meaning as the German sources, which can be seen as an external knowledge to improve translation quality (Zoph and Knight, 2016), as the third constraint.
Evaluation. We adopted SACREBLEU6 (Post, 2018) to calculate BLEU scores (Papineni et al., 2002) to evaluate the translation quality. Similar to text generation (§5.1), we used CSR (Chen et al., 2020) and tense accuracy to evaluate lexical and
5https://github.com/rtmdrr/testSignificanceNLP 6The signature is “BLEU+case.mixed+numrefs.1+smooth
.exp+tok.13a+version.2.0.0”.
temporal constraints, respectively.
Baselines. Besides PROMPT-TUNING (Lester et al., 2021) and PREFIX-TUNING (Li and Liang, 2021) (§5.1), we adopted another two representative parameter efficient methods as baselines:
• LORA (Hu et al., 2021): a method that adds trainable rank decomposition matrices into attention sublayers.
• PARALLEL ADAPTER (Houlsby et al., 2019): a method that parallelly inserts feed-forward sublayers between pre-trained sublayers.
Similar to PREFIX-TUNING, for both LORA and PARALLEL ADAPTER, each plugin is trained separately, and multiple plugins are simply concatenated for multi-aspect setting during inference.
Results. Table 4 shows the results on controllable machine translation. Unlike text generation, constraints in machine translation do not merely contain attribute-based constraints. Therefore, methods specially designed for attribute-based constraints cannot be applied to this task. Surprisingly, PROMPT-TUNING achieves the highest constraint accuracies and translation quality among baselines because it largely retains the capabilities of plugins to satisfy constraints. PREFIX-TUNING faces the severe degeneration of both accuracies of constraints and BLEU scores, which might be attributed to the more complicated model structure in machine translation than text generation. Our approach outperforms all baselines in machine translation, and the consistent superiorities on both tasks show its generalizability.

heading: 5.3 Analysis
text: Mutual Interference. Similar to empirical analysis on mutual interference for PREFIX-TUNING
and PROMPT-TUNING (see §3), we also plotted the variation of the mutual interference with the number of injections of our approach in Figure 2. With the gates to selectively rescale the interventions of plugins, the growth of interference is restrained.
Ablation Study. Table 5 shows the ablation study and comparison with the variant of our approach. According to the performance gaps corresponding to the changes, we can find that the textual context of constraints slightly affects the constraint accuracies, and the normalization of the trainable gates is a key point for good performance. Moreover, the trainable gates should be placed where the interactions have just happened (i.e., after attention sublayers). Please refer to Appendix C and D for more results, analyses, and cases.

heading: 6 Related Work
text: Multi-aspect controllable text generation (MCTG) (Qian et al., 2022; Yang et al., 2022; Gu et al., 2022) that simultaneously satisfies multiple constraints is a challenging task for which highly extensible methods make more practical sense. Approaches to it can be roughly divided into the following three categories.
Dedicated Model. The dedicated conditional generative models (Keskar et al., 2019; Dou et al., 2021; Huang et al., 2021; Chen et al., 2020) can accept multiple constraints by training from scratch or full-parameter finetuning on the multi-aspect labeled data. However, the multi-aspect labeled data is hard to obtain, and the constraints that can be satisfied are already determined during training. Thus it is usually too expensive to apply dedicated models to MCTG.
Decoding-Time Regulation. Although multiaspect controlling can be achieved by interpolating distributions of multiple discriminators (Dathathri et al., 2019; Chen et al., 2021; Krause et al., 2021; Lin and Riedl, 2021) or optimizing towards multiple objectives (Qin et al., 2022; Kumar et al., 2021), they usually significantly impair text fluency because of the intervention in the decoding stage (Gu et al., 2022).
Parameter Efficient Tuning. Unlike the above two branches, PET introduces plugins trained with fixed pretrained models for generating required text (Li and Liang, 2021; Lester et al., 2021; Wang et al., 2022). Because of its potential to achieve
high extensibility in a plug-and-plug manner, our work also falls in this line. However, when multiple constraints are required, joint training of plugins is introduced to alleviate the mutual interference of plugins (Chan et al., 2021; Qian et al., 2022; Yang et al., 2022; Gu et al., 2022), which hurts extensibility. Differently, our work aims at reducing mutual interference while maintaining separate training. Similar to our work, Yang et al. (2022) proposes preventing two prompts from interactions in attention layers by modifying attention masks. Nevertheless, their method like prompt-tuning (Lester et al., 2021) only introduces trainable parameters to the model input, leading to insufficient trainable parameters and dissatisfied constraints. In contrast, we propose a novel PET method that attaches trainable gates to the pretrained model, alleviating mutual interference while providing sufficient interventions, leading to both desired extensibility and effectiveness.

heading: 7 Conclusion
text: In summary, we propose an extensible plug-andplay method for multi-aspect controllable text generation. By replacing trainable prefixes with trainable prompts and gates, our approach alleviates the mutual interference of multiple plugins while providing sufficient interventions. Experiments on text generation and machine translation show its superiorities over baselines on the cost of extending to new combinations of aspects, the fulfillment of constraints, and text fluency.
Limitations
First, although our approach and existing methods for controllable text generation can improve the constraint accuracies, they are currently unable to achieve 100% accuracies in the vast majority of aspects (e.g., sentiment or topic). This makes them not yet applicable in scenarios with requirements of 100% control fulfillment. Second, there is still a gap between the automatic and human evaluation of text generation, which makes there a trade-off between precision and efficiency in the evaluation of controllable text generation. Third, although our approach reduces the mutual interference of plugins so that multiple plugins can be combined at a relatively small cost (a decrease in constraint accuracy), this cost will not be zero, which puts an upper limit on the number of plugins can be applied simultaneously. Fortunately, for controllable text
generation, the number of controls applied simultaneously is generally not too large (e.g., four or five aspects).
Ethics Statement
Since the text generation model is trained on data collected from the web and often not thoroughly cleaned, it can generate offensive or toxic text. We must state that the texts generated by our approach do not represent our opinion. To alleviate these issues, we can take detoxification and politeness as the default aspects of constraints in our multiaspect controllable method.

heading: Acknowledgments
text: This work is supported by the National Key R&D Program of China (2022ZD0160502), the National Natural Science Foundation of China (No. 61925601, 62276152), the National Social Science Fund of China (20&ZD279), and a grant from the Guoqiang Institute, Tsinghua University. We thank Kaiyu Huang, Fuchao Wei, Yuanhang Zheng and all the anonymous reviewers for their valuable comments and suggestions on this work, as well as all the volunteers who participated in the human evaluation.

heading: A Theoretical Analysis
text: In this section, we theoretically analyze mutual interference (MI) and derive a lower bound of MI for prefix-tuning (Li and Liang, 2021). As Feed Forward and Layernorm sublayers are position-wise operations (Vaswani et al., 2017) which would not introduce the interference of plugins, we focus on analyzing the multi-head attention (MHA) sublayers.
According to the previous study (He et al., 2021), the output of a single head of attention with prefixes of the i-th plugin, which is represented by hi, could be described as
hi = λ(xi)hi + ( 1− λ(xi) ) ∆hi
= sihi + ti∆hi, (8)
where hi denotes the original output of the pretrained generative model with xi as input. λ(xi) is a scalar related to the attention weights, where λ(xi) = si = 1 − ti ∈ (0, 1). In addition, ∆hi is an offset determined by the i-th plugin, and its magnitude is positively correlated with the magnitude of ϕi, where ϕi is the set of parameters of the i-th plugin.
Following the pattern above, when the i-th and jth plugins are inserted at the same time, the output of the head (i.e., hi,j) turns to be
hi,j = γhi,j + α∆hi + β∆hj , (9)
where hi,j is the output of pretrained generative model, and α, β, γ ∈ (0, 1), α < ti, β < tj , γ < si, γ < sj . Similarly, ∆hi and ∆hj are determined by the i-th and j-th plugins.
According to the definition in Eq. (4), let h̃i,j and ĥi,j be the outputs like hi,j after training on multi- and single-aspect labeled data, respectively. The mutual interference of two plugins in a single head (i.e., MIs) can be measured by the norm of the gap between outputs under supervised and zeroshot inference:
MIs = ∥∥h̃i,j − ĥi,j ∥∥
= ∥∥h̃i,j − (γhi,j + α∆ĥi + β∆ĥj) ∥∥ ≥ ∥∥h̃i,j − γhi,j ∥∥− ∥∥α∆ĥi + β∆ĥj
∥∥, (10)
where ∆ĥi and ∆ĥj correspond to offsets that plugins are trained on single-aspect labeled data.
Considering that the intervention caused by two plugins simultaneously should larger than the sum
of two interventions caused by two plugins respectively because of the interaction between two plugins, we assume that there is ∥∥h̃i,j − hi,j ∥∥ > ∥∥ĥi − hi ∥∥+ ∥∥ĥj − hj
∥∥. (11) Based on this, we can derive
MIs > ∥∥ĥi − γhi ∥∥+ ∥∥ĥj − γhj ∥∥
− ∥∥α∆ĥi + β∆ĥj ∥∥. (12)
Given that si > γ, sj > γ, and ĥi = sihi + ti∆ĥi (Eq. (8)), MIs satisfies
MIs > ∥∥ĥi − sihi ∥∥+ ∥∥ĥj − sjhj ∥∥
− ∥∥α∆ĥi + β∆ĥj ∥∥
= ∥ti∆ĥi∥+ ∥tj∆ĥj∥ − ∥α∆ĥi + β∆ĥj∥ ≥ (ti − α)∥∆ĥi∥+ (tj − β)∥∆ĥj∥, (13)
where 1 > ti − α > 0 and 1 > tj − β > 0. Therefore, the mutual interference of two plugins in a single head has a positive lower bound, and it is positively correlated with the magnitude of ϕ̂i and ϕ̂j .
To step further, we derive the lower bound of MI in the multi-head scenario. Assume that K denotes the number of heads, Wo denotes the fixed output projection matrix in the MHA, Wo = QoRo is the QR-decomposition format of Wo, λ̂o is the average of absolute eigenvalues. Specifically, ĥki,j and h̃ k i,j denotes ĥi,j and h̃i,j in the k-th head, respectively. Then, the lower bound of MI in MHA (i.e., MIm) can be derived as (viewing Ro as a diagonal matrix for simplicity)
MIm = ∥∥concat(h̃ki,j − ĥki,j)Kk=1Wo ∥∥
= ∥∥concat(h̃ki,j − ĥki,j)Kk=1QoRo ∥∥ ≈ λ̂o ∥∥concat(h̃ki,j − ĥki,j)Kk=1Qo ∥∥
= λ̂o
√√√√ K∑
k=1
∥∥h̃ki,j − ĥki,j ∥∥2
≥ λ̂o√ n
K∑
k=1
∥∥h̃ki,j − ĥki,j ∥∥
> λ̂o√ n
K∑
k=1
( (tki − αk)∥∆ĥki ∥
+ (tkj − βk)∥∆ĥkj ∥ ) , (14)
where 1 > tki − αk > 0 and 1 > tkj − βk > 0, and ∆ĥki and ∆ĥ k j are also positively correlated with the magnitude of ϕ̂i and ϕ̂j , respectively.
Therefore, the mutual interference of multiple plugins has a theoretical positive lower bound, which implies concatenating prefixes that are separately trained has an irreparable gap against supervised-trained prefixes. As a result, MI might accumulate along with the depth of the model, like in Figure 2. Intuitively, introducing gates, which contain trainable coefficients between 0 to 1, to ϕ̂i is helpful for decreasing the offsets in Eq. (14) and thus mutual interference.

heading: B Reproducibility
text: B.1 Data Preparation For text generation, we adopted the widely-used Yelp dataset7 (Lample et al., 2019), which contains restaurant reviews with sentiment (positive and negative) and topic (American, Mexican, and Asian) labels. Specifically, following previous work (Yang et al., 2022), we randomly sampled 30K/3K sentences for each attribute for training/validation while ensuring the balance of different attributes in the final dataset (Table 6). For evaluation, we sampled 25 sentences for each given textual prefix and combination of aspects. In addition, we eliminated the sentences rated 3 in sentiment. To evaluate the extensibility of methods, we added two additional aspects of constraints: keywords (He, 2021) and tense (past and present) (Ficler and Goldberg, 2017), where their labels are automatically extracted from the reviews. More precisely, we randomly extracted 1 to 3 words as keywords for each sentence (Post and Vilar, 2018), and the tenses of sentences are labeled by an open-source toolkit8 that is based on a POS tagger (Nguyen and Verspoor, 2018).
For machine translation, we adopted the WMT14 German → English benchmark9. Specifically, the training, validation, and test sets contain 4,500K, 3K, and 3K sentences, respectively. We adopted three aspects of constraints in machine translation, and they are all automatically obtained from target sentences. We use keywords (Post and Vilar, 2018) and tense (Ficler and Goldberg, 2017) like the text generation task to control translations. For the third constraint, we adopt French synonymous sentences as external knowledge, which is
7https://github.com/shrimai/ Style-Transfer-Through-Back-Translation
8https://github.com/ajitrajasekharan/simple_ tense_detector
9https://statmt.org/wmt14/translation-task. html
beneficial to disambiguation. Note that it does not directly control any attribute of translations but will improve the translation quality (Zoph and Knight, 2016). The French synonymous sentences are given by a Transformer-based English→French translation model (Vaswani et al., 2017).
B.2 Evaluation Metrics
For text generation, following previous work (Lyu et al., 2021; Dathathri et al., 2019; Gu et al., 2022), we adopted automatic and human evaluation for constraint accuracy and text quality. For the evaluation of sentiment and topic, we finetuned two RoBERTa-based (Liu et al., 2019) classifiers on the Yelp dataset. Specifically, we randomly over-sampled 1,500K/15K/15K sentences for training/validation/test set of topic and 1,380K/1K/1K sentences for training/validation/test set of sentiment. The F1 scores for sentiment and topic are 98.71 and 89.62, respectively. The same toolkit as training evaluated the accuracy of tense, and we used word-level Copy Success Rate (CSR) (Chen et al., 2020) to evaluate the lexical constraint. In addition, we used the perplexity (PPL) given by GPT-2medium (Radford et al., 2019) and averaged distinctness (Li et al., 2016) to evaluate the fluency and diversity of the generated text, respectively. Similar to previous work (Dathathri et al., 2019; Yang and Klein, 2021), we used 15 textual prefixes10 and asked models to start writing from them for each combination of constraints during inference. For human evaluation, each sentence received a score of 1 to 5 on sentiment and topic relevance as well as fluency given by three evaluators. The final scores are averaged over three ratings.
Specifically, the 15 textual prefixes are: “Once upon a time”,“The book”,“The chicken”,“The
10https://github.com/uber-research/PPLM
city”,“The country”,“The lake”,“The movie”,“The painting”,“The weather”,“The food”,“While this is happening”,“The pizza”,“The potato”,“The president of the country”,“The year is 1910.”.
For machine translation, we adopted SACREBLEU11 (Post, 2018) to evaluate the translation quality. Similar to text generation, we used CSR (Chen et al., 2020) and tense accuracy to evaluate lexical and tense constraints, respectively.
B.3 Model and Hyper-parameters As our approach has both encoder and decoder, we adopted BART-large12 (Lewis et al., 2020) for text
11The signature is “BLEU+case.mixed+numrefs.1+smooth .exp+tok.13a+version.2.0.0”.
12https://huggingface.co/facebook/bart-large
generation and mBART-large-cc2513 (Liu et al., 2020) for machine translation.
For GEDI (Krause et al., 2021), DIST. LENS (Gu et al., 2022), and TAILOR (Yang et al., 2022), we follow the settings in their paper. Specifically, we found that the weights for attribute balance and the number of candidates in the decoding stage of DIST. LENS significantly affect constraint accuracies. For the weights for attribute balance and the number of candidates, we searched in {0.1, 0.2, 0.5, 1, 1.5, 2,
13https://huggingface.co/facebook/ mbart-large-cc25
3, 4, 5, 8, 10, 25, 50} and {1, 2, 5, 10}, respectively. For the other methods, we demonstrate their hyperparameters in Table 7. Table 8 shows the number of trainable parameters of each method. The learning rates were determined by searching in {1× 10−5, 2×10−5, 4×10−5, 8×10−5, 1×10−4, 2×10−4, 4 × 10−4, 1 × 10−3} on the development set. In our approach, the textual contexts used for attributebased constraints (see §4.2) are:
• Sentiment: “This is a {} review.” for “positive/negative”.
• Topic: “The following is about {} food.” for “Asian/American/Mexican”.
• Tense: “The tense of this sentence is the {} tense.” for “past/present/future”, and “The tense of this sentence is undecided.” for sentences that do not have an explicit tense.
Note that our use of existing artifact(s) was consistent with their intended use. The source code of this work is available at https://github.com/ THUNLP-MT/PromptGating4MCTG and it was developed based on THUMT (Tan et al., 2020), an opensource toolkit for machine translation.

heading: C Experimental Results
text: C.1 More Analyses
Visualization of PROMPT GATING. To further investigate how our approach alleviates the mutual interference of plugins, we visualized the trainable parameters in PROMPT GATING. Specifically, we first extracted P(j)i and σ(G (j) i ) in Eq. (7) from each layer j for every single aspect. Then we calculated the average of σ(G(j)i ) and the L1 norm of P (j) i
over all the layers, which are represented by red and blue bars respectively in Figure 4 and Figure 5.
We can find that when the magnitude of P(j)i (i.e., trainable vectors added to hidden states) becomes larger, the values of σ(G(j)i ) (i.e., trainable gates) tend to become smaller. In other words, these trainable gates attempt to normalize or stabilize the magnitude of hidden states and thus alleviate mutual interference.
Efficiency. Table 9 and Table 10 show the training and inference speeds of each method in text generation and machine translation. All training and inference were run on a single GeForce RTX 3090 GPU.
C.2 Detailed Results
Table 11 and 12 are the detailed versions of Table 1 and 4, respectively. We provide detailed results of both single- and multi-aspect models. For text generation, we further demonstrate the accuracy of each attribute.

heading: D Case Study
text: To further investigate the fulfillment and text quality of each combination of constraints of these methods, Table 13 and Table 14 demonstrate examples of text generation and machine translation, respectively. Models only trained on single-aspect data are required to give results satisfying multiple aspects of constraints.

heading: E Details in Human Evaluation
text: In this section, we show more details about the human annotation adopted for evaluating model performance on text generation. We recruited three volunteers from schools, shuffled the output of models and provided it to them for scoring. Since
they are volunteers, they were not paid. Their average age is 25 years old and they have enough daily English communication skills. The instruction we provided to them like “This human evaluation aims to evaluate the model-generated review texts in three aspects: sentiment and topic relevance, and text fluency. All three integer scores are on a scale of 1-5, with a higher degree of topic/sentiment relevance representing a more consistent theme/sentiment, and a higher degree of text fluency representing a more fluent text. Your personal information will not be retained and these scores will only be used for human evaluation in research”.
Method Constraint Sentiment↑ Topic↑ PPL↓Positive Negative Asian American Mexican
Decoding-Time Regulation Method
GEDI
single-aspect
98.67 / / / / 227.87 / 100.00 / / / 839.69 / / 94.93 / / 206.97 / / / 99.73 / 246.36 / / / / 97.33 297.54
multi-aspect
98.67 / 28.27 / / 363.51 99.20 / / 87.73 / 1834.65 99.47 / / / 37.87 378.73
/ 100.00 44.53 / / 329.38 / 99.73 / 97.87 / 423.21 / 99.73 / / 11.87 372.03
single (avg.) 98.67 100.00 94.93 99.73 97.33 363.69
multi (avg.) 99.11 99.82 36.40 92.80 24.87 616.92
Parameter Efficient Tuning Method with Joint Training
DIST. LENS
single-aspect
91.33 / / / / 28.48 / 97.95 / / / 28.70 / / 77.33 / / 39.01 / / / 88.98 / 29.87 / / / / 79.47 38.26
multi-aspect
36.27 / 43.73 / / 45.89 57.87 / / 71.73 / 49.84 74.67 / / / 54.67 47.59
/ 99.73 70.13 / / 56.20 / 97.60 / 78.93 / 59.24 / 98.67 / / 82.67 56.77
single (avg.) 91.33 97.95 77.33 88.98 79.47 32.86
multi (avg.) 56.27 98.67 56.93 75.33 68.67 52.59
Parameter Efficient Tuning Methods without Joint Training
PROMPT-TUNING
single-aspect
52.00 / / / / 136.10 / 56.27 / / / 26.26 / / 46.67 / / 25.83 / / / 84.53 / 26.07 / / / / 39.60 24.36
multi-aspect
57.07 / 40.80 / / 65.96 59.47 / / 83.87 / 30.45 45.87 / / / 20.13 47.43
/ 49.73 30.27 / / 40.78 / 38.67 / 84.00 / 33.43 / 38.93 / / 29.60 27.30
single (avg.) 52.00 56.27 46.67 84.53 39.60 47.72
multi (avg.) 54.13 42.44 35.53 83.93 24.87 40.89
PREFIX-TUNING
single-aspect
70.67 / / / / 23.53 / 98.93 / / / 21.55 / / 77.87 / / 21.89 / / / 84.00 / 21.99 / / / / 73.60 22.55
multi-aspect
64.53 / 70.27 / / 141.71 67.20 / / 80.00 / 243.93 51.20 / / / 65.73 125.30
/ 33.87 60.67 / / 118.65 / 27.32 / 78.13 / 138.91 / 41.07 / / 59.87 116.34
single (avg.) 70.67 98.93 77.87 84.00 73.60 22.30
multi (avg.) 60.98 34.08 65.47 79.07 62.80 147.47
WARNING: Next may contain contents that are offensive in nature. Constraints Method Generation Result
While this is happening to people of colour over and over again in schools and hospitals regardless of how kids should behave, let alone thats happening here? Is we as a society humane towardsGEDI minks that poo on each other before death leaves them.
While this is happening i was very impressed. food is average and the service is great. DIST. LENS
Advertisements: -) ***EDIT: The vegetarian menu is amazing!
While this is happening in my hometown. In New York, New York, and the UK , are in need of something, they say. It’s a good year in my blood. In New Orleans, is it is good for the heart to get rid of it.In New York is getting all of its emotions out because we don’t want to make the traditional church PROMPT-TUNING
in-state.It was also all the land that is, and it is not good is all are for me is all a rock, and we don’
PREFIX-TUNING While this is happening food. is about Asian. isaboutAsian.is-foodsfood-trending.s.e...trend-stings-food-trends
While this is happening in my hometown, i will say that it’s a little pricey but worth it for a good meal. the food was good, the service was good, the service was fast and the food was tasty. the food arrived on time, so i could eat it. i did get a couple of things wrong with this place. the rice is bland. i think the rice is too salty. my boyfriend ordered the fried squid. it was a bit greasy. the shrimp had TAILOR
a good flavor and it was fresh. i would give it a try again but it won’t be here for at least 3 years.
While this is happening, i’ve never been here to the japanese section and have never eaten here before. the place is small and the place is crowded. the food is small, and the service isn’t very friendly. i did not expect to spend a lot of money for a sushi place because it’s pretty small. they have a few places where they can take a large order and they give it to those who pay
Neg., Asian
PROMPT GATING (Ours)
for the food. also the staff seems to ignore you if you come for sushi.
The food sucks!! Luckily we’re Gilmore Girls and I still feel pretty strong GEDI
from an adrenaline rush, not being adequately clothed. Return to Content
The food is great and a little overpriced. service was great though. i will never go back. DIST. LENS
Advertisements for the sake of getting good food
The food you don’t need to worry about it. This is the most important food. This could be years. This would also have you look for new business to move to. This is a big one and we’re not looking for one.This might get more people. It may get people PROMPT-TUNING
looking for work elsewhere, it could get people lookin at work in the area.
The food. is about negative aspects, and about negative food.” – Michael Siegel. PREFIX-TUNING
following is a negative review? –@siegel.
The food was good for what you paid for it, but the service was terrible, the food tasted bad. the waiter was rude and was constantly taking orders that were not on the table. when he asked why the food was so hot or that they were out of food. he said that the menu was out of order because of a busy night. after that he told me to call him back to make sure that the order was in order. the food itself was not that great, TAILOR
but that’s ok because it was only $6.
The food was very bland. i had a burger that was really dry. they had the "tender beef patty".
Neg., American
PROMPT GATING (Ours) the beef was dry so i didn’t like the "tasty" flavor and the burger tasted stale. i don’t eat that type of beef.
GEDI The country didn’t learn its lesson soon.
The country fried is great, not the best. service was decent and we’ll be back! DIST. LENS
Advertisements are good! Related Posts: (
The country. hell yes I am. and I am off. I am off to do some drugs and drink with a woman. I am off. and I am also off on drugs with the women! I can see I could go. and have sex with someone. I don’t think I am a homa, though I could have sex sometime, which wouldn’t be good with the men.I PROMPT-TUNING
am off my period and I’m not ready yet for that, except sex, it is good with the women
The country and Mexican food. is about Mexican food. for a reason. It’s about Mexican food. following is about Chinese food.is...Read Full Article......and more »Zachary: Food is delicious...!}>PREFIX-TUNING More...</show>">Chef-in-chief”...Zach!</show><Chev...">ZACH!....!"</show>
The country style, fresh, delicious tacos, and fresh tortilla chips. the salsa, the burrito and rice were good. the salsa was very good, but the rice was a bit dry. i’m going to give the burrito a 10/10. the burrito was not as fresh as i would like, and the salsa was lacking, not fresh enough. my brother got TAILOR
the burrito with the green salsa, he liked it.
The country food at this place has become so bland that i have stopped coming here. the service was slow and the food didn’t have any flavor to it at all. it just felt stale to the touch. i have never been in the place. i have been to many different mexian restaurants in my time in vegas,
Neg., Mexican
PROMPT GATING (Ours) however i can only say they are the worst in vegas, at least to this point, in my memory
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section* Limitations after the conclusion.
3 A2. Did you discuss any potential risks of your work? Section* Ethics Statement after the limitations.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section 1 Introduction
3 A4. Have you used AI writing assistants when working on this paper? Grammarly, grammar error correction, the whole paper.
B 3 Did you use or create scientific artifacts? Section B Reproducibility
3 B1. Did you cite the creators of artifacts you used? Section B Reproducibility
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Section B Reproducibility
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section B Reproducibility
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section B Reproducibility
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section B Reproducibility
C 3 Did you run computational experiments? Section 5 Experiments
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section C Experimental Results
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section B Reproducibility
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section C Experimental Results
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section B Reproducibility
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Seciton E Details in Human Evaluation
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Seciton E Details in Human Evaluation
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Seciton E Details in Human Evaluation
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Seciton E Details in Human Evaluation
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Seciton E Details in Human Evaluation

heading: Future Work
text: ” for “past/present/future”, and “The tense of this sentence is undecided. ” for sentences that do not have an explicit tense. Note that our use of existing artifact(s) was consistent with their intended use. The source code of this work is available at https://github. com/ THUNLP-MT/PromptGating4MCTG and it was developed based on THUMT (Tan et al. , 2020), an opensource toolkit for machine translation