Controlled Text Generation for Black-box Language Models via Score-based Progressive Editor (2024)

Abstract: Controlled text generation is very important for the practical use of language models because it ensures that the produced text includes only the desired attributes from a specific domain or dataset. Existing methods, however, are inapplicable to black-box models or suffer a significant trade-off between controlling the generated text and maintaining its fluency. This paper introduces the Score-based Progressive Editor (ScoPE), a novel approach designed to overcome these issues. ScoPE modifies the context at the token level during the generation process of a backbone language model. This modification guides the subsequent text to naturally include the target attributes. To facilitate this process, ScoPE employs a training objective that maximizes a target score, thoroughly considering both the ability to guide the text and its fluency. Experimental results on diverse controlled generation tasks demonstrate that ScoPE can effectively regulate the attributes of the generated text while fully utilizing the capability of the backbone large language models. Our codes are available at https://github.com/ysw1021/ScoPE.

heading: 1 Introduction
text: Modern language models can now generate text as fluently as humans in response to any given sequence or instruction (Radford and Narasimhan, 2018; Radford et al., 2019; Brown et al., 2020; Ouyang et al., 2022; Zhang et al., 2022b; Touvron et al., 2023a; Touvron et al., 2023b; Jiang et al., 2023). However, the generated text may carry potential risks, such as harmful expressions or inappropriate content (Gehman et al., 2020; Liu et al., 2021; Lu et al., 2022). If the previous context includes non-preferred attributes, these attributes can potentially appear in the generated text. Therefore, controlled text generation, which aims to generate
* Corresponding author
text constrained to target domain attributes regardless of the given context, is crucial for addressing the current issues in language models (Dathathri et al., 2020; Khalifa et al., 2021; Qian et al., 2022; Meng et al., 2022; Qin et al., 2022; Ma et al., 2023).
Recently, many large language models, especially those exceeding hundreds of billion parameters, are presented as de facto black-box models with limited access to model parameters (OpenAI, 2022; OpenAI, 2023; Team et al., 2023; Anthropic, 2024). Consequently, most existing approaches to controlled text generation that require access to the model’s parameters are either inapplicable or have limitations in these black-box scenarios. While access to the language model parameters is unavailable, some previous works assume access to the output token distribution and have studied controlled generation by manipulating this distribution without tuning the parameters (Krause et al., 2021; Yang and Klein, 2021; Arora et al., 2022). However, this approach significantly diminishes the fluency of the generated text, as it manipulates the output distribution based on the previous context. Therefore, there is a requirement for a novel approach to effectively leverage the generation performance of black-box large language models.
In this paper, we propose Score-based Progressive Editor (ScoPE) to address controlled generation for black-box language models and tackle the trade-off between control and fluency. ScoPE modifies the intermediate output text token blocks during the generation process of a backbone language model, ensuring that the edited text tokens contain the target attributes. This approach effectively guides the subsequent generation to naturally include the target attributes. Since it does not access the model parameters, including the output distribution, ScoPE can be adapted to any black-box model. By modifying the previously generated context tokens while maintaining fluency, ScoPE ensures that the text generated
14215
1
afterward includes the target corpus’s attributes without compromising fluency. To effectively incorporate target attributes during the editing process, we introduce a score aligned with the attributes in the target corpus, which is computed by the score model based on a pre-trained masked language model. To further assist ScoPE’s text modification, we additionally consider a repetition score and a task-specific score. Given the overall target score, ScoPE is trained with the objective of maximizing the score on text samples from a training set composed of sampling from the language models.
In our experiment for the diverse controlled generation tasks, using various corpora constructed from the Amazon Customer Reviews dataset, we comprehensively evaluate ScoPE in terms of control and fluency. We discover that ScoPE, using various black-box language models as its backbone, effectively regulates target attributes while preventing a decline in text fluency through experiments including GPT-4 evaluation. Additionally, our research confirms the compatibility of ScoPE with instruction prompting for control purposes. In tasks requiring sentiment control, ScoPE offers an effective solution to the tradeoff between fluency and control, a challenge often encountered in existing baseline models. Furthermore, through exploring various settings of ScoPE, we affirm its proficiency in managing multi-attribute control.
Our work presents a distinctive contribution by facilitating fluent controlled text generation utilizing black-box language models, thereby demon-
strating their effectiveness and versatility for a wide range of controlled text generation tasks within the current context.

heading: 2 Related Work
text: The elements addressed by previous approaches for controlled text generation can be primarily divided into three categories: input context, weights of language models, and decoding strategy. When handling input context, the objective is to effectively incorporate target attribute information into the input of the language model (Li and Liang, 2021; Lester et al., 2021; Qian et al., 2022; Ma et al., 2023). In the case of weights of language models, the approach involves fine-tuning the weights of the model, either partially or entirely, with data from the target domain (Keskar et al., 2019; Ziegler et al., 2019; Lu et al., 2022). Lastly, the approach to decoding strategy entails using adaptive modules, such as discriminators, that are tailored to the target domain (Dathathri et al., 2020; Qin et al., 2020; Yang and Klein, 2021; Krause et al., 2021; Liu et al., 2021; Arora et al., 2022), or employing distributional approaches (Deng et al., 2020; Khalifa et al., 2021; Meng et al., 2022) to perform weighted decoding of the language model. The practical applicability of fine-tuning approaches under black-box conditions is limited due to their dependence on model parameters. While methods of the decoding strategy can be adapted to loose black-box conditions when output distribution is approachable, they still suffer from the decreasing fluency of generated texts.
The most prominent novelty of our work, compared to these methods, is that it secures controllability for entirely black-box LLMs, such as the ChatGPT API, where both model parameters and output logits are inaccessible. Additionally, whereas previous works have encountered a decrease in the fluency of generated text when securing controllability, our approach addresses this issue by ensuring that the text generated after tokenwise editing naturally continues from the edited previous text.
There exist studies that perform controlled generation through iterative sampling, including energybased models, aiming to maximize scores computed from various domain-specific modules, starting from text sampled from the initial distribution (Mireshghallah et al., 2022; Qin et al., 2022). However, due to numerous iterative sampling steps, they result in significantly slower generation speeds compared to standard language model generation, reducing its practicality. About this issue, our work achieves a significant improvement in generation speed by utilizing the trained editor that only requires a few editing steps.
Recent studies introduced the method for adapting gray-box language models which only permit access to the generative distribution of the model (Ormazabal et al., 2023; Liu et al., 2024). Additionally, Sun et al. (2024) proposed a lightweight module that controls black-box language models for specific tasks. Our approach effectively adapts the black-box model from a different perspective of this concurrent work.

heading: 3 ScoPE: Score-based Progressive Editor

heading: 3.1 Approach for Controlled Text Generation
text: Overview In this work, we introduce ScoPE: Score-based Progressive Editor, a novel approach for controlled text generation. The primary goal is to generate a fluent continuation ŷ = (ŷ1, ..., ŷl) that incorporates target attributes given an input sequence x = (x1, ..., xk). The input sequence x can either contain the target attributes (in-domain) or have attributes that are orthogonal or adversarial to the target (out-of-domain). Language models, when given x as an input, generate a continuation ỹ that inherits the attributes of x due to the autoregressive nature of language modeling. This means that the generated output (x, ỹ) may not always possess the desired target attributes.
To address this, we aim to perform controlled
generation by editing the output ỹ generated by the language model to incorporate the target attribute, resulting in ŷ. Editing the entire ỹ at once is challenging, so we divide the task into progressive block-wise editing during the generation process. This means that we repeatedly edit the token block generated by the language model with a short b token length, and then reinsert the previously generated tokens that include the edited token block back into the input. This method allows us to steer the language model’s generation step-by-step towards incorporating the target attributes effectively.
Block-Wise Editing The progressive block-wise editing approach involves dividing the continuation into blocks of b tokens. Each time b generation steps are completed at time step t, the editor takes the generated sequence (x, ŷ:t−b, ỹt−b:t) as input and modifies the current block of tokens ỹt−b:t = (ỹt−b+1, . . . , ỹt) to incorporate the target domain attributes. This iterative process ensures that the language model progressively incorporates the desired attributes in subsequent steps.
One of the key challenges in editing a token block ỹt−b:t is the absence of gold labels that can act as ŷt−b:t. In general causal language modeling, the gold label for generating a token can be considered the next token in the input x, which is available in the training set. However, retrieving the next b tokens after x from the training set as ŷ can be suboptimal. This is because ỹ generated by the backbone language model often diverges from the continuation present in the training set, leading to a significant difference from ŷ. In this case, ŷ is not a natural continuation from the perspective of the backbone model, leading to a decrease in the generation quality.
To overcome this, we introduce a target score that measures how well the target attribute is incorporated into the text. The objective of ScoPE is to enhance this target score while maintaining the overall structure of ỹ for fluency. Instead of relying on labels from the training set, the target score guides the editing process, ensuring that only the essential attributes of ỹ are transformed to align with the target domain. This approach minimizes the editor’s workload and preserves the content’s coherence.
The subsequent sections will delve into the training methodology for ScoPE and the formulations of the target score. Figure 1 provides an overview of our approach.

heading: 3.2 Training ScoPE

heading: 3.2.1 Preparation for ScoPE Training
text: Training set for offline learning framework The inference process in ScoPE is a block-wise auto-regressive process where ỹ = (ỹ1, ..., ỹb), generated to continue given short sequence x, is edited to serve as the input x for the next step. If we apply the online learning strategy to the training, x in the training sample (x, ỹ) for the editor is constructed from the editor’s output at the previous time step. This approach, especially during the early stages of learning, cannot ensure that x contains the target domain attributes. Therefore, to ensure stable training, we employ the offline learning strategy. This strategy utilizes the ground truth data from the target domain as input for the next step, instead of the model’s output during training like teacher-forcing strategy (Sutskever et al., 2014). In other words, to guarantee that x already possesses the target domain attributes, we sample x from the target domain data when constructing a training set for the ScoPE training. Algorithm 1 shows an algorithm about the concrete process to construct the training set.
Fine-tuning pre-trained MLM In addition to constructing a training set for the offline learning framework, the preparation phase before training involves further tuning a pre-trained masked language model (MLM) on the target corpus. This fine-tuned MLM serves two purposes. First, it is used as a scoring model to calculate a score that measures both the similarity to the target corpus and the level of fluency. Second, it serves as the base model for training the ScoPE model. That is, the parameters of ScoPE are initialized with the parameters of the fine-tuned MLM. This initialization positions the initial generative distribution of ScoPE closer to the target corpus, significantly improving training stability.

heading: 3.2.2 Maximizing Score Disparity between Input and Edited Texts
text: To ensure that the edited text achieves a higher target score than the input text, we set the objective of editor training to maximize the difference between the scores of the edited text and the input text, rather than just maximizing the score of the edited text. To provide a more refined training signal during the learning phase, we decompose the target score of the text sequence at the token level. This token-wise target score allows the editor
Algorithm 1 Training set Construction 1: Input: target corpus X , backbone LM PLM,
maximum edit block size bmax, maximum input length lmax, training set size N
2: Define training set T = {} 3: while |T | < N do 4: Sample x = (x1, . . . , xlx) from X , where lx ∈ [1, lmax] 5: Sample ỹ = (y1, . . . , yly) ∼ PLM(·|x), where ly ∈ [1, bmax] 6: Construct train sample (x, ỹ) 7: Append (x, ỹ) to T 8: return T
to receive training signals only for the token positions where edits occur, thereby enhancing training stability. We empirically observe that the training becomes unstable when the editor receives training signals for all token positions. The training objective function J(θ)t that should be maximized for the distribution of the editor θ at position |x|+ t of (x, ŷ) is as follows:
J(θ)t = Eŷt∼pθ(ŷt|x,ỹ)d(x, ỹ, ŷ)t = ∑
ŷt∈V pθ(ŷt|x, ỹ)d(x, ỹ, ŷ)t, (1)
where d(x, ỹ, ŷ)t is defined as follows:
d(x, ỹ, ŷ)t
= { 0 if ŷt = ỹt st′((x, ŷ))− st′((x, ỹ)) else, (2)
where st′ denotes the decomposed score at position t′, t′ = |x|+ t, and V is a vocabulary of editor. We clip d(x, ỹ, ŷ)t within to the pre-defined range to prevent the gradient from exploding. J(θ)t requires computing the score for all ŷt ∈ V , which is computationally expensive and impractical. To address this issue, we approximate J(θ) by computing the expectation only for those ŷt corresponding to the top-k probabilities. This approximation is viable because when the editor is initialized from a fine-tuned MLM, it already possesses a reasonably sharp generative distribution from the early stages of training. Tokens with small probabilities, except for a few tokens with large probabilities, can be ignored in the calculation of expectation. In practice, we sample only one ŷt from the pθ(ŷt|x, ỹ) during training, and it shows successful results in both
training and inference. When k = 1, the gradient of J(θ)t about θ, ∇θJ(θ)t can be approximated as follows:
∇θJ(θ)t ≈ d(x, ỹ, ŷ)t∇θpθ(ŷt|x, ỹ) = w(x, ỹ, ŷ)t∇θ log pθ(ŷt|x, ỹ), (3)
where w(x, ỹ, ŷ)t is defined as follows:
w(x, ỹ, ŷ)t = d(x, ỹ, ŷ)tpθ(ŷt|x, ỹ). (4)
The approximated ∇θJ(θ)t is the same as the gradient of the weighted log-likelihood as follows:
w(x, ỹ, ŷ)t∇θ log pθ(ŷt|x, ỹ) = ∇θ[w(x, ỹ, ŷ)t log pθ(ŷt|x, ỹ)],
(5)
where the weighting factor w(x, ỹ, ŷ)t is considered as constant. We can simply implement this by detaching the pθ(ŷt|x, ỹ) in the weighting factor during the backward process.
The editor can achieve more refined edits through an iterative editing process for a token block. However, when performing the n-th step of iteration where n > 1, the input for the editor, (x, ŷ(n−1)), is generated from the editor distribution pθ(·|x, ŷ(n−2)), while the input for the 1st step, ỹ = ŷ(0), is generated from the language model distribution pLM(·|x). As a result, these two texts can be situated in different distributions. To address this distributional mismatch, we perform N iterations of the iterative process during training. This involves sampling the edited text and using it as input to the editor again, allowing for the refinement of edits over multiple iterations. Finally, the loss function for a training ScoPE with training sample (x, ỹ) = (x, ŷ(0)) can be expressed as follows:
LScoPE
= 1
N
N−1∑
i=0
|ỹ|∑
t=1
wt log pθ(ŷ (i+1) t |x, ŷ(i)),
(6)
where wt is detached during back-propagation and refers to as follows:
wt = w(x, ŷ (i), ŷ(i+1))t. (7)
Considering the increasing training cost as N grows, we fix N = 2 in practice.
Techniques for stable training In this work, we employ four techniques for training stability: 1) offline learning framework, 2) parameter initialization with MLM fine-tuned for the target corpus, 3)
providing training signals only at the positions of edited tokens, 4) clipping the score disparity between input and edited texts. Appendix A contains training algorithms of ScoPE.

heading: 3.3 Score Formulation for Target Attributes

heading: 3.3.1 Target Score from Fine-tuned MLM
text: We calculate the major target score using a masked language model (MLM) fine-tuned with target corpus. Previous research has demonstrated that an MLM trained with the objective of masked language modeling, predicting masked tokens at masked positions, can be parameterized as an implicit energy-based model (Wang and Cho, 2019; Clark et al., 2020; Goyal et al., 2022). Also, it has been shown that fine-tuning the pre-trained MLM to the target domain improves the end-task performance in the domain (Gururangan et al., 2020; Ke et al., 2023). By integrating the findings of these studies, we present the target score smlm(x) for a given sequence x = (x1, ..., xT ) and a target domain-specific MLM ϕ as follows:
smlm(x) = T∑
t=1
smlm,t(x)
= T∑
t=1
fϕ(xt, hϕ(x\t)),
(8)
where x\t is the sequence obtained by masking the position t of x, hϕ(x\t) is the representation at the position t of x\t computed by ϕ, and fϕ(·) is the language modeling head function of ϕ. In practice, the score smlm,t(x) is computed as the raw logit before entering the softmax activation function for the original token xt at the masked position, following the forward pass of the MLM.

heading: 3.3.2 Repetition Score
text: Since the base architecture of the editor is MLM, it bidirectionally and non-autoregressively edits ỹ. However, in the non-autoregressive generation, repetitive generation remains a problem due to the multi-modality issue arising from the conditional independence assumption (Gu et al., 2018; Zhang et al., 2022a). Moreover, as discussed in the existing works (Goyal et al., 2022; Mireshghallah et al., 2022), we observe that MLM may assign high scores to repetitive text, which means relying solely on the MLM score is insufficient to handle repetitive generation. To address this problem, regardless of the target attributes, we introduce a repetition score for an arbitrary sequence
x = (x1, ..., xT ). Repetition at the position t of the sequence x occurs when there exists the same token as xt at different position i. In this case, as the distance between positions, |i− t|, decreases, the likelihood of an unnatural repetition increases. From this perspective, the repetition score for x, srep(x) is described as follows:
srep(x) = T∑
t=1
srep,t(x)
= T∑
t=1
T∑
i=1
−1(xi ̸= xt)|i− t| , (9)
where 1(·) denotes the indicator function.

heading: 3.3.3 Task-specific Score
text: In addition to the MLM score, other task-specific scores such as raw logits from a discriminator for the target attribute, automatic evaluation metric, or human feedback can also be combined with the target score. Motivated by the previous work (Mireshghallah et al., 2022), the target score is augmented through a linear combination of domainspecific scores. For example, the integration of smlm, srep, and sdisc, is computed as follows:
stotal,t = smlm,t + α · srep,t + β · sdisc, (10)
where α and β are the scaling factors for scores, and stotal,t is the overall score at position t. As sdisc is a sequence-wise score, we assign the same score sdisc for all positions.

heading: 4 Experimental Setup
text: We conduct a controlled generation evaluation using the Amazon Customer Reviews dataset1. The dataset includes multiple attributes related to the review style, the categories corresponding to the type of reviewed product, and the sentiments aligned with the product rating. Consequently, this dataset is well-suited for the creation of diverse corpora that contain specific attributes. We classify the dataset corpus based on category and sentiment. For categories, we construct four distinct corpora: Camera, Videogame, Grocery, and Music. In terms of sentiment, samples rated with 5 stars are compiled into a positive corpus, while those rated with 1 star formed a negative corpus. Details on the dataset statistics are discussed in Appendix B.
1https://s3.amazonaws.com/amazon-reviewspds/readme.html
Using these corpora, we evaluate controlled generation tasks for both category and sentiment attributes. In each task, we test the controllability and fluency of generated continuations when fixing one target attribute among the attributes relevant to the task, starting from an input text with arbitrary attributes. Experimental details for our experiments are mentioned in Appendix C.

heading: 4.1 Category Controlled Generation
text: In the category controlled generation task, we employ Perplexity (PPL) calculated from LLaMA213B as the metric for measuring fluency. For assessing controllability, we employ the MAUVE (Pillutla et al., 2021) metric. MAUVE calculates the distance between the approximated distributions of generated texts and reference texts. A detailed description of the empirical study examining the suitability of MAUVE as a metric for controllability is presented in the Appendix D. Recently, it has become increasingly recognized that using GPT-4 to assess the quality of text correlates more closely with human preferences compared to existing evaluation metrics (Liu et al., 2023; Dekoninck et al., 2023). Consequently, in addition to automatic metrics, we conduct a comparative evaluation between ScoPE and the backbone LM using GPT-4 to provide a more comprehensive assessment that aligns with human preferences. The prompts injected into GPT-4 for the evaluation can be found in the Appendix E.
In this task, control over the target attributes was executed under the corresponding target corpus, not under the specific keywords. Consequently, traditional keyword-focused topic control baselines are not suitable for comparison in this task. Hence, our evaluation in the category controlled generation task explores adaptability with various black-box models rather than comparison with existing baselines. We compose a training set from the GPT2XL and apply it to controlled generation using two black-box API backbones: davinci-002 and babbage-002. Moreover, we further refine ScoPE, which was initially trained with a GPT2-XL composed training set, by tuning it on a smaller training set derived from the black-box instruct LLM API: gpt-3.5-turbo-0613. We then evaluate various settings, including scenarios where ScoPE is combined with the instruction prompting of the specified backbone model.

heading: 4.2 Sentiment Controlled Generation
text: In the sentiment controlled generation task, we use Perplexity (PPL) as the metric for measuring fluency and employ two pre-trained sentiment classifiers as metrics for assessing controllability. For evaluating controllability, the accuracy concerning the target sentiment of each classifier is utilized. Among these classifiers, one is used for score calculation during the training of ScoPE (Hartmann et al., 2023). We utilize the raw logits of the classifier as the additional target score sdisc, which is denoted as EDG (External Discriminator Guidance). Since this might result in an overfitted outcome for this classifier, we adopt an additional pre-trained sentiment classifier of a different type to check overfitting (Hartmann et al., 2021).
We conduct the comparison of ScoPE with existing baselines that manipulate the output distribution of the backbone language model: GeDi, DExperts, (Krause et al., 2021; Liu et al., 2021) or sample the generated text through iterative steps: Mix&Match (Mireshghallah et al., 2022). To en-
sure fair performance comparison in terms of the backbone model, we use GPT2-XL both for the training set composition and the backbone language model in the generation process.

heading: 4.3 Multi-Attribute Controlled Generation
text: Our proposed method, ScoPE, can be designed to control multiple attributes simultaneously. In this work, we explore a multi-attribute control methodology with below three variants of ScoPE. • ScoPE-MS (Mixed Score): During the training
process of ScoPE, scores for various attributes are combined in the form of Eq. 10 to construct an overall target score. • ScoPE-ML (Mixed Logit): During the generation process, when calculating logits for editing, the logits from editors targeting each attribute are combined through linear combination, resulting in a new calculated logit. • ScoPE-SE (Sequential Editing): During the generation process, various editors sequentially perform token-level edits in the editing phase.
Fig. 2 illustrates the overall editing mechanisms of ScoPE for multi-attribute controlled generation using both the ML and SE approaches for handling K different attributes. We evaluate ScoPE’s ability to control specific categories and sentiments simultaneously. Detailed descriptions of ML and SE can be found in the Appendix G.

heading: 5 Results

heading: 5.1 Category Controlled Generation
text: Table 1 presents the evaluation results of controlled generation for two GPT3 base model APIs, targeting the Camera attribute. The improvements in both PPL and MAUVE, regardless of the type of backbone model, demonstrate ScoPE’s adaptability to various black-box models. Moreover, the ability to enhance the fluency of backbone models while ensuring controllability over target attributes indicates that ScoPE’s controllability is not achieved at the expense of fluency. This highlights the balanced effectiveness of ScoPE in managing both aspects simultaneously.
Table 2 presents a comparative experiment evaluation for fluency and controllability. Regardless of the input text’s attributes, texts generated via ScoPE were evaluated as more fluent than those produced solely by the backbone LLM ("Win" > 50). This suggests that despite utilizing an editor with a significantly smaller parameter size compared to the backbone model, ScoPE’s generation
is indeed more fluent. This trend aligns with the results of PPL. In terms of controllability, ScoPE also received better evaluations in all settings compared to the backbone model. In particular, ScoPE generation was evaluated as superior even when the input attribute was Camera, so that the continuation generated by the backbone model was likely to belong to the camera category.
Table 3 presents the evaluation results of controlled generation for the ChatGPT API, targeting the Camera attribute. Focusing on the fact that ChatGPT is an instruct LLM, we confirm that the ScoPE generation method can be effectively combined with instruction prompting. It demonstrates significantly higher performance in terms of control while preserving the fluency of a large-scale model, as compared to when each method is used independently. This underscores the compatibility between the commonly used prompting methods in LLMs and ScoPE. The instruction prompts for ChatGPT can be found in Appendix F.

heading: 5.2 Sentiment Controlled Generation
text: Table 4 presents the evaluation results of comparisons with prior works in sentiment controlled generation targeting the positive attributes. Although control over sentiment is not always predominant, ScoPE shows significantly improved results in terms of fluency compared to DExperts and GeDi. This can be seen as a meaningful resolution to the
trade-off that typically arises in existing methodologies manipulating the output distribution of the backbone models, where increasing controllability often leads to a reduction in fluency.
When compared with Mix&Match, this approach also preserves a degree of fluency, as it employs token-level iterative sampling instead of directly altering the distribution. However, it is observed that while control performance is ensured for the used sentiment classifier, it declines for other classifiers, suggesting that overfitting has occurred specifically for the chosen sentiment classifier. The improved accuracy of the external discriminator not used in ScoPE training shows that the ScoPE is not overfitted to the discriminator used in training. The results of targeting the negative attributes can be found in Appendix H.

heading: 5.3 Multi-Attribute Controlled Generation
text: Table 5 presents results from utilizing ScoPE in multi-attribute control settings, targeting both Camera and Positive attributes simultaneously. The outcomes suggest that all three settings successfully target both attributes. In the case of MS, it particularly demonstrates superior control over the sentiment attribute compared to the other two settings. While ML and SE show slightly lower performance
than MS, they hold an advantage in that they can freely use pre-tuned editors for single attribute targeting in a plug-and-play manner. Notably, ML offers the highest degree of freedom by allowing the adjustment of each attribute’s proportion.
Additionally, the control explorations using LLaMA2-7B model as the black-box backbone model and the the diversity evaluation of the texts are presented in Appendix H. Including ablation studies, the analysis of the inference cost and fluency are presented in Appendix I. Appendix J shows the generated samples of ScoPE.

heading: 6 Conclusion
text: We present ScoPE which guides the generation process of a backbone language model to improve target domain scores, enabling fluent controlled text generation. ScoPE effectively addresses the challenges associated with the black-box scenario and the trade-off between controllability and fluency. Furthermore, it demonstrates its pragmatic utility when incorporated within various large language model APIs, manifesting its tangible applicability in real-world contexts. We believe ScoPE, combined with improved functions, can be effectively utilized for the adaptation of large language models to specific domains or tasks in future works.
Limitations
In this section, we mention several limitations of our method. During the training process, there is a memory and time cost incurred due to the need for masking at each position within the sequence when calculating the MLM score. Additionally, to understand the target domain attributes and analyze the distribution of text generated by the backbone language model, a certain amount of target domain data and samples from the backbone language model are required. In a few-shot setting, additional methods would be necessary for future work. If the target domain and the domain to which the input belongs are too different, the burden on the editor during editor training becomes significant. As a result, the frequency of modifications by the editor increases significantly, leading to instability in the training process and a substantial increase in cost. In conclusion, considering the challenging nature of training and inference, addressing the issue of editor instability should be a key topic for future work. Furthermore, ScoPE uses only the replacement option among the possible edition options (replacement, insertion, deletion). Architecture or training methods for performing various edition options could be improvement directions of future works. We discuss additional limitations about the inference cost and fluency of the generated text in Appendix I.
Ethics Statement
In this section, we aim to address the ethical issues we perceive in our work. The most significant concern pertains to the potential of our controlled text generation task to target morally objectionable attributes. The ability of ScoPE to generate text limited to a desired target domain, regardless of the input’s originating domain, is an important issue related to the misalignment of large language models. Referred to as "Jailbreak," this topic deals with the phenomenon where language models become misaligned by specific contextual inputs, resulting in the generation of non-preferred and potentially harmful text. Our research involves guiding the context towards the target domain during the generation process of the language model, which could be exploited to induce misalignment in the language model by designating the target domain as the domain of misalignment. To mitigate such misuse, we propose leveraging misalignment as a shield against misalignment attacks by establishing
ScoPE’s target as the generation of only preferred text, thereby preventing the occurrence of misalignment.

heading: Acknowledgements
text: This work was supported by the National Research Foundation of Korea (NRF) grants funded by the Korea government (Ministry of Science and ICT, MSIT) (2022R1A3B1077720), Institute of Information & Communications Technology Planning & Evaluation (IITP) grants funded by the Korea government (MSIT) (RS-2021-II211343: Artificial Intelligence Graduate School Program (Seoul National University) and 2022-0-00959), Samsung Electronics Co., Ltd (IO221213-04119-01), and the BK21 FOUR program of the Education and Research Program for Future ICT Pioneers, Seoul National University in 2024.

heading: A Training Algorithms for ScoPE
text: Algorithm 2 MLM Fine-Tuning 1: Input: pre-trained MLM ψ,
fine-tuned MLM ϕ, target corpus X , MLM loss function LMLM
2: Initialize ϕ with ψ 3: for x in X do 4: Update ϕ by ∇ϕLMLM(x) 5: return ϕ
Algorithm 3 Training ScoPE 1: Input: training set T , fine-tuned MLM ϕ,
ScoPE θ, clip scale c 2: Initialize θ with ϕ 3: for (x(i), ỹ(i)) in T do 4: Sample ŷ(i) ∼ pθ(·|x(i), ỹ(i)) 5: Compute d(x, ỹ, ŷ)t for 0 ≤ t < |ỹ(i)| 6: Clip d(x, ỹ, ŷ)t within the range (−c, c) 7: Compute LScoPE 8: Update θ by ∇θLScoPE 9: return θ

heading: B Data Statistics
text: Table 6 shows the data statistics for all domains used in our work. When training and evaluating ScoPE combined with gpt-3.5-turbo-0613 API, we utilize about 5% of the data corpus of the Camera corpus.

heading: C Experimental Details
text: The test set created for evaluation consists of input texts of 32 tokens and generated continuation texts of 128 tokens. During test set creation, the block size b for editing was fixed at 16. For MAUVE evaluation, we compute the performance of MAUVE using five seeds for k-means clustering and evaluate it as the average. A reference set for MAUVE evaluation is constructed from the target corpus.
Table 14 presents the hyperparameters for finetuning the pre-trained MLM. Table 15 presents the hyperparameters for training ScoPE. All training and fine-tunings are conducted with four GPU on our machine (GPU: NVIDIA V100). Irrespective of the domain, during the first stage of ScoPE training, the additional fine-tuning of the pre-trained MLM has been observed to transpire at a rate of approximately 1.8 updates per second. The subsequent
stage, involving ScoPE loss training, proceeds at an approximate rate of 0.45 updates per second. Given that the camera domain, characterized by the largest target dataset size, requires the most extensive training, the complete training process for this domain takes approximately 29h 52m time. Our code is based on FairSeq (Ott et al., 2019). The pre-trained MLM used for acquiring the base models and score models of ScoPE in all tasks is RoBERTa-base (Liu et al., 2019).We fix the output text decoding strategy of the ScoPE editor to ancestral sampling in all evaluations.

heading: D Empirical Study for MAUVE as Controllability Metric
text: In this section, we present our empirical studies on MAUVE, which are evaluation metrics used for scoring target attributes. Referring to the previous work which already demonstrated that MAUVE effectively indicates sentimental features in the text (Pimentel et al., 2022), we investigate whether it also effectively indicates categorical features. Since MAUVE considers the comprehensive attributes of the target corpus, it evaluates not only the category attribute but also other attributes like review style. To tackle this issue, we split the test set for each domain into two parts, designating one as the source set for MAUVE calculation and the other as the reference set. We measure the scores for all
combinations of domains. Table 7 demonstrates that MAUVE effectively indicates categorical attributes by showing significantly higher values only when the source and reference domains are the same. Notably, even though sets with different category attributes share commonalities in other attributes, the differences in MAUVE scores are pronounced. This suggests that the category attribute, compared to style or other attributes, possesses more distinct characteristics, allowing for effective differentiation through MAUVE measurements.
In addition to MAUVE, we also used GPT-4 as a means to verify the controllability of ScoPE by comparing which of the ScoPE-generated texts or the backbone model (Davinci-002) generated texts are closer to the target category for the same input text in terms of continuations. The results of this evaluation align with the MAUVE results, particularly highlighting that the control over the Music category is relatively lacking, which is corroborated by the GPT-4 evaluation results. This adds further justification for using the MAUVE metric as a measure of controllability in our work.

heading: E Prompts for Comparative Evaluation using GPT-4
text: Table 16 shows the content of the prompts injected to GPT-4 for the comparative evaluation of fluency. Table 17 shows the content of the prompts injected to GPT-4 for the comparative evaluation of controllability.

heading: F Prompts for Controllability Instruction of ChatGPT
text: Table 18 shows the content of the prompts injected to ChatGPT for the instruction of controllability to Camera domain.

heading: G Detailed Descriptions of ScoPE-ML and ScoPE-SE for Multi-attribute Controlled Generation
text: In the ML method, the final edit logits are calculated through a linear combination of the logits from editors tuned to target different attributes. This process allows for the adjustment of the relative importance of each attribute by manipulating the weights in the linear combination. For instance, in the Camera-Positive multi-attributes control scenario, we set the weight for the Camera attribute at 10.0 and for the positive attribute at 1.0.
The SE method involves using editors, each tuned for different attributes, in a sequential manner for editing. Consequently, the overall editing process consists of K editing sub-processes. In this approach, the order in which editors are applied significantly impacts the overall editing performance. Through empirical studies, we have found that placing broader attributes like sentiment earlier in the sequence enhances performance. In practice, for the Camera-Positive control task, we first used the editor for the positive attribute, followed by the editor for the Camera attribute.

heading: H Additional Results
text: Results utilizing LLaMA2-7B To explore the most basic setting where the backbone language models used for training set composition and in the generation process are the same, we use the relatively large-scale model LLaMA2-7B to construct the training set for ScoPE, then apply this ScoPE to controlled generation using the same type
of backbone language model. Table 19 presents the evaluation results of controlled generation targeting the Camera attribute. As the number of iterations N increases, the controllability of ScoPE (MAUVE) usually improves along with the fluency (PPL). These results indicate that the ScoPE framework is capable of generating texts that are not only fluent but also precisely controlled. Table 8 and 9 present the experimental results for sentiment controlled text generation tasks targeting each sentiment domain. To examine the impact of using an external discriminator, we denote the cases where only MLM score and repetition score are used, as in the baseline, as ScoPE, and the cases where the external discriminator is included in score computation as EDG (External Discriminator Guidance). In all cases using ScoPE, we observe improved scores and MAUVE, demonstrating the successful steering of the language model’s controlled generation in the target attributes. Moreover, the improved accuracy of the external discriminator not used in ScoPE training shows that the ScoPE is not overfitted to the discriminator used in training.
Diversity evaluation We assess the diversity of the texts generated by ScoPE using Davinci-002 as the backbone model and those directly generated by Davinci-002, employing the distinctness metric. Distinctness was calculated by measuring the unique number of unigrams, bigrams, and trigrams in each generated sample and dividing this by the total number of (uni- or bi- or tri-)grams in the sample (Li et al., 2016). Table 10 displays the results of this evaluation. While diversity was slightly reduced in the texts generated through ScoPE during the process of achieving controllability compared to those generated solely by the backbone model, the difference is marginal. Particularly, considering that over 90% of bigrams and trigrams remain non-duplicative, we can assert that diversity is still well-preserved in the ScoPE generation process.

heading: I Analysis of ScoPE
text: Ablation study on the repetition score Table 20 demonstrates a significant decrease in the repetition score when it is not utilized during training. In terms of PPL, the repetitive results without repetition score training show improved results. However, this is because language models tend to score higher for repetitive texts. In terms of MAUVE, the presence or absence of the repetition score has a negligible impact on performance for relatively easy in-domain input conditions. However, for outof-domain conditions, the absence of the repetition score results in a significant performance drop.
Ablation study on the edit block size Table 21 presents the results of controlled text generation with four different token block sizes, b = 4, 8, 16, 32. The combined score comprising PPL demonstrates that the small block size setting can guarantee fluency. For out-of-domain input conditions, especially in challenging generation scenarios with lower relevance to the target domain, smaller block sizes show significantly superior MAUVE to larger black sizes. This suggests that reducing the block size for more delicate editing can be beneficial in challenging scenarios.
Analysis of the inference cost of ScoPE In the process of generating a total of T tokens, each token undergoing E iterations in the Transformer model, the cost associated with passing a sequence through the model can be denoted as C. If we take into account the computational complexity presented by Mix&Match, it is expressed
as O(T 2EC) (Goyal et al., 2022). By modifying the computation of the MLM (Masked Language Model) energy calculation, which inherently holds a complexity of O(TC), to be executed in a parallel fashion through a trade-off with memory cost, an optimization towards a time complexity of O(TEC) becomes feasible.
In the scenario of ScoPE, which operates through block-wise edits, the computational complexity is more optimal. During the inference phase, the necessity for MLM energy computation is obviated, and as the generation of b tokens occurs within the context of producing the overall T tokens, the editor conducts E edit operations. Consequently, the computational complexity associated with ScoPE becomesO(TEC/B). This implies that during the token generation process, ScoPE demonstrates advantages over Mix&Match in both memory and time complexity realms. Table 11 shows the comparison of the runtime of ScoPE with other baselines.
During the generation process, the backbone model undergoes a step where it edits the previous text via ScoPE each time b tokens corresponding to the block size are generated. Consequently, there exists a genuine issue of having to recompute hidden states that are usually cached during autoregressive generation each time. However, the cost incurred by this recomputation impacts the overall inference time to a relatively minor degree, as recomputation occurs the times of the 1/b of the number of tokens generated. For instance, in the setting of our evaluation with a block size of b = 16, generating 128 tokens through ScoPE generation, the recomputation of the previous text’s hidden states occurs no more than 8 times throughout the entire generation process.
Nonetheless, since the frequency of recomputing varies with b, it is necessary to measure the inference time for various b values. Therefore, we have measured the ScoPE generation time using LLaMA2-7B as the backbone model across different b values and the results are presented in Table 12. As expected, a decrease in b leads to an increase in inference time. In extreme cases where b approaches 1 or when the overall length of the gen-
erated text becomes very long, a significant issue with inference time may indeed arise.
However, considering that the scope of text edited by ScoPE is limited to the most recently generated b tokens (1) and that tokens generated prior remain unchanged (2), it would be possible to limit the number of tokens whose hidden states need to be recomputed to a maximum of b during the ScoPE generation process. This approach is expected to significantly reduce the cost associated with recomputing hidden states.
Analysis of the fluency of the text generated by ScoPE Despite ScoPE generation achieving improved fluency (as shown by PPL) over existing baselines (DExperts, GeDi, Mix&Match), we observed an increase in PPL when using ChatGPT and LLaMA2 as backbone models. We interpret this as follows: ScoPE generates text constrained by the overall characteristics of the target corpus including the general fluency of the target corpus.
Table 13 presents the average PPL across different subsets of the Amazon review corpus which corresponds to the target corpus on which ScoPE was trained. We note that for category-related subcorpora, the values are generally around 15, which is consistent with the PPL scores of the text generated through ScoPE as seen in Table 1. Additionally, it is observed that for sub-corpora related to sentiment, the Positive corpus generally has a lower PPL than the Negative corpus. This trend also appears consistently in the generation results, where outputs targeting positive sentiment generally have lower PPL than those targeting negative sentiment (as shown in Tables 4, 8, 9). Therefore, while the target score (smlm(x)) learned by ScoPE includes overall fluency-related information, leading to an improvement in fluency as the number of edit iterations (N ) increases, the text generated through ScoPE is inherently constrained by the fluency of the target corpus it was trained on.
This interpretation suggests that for the LLaMA2 and ChatGPT backbone models, which generate text with a lower PPL than the original target corpus, it is challenging for ScoPE to produce text with a lower PPL than that of the backbone models.
However, as the overall low PPL results in Table 3 suggest, ScoPE significantly benefits from the fluent text generation capabilities of the backbone model, effectively utilizing its capacity.

heading: J Generated Samples
text: We provide examples of the text generated using ScoPE for various controlled text generation tasks in Tables 22 to 26.

heading: K Usage of AI Writing Assistance
text: This paper was written with linguistic support from the AI assistant ChatGPT, which offered paraphrasing, spell-checking, and polishing of the author’s original content. No other assistance was received beyond this support.

heading: Future Work
text: , Ltd (IO221213-04119-01), and the BK21 FOUR program of the Education and Research Program for Future ICT Pioneers, Seoul National University in 2024