Critic-Guided Decoding for Controlled Text Generation
Year: 2023

Abstract:
Steering language generation towards objectives or away from undesired content has been a long-standing goal in utilizing language models (LM). Recent work has demonstrated reinforcement learning and weighted decoding as effective approaches to achieve a higher level of language control and quality with pros and cons. In this work, we propose a novel critic decoding method for controlled language generation (CriticControl) that combines the strengths of reinforcement learning and weighted decoding. Specifically, we adopt the actor-critic framework and train an LM-steering critic from reward models. Similar to weighted decoding, our method freezes the language model and manipulates the output token distribution using a critic to improve training efficiency and stability. Evaluation of our method on three controlled generation tasks, topic control, sentiment control, and detoxification, shows that our approach generates more coherent and well-controlled texts than previous methods. In addition, CriticControl demonstrates superior generalization ability in zero-shot settings. Human evaluation studies also corroborate our findings.

heading: 1 Introduction
text: With recent advances in large language models (LMs), generating natural-sounding text has become feasible (Radford et al., 2019; Brown et al., 2020; Kim et al., 2021). However, such text can still be undesirable; for instance, it may be off-topic or biased and otherwise offensive, reflecting the harms in the real-world data (Keskar et al., 2019; Liu et al., 2021; Gehman et al., 2020; Hosseini et al., 2017; Krause et al., 2020; Lin and Riedl, 2021; Qian et al., 2022; Meng et al., 2022). To address this issue, several controlled text generation methods have been proposed (Lu et al., 2022; Yang and Klein, 2021; Dathathri et al., 2019).
∗ Work done during an internship at NAVER AI Lab. † Co-corresponding authors.
As shown in Figure 1, two major categories of approaches to controlled text generation currently exist: (a) reinforcement learning (RL) and (b) weighted decoding. In the RL approaches, generating a word at each time step is formulated as sequential decision-making leveraging an LM’s probability distribution over words (Wu et al., 2016; Paulus et al., 2017). For example, in the widely used Actor-Critic framework, the actor—a pretrained LM—predicts the next word, and the critic— a value network—evaluates the state, i.e., text generated thus far (Stiennon et al., 2020; Wu et al., 2021). Unfortunately, LM fine-tuning through RL often suffers from noisy gradient estimation (Greensmith et al., 2004), which may lead to unstable training, and eventually mode collapse (Upadhyay et al., 2022). Even when fine-tuned correctly, optimizing an entire LM for each target attribute, e.g., detoxification, is computationally expensive and memory
4598
inefficient for real-world applications (Guo et al., 2021).
In the weighted decoding approaches, the underlying LM is kept frozen, and only the final output probability distribution is adjusted (Holtzman et al., 2018; Dathathri et al., 2019; Kumar et al., 2021). More specifically, to condition an LM’s output p(x) on a target attribute, an approximation of Bayesian decomposition p(x|a) ∝ p(x)p(a|x) is computed with p(a|x) from an external discriminator (Yang and Klein, 2021; Krause et al., 2020). Note, these approaches involve a frozen LM with an independent discriminator for each target attribute; In comparison to the RL approaches, this plug-and-play structure allows more efficient training and memory use (Gu et al., 2022), but degrades the text quality in terms of fluency, diversity, etc. (Lu et al., 2022).
In this paper, we propose a novel controlled text generation algorithm, Critic-Guided Decoding for Controlled Text Generation (CriticControl), that re-weight the word distribution from an LM with predicted state-values from the critic network of RL. As a result, CriticControl raises the likelihood of words that increases the value of the next state over the current state while lowering that of others. Since the critic is trained with a frozen LM, the training is stable and efficient as in weighted decoding. Moreover, different critics can be used in a plug-and-play manner depending on targeted attributes or reward models. In other words, CriticControl combines the strengths of both the RL and the weighted decoding approaches.1
We demonstrate the efficacy of CriticControl through experiments on three controlled text generation tasks: topic, sentiment, and toxicity control. For all tasks, we find that CriticControl consistently outperforms previous methods in terms of control success, fluency, and diversity. Also, CriticControl exhibits strong zero-shot controllability on unseen topics. Finally, CriticControl is compatible with widely used sampling methods like top-k and top-p sampling (Holtzman et al., 2019; Fan et al., 2018) for improved text quality.

heading: 2 Related Works
text: Reinforcement Learning RL and the adversarial training formulation were first proposed in the context of language generation as an auxiliary algorithm to mitigate exposure bias in the teacher-
1Code will be available at https://github.com/ minbeomkim/CriticControl.
forcing training of sequences (Ranzato et al., 2015; Wu et al., 2016; Hu et al., 2017). The main motivation is to incorporate readily-available sequencelevel reward signals into training, such as BLEU or ROUGE (Paulus et al., 2017). The success of utilizing RL has been observed in a wide range of tasks, including summarization (Paulus et al., 2017; Wu and Hu, 2018; Stiennon et al., 2020; Ziegler et al., 2019), dialog modeling (Li et al., 2016; Yi et al., 2019; Jang et al., 2021; Upadhyay et al., 2022), neural machine translation (Wu and Hu, 2018; Nguyen et al., 2017), and style transfer (Gong et al., 2019; Ziegler et al., 2019). Furthermore, RL has allowed models to capture high-level human feedback (Paulus et al., 2017; Stiennon et al., 2020; Sharma et al., 2021; Ramamurthy et al., 2022), which is out of the current work’s scope, however. Various RL approaches have been explored so far, such as REINFORCE (Sutton et al., 1999; Ranzato et al., 2015; Wu et al., 2016; Sharma et al., 2021; Upadhyay et al., 2022), the actor-critic framework (Bahdanau et al., 2016; Nguyen et al., 2017; Jang et al., 2021), and PPO (Schulman et al., 2017; Nakano et al., 2021; Snell et al., 2022). More recently, Critic-guided methods have been studied to avoid the risk of RL’s policy learning collapse. GPTCritic (Jang et al., 2021) generates critic-guided texts for enriching task-oriented dialogue datasets. ILQL (Snell et al., 2022)’s critics guide supervised models to mimic the behavior of reinforced models. Our work is the first to incorporate weighted decoding for sequence-level reinforcement learning.
Weighted Decoding Freezing the language model and controlling the output probability distribution to suit the purpose is being actively researched (Holtzman et al., 2018; Ghazvininejad et al., 2017; Keskar et al., 2019; Sudhakar et al., 2019). This mainstream approach can be used in various scenarios, such as forcing the model to generate text that conforms to certain stylistic or content-based constraints or mitigating aggressive and toxic expressions (Kumar et al., 2021; Gu et al., 2022; Mireshghallah et al., 2022). The PPLM (Dathathri et al., 2019) generates texts by plugging a steering layer into the top of language models p(x). Then, the gradient from p(a|x) updates iteratively the last hidden representation to desired attributes. It only needs a few layers per each attribute but requires a lot of iterative computations; FUDGE (Yang and Klein, 2021) economizes p(x|a) by bayesian decomposition p(x|a) ∝ p(x)p(a|x)
with p(a|x) computing classifier instead of gradient update methods. For better linguistic quality, GeDi (Krause et al., 2020) and DExperts (Liu et al., 2021) take generative discriminator approaches with two LMs conflicting polarly on the desired attribute(e.g., positive vs. negative). They interpret p(a|x) as the degree of disagreement between two conflicting LMs. In contrast, CriticControl is advantageous in taking a generative manner with only a steering layer, like PPLM.

heading: 3 CriticControl
text: CriticControl is a controlled text generation framework that consists of a frozen pre-trained language model (Actor network) and an extra network for the state-value prediction (Critic network). Given a part of the sentence, the actor tries to complete the attribute-aware continuations with the critic’s support.
As an RL formulation, at each step t, states are given as a set of tokens x1:t−1 = {x1, ..., xt−1} and the attribute a. The policy πθ of the actor samples xt ∈ V from the next tokens probability of πθ(xt|xt−1) as
Pπ(xt|x1:t−1) = softmax (yt T ) , (1)
which output logits yt for words in dictionary V with temperature T to experience diverse trajectories. To train the policy network (i.e., LM), we use widely used optimization methods (Sutton et al., 1999) that take policy loss as ∇θJ(θ) = Eπ[ ∑end t=1At∇θ lnπθ(xt|x1:t−1)]. Note that At is the advantages function that measures how much the choice xt is better than the critic predicted, and we train the critic to minimize At. In our algorithm,
we freeze the actor model and only train the critic for flexible control, as described in the following section.

heading: 3.1 CriticControl Training
text: Different from the previous supervised training, we design a simple text generation framework via reinforcement critic learning. As shown in Figure 2, when the actor completes a given prompt, we let the reward model evaluate how well the completed sentence correlates with the attribute a. Then we give critic a reward rend = Pπ(a|x1:end) to calculate the temporal difference (TD) error δt = rt+γVπ(x1:t+1)−Vπ(x1:t), consisting of At. This TD error generalizes Vπ(x1:end) = Pπ(a|x1:end) to Vπ(x1:t) = Pπ(a|x1:t) (Sutton and Barto, 2018). Using the evaluation result rend and the text generation history x1:end, CriticControl minimizes the generalized advantage estimation loss (Schulman et al., 2015) to train critic network as follows:
Lcritic = end∑
t=1
( end−t∑
i=0
(γλ)iδt+i) 2, (2)
where γ is the reward discount factor and λ is the re-weighted averaging factor. We backpropagate this loss to align the critic’s prediction Vπ(xt) with unbiased empirical future returns to train the critic network. For the exploration in the training phase, we adopt a highly stochastic actor strategy. Specifically, an actor generates diverse outputs from language models by using high temperatures T > 1 as in Equation 1, which increases the entropy of output distributions for logits yt over a vocab of tokens V . And this leads to diverse text generation, and the critic can experience more diverse samples in
each episode. By repeating this simulation, we expect the critic to learn which decisions of language models will lead to a promising future.

heading: 3.2 Text Generation with CriticControl
text: Generating human-like text often requires stochastic decoding strategies. They truncate the unreliable long-tail on the probability distribution for sampling only on realistic token candidates (Holtzman et al., 2019; Fan et al., 2018). However, adjusting all probabilities in the vocabulary is very computationally inefficient. Therefore, to achieve both stochasticity and computational efficiency simultaneously, CriticControl steers the subset of vocabulary V ′ ⊂ V , consisting of top-k probability tokens from frozen language models with exact probability re-weighting
Pπ(xt|x<t, a) = Pπ(xt, x<t, a)
Pπ(x<t, a)
= Pπ(a|x≤t) Pπ(a|x<t)
Pπ(xt|x<t), (3)
to align the probability scale of adjusted words x ∈ V ′ and non-adjusted words x /∈ V ′. After adjusting the distribution, CriticControl can combine beam search and various sampling methods. In experiments, CriticControl adopts both top-k sampling and nucleus sampling by adjusting the top-10 word probabilities of V ′ ⊂ V .

heading: 4 Experiments
text: To evaluate the effectiveness of CriticControl, we conduct experiments on a variety of controlled text generation tasks, including topic control, sentiment control, and detoxification.

heading: 4.1 Topic Control
text: We conduct experiments on topic control tasks to generate topic-related text, starting with a prompt consisting of natural plain text independent of any topic. When implementing our framework, we use BART-large-MNLI (Lewis et al., 2019)2 as a reward model that computes the relevance between the texts and the topics. During the training, the critic is randomly given 1 of 7 topics (computers, space, military, legal, politics, science, and religion) (Dathathri et al., 2019) and learns to predict the semantic relevance between given topics and generated texts. For the baselines, we report
2https://huggingface.co/facebook/ bart-large-mnli
the results of pure GPT-2-medium (Radford et al., 2019), WDEC (Yang and Klein, 2021), PPLM, and FUDGE. Additional implementation details can be found in Appendix A.1.

heading: 4.1.1 Metrics and Evaluation
text: We evaluate the quality of the generated text in terms of success in controllability, fluency, and diversity. All baselines generate 80 tokens on 20 prompts × 7 topics = 140 comparisons. Previous works (Dathathri et al., 2019; Yang and Klein, 2021) measure controllability as the usage rate of their predefined topic-related words used for both training and evaluation. In contrast, our approach performs more general optimization and does not use any predefined topic-related words. Hence, we measure this ’success on control’ with human evaluation. Annotations answer the question, ‘Is the text relevant to a given topic?’ for all generated texts. Additionally, we also run the human evaluation of topic control success on the unseen topic (i.e., zero-shot setting), not directly related to training topics. We measure the fluency using two metrics: perplexity, which is calculated using the GPT-2-XL language model, and grammaticality, which is determined using the Roberta-based CoLA model (Warstadt et al., 2019) 3. Finally, we measure diversity (Li et al., 2015) using distinct n-grams normalized by text length, reporting distinct unigrams, bigrams, and trigrams as Dist-1, Dist-2, and Dist-3 scores, respectively.
Results As shown in Table 1, CriticControl significantly outperforms other baselines on topic controllability, fluency, and diversity. CriticControl achieves a superior control success rate, and this tendency is proportional to the size of the GPT-2 model. And even small-sized CriticControl beats the FUDGE in terms of topic controllability. Furthermore, CriticControl is able to well preserve the linguistic characteristics of the baseline GPT-2 as verified by the lower perplexity and the higher grammar scores. We argue that the sequential decisionmaking approach of CriticControl helps the system preserve the linguistic characteristics of the original GPT-2 during the training process. Additionally, CriticControl generates texts without any pre-defined "bag of words" about topics, unlike WDEC, PPLM, and FUDGE. This property makes the system freely choose the topic-related words and
3https://huggingface.co/textattack/ roberta-base-CoLA
results in a higher Dist-1 score. These experimental results verify the effectiveness of this reward-driven controllable text generation system as sequential decision-making.
Study Topic
Studied Topic
New Topic
Politics Military Science Religion Computer Legal
Space
Foods Climate Education Games
Cars Sports Music
Worlds Donald Trump
Awards
0 0.25 0.5 0.75 1
Figure 4: The CriticControl success rate per topic. It is examined for both the topic used for training and a new, previously unseen topic. The results indicate consistent performance on zero-shot topics.

heading: 4.1.2 Generalization on unseen Topics
text: Since the critic learns to predict semantic relevance between the given random topic and texts to generate, CriticControl is also able to generalize toward unseen topics other than seven training domains. In zero-shot results of table 1, CriticControl shows a high topic control success rate on new topics. As shown in Figure 4, CriticControl is even able to
control zero-shot topics such as ‘food, cars, sports, and music’ that are not seen during training. These results indicate that the usage of the general reward model makes it possible to evaluate the semantic relevance of unseen topic codes to the current texts. For example, in Table 3, CriticControl generates texts about Paul McCartney and the FDA from unseen topics music and food respectively, without using pre-defined dictionaries. These results prove that CriticControl obtains generalization ability by taking universal reward models, and this free setting of the reward model will enable various promising future research.

heading: 4.1.3 Human Preference Tests
text: There is a limitation to solely relying on automated evaluation for measuring text quality. Therefore, we run preference tests on CriticControl against GPT2 and FUDGE to validate that the text quality of CriticControl outperforms previous baselines while being on topics. For the preference test, we hire three annotators for each comparison pair and ask two questions Success Rate: 1) Which sentence is more related to the given topic? and Fluency: 2) Which sentence is more fluent?.
Results In the preference test, CriticControl also outperforms baselines on both topic control success rate and fluency, as shown in Figure 3. In the fluency comparison with GPT-2, CriticControl also wins, resulting from generating sentences within the area guaranteed by the reward model. Both automatic and human evaluation results indicate that CriticControl achieves a state-of-the-art topic control ability while preserving the original GPT-2’s text quality.

heading: 4.2 Sentiment Control
text: Next, we explore CrilticControl’s ability to steer text generation toward a specific sentiment. This sentiment control task aims to steer the model to complete positive movie reviews with any emotional prompt. We adopt the IMDB movie review dataset, containing highly polar (positive or negative) reviews of 2.5K for training and 2.5K for testing. We use the 8 starting tokens for each sentence to make prompts and generate 25 continuations for each prompt using all of the baseline systems. In this task, we additionally test ‘Could CriticControl enhance the reinforcement-learned LMs to achieve goals more appropriately?’. Additional implementation details are in Appendix A.2.

heading: 4.2.1 Metrics and Evaluation
text: We evaluate total 2.5K generations of each baseline in terms of positiveness, fluency, and diversity. We define the positiveness as the percentage of generations classified to ’positive’ from distilBERT classifier finetuned on the IMDB dataset, used as our reward model. In the same way as the topic control task, we measure fluency through perplexity and grammatically, using Dist-n for diversity metrics.
Results As shown in Table 2, CriticControl significantly outperforms the other baselines on the success rate of sentiment control, fluency, and diversity metrics. Also, we observe that generative controllers such as GeDi and CriticControl demon-
strate better performance than PPLM. Among them, GeDi outperforms CC-LM’s control performance by generating guided texts through the use of contradicting positive and negative CC-LMs. However, the different text generation strategies of GeDi and CC-LMs lead to a reduction in text quality, as indicated by the perplexity score. On the other hand, CriticControl trains the critic in a sequential decision view and allows the actor and critic to share the same experience, resulting in the best performance on metrics. And both the topic and sentiment control experiments show the effectiveness of CriticControl in improving grammatical correctness compared to naive GPT-2. We explain that this is because CriticControl increases the amount of information within the region identified by the reward model, whereas naive GPT-2 does not. Furthermore, our additional experiment on PPO in Table 2 shows that CriticControl even improves the performance of RL-finetuned language models, not just freezing language models. Overall, our results show that CriticControl is promising for extending the use of RL in downstream tasks.

heading: 4.2.2 Human Preference Tests
text: For human evaluation, we conduct preference tests by comparing CriticControl with GPT-2-medium and GeDi. We randomly select 200 samples from the test set and ask annotators to indicate 1) Positiveness: Which sentences are more positive, and 2) Fluency: Which are more linguistically fluent. As in the topic control experiment, we take the majority vote of 3 annotators for each comparison.
Results As shown in Figure 5, CriticControl is successful at generating positive text compared to other baseline systems. However, since CriticControl should force negative prompts to be positive, the fluency of the generated text is poorer than that of a naive GPT. On the other hand, when we compare CriticControl to GeDi, we observe that CriticControl has much better fluency while still maintaining a high level of positivity. We find that people show the same preference for positivity among the two models and choose equality for considerable samples, different from the automatic evaluation results. We explain that this is because people have difficulty choosing more positive sentences among two positive sentences, unlike distinguishing between positive and negative sentences.

heading: 4.3 Detoxification
text: LMs might generate offensive or biased responses that are risky. To remedy this issue, we conduct experiments on reducing the toxicity of LMs as another controlled text generation task. We use GPT2-Large as base LM and train the reward model using BERT-based classification (Devlin et al., 2018) models on a dataset from the Jigsaw Unintended Bias in Toxicity Classification Kaggle challenge 4. We use the prompts in RealToxiciyPrompts dataset (Gehman et al., 2020) which consists of 100K prompts. During the training, we use 90K prompts in this dataset to train the critic. We use the remaining 10K non-toxic test prompts as DEx-
4https://www.kaggle.com/c/ jigsaw-unintended-bias-in-toxicity-classification
perts and generate 20 tokens. We compare our approach with naive GPT-2-Large, PPLM, DAPT (Gururangan et al., 2020), GeDi, and DExperts (Liu et al., 2021). Detailed implementations are in Appendix A.3.

heading: 4.3.1 Metrics and Evaluation
text: We evaluate a total of 10k generated sentences for toxicity, fluency, and diversity. We use the Perspective API (Hosseini et al., 2017) 5 to classify sentences based on the most toxic token. We measure fluency and diversity as the same way in the previous section on topic control 4.1.
Results The experimental results in table 4 show that CriticControl outperforms other existing baselines. CriticControl effectively avoids using toxic language while maintaining the natural flow of the text. Since this task aims to remove toxic elements while preserving the amount of information, the diversity scores are all similar between baselines.

heading: 4.3.2 Human Preference Tests
text: For human evaluation, we conduct preference tests comparing CriticControl to GPT-2-large and DExperts. We randomly draw 200 samples from the results of the test set and ask annotators to indicate 1) Less Toxic: Which one is more rude or disrespectful (toxic comparison) and 2) Fluency: Which one is more grammatically correct and coherent? As in the topic and sentiment control experiment, we adopt the majority vote of three annotations for each comparison.
Results Human evaluation results in Figure 6 state that CriticControl is almost equivalent to the fluency of GPT-2 and exceeds the DExperts. Furthermore, the results verify that when evaluating the toxicity of the text generated from the same prompts, CriticControl consistently outperforms both GPT-2 and Dexperts.

heading: 4.4 Qualitative Analysis and Discussion
text: We analyze how the critic in our proposed method controls language models to generate the desired texts. As a qualitative analysis, we identify the probability change statistics of each word during the decoding step of CriticControl. We then discuss the most critic-favored and most critic-rejected words in each task. In topic and sentiment control, each word whose probability is quite improved through
5https://perspectiveapi.com/
the critic is highly related to the given topic or sentiment. For example, ‘terrorism, crime, and guns’ come up for political topics, ‘immunity and flu’ for science topics, and ‘play, stylish, and emoji ><’ for positive sentiment. One of the interesting findings is that the critic strongly rejects emotional words, such as ‘felt and depressed’, to generate sentences suitable for science topics. We explain this because language models struggle to generate scientific texts starting from emotional prompts. We also observe another notable point to discuss in the detoxification experiment. The detoxifier-critic recommends words related to science or technology to the language model while rejecting offensive and insulting words. The frequently recommended words such as ‘sci, application, and apps’ are factual words because the opportunity for the language model to make an aggressive remark disappears when a prompt for objective facts are created in the first place. These analyses verify how well CriticControl corrects the word’s probability to achieve its goal in terms of language models.

heading: 5 Conclusion
text: We propose CriticControl, a controlled text generation method that takes advantage of both reinforcement learning and weighted decoding. Through experiments on various controlled text generation tasks, we demonstrate that CriticControl can effectively guide language models toward desired attributes while producing high-quality texts. Additionally, we show that CriticControl has a strong generalization ability in zero-shot attribute control tasks by using a general reward model. One of the limitations of this approach might be its high computational cost to explore with ‘GPT3-scale’ lan-
guage models (Brown et al., 2020), and we expect that this can be addressed through offline reinforcement learning (Fujimoto et al., 2019) techniques in future research.
Limitations
Large language models over the GPT-3 have made significant progress in natural language generation, but applying the CriticControl method, and exploring through these large language models are computationally too expensive. To address this, offline reinforcement learning (Fujimoto et al., 2019) may be a promising option to minimize training costs. CriticControl also has inference speed degradation because additional inference costs are needed like other controlled text generation methods (Dathathri et al., 2019; Yang and Klein, 2021). The potential solution may be to use the action-value predicting critic (Yue et al., 2020), which would allow for real-time control of various attributes without affecting the inference speed of the language model. Recently, the impact of instruction models (Chung et al., 2022; Ouyang et al., 2022) on text generation has recently been highlighted in academic research. These models, which allow for control over the generated text via input manipulation, have become widely accessible on various attributes without extra computational costs. Future works will investigate the synergistic potential between the ‘inputside’ control of instruction-based models and the ‘output-side’ control of CriticControl.
Ethical Statement
We acknowledge that our reward-driven text generation system may lead to generating harmful or misleading content when used with undesired reward models. However, controlled text generation methods have the potential to address these ethical issues present in large-scale pretrained language models, for example, through the detoxification of language. Therefore, we emphasize the proper use of reward models to pursue the public good and believe that it is important to continue research in this area as these techniques can offer significant benefits.

heading: Acknowledgements
text: K. Jung is with ASRI, Seoul National University, Korea. This work has been financially supported by SNU-NAVER Hyperscale AI Center. This
work was partly supported by Institute of Information communications Technology Planning Evaluation (IITP) grant funded by the Korea government(MSIT) [NO.2021-0-01343, Artificial Intelligence Graduate School Program (Seoul National University)]

heading: B Human Evaluation
text: During human evaluation, the end of the sentence is almost incomplete because baselines generate a fixed number of tokens from the prompt. Therefore, we added ‘[...]’ for the end of all generated texts for comfortable reading. Then, we give instructions about ‘Don’t care about the incomplete last sentence for evaluating fluency, which is marked with [...] due to length limitation.’ to annotators as in Figure 8. Also, we notice ‘Please select equal only when it is really difficult to judge’ for more accurate human evaluation results in the preference test. We pay MTurk workers a competitive pay of more than $10
6https://www.kaggle.com/c/ jigsaw-unintended-bias-in-toxicity-classification
an hour. We hire the workers whose nations in one of the US, CA, UK, AU, NZ. We restrict the annotators whose HIT minimum hits are over 1000 and HIT rates are higher than 96%.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section ‘Limitations’

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section ‘Ethical Considerations

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section ‘Abstract’ and ‘1. Introduction’
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section ‘4 Experiments’

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section ‘4 Experiments’
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No, all artifacts we used are based on open-source and have no restriction for research

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section ‘4 Experiments’ and ‘Appendix’

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Section ‘4 Experiments’ and ‘Appendix’

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section ‘4 Experiments’ and ‘Appendix’
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section ‘4 Experiments’ and ‘Appendix’
C 3 Did you run computational experiments? Section ‘4 Experiments’

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section ‘Appendix’
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section ‘Appendix’

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section ‘4.4 Qualitative Analysis and Discussion’

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section ‘4 Experiments’. We report used pre-trained models served by Huggingface
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section ‘4 Experiments’ and ‘Appendix’

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Section ‘Appendix’

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section ‘4 Experiments’ and ‘Appendix’.

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Section ‘4 Experiments’ and ‘Appendix’.

heading: 3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?
text: Section ‘4 Experiments

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Section‘Appendix’

heading: Future Work
text: , 2019) techniques in future research. Limitations
Large language models over the GPT-3 have made significant progress in natural language generation, but applying the CriticControl method, and exploring through these large language models are computationally too expensive. To address this, offline reinforcement learning (Fujimoto et al. , 2019) may be a promising option to minimize training costs. CriticControl also has inference speed degradation because additional inference costs are needed like other controlled text generation methods (Dathathri et al. , 2019; Yang and Klein, 2021). The potential solution may be to use the action-value predicting critic (Yue et al. , 2020), which would allow for real-time control of various attributes without affecting the inference speed of the language model. Recently, the impact of instruction models (Chung et al. , 2022; Ouyang et al. , 2022) on text generation has recently been highlighted in academic research. These models, which allow for control over the generated text via input manipulation, have become widely accessible on various attributes without extra computational costs. Future works will investigate the synergistic potential between the ‘inputside’ control of instruction-based models and the ‘output-side’ control of CriticControl