GRACE: Gradient-guided Controllable Retrieval for Augmenting Attribute-based Text Generation
Year: 2023

Abstract:
Attribute-based generation methods are of growing significance in controlling the generation of large pre-trained language models (PLMs). Existing studies control the generation by (1) finetuning the model with attributes or (2) guiding the inference processing toward control signals while freezing the PLM. However, finetuning approaches infuse domain bias into generation, making it hard to generate out-of-domain texts. Besides, many methods guide the inference in its word-by-word generation, pushing the word probability to the target attributes, resulting in less fluent sentences. We argue that distilling controlling information from natural texts can produce fluent sentences while maintaining high controllability. In this paper, we propose GRAdient-guided Controllable rEtrieval (GRACE), a retrievalaugmented generation framework to facilitate the generation of fluent sentences with high attribute relevance. GRACE memorizes the semantic and attribute information from unlabeled corpora and applies a controllable retrieval to obtain desired information. For the generation, we design techniques to eliminate the domain bias from the retrieval results and integrate it into the generation model. Additionally, we propose a gradient-guided generation scheme that iteratively steers generation toward higher attribute relevance. Experimental results and quantities of examples verify the effectiveness of our method.

heading: 1 Introduction
text: Controlling the text generation model toward a specific direction remains an active research area, covering many tasks, including storytelling, text debiasing, and attribute-based generation (Xu et al., 2020; Liu et al., 2021; Dathathri et al., 2019). Attribute-based text generation requires generating text that satisfies the given attribute, which is a control code for a specific topic, sentiment, or
∗Corresponding Authors.
style (Prabhumoye et al., 2020; Zhang et al., 2022). Pre-trained language model (Radford et al., 2019) (PLM) can generate fluent texts by learning on large corpora but is difficult to control because it does not learn to adapt controlling signals.
Some researchers re-train a PLM supervised with control signals (Keskar et al., 2019; Zhang et al., 2020) or fine-tuning on domain-specific data (Bakker et al., 2022). CTRL (Keskar et al., 2019) pre-trains with texts from the Internet and extracts control code from URLs. PPVAE (Duan et al., 2020) fine-tunes part of the parameters for the target condition to bridge the conditional latent space and the global latent space. These methods bring high controllability and fluency to the generated text by modeling the relationship between the attribute and its contexts from supervised data. However, attribute-based supervised datasets usually derive from some specific domains (see App. F). Fine-tuning on those datasets brings in not only attribute information but also domain bias. The generated texts, without eliminating the domain bias, likely fall into the specific domain and lack the generalization ability across domains. Besides, the computational overhead of re-training a large PLM is becoming increasingly expensive (Liu
8377
et al., 2021). To address the above issues, researchers develop
inference-based approaches that freeze the PLM and affect the generation preference at the inference stage (Zhang et al., 2022). Many studies influence the preference of words according to a discriminator (Krause et al., 2021; Yang and Klein, 2021) or bag-of-words (Pascual et al., 2021; Dathathri et al., 2019). FUDGE (Yang and Klein, 2021) adjusts word probabilities with the discriminator’s prediction of whether the future generation satisfies the attribute. K2T (Pascual et al., 2021) encourages generating words similar in semantics to the attribute. As prevailing auto-regressive inference is decomposed into multiple steps to conduct wordlevel generation, the above inference-based methods always push the word-level probability toward the target attribute. It may break the natural inference processing, leading to less fluent sentences.
We argue that inference-based methods require guiding information that satisfies both attribute and common language patterns to achieve attributebased text generation. The patterns derived from natural language ensure fluency and grammaticality. Accordingly, it would be better if the controlling information comes from a natural text span.
In this paper, we propose to augment attributebased generation through gradient-guided controllable retrieval (GRACE)1, considering the target attributes (see Fig. 1). Specifically, we train a discriminator to compute the attribute distribution of a given context. We build a retrieval repository storing natural text with its semantic and attribute information distilled from unlabeled data. The generation model extracts attribute-related information with similar semantics through a controllable retrieval. We design strategies to disentangle the irrelevant attributes from the retrieval results and fuse the PLM representations into the generation process. Additionally, we propose an algorithm that iteratively revises the stepwise generation based on gradients. By optimizing toward the target attribute, the algorithm retrieves information with more vigorous attribute intensity, thus improving the attribute relevance of the generated text.
Our contributions are threefold: 1) We propose an attribute-based generation framework that leverages unlabeled corpora with controllable retrieval. 2) We design a gradient-guided generation algorithm that iteratively guides the retrieval to gen-
1Our code is available at github.com/araloak/grace
erating with suitable attributes. 3) Our method surpasses strong baselines in the sentiment- and topic-controlled generation on attribute controllability and fluency.

heading: 2 Related Work

heading: 2.1 Attribute-based Generation
text: Researchers focus on attribute-based generations in two directions: training-based and inference-based approaches. The training-based methods either update the entire model or attach the model with additional parameters. They explore different methods, including pre-training conditional language models (Keskar et al., 2019; Zhang et al., 2020) and fine-tuning the PLM to incorporate desirable attributes (Bakker et al., 2022). Cocon (Chan et al., 2020) conditions on word- and phrase-level content to steer generation. Bakker et al. (2022) finetune through reinforcement learning and design a reward function for evaluating whether the generation agrees with the constraint. Besides, Qian et al. (2022) propose to learn attribute-specific prompts and Yu et al. (2021) train attribute-agnostic alignment functions. These approaches are becoming increasingly expensive due to the growing size of recent PLMs (Liu et al., 2021).
Many studies investigate inference-based strategies that affect the generation probability while freezing the PLM. PPLM (Dathathri et al., 2019) updates the hidden states toward the target tokens. GeDi (Krause et al., 2021) and FUDGE (Yang and Klein, 2021) alter the next word probability according to a step-wise attribute discriminator or bag of words. DEXPERTS (Liu et al., 2021) combines the output distributions from attribute-specific expert and anti-expert models. There are also studies that either consider attributes in energy-based models (Khalifa et al., 2020; Mireshghallah et al., 2022) or propose attribute-sensitive decoding algorithms (Kumar et al., 2021; Gu et al., 2022). Nevertheless, these studies guide the off-the-shelf PLMs implicitly with signals from other models and do not explicitly leverage retrieval systems. Therefore, as an inference-based approach, our method constructs a retrieval repository to augment attributebased generation.

heading: 2.2 Retrieval-augmented Text Generation
text: Retrieval-augmented text generation assists the generative model with the information retrieval technique. It achieves state-of-the-practice results in
many tasks, including dialogue generation (Wu et al., 2021; Zhang et al., 2021), machine translation (Khandelwal et al., 2021; Meng et al., 2022), and language modeling (Khandelwal et al., 2020).
The community explores different ways to integrate the retrieved data into text generation. One line of work requires training models to learn to use retrieval knowledge. Bulte and Tezcan (2019); Xu et al. (2020) augment the model inputs by retrieving and concatenating similar samples. Hua et al. (2019); Bapna and Firat (2019); Izacard and Grave (2021) encode the retrieved texts and fuse them with attention mechanisms. Another line of studies explicitly extracts a skeleton from the retrieved data and trains the model to complete or revise it (Guu et al., 2018; Cai et al., 2019a,b). Another group is training-free methods that directly incorporate the retrieval results at the inference stage. Wang et al. (2022) prompt PLM with retrieved similar samples. Khandelwal et al. (2020); He et al. (2021); Khandelwal et al. (2021) facilitate inference with cached PLM context representations.
Our work belongs to the training-free approach. To the best of our knowledge, the existing methods do not conduct controllable retrieval in attributebased generation, which is the target of this paper.

heading: 3 Method
text: Our framework consists of three parts (see Fig. 2): (1) Attribute Discriminator conducts attribute classification with a discriminator D to evaluate if a given context satisfies the target attribute. (2) Retrieval Repository builds a repository R with unlabeled corpora, which carries a mapping of a context Xn to its next word xn+1. R supports reading operations to provide information that is semantically similar to the query and related to the target attribute. (3) Generator generates a sentence based on a prefix with a PLM G. At each step, G retrieves (read) information from R, reduces the effect of domain-specific vocabulary, and integrates it into a neural network model to generate the next word.
The above modules collaborate to conduct attribute-based generation. We design a gradientguided retrieval-generation framework that steers generation toward the target attribute at each step and polishes the retrieved text guided by the gradient, where the gradient respects the target attribute (mentioned in Sec 3.4).

heading: 3.1 Attribute Discriminator
text: D consists of a context encoder, a classification layer, and a language modeling layer. The encoder transfers texts to context representations. The classification layer maps a context representation to an attribute vector, which can be used for attribute classification with an additional softmax layer. The language modeling layer maps a context representation to word probability distribution. We perform the classification with the encoder and classification layer. To obtain D, we initialize our encoder and language modeling layer with a pre-trained language model. Then, we fine-tune the encoder and the classification layer on a classification dataset.

heading: 3.2 Retrieval Repository

heading: 3.2.1 Repository Construction
text: We construct a retrieval repository R on unlabeled corpora via our discriminator D and generator G. The repository comprises numerous items, each containing three vectors (rs, rc, vc) representing the semantics, attribute-augmented semantics, and attribute distribution of a given context.
For a sentence Xn = {x1, x2, ...xn}, a subsequence is Xi = {x1, x2, ...xi} for any i <= n. To construct the repository R, for every subsequence Xi of every sentence in the corpora, we take the following steps: 1) G is a frozen PLM, we calculate Xi’s context representation rs by the text encoder in G. 2)We feed Xi to D’s encoder to obtain its attribute-augmented context representation rc. 3) We feed Xi+1 to D’s encoder and then the classification layer to obtain its attribute vector vc. Finally, we define (rs, rc, vc) as a repository item for Xi (see the repository items in Fig. 2). Notice that vc
measures the attribute distribution considering the next word of the current subsequence.

heading: 3.2.2 Repository Retrieval
text: A controllable retrieval finds the repository items similar to the query and concerning the target attribute. To retrieve for a given query text, we feed the context to the generator G to obtain a context representation rs. Then, we search for the items whose rs are highly similar to the query’s rs mentioned above from the repository. Further, we retrieve two sets of items with high attribute relevance as retrieval results.
PkNN (xi+1|c,Xi) ∝ (1) PkNN (xi+1|Xi) ∗ P (c|Xi, xi+1)
The idea of retrieval follows the heuristic in Eq. 1 inspired by Krause et al. (2021) (see deductions in APP. A). In the retrieval results, higher PkNN (xi+1|c,Xi) indicates (1) a better semantic coherence between the next word xi+1 and subsequence Xi, and (2) a higher attribute relevance with the attribute c. We design the following strategies to model the two probabilities accordingly:
• Semantic Retrieval. To boost PkNN (xi+1|Xi), we search for items similar to the context Xi in semantic. As step 1 in Fig. 2, we take Xi’s context representation rsXi from G as the input. We search for K nearest items from the repository according to the similarities between the stored items’ context representations rs and rsXi . The algorithm returns a set N , which provides auxiliary semantic information to facilitate the next word prediction (Khandelwal et al., 2020).
• Attribute Retrieval. We select two subsets of highly attribute-relevant items to increase P (c|Xi, xi+1). First, we select items from N to compose a subset N+, where similarities between the items’ attribute vectors vc and the target attribute c excel a threshold p. The similarity is the cosine similarity between the item’s attribute
vectors vc and the one-hot representation of c. vc
measures the attribute distribution considering the next word of a subsequence. Thereby, we increase the possibility of c by considering the next step generation preference. We denote ¬c as the antitarget attribute 2 and obtain N− following the above procedure considering ¬c. (The following Sec. 3.3.1 employs N− to remove the domain bias from the retrieved information). In this way, we acquire items whose attributes are the most relevant to the target attribute.
Finally, the retrieval operation returns N+ and N−. The two sets contain items that are highly correlated with the target attribute and non-target attributes, respectively. Notice that the context representations in both N+ and N− are semantically consistent with the current subsequence.

heading: 3.3 Generator
text: The generator G generates texts based on a given prefix and considers the target attribute. At each generation step, G retrieves from R, removes the
2¬c includes all the undesirable attributes. For example, if the target attribute c is "Technology" and there are four attributes in total, all the remaining attributes are ¬c (i.e. Business, World News, Sports in the Agnews dataset).
irrelevant domain bias from the retrieval results, and integrates them into the generation model to produce the next token.

heading: 3.3.1 Representation Debiasing
text: We resolve the domain bias from the retrieved information and aim to eliminate domain-specific information from the generated sequences. We call the processing “debiasing”. In most existing attribute-based generation methods, domain bias exists in the generated text where its attribute entangles with the text domain since the attribute-based training corpora usually come from a limited set of domains (Yu et al., 2021) (see App. F).
At the i-th generation step, G encodes the current subsequence to query the repository R to obtain two sets of items: N+ and N−. Afterward, we feed the attribute-augmented context representations rc from each set into D’s language modeling layer to obtain the next word probability distributions. Then, we average the values within each set and acquire P+kNN (xi+1|c,Xi) and P−kNN (xi+1|¬c,Xi) for N+ and N−, respectively. Lastly, we calculate their difference to obtain ∆P (xi+1|c,Xi) = P+kNN (xi+1|c,Xi) − P−kNN (xi+1|¬c,Xi).
The intuition is that when the retrieval repository is rich in domain-specific expressions, the retrieval results of c alone may produce many domain-specific language patterns that are not necessarily relevant to the desirable attribute. However, if a word has a high probability on both P+kNN (xi+1|c,Xi) and P−kNN (xi+1|¬c,Xi) by retrieving with both c and ¬c, the word is likely critical to the domain instead of the target attribute c. If P+kNN (xi+1|c,Xi) is high while the P−kNN (xi+1|¬c,Xi) is relatively low, it indicates that the word xi+1 is insignificant to the domain but essential to the target attribute. Therefore, the above operation eliminates the domain bias in Xi’s semantic-similar neighbors originating from R’s repository corpora.

heading: 3.3.2 Representation Integration
text: We design a strategy to integrate the debiased information into PLM’s probability to produce the next word. Intuitively, a token is desirable if it is consistent with the given context and closely related to the target attribute. We denote the word probability of the PLM in G as PLM (xi+1|Xi) and integrate it
with ∆P (xi+1|c,Xi) as:
Sfuse(xi+1, c,Xi) = (2) λ∆P (xi+1|c,Xi) + (1− λ) ∗ PLM (xi+1|Xi)
In Eq. 2, λ is a factor measuring the controllability of the target attribute c in predicting the next word. We consider λ(i) as a step-dependent control signal that decreases linearly with the generation step i:
λ(i) =    λmin − λ0 I
∗ i+ λ0 i <= I λmin i > I,
where λ0 is the initial rate at the 0-th step, λmin is the minimum rate, and I is a pre-defined step number. With fixed λ0 and λmin, a larger I allows more steps to receive higher controllability.
After the integration, we normalize the score and use the existing decoding strategy (e.g., top-k sampling) to generate the next token.

heading: 3.4 Gradient-guided Generation
text: We propose a gradient-guided generation to pilot the generation toward the target attribute. We iteratively evaluate the current subsequence at each step and revise it until its attribute becomes satisfactory.
• Subsequence Evaluation. We evaluate if the current subsequence satisfies the target attribute. We concatenate the generated word xi+1 at the i-th step with the current subsequence Xi to obtain Xi+1. Then, we feed Xi+1 into the discriminator D and determine whether it matches the target attribute (in Sec. 3.1). We accept Xi+1 for the next generation step if it satisfies the target attribute. Otherwise, we save the gradient of D’s encoder and classification layer ∆Θ to help update Xi+1.
• Gradient-guided Subsequence Update. We enhance the subsequence’s relevance with the target attribute to help generate words with stronger attribute intensity. We optimize D’s encoder and classification layer according to ∆Θ to obtain DΘ−∆Θ. We feed Xi into the encoder of DΘ−∆Θ to acquire the updated attribute-augmented context representation r
′c i . Furthermore, based on r ′c i ,
we employ the retrieval steps 3.2.2 and generation steps 3.3 introduced in the above modules to obtain new retrieval results and generate a new word x
′ i+1. So far, we have completed an iteration of
the gradient-guided generation.
When DΘ−∆Θ is optimized toward the target attribute, its encoder produces context representations containing richer attribute-related information, which is helpful for retrieving texts with the target attribute. Hence, rci matches with items more related to the target attribute during retrieval, which in turn helps generate the next token x
′ i+1
more related to the desirable attribute.
In our framework, we first train the discriminator D (Sec. 3.1) and build the retrieval repository R (Sec. 3.2.1). At the i-th generation step, we follow the retrieval steps in Sec.3.2.2 and use G to generate a new word (Sec. 3.3). Afterward, gradientguided generation (Sec. 3.4) optimizes the generation results in iterations, which calls retrieval (Sec. 3.2.2) and generation (Sec. 3.3) until it satisfies the attribute requirement. 3

heading: 4 Experiments

heading: 4.1 Experimental Settings
text: Hyperparameters. We experiment on sentimentand topic-controlled generation tasks. We initialize the D and G with GPT2-medium (Radford et al., 2019). We build our repository with FAISS (Johnson et al., 2021) for fast retrieval. To evaluate GRACE in different control intensities, we experiment on GRACE-20, GRACE-40, and GRACE-80, whose threshold step numbers I are set to 20, 40, and 80, respectively. We follow the reported settings for the baselines. More details are in App. E. Datasets. We use one-half of the IMDB (Maas et al., 2011) dataset to train our discriminator for the sentiment-controlled generation and use another half of the IMDB, the DailyDialog (Li et al., 2017), and the Amazon (Ni et al., 2019) dataset to build the retrieval repository. Following Yu et al. (2021), We use one-half of the Agnews dataset (Zhang et al., 2015) to train a topic classifier for evaluation in the topic-controlled generation. We use another half of the Agnews dataset to train our discriminator and build the retrieval repository, which also contains the target sentences of the Xsum (Narayan et al., 2018) dataset. We follow the prefixes in Dathathri et al. (2019) to prompt the sentiment- and topic-controlled generation. Evaluation Metrics. We follow the standard practice in attribute-based generation for automatic evaluation (Dathathri et al., 2019; Liu et al., 2021).
3The gradient update in Sec. 3.4 only affects the current generation output and does not influence other generated sentences.
(1) Following Liu et al. (2021), we measure Attribute Relevance with a HuggingFace’s sentiment classifier 4 that is trained on SST-2 dataset (Socher et al., 2013) to evaluate whether the generation results satisfy its target sentiment. Following Yu et al. (2021), we train another BERT-based topic classifier with the above subset of the Agnews dataset. (2) Following (Dathathri et al., 2019), we evaluate the fluency of the generated text with model-based perplexity (PPL) via GPT2-large.
For human evaluation, we evaluate the generated text on overall quality (Qual), attribute relevance (Attr), and domain resemblance (Domain) with a 5-point rating scheme. Qual measures whether the generated text is grammatically correct and semantically appropriate. Attr evaluates whether the generation output agrees with the desirable attribute. Domain evaluates how likely the generation result belongs to the domain of the data that trains the discriminator5. For GRACE, Domain also evaluates whether its generation seems like the text from the repository corpora. Baselines. GPT2-F concatenates attribute with the generation prefix and fine-tunes GPT2-medium. PPLM (Dathathri et al., 2019) perturbs a PLM’s hidden states based on gradients from the discriminator or bag of words to control the generation. FUDGE (Yang and Klein, 2021) trains a discriminator to determine whether the future generation satisfies the target attribute. GeDi (Krause et al., 2021) uses GPT2-XL for generation and increases the probability of attribute-related words with Bayes’ Rule. For a fair comparison, we also implement GeDi-M with GPT2-medium and finetune it on the retrieval corpora to obtain GeDi-M-F. AA (Yu et al., 2021) learns an attribute alignment to guide the PLM for attribute-based generation. Based on BERT, MM (Mireshghallah et al., 2022) samples attribute-related texts according to a combination of scores from off-the-shelf PLMs. Except for GeDi and MM, our baselines are based on GPT2-medium for a fair comparison.

heading: 4.2 Overall Performance
text: Fig. 3 and Tab. 1 show the results of all methods on automatic and human evaluations in both sentiment- and topic-controlled generation. En-
4huggingface.co/gchhablani/bert-base-cased-finetunedsst2
5Domain is inapplicable for methods that control with a bag of keywords (e.g., FUDGE, MM, and PPLM in the topic-controlled generation).
hancing the controllability of attributes tends to result in a less fluent generation, and vice versa (Liu et al., 2021). Therefore, we demonstrate the automatic evaluations in Fig. 3 to show that GRACE achieves a better trade-off between attribute controlling (accuracy) and generation fluency (PPL). Except for GPT2-F, GRACE achieves the best performance in both automatic and human evaluations. While GPT2-F excels GRACE in attribute accuracy, it is the worst in domain resemblance, indicating that fine-tuning on domain-specific data makes the PLM malfunction in the other domains. (see cases in Tab. 13). Existing labeled datasets for attribute-based generation only cover very few text domains (e.g., movie and restaurant reviews), thus limiting PLM’s generation ability in other domains (Krause et al., 2021; Yu et al., 2021). GRACE-20 outperforms FUDGE, PPLM, and AA in attribute accuracy when GRACE-20 achieves similar or better PPL in the sentiment-controlled generation. PPLM and FUDGE behave similarly in the topic-controlled generation. Although PPLM and FUDGE achieve low PPL, their Qual is worse than GRACE-20. The reason is that PPLM may degenerate toward repeating the same word when the
PLM’s latent representations are not properly updated (Dathathri et al., 2019) (see cases in Tab. 10). Similarly, FUDGE may repeat specific keywords because it increases the possibilities of a limited number of keywords (see cases in Tab. 9).
In both sentiment- and topic-controlled generation, GeDi, GeDi-M, GeDi-M-F, and MM have higher PPL when their attribute accuracy is similar to or worse than GRACE-80. GeDi underperforms GRACE-80 in automatic and human evaluation even with a larger PLM. Notice that GeDi-M-F performs worse than GRACE-80, meaning that fine-tuning PLM on the retrieval corpora is suboptimal in incorporating the attribute information into generation. MM’s attribute accuracy drops in the topic-controlled generation. However, it maintains a high PPL and is low on Qual, meaning that its decrease in controlling accuracy does not lead to the gain of text fluency.
By adjusting the threshold step number I , GRACE can control the trade-off between text fluency and attribute accuracy. GRACE allows near 100% attribute accuracy and can achieve a low PPL of 12.99, which is − R (equivalent to generating with GPT2 only) in Fig. 4. With the increase of retrieving steps in GRACE, its attribute accuracy improves. Notice that the accuracy improvement between retrieving 20 and 40 steps is more significant than the improvement between 40 and 80 steps. The trade-off between perplexity and attribute accuracy is more efficient during the early generation steps (i.e. GRACE-20). App. 4.6 exemplifies that GRACE excels the baselines in a case study. We report the exact PPL and attribute accuracy in App. B and show that GRACE still outperforms the baselines when we adjust their hyperparameter to re-balance the attribute accuracy and generation fluency during generation.

heading: 4.3 Ablation Study
text: Fig. 4 and Tab. 2 show the ablation studies on our model components6. Compared to the model variants, our full model with different retrieving steps (GRACE-20, GRACE-40, and GRACE-80) achieves the best attribute accuracy under similar perplexity. − R generates without retrieving from the repository and is equivalent to generating with PLM only. − Attr R discards the attribute retrieval stage and integrates the outputs from the semantic retrieval into text generation. − R and − Attr R perform poorly on attribute accuracy, indicating that the controllable retrieval with the attribute is crucial in controlling the generation direction (see App. C for analysis on retrieval results). − Semantic R retrieves from the repository ignoring the context representations and only considers attribute vectors. GRACE-80 outperforms − Semantic R in both metrics, showcasing that semantic retrieval helps generate fluent texts (Khandelwal et al., 2020). As − Semantic R produce unreadable texts, its domain resemblance is relatively low.−D Rep integrates the context representation distilled from the frozen PLM instead of the discriminator to predict word probability. −D Rep achieves a certain degree of controllability; however, the accuracy is still lower than GRACE-40. Information distilled from the attribute-agnostic PLM is less sensitive to the attribute than the that from the attribute discriminator, resulting in poor controllability. It verifies that the attribute-sensitive discriminator produces biased context representation toward attributes. − Debias does not consider the anti-target control signal while integrating word probabilities and re-
6We set I = 80 for our model variants.
tains domain bias in retrieval results that leads the generation toward a fixed style. − Debias is the highest in domain resemblance and underperforms GRACE-40 in attribute accuracy, rendering the effectiveness of our debiasing method. − Revision generates without the gradient-guided generation scheme to revise the poorly controlled generation and achieves low attribute accuracy. Its poor performance indicates that our gradient-guided generation is crucial to accurately steer the generation toward the target attribute.

heading: 4.4 Analysis of the Attribute-augmented Context Representation
text: To visualize the improvement of our variants, we show the entanglement of context representations with different attributes in Fig. 5, which analyzes the attribute information encoded in the attributeaugmented context representation. Given the same prefix under different attributes, we display the context representation rs from the generator G, the attribute-augmented context representation rc from the attribute discriminator D, and the updated r ′c
from the gradient-guided generation at each generation step in Fig. 5 using t-SNE. From left to right of the figure, the distribution of representations in the vector space with the same attribute becomes less sparse. Besides, the representations with different attributes are more clearly dispersed. Trained on the attribute-sensitive dataset, D encodes attribute information into rc, making it more distinguishable in the vector space. Therefore, it encourages the generation to favor attribute-related words. rc is further optimized toward the target attribute in the gradient-guided generation. Therefore, the updated context representation r
′c concerning the same attribute is more concentrated, and the r
′c with different attributes is more separable. Hence, r
′c can match with more attribute-related items and help the gradient-guided generation to update the subsequence toward the desired direction.
4.5 sectionAnalysis of Inference Speed We analyze the time overhead of our method against other inference-based approaches. For a sentence of 80 words, GRACE requires 10 seconds per generation, while GeDi, FDUGE, and PPLM take 4, 6, and 30 seconds per generation, respectively. MM takes more than 360 seconds to generate and optimize a sentence. GRACE is slightly slower than GeDi and FDUGE but much faster than PPLM and MM.
In our method, the retrieval and gradient backpropagation are the most time-consuming operations. In the experiments, we find that the early generation sets the tone for the entire generation and is the key to achieving a controlled generation. For example, if the generation starts with “The pizza is awful”, the generated result tends to imply a negative sentiment. Therefore, we provide the strongest control signal in the early stage through the stepdependent λ that declines with generation and stop retrieving after a few steps. Based on the same observation, we also limit the number of iterations of the gradient-guided generation to save more time. Our generation speed can be further reduced with other speed-up strategies and better hardware support. In the future, we will explore faster generation schemes.

heading: 4.6 Case Study
text: We demonstrate cases of each attribute in both sentiment- and topic-controlled generation in tables from Tab. 8 to Tab. 12. GRACE produces fluent and attribute-related sentences in all cases. PPLM sometimes degenerates when its update size is inappropriate (see Tab. 10). FUDGE increases the possibilities of the given attribute-related bagof-words, thus tends to repeat specific keywords despite their incoherence (see Tab. 9 and Tab. 11). GeDi may generate unsatisfying sentences that are seemingly fluent but irrelevant in semantics. AA is
relatively inefficient in controlling the generation toward the target attribute (see Tab. 10). MM is likely to produce less fluent sentences with grammatical mistakes. Augmented by the retrieval corpora, GRACE produces text with few repetitions and is semantically consistent among sentences.

heading: 5 Conclusion
text: We propose GRACE, an attribute-based generation framework that controls the generation through controllable retrieval. We train a discriminator to distinguish attributes and build a retrieval repository with unlabeled corpora. We design strategies to remove the domain bias from the retrieval information. Moreover, we propose a gradient-guided generation scheme that iteratively updates the retrieval toward higher attribute relevance. Experimental results on two attribute-based generation tasks show that GRACE outperforms strong baselines in generation quality and attribute relevance.

heading: 6 Acknowledgement
text: This work is supported by the following foundations: the National Natural Science Foundation of China under Grant No.62025208, the Xiangjiang Laboratory Foundation under Grant No.22XJ01012, and 2022 International Postdoctoral Exchange Fellowship Program (Talent-Introduction Program) under Grant No. YJ20220260.

heading: A Deduction of the Retrieval Heuristic
text: By Bayes’ Theorem,
P (c, xi+1, Xi) (3)
= P (c|xi+1, Xi) ∗ P (xi+1|Xi) ∗ P (Xi)
P (c, xi+1, Xi) (4)
= P (xi+1|c,Xi) ∗ P (c|Xi) ∗ P (Xi),
so that we have:
P (xi+1|c,Xi) = P (xi+1|Xi) ∗ P (c|Xi, xi+1)
P (c|Xi) .
(5)
Given the current subsequence Xi and the target attribute c, P (c|Xi) is determined by the discriminator. Therefore, we obtain:
P (xi+1|c,Xi) ∝ (6) P (xi+1|Xi) ∗ P (c|Xi, xi+1),
which is Eq. 1

heading: B Details of Overall Perforamnce
text: We tune the hyperparameters in baseline models that affect the controllability of the target attribute to show that GRACE allows a more efficient tradeoff between attribute accuracy and generation fluency. We conduct experiments on our baselines except for GPT2-F since its generation lacks a clear signal to measure the controllability of the attribute. Besides, GPT2-F generates texts like its source domain, which are inapplicable in most situations (see examples in Tab. 13). Within GeDibased baselines, GeDi outperforms GeDi-M and GeDi-M-F in Fig. 3. Therefore, we tune GeDi to compare GRACE. As shown in Fig. 6, GRACE outperforms all baseline approaches under different settings. We show the baseline models’ best performance in Tab. 3.

heading: C Analysis of Retrieval Results
text: To exemplify the effectiveness of our retrieval method in providing semantically appropriate and attribute-related information, we display cases of the retrieval results with their future generations in Tab. 6 and Tab. 7. As each retrieved item comes from a piece of context Xi in the retrieval corpora, we collect the context’s next word xi+1 in a set N-BOW. Besides, we show the top-100 highprobability words in P-BOW from the word distribution P+kNN (xi+1|c,Xi), which is interpreted
11www.yelp.com/dataset
from the retrieved context representations. We observe that both sets provide many semantically consistent and attribute-relevant word candidates. Besides, we find that P-BOW contains more diverse word candidates that are more intensively co-related with the desirable attribute than N-BOW. For example, in Tab. 6, many words like “NASA", “Mars”, and “Pluto”, are unique in P-BOW. The reason is that N-BOW collects words that appear in the retrieval corpora, while P-BOW derives from the PLM’s generations, thus generalizing well to other potential preferable words. It verifies that our retrieval method can supply many coherent and attribute-relevant word candidates for each generation step.

heading: D Multi-attribute Controlled Gereation
text: Although we build GRACE considering a single attribute, our method can satisfy multiple attributes by retrieving with multiple attributes. Specifically, after the Semantic Retrieval in Sec. 3.2.2, we retrieve items that satisfy the desirable attributes. Apart from this, we follow the settings in the singleattribute controlled generation to generate texts. We showcase that apply GRACE to multi-attribute controlled generation in Tab.4.
E Implementation Details
We implement GRACE based on an open-source text generation framework Fairseq 7. We perform our experiments on a single GeForce RTX 3090 GPU with 24GB memory. To train our discriminator and the topic classifier, we set the dropout rate to 0.1 and use the Adam optimizer with a learning rate of 1e-5. The dimension of the context representations in the retrieval repository is 1024. We
7github.com/facebookresearch/fairseq
set the K and p in Sec. 3.2.2 as 1000 and 0.9 for retrieval and set the λ0 and λmin in Sec. 3.3.2 as 0.8 and 0.4 for representation integration. We set the maximum iteration number in Sec. 3.4 to 1. We freeze the GPT2-medium for the generation and use top-k sampling as the decoding scheme with k = 10. We run GRACE five times to obtain the evaluation results.
In the topic-controlled generation, PPLM and FUDGE control with a bag of topic-related words and experiment on topics that are different from GRACE. We collect keywords with a similar number on our topics and implement them to compare with our method. FUDGE does not experiment with the sentiment-controlled generation in its work. Similarly, we collect keywords representing different sentiments and follow the setting in its topic-controlled generation to guide the generation. As MM also controls generation with keywords, we implement MM with the above lists of words. Following Yang and Klein (2021), we set the maximum sentence length to 80 for all models. For each model, we run the generation 3 times on each prefix, thus obtaining 240 sentences (4 topics × 20
prefixes × 3) for the topic-controlled generation. Similarly, we obtain 90 sentences (2 sentiments × 15 prefixes × 3) for the sentiment-controlled generation.

heading: F Discussion of Existing Attribute-sensitive Datasets
text: We display several labeled datasets commonly utilized for attribute-based generation tasks in Tab. 5. All these datasets imply attributes in specific text domains. Fine-tuning on these texts makes the PLM entangle the attributes with domain-specific characteristics. Thus, the generation results of finetuned PLM tend to be biased toward the training data domain. For example, when fine-tuned on IMDB datasets for sentiment-controlled generation, GPT2 tends to generate texts thet seem like movie reviews (see Tab. 13 for the generation results).

heading: G Limitations
text: Our approach requires training a discriminator with an attribute classification dataset, which may be expensive in some scenarios. However, it is still applicable by collecting a small set of attribute-sensitive training instances and applying data augmentation techniques.
Our method is hard to achieve fine-grained control. We aim to address attribute-based generation that conditions on a given style, sentiment, toxicity, or topic. However, it cannot condition on a piece of content to control the generation. We encourage future works to explore retrieval-augmented generation with fine-grained control signals.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
G
7 A2. Did you discuss any potential risks of your work? The ethical impact of our research is the same as other text generation papers, whose ethical impact is widely discussed.
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 3
3 B1. Did you cite the creators of artifacts you used? 4
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? E
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? E
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? E
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? F
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. E
C 3 Did you run computational experiments? 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? E
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? E
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? E
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? E
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 4
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? E
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? E
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: We encourage future works to explore retrieval-augmented generation with fine-grained control signals. ACL 2023 Responsible NLP Checklist