Inference-Time Policy Adapters (IPA): Tailoring Extreme-Scale LMs without Fine-tuning
Authors: Ximing Lu
Faeze Brahman
Peter West
Jaehun Jung
Khyathi Chandu
Abhilasha Ravichander
Prithviraj Ammanabrolu
Liwei Jiang
Sahana Ramnath
Nouha Dziri
Jillian Fisher
Bill Yuchen Lin
Skyler Hallinan
Lianhui Qin
Xiang Ren
Sean Welleck
Yejin Choi
Venue: EMNLP 2023 Main (2023)
Keywords: Language Model
Text Generation
Reinforcement Learning
Inference-time Algorithm
TL;DR: We propose a lightweight policy adaptor to tailor extreme-scale language models to reflect arbitrary user objectives.

Abstract: While extreme-scale language models have demonstrated exceptional performance on a variety of language tasks, the degree of control over these language models through pure prompting can often be limited. Directly fine-tuning such language models can be effective for tailoring them, but it can be either extremely costly (e.g., GPT-3) or not even feasible for the broader community (e.g., GPT-4).

We propose Inference-time Policy Adapters (IPA), which efficiently tailors a language model such as GPT-3 without fine-tuning it. IPA guides a large base model during decoding time through a lightweight policy adapter trained to optimize an arbitrary user objective with reinforcement learning.

On five challenging text generation tasks, such as toxicity reduction and lexically constrained generation, IPA consistently brings significant
improvements over off-the-shelf language models. It outperforms competitive baseline methods, sometimes even including expensive fine-tuning. In particular, tailoring GPT-2 with IPA can outperform GPT-3, while tailoring GPT-3 with IPA brings a major performance boost over GPT-3 (and sometimes even over GPT-4). Our promising results highlight the potential of IPA as a lightweight alternative to tailoring extreme-scale language models.

Inference-Time Policy Adapters (IPA):
Tailoring Extreme-Scale LMs without Fine-tuning
Ximing Lu
Faeze Brahman V Khyathi Chandu +
Peter West 8 Jaehun Jung
Abhilasha Ravichander
Lianhui Qin
Prithviraj Ammanabrolu * Liwei Jiang * Sahana Ramnath
Nouha Dziri
Jillian Fisher
Bill Yuchen Lin Skyler Hallinan
Xiang Ren +
Sean Welleck & Yejin Choi
Allen Institute for Artificial Intelligence
University of Washington University of Southern California
Abstract
Frozen
Parameters
Learning
While extreme-scale language models have demonstrated exceptional performance on variety of language tasks, the degree of con- trol over these language models through pure prompting can often limited, Directly fine- tuning such language models can be effective for tailoring them, but can be either extremely costly (e.g., GPT-3) or not even feasible for the broader community (e.g.. GPT-4).
Base Policy (Extreme-Scale LM)
Prediction
Approximate Policy
Reward
Policy Adapter
We propose Inference-time Policy Adapters (IPA), which efficiently tailors a language model such as GPT-3 without fine-tuning IPA guides large base model during decod- ing time through lightweight policy adapter trained to optimize arbitrary user objective with reinforcement lcarning.
Tailored Policy Prediction
Adapted Inference
Figure I: Inference-time Policy Adapters (IPA) effi- ciently steer large-scale language model (such as GPT- 3) during decoding-ting -ime through lightweight policy adapter trained to optimize any arbitrary user objective with reinforcement learning.
On five challenging text generation tasks, such toxicity reduction and lexically constrained generation, IPA consistently brings significant improvements over off-the-shelf language mod- els. outperforms competitive baseline meth- ods, sometimes even including expensive fine- tuning. In particular, tailoring GPT-2 with IPA outperform GPT-3, while tailoring GPT- 3 with IPA brings major performance boost over GPT-3 (and sometimes even over GPT-4), Our promising results highlight the potential of IPA as a lightweight alternative to tailoring extreme-sed scale language models.'
brittle process due to LLMs being overly sensi- tive to the surface-fone form of the instructions (Perez et al., 2021; Lu al., 2022c). Furthermore, even with carefully written prompt, LLMs may still struggle fulfill certain task requirements due their inherent limitations (Liu et al., 2022a; Zong and Krishnamachari, 2022).
1 Introduction
Resturve-intensivve fine-tuning, through super- vised learning, and more recently reinforcement learning (RL) (Lu et al., 2022a) have shown promise in tailoring language models to arbitrary user-given objectives. RL, in particular, known for its generalizability and flexibility, allows mod- cls to learn from desired rewards. However, these methods require accessing and updating models parameters, which can be extremely large inac- cessible in state-of-the-ar models like GPT-4 (Ope- nAI, 2023b). This limitation makes fine-tuning unfeasible for the broader community.
Large language models (LLMs) have recently shown remarkable progress in various text gener- ation tasks by adapting to instructions exam- ples (Ouyang al., 2022: Brown al., 2020). However, the degree of control (e.g., the inclu- sion of keywords, avoiding harmful language) of- fered by these extreme-scale models through pure prompting is still limited (Lou et al., 2023; Webson and Pavlick, 2021). Moreover, prompting can be
OOr code is publicly available at: https:://githbb com/ CXimingLU/IPA

---

Alternatively, inference-time algorithms can tai- lor a language model without accessing its pa- rameters. These algorithms align language mod- els' outputs with desired lask/user-specifee prop- erties by adjusting the model's s output distribution based on certain task-specific heuristics, while leav- ing the underlying model untouched. Despite the progress, these approaches are either restricted specific tasks (Lu al., 2021, 2020), require domain- domain-specific knowledge (Liu et al., 2021a; Yang and Klein, 2021), suffer from expensive run-time at inference (Qin et al., 2022, 2021; Dathathri et al., 2020a), or have shown to be less effective com- pared to direct RL optimization (Lu et al., 2022a).
Background
In this section, we introduce our text generation set- ting ($2.1) and brief background on tailoring lan- guage models with reinforcement learning ($2.2). We then introduce our IPA algorithm for tailoring large language models without fine-tuning ($3).
2.1 Problem Setting
Text generation is the task of generating out- put sequence y given an input sequence x. We consider standard autoregressive language mod- els, which decompose a sequence's probability as ( -17 p x), where Pe is neu- ral network with parameters B. Intuitively, our goal to 'tailor' a pretrained model pe towards user- specified objective (e.g.. safety). Concretely, we assume that the objective quantified by a reward function R(y) E R. We then aim to adjust pe so that its generated sequences have high reward and reasonable language quality (e.g., fluency).
Drawing inspiration from RL and inference- time techniques, we propose Inference-time Pol- icy Adapters () (in) IPA), efficient and general- izable algorithm, which tailors large language model decoding-time toward desired objectives without linettunig it. To do so, IPA combines a large base LM's ss output distribution with that of a smaller-sized model (a lightweight adapter pol- icy), and optimizes the combined distribution to- wards a given objective with RL (Figure 1). IPA uses two key ideas to make learning efficient. First, IPA only updates the adapter's parameters, avoid- ing the need update the large base LM. Second, IPA replaces the large base model with an approx- imate policy-a smaller model that approximates the base model's distribution. The approximate policy is either a smaller model from the same lan- guage model family or distilled version of the base model. At inference time, we decode with the combined distribution of the base model and the trained policy adapter.
2.2 Preliminary: Tailoring LMs with RL
Online policy-based reinforcement learning has emerged effective way to adjust a language model towards reward function. Formally, these algorithms (c.g.. PPO (Stiennon al., 2022), Quark (Lu al., 2022b), or NLPO (Ramamurthy" et al., 2023)) optimize a language model pa towards gen- erating outputs y that maximize a given reward R:
0 = arg max R(y),
often along with regularization maintain lan- guage quality. At high-level, these algorithms use a policy pe to collect input-output examples, score the outputs with a reward function R, and update parameter to maximize the expected reward. Al- though the exact optimization may differ, we can view any online policy-based RL algorithms as a functions JRI. that take policy pe and reward function R the inputs and outputs optimized policy pa- with respect to R. Formally,
Experiments across five challenging text gen- eration tasks show that IPA brings consistent im- provements over off-the-shelf language models, outperforming competitive baselines - sometimes even including expensive fine-tuning. In particu- lar, tailoring GPT-2 with IPA can outperform GPT- 3, while tailoring GPT-3 with IPA brings ma- jor performance boost over GPT-3 (and sometimes even GPT-4). Our compelling highlight the promise of IPA lightweight alternative for tai- loring large language models wide range of objectives. IPA opens new ways to augment or cus- tomize extreme-seale language models using only academic-level resources.
fRI. (po. R; 0') 6*
(1)
Here 8' C denotes the subset of po parameters that are updated by the algorithm. The key idea behind IPA is to use full model pa to collect examples, but update small set of parameters 0',
3 Inference-time Policy Adapters (IPA)
We introduce Inference-time time Policy Adapters (IPA), a lightweight approach to tailor language models

---

towards a user-specified objective. IPA trains small adapter policy that adjusts the outputs of a (larger) base model at inference-time in order to maximize a reward. In doing so, IPA avoids the cost of updating the large base model, without the need to hand-design inference-time heuristics.
framework does not depend on a specific RL al- gorithm, but rather treats RL flexible plug-in optimization tool. As we will demonstrate later, IPA proves to be effective when paired with three different RL algorithms (Lu et al., 2022b; Schul- man et al., 2017; Ramamurthy et al., 2023), and in principle, it can easily integrate with others.
3.1 Policy Adaptation
We introduce the notion of 'tallorin'' used by IPA, which mainly involves three policies. First, IPA starts with base policy pg. which is the language model to tailor. Second, IPA introduces an adapter policy Ps. which a language model with the same output space as the base policy (i.e., vocabulary), but different parameters Φ. Finally, IPA combines the base and adapter policies into a tailored policy: Definition 1 (Tailored policy). The tailored policy Po-6 combines the distributions of the base policy po and the adapter policy po.
Approximate Policy. When the base model is extremely large (e.g., GPT-3), its forward pass is too costly to be used in the RL training loop. To overcome this, we propose using an approximate policy in IPA.
Definition 2 (Approximate policy). The approx- imate policy desined as a smaller-sized neural model parameterized by i that approximates the distribution of the base policy and is used replace the base policy in the RL-based adapter training:
x² = SRL (Pô- Rí 6)-
1
poc (b,l) = (ep0
In practice, we obtain approximate pol- icy in two different ways. First, we can use smaller pre-trained language model from the same model family. We do this if the smaller model has similar conditional generation behavior as the base policy. For instance, we use an off-the-shelf GPT2-XL as the approximate policy to tailor GPT- 3 in an open-ended generation. Alternatively, we can use distilled base policy as the approxi- mate policy. A distilled base policy is a language model trained generations from the base pol- icy, θ arg max (-y-po--x)) [oo Pg(y)], known as sequence-level knowledge distillation (Kim and Rush, 2016; West et al., 2022). For example, to tai- lor GPT-3 for lexically constrained generation, we tune GPT2-XL on prompt- seenp-tgnnertioon pairs from GPT-3 to get a distilled base policy.
where Z is a normalization factor.
The tailored policy is product-of-expert (Hin- ton, 2002), which amounts to multiplying the next- token probabilities from the base and adapter poli- cies, then normalizing the result. IPA's 's tailored policy has two key properties. First, it allows for adjusting the base policy's output without direct ac- cess the base policy's parameters. This critical for tailoring modern LLMs that provide access to the model's output distribution but not the model's parameters. Second, the policy adapter can use a much smaller model (i.e., ø < 0). This provides an efficient way to tailor a large base model.
3.2 Adapter Training with RL
Our goal is to adjust the tailored policy towards user-specified objective. The key idea in IPA is to train the tailored policy to optimize a given reward with reinforcement learning, while only updating the parameters of the adapter policy.
IPA at Inference Time. At inference time, IPA uses the tailored policy Pa-ó for decoding. Namely, each time-step we obtain the next-token distribution from the tailored policy ( yy which can then be used with a standard decoding algorithm (e.g. nucleus sampling).
Concretely, we use a reinforcement learning al- gorithm fRL (Eqn. 1) to optimize the tailored policy Po-6 with reward function R. Notably, we keep the base policy's parameters (0) frozen, and only update the adapter policy's 's parameters (Ø). That is,
Experiments
We evaluate IPA on a diverse range of tasks: toxicity reduction ($4.1), lexically constrained generation ($4.2), open-ended generation ($4.3), dialogue safety control ($4.4), and knowledge- grounded dialogue ($4.5). In all benchmarks, IPA
6² fRI. (po. d R; Φ)-
Intuitively, the adapter policy Ps learns to rescale the frozen base policy Po. yielding a tailored pol- icy that is 'tailored to' the reward. Notice that our

---

consistently improve upon LLMs such as GPT-
Toxicity
Fluency
Diversity
Models
3 (text-davnci-0033) surpassing competitive baselines and sometimes even outperforming ex- pensive fine-tuned GPT-3 at a fraction of the cost.
Avg Max.
Prob.
PL.
Dist-2.
Dist-3.
base policy; GPT2-L
GPT-2
0.527
0,520
11.31
0.85
0.85
PPLM
0.520
0,518
32.58
0.86
0.86
GeDi
0.363
0.217
60.03
0.84
0.83
4.1 Toxicity Reduction
DEXPERTS
0,314
0.128
32.41
0.84
0.84
DAPT
0.428
0.360
31.21
0.84
0.84
LMs are susceptible to generating toxic comple- tions, even when prompted with seemingly innocu- ous text (Gchman et al., 2020). Here, we assess IPA's efficacy in reducing toxicity from LMs.
PPO
0.218
0.044
14.27
0.80
0.84
QUARK
0.196
0.035
12.47
0.80
0.84
(GPT-2)
base policy: GPT-3
GPT-3
0.275
0.197
10.65
0.78
0.81
Datasets and Metries. The task generate fluent continuation y while avoiding offensive con- tent for given prompt z. We evaluate this on RE- ALTOXICITYPROMPI benchmark (Gehman et al., 2020), which contains 100k prompts designed to elicit toxic generations. Following the experimen- tal setup of Liu et al. (2021b), we use Perspective API to determine the average maximum toxicity across 25 sampled generations and the (empirical) toxicity probability of at least one toxic generation. addition, we report fluency as the perplexity of generated output based on an off-the-shelff GPT2- XL model, and diversity as the count of unique n-grams normalized by the length of text. We also perform human evaluations; see Appendix A.1 for more details.
DEXPERTS
0.223
0,112
23.41
0.79
0.82
DAPT
0.254
0.176
20.19
0.80
0.83
IPA' (GPT-3)
0.150
0.056
10.34
0.79
0.81
IPA (GPT-3)
0.101
0.028
12.68
0.79
0.83
Table 1: Automatic evaluation for Toxicity Reduction with off- she-lh GPT22-arge (top) and GPT-3 (bottom) as the base policy to tailor.
Toxicity
Fluency
Diversity
RI Algo
Avg Max.
Prob.
PL.
Dist-2.
Dist-3.
Quark PPO NLPO
0.138
0.031
11.94
0.80
0.84
0.125
0.029
12.47
0.80
0.84
0.136
0.032
12.13
0.80
0.85
Table 2: Comparison of using different RL algorithm for training IPA for Toxicity Reduction with off-the-shelf GPT2-large as the base policy to tailor.
Setup and Baselines We apply IPA to tailor off- the-shelf GPT-2 and GPT-3?. To tailor GPT-2, directly apply the base policy in the adapter train- ing, denoted as IPPT2))- For tailoring GPT-3, we use an off-the-shelf GPT-2 and distilled GPT-3 as the approximate policy for the adapter training, labeled as IPA (GPT-3) and IPA* (GPT(3) respec- tively. Notice that IPA (GPT-3) is equivalent to directly applying the policy adapter trained to tai- lor GPT-2 top of GPT-3. We initialize all the policy adapters with pre-trained GPT2-L model.
2021). DExpert (Liu et al., 2021a), and learning- based methods: DAPT (Gururangan et al., 2020), PPO (Schulman et al., 2017), and QUARK (Lu et al.. 2022a). For tailoring GPT-3, we compare IPA the baselines described above that are compatible with GPT-3's limited accessibility: DExpert (Liu et al., 2021a) and DAPT (Gururangan et al., 2020). We also provide runtime analysis in Appendix B.
As shown in Table 1, IPA outperforms all barning-basee and decoding bassed methods in tai- loring GPT-2 and GPT-3, significantly reduces the toxicity while maintaining language quality. Inter- estingly, we found that applying the policy adapter optimized for GPT-2 directly on top of GPT-3 (i.c., IPA') ) highly effective, showcasing the adaptabil- ity and reusability of TPA. Notably, when tailoring GPT-3, IPA outperforms the costly domain adap- tive training (DAPT), which exhaustively fine-tune GPT-3 on a non-toxic corpus. This further em- phasizes the promise of the IPA as a cost-efficient approach to align LLMs. Our findings are further confirmed by human evaluation (Appendix A. Finally, we conduct ablations on the effect of
Results
We use QUARK as the RL algorithm in adapter optimization, and provide additional ablation stud- ies to assess the effects of different RL algorithms. We use the Perspective API as the reward func- tion, which provides score ranging from 0 to 1 to indicate the degree of toxicity.
For tailoring GPT-2, we compare IPA with previously reported baselines from Lu et al. (2022a), including decoding-based methods: PPLM (Dathathri et al., 2020a), GeDi (Krause et al.,
We refer text-davinci-eee as GPT-3 in this paper *Wo finetune GPT2-XIL with prompt-output pairs from GPT-3 on REALTOXICITY as the distilled GPT-3.

---

0.23
toxic prob.
12.8
perplexity
0.83
dist-3
Setup and Baselines. As we will demonstrate later, zero-shot GPT-3 is surprisingly poor satis- fying ordered lexical constraints, even with explicit instructions. Our goal is to make GPT-3 more re- liable constraint satisfaction. We use distilled GPT3 $ as the approximate policy for adapter train- ing, since an off-the-shelf GPT-2 cannot perform constrained generation out of the box. We initial- ize the policy adapter with pre-trained GPT2-LL model. We use QUARK as the RL algorithm and choose our reward to be the product of the coverage score and the fluency score, as this promotes con- straint satisfaction and fluency preservation. Please see Appendix A.4 for more reward analysis.
0.18
12:1
0.82
0.13
11.4
0.81
0.08
0.80
0.03
XL 10.0 8
0:79
XL
Figure 2: Performance of IPA" (blue line) with respect to the size of the adapter model (distill-GPT2, GPT2- small, GPT2-medium, GPTe-large, GPT2-XL) on top of off-the-shelf GPT-3 as the base policy. The grey line denotes the performance of the off-the-shelf GPT-3.
Automatic
Human
Models
Cov.
FL
Qu.
PL.
Overall
GPT-3
37,01
94.89
2.84
2.81
2.60
GPT-3.5
65.17
95,89
2.93
2.88
2.90
GPT-4
84.81
95.49
2.95
2.97
2.96
We compare IPA with its base policy GPT-3, well as more advanced LLMs: GPT-3.5 and GPT-4 (OpenAI. 2023a). As a strong supervised baseline, we also fine-tune GPT-3 on the COMMONGEN train set, which contains -umannwrittenn outputs with the correct lexical order, denoted as GPT-3
GP
72.89
73.96
2.56
2.60
2.50
IPA" (GPT-3)
88.54
92.58
2.90
2.87
2.88
Table 3: Automatic and human evaluation results for Lexically Constrained Generation. Human evaluation scores are 3-point Likert Scale.
Results. As shown in Table 3, powerful LMs such GPT-3 often struggle satisfy ordered lex- ical constraints even with explicit instructions. IPA leads to remarkable improvement on top of GPT- and surpasses more advanced models such as GPT-3.5 and GPT-4 in terms of constraint coverage, while achieving better or comparable generation quality. Noticeably, IPA outperforms fine-tuned GPT-3 in both constraint coverage and generation quality at a fraction of its cost: while fine-tuning GPT-3 costs $156.82, training distilled GPT-3
RL algorithms. As shown in Table 2, IPA is ef- fective with various RL algorithms, all of which lead to state-of-the-art performance. Additional ablation in Figure 2 shows that policy adapter as small distilled GPT-2 can effectively tailor the x 1000 larger GPT-3 model, achieving comparable performance with our main result.
4.2 Lexically Constrained Generation
Next, we test IPA in lexically constrained gener- ation. We consider more challenging setup of ordered lexical constraints, where the generation is considered correct if it includes all the keywords with the correct order specified the input prompt.
the approximate policy requires only $28.59 for generating outputs from GPT-3, Our results high- light the potential of the IPA as cost-efficient way to enhance the capabilities of LLMs.
Datasets and Metrics. We use COMMONGEN (Lin al., 2020), a dataset for generative com- monsense reasoning. We deliberately instruct the models to generate a sentence with the given key- words while following the order they appear in the input prompt. For automatic evaluation, we gauge the constraint satisfaction with coverage, binary metric that evaluates a generation to be correct only when it includes all the keywords and also matches the specified order. We also measure the fluency using critic model fine-tuned on CoLA (Warstadt et al., 2019). For human evaluation, we assess the quality and plausibility of model generations for 100 randomly sampled test examples based on a 3- 3-point Likert Scale; see details in Appendix E.
4.3 Open-ended generation
We further evaluate IPA on open-ended genera- tion, following the experimental setup in (Li et al., 2022b). The goal is to make machine-generane content more fluent, coherent, and human-likk
Datasets and Metrics. We experiment on the news domain using XSum dataset (Narayan et al., 2018). Following Li et al. (2022b), we the first 32 words as our input prompt, and generate 84 tokens as continuations. We evaluate using both
"Human pairwise agreements are 0.97, 0.94, and 0.93 for quality, plausibility and overall, respectively.
*We finetune a GPT2-XI. with prompt-output pairs from GPT-3 on COMMONGEN train set as the distilled GPT-3

---

Decoding Method Diversity Coherence Critic Mauve
109.p
0.71
0.28
Contrastive
0.60
0.36
base policy: GPT2-XL
0.57
0.35
SimCTG
0.61
0.31
greedy
55.05
15.32
top-k (k=50)
92.60
48.53
10.72
53.13
greedy
0.55
0.36
typical
0.59
0.37
top-p (p=0.95)
95.85 25.85
47.61
13.24
56,42
(GPT-2)
Baseline
lypical (r=0.95)
95.80
46.08
23.49
63.92
SimCTTG
95.67
46.12
23.67
62.21
IPA
0.54
0.43
IPA-
0.54
0.41
Contrastive
95.99
49.42
36.73
61.95
IPA (GPT-3)
GP13
(GPT2-XI.)
96.12
base policy: GPT-3
Figure 3: Pairwise human evaluation in terms of overall quality for Open-ended Generation on XSum with off- the-shelf GPT2-XL (top) and GPT-3 (bottom) as the base policy to tailor."
top-p (p=0.95)
95.63
56.16
18.58
63.73
IPA" (GPT-3)
95.35
57.26
22.62
71.40
IPA (GPT-3)
96.26
61.94
32.84
73.17
Table Automatic evaluation for open-domain genera- tions XSum with off-the the- GPT2-XL (top) and GPT-3 (bottom) as the base policy to tailor. Critic scores refer to human-likeness according to OpenAl detector.
Automatic
Human
Models
Safety
Safety
Coherence
DialoGPT
1.34
2.45
Godel
0.49
1.40
2.53
Blenderbot
0.53
1.43
2.60
automatic and pairwise human evaluation. For au- tomatic ovaluation, we use aggregate n-gram di- versity and coherence scores (Li et al., 2022b) as well as MAUVE (Pillutla et al., 2021), which mea- sures the distribution similarity between the set of numan-writter and machine-generated texts. To measure the human-likeen likeness of generated texts, employ OpenAl detectors, classifier for distin- guishing AI vs. human-written text. We use the classifier's ss probability assigned to 'human' text to serve as an additional metric, denoted as Critic. For human evaluation, we randomly sample 100 test examples and perform pairwise comparisons of our method against baselines on coherence and fluency using AMT; see details in Appendix E.
CharGPT
0.74
1.60
2.68
IPA (BlenderfBot-333)
0.78
1.57
2.75
Table 5: Automatic and human evaluation results for Dialogue Safety Control. Human evaluation scores are a 3-point Likert Scale.
specifically designed to improve the coherence and naturalness of the generated text. For tailoring GPT- 3, we compare IPA with GPT-3's default generation technique: decoding with nucleus sampling (p 0.95). other decoding methods are not applicable to GPT-3 due its limited API access.
Setup and Baselines. We apply IPA to tailor off-the-shelf GPT2-XL and GPT-3, following the same setup as toxicity reduction task (section 4.1). Same as before, the tailor policies are denoted as IPA(G))-2), IPA (GPT-3) and IPA (GPT33),, re- spectively. We use QUARK the RL algorithm and the product of diversity, coherence, and critic scores the reward function. We found critical to combine multiple metrics as the reward function to improve the overall generation quality; see Ap- pendix A.4 for more analysis on reward functions. For tailoring GPT-2, we compare decoding with IPA with six different decoding strategies: greedy, top-k sampling (k: 50), nucleus sampling (p 0.95), typical sampling
Results. As shown Table 4, IPA significantly outperforms all previous baselines in tailoring GPT- 2 and GPT-3 across all automatic metrics. Notably, it achieves an absolute improvement of 20.26% the best-performing baseline in the Mauve score. Our pairwise human evaluation in Figure also verify the results. IPA generates signifi- cantly more coherent and fluent texts compared to other baselines. Overall, on average, human evalu- ators preferred IPA 1.8x more than other baselines. Interestingly, we found that directly applying the policy adapter optimized for GPT-2 on top of GPT- 3 (i.c., IPA') significantly improves the generation quality, highlighting the adaptability and reusability of IPA. We observed further improvement when ing distilled GPT-3 as the approximate policy (i.e., IPA*). Our promising results once again showcase the effectiveness and efficiency of IPA.
0.95) (Meister al., 2023), SimCTG (Su et al., 2022), and Contrastive decoding (Li ct al., 2022b). The latter three are
"https://githur ctt htmm/ppmmptsllab/ demai-deteccor

---

4.4 Dialogue Safety Control
on both automatic and human evaluation while showcasing improved coherence. Upon further in- vestigation, we found that ChatGPT often generates canned responses like "I'm a language model; I'm not allowed..." " as hard safeguards, which hurts the coherence and naturalness of the dialogue flow. On the other hand, Blenderbot tailored by IPA can gen- erate safe responses that are coherent, natural, and human-like. Our results demonstrate the potential of IPA to enhance controllability in various NLP applications beyond conditional text generation.
Existing dialogue systems often fail to respond safely to potentially unsafe user utterances (Kim et al., 2022), limiting their deployment in real- world applications. Here, we aim to evaluate IPA for controlling the safety of a dialogue model.
Datasets and Metrics. We experiment on D1- ASAFETY (Sun al., 2022), challenging dataset containing 54K context-sensitive unsafe examples. The task is to generate a coherent response to a po- tentially unsafe utterance while avoiding offensive, harmful, toxic or biased language. DIASAFETY contains numan--riited safe and unsafe responses which we use to train a dialogue safety classifier. We use the classifier score as an automatic mea- sure of safety. In addition, we conduct human evaluation of safety and coherence (3-point Likert scale) on 200 examples through Amazon Mechani- cal Turk; see Appendix E Figure 4 for details.
4,5 Knowledge-grounded Dialogue
Idcally, knowledge-groune dialogue systems should generate responses that are faithful the given knowledge. However, models tend to gen- erate hallucination containing unverifiable infor- mation (Dziri et al., 2022a; Rashkin et al., 202 la; Dziri et al., 2022c). To address this undesirable behavior, we use IPA to tailor dialogue model to- wards generating more faithful content. Given the knowledge K and the conversation history H, the task is to generate a response that's faithful to K and coherent with H.
Setup and Baselines. We apply IPA tailor the Blenderbot family models (Roller et al., 2021), which are pretrained dialogue agents. Specifically, we use Bienderbot-3B-diiae as the frozen base pol- icy, a samller Blenderbot- Biteerroot----dsttiee IB-distill as the approxi- mate policy and initialize the policy adapter with a Blenderbot- | BBdistill model. We use QUARK as the RL. algorithm for adapter training. To preserve the dialogue quality while controlling the response safety, choose our reward to be the product of the safety score from our dialogue safety classifier, as well as coherence and engagingness scores from UniEval-Dialogue (Zhong et al., 2022).9
Dataset and Metrics We evaluate on the Wizard of Wikipedia (WoW) data. WoW (Dinan et al.) involves Wizard and an Apprentice engaging in conversation. The Wizard's role is to provide information
specific topic, while the Appren- tice's task is seek further details. WoW has been shown to suffer from hallucinations (Dziri et al., 2022b), in more than 60% of the turns, making it a valuable dataset for studying hallucination issues. FaithDial (Dziri et al., 2022a) is a hallucination- free benchmark created by modifying the halluci- nated responses within the WoW dataset. We use the FaithDial test data at test time evaluate the faithfulness of responses and compare them against the knowledge snippets and gold responses.
We compare IPA with its base policy, i.e.. Beanderbot-3B-distiie and other off-the-shelf di- alogue models including DialoGPT (Zhang et al., 2020), GODEL (Peng et al., 2022) well Chat- GPT (OpenAI, 2022). ChatGPT is known to have safeguards through content filtering and is consid- ered a strong baseline.
To measure faithfulness, we use the critic model (Dziri al., 2022a), which returns the probability of an given utterance being identified as faithful. Additionally, we use BERTScore to measure the semantic similarity between the generated response and the knowledge K, and the token-level F1 score to rate the lexical overlap between and K. To measure coherence and engagingness, use the UniEval model (Zhong et al., 2022).
Results. As shown in Table 5, IPA significantly improves dialogue safety and coherence compared to its base policy Blenderbot-3B--dsiin surpass- ing other dialogue models including DialoGPT and GODEL. In comparison with ChatGPT, IPA achieves comparable performance on safety based
Average pairwise agreements 0.88 and 0.82 with GPT2-XL and GPT-3, respectively. 'Human pairwise agreements are 0.84 and 0.87 for safety and coherence.
Setup and Baselines Similar to the dialogue safety experiment, we use the Blenderbot-(3, 1 I-B- distill model (Roller et al., 2021) as our base policy
'https:/ htw.t/gttttttteeeeee http://gttuucc.mmmaa/r///ae/ htmm/mmazzhoonminnllniiiaaaaaa

---

Dialogue Model
Critic BERTScore
Coherence Engaging
5
Related Work
supervised baseline
GPT-2
39.9
0.29
47.7
0.77
1.26
Controlled Decoding Recent studies have ex-
DIALOGPT
40.6
0.34
53.5
0.83
1.32
plored controlled generation at inference time by designing new decoding algorithms (Keskar et al..
DOHA
46.8
0.32
56.1
0.88
1.33
T5
53.5
0.41
61.7
0.86
1.28
T5-CTRL
54.8
0.45
65.2
0.83
1.21
2019; Mireshghallah al., 2022; Li et al., 2022a;
T5-LT
58.6
0.43
65.0
0.83
1.21
Chen et al., 2022; Zhang et al., 2022). For example,
off-the-shelf dialogue model
Neurologic decoding (Lu et al., 2020), and GBS (Hokamp and Liu, 2017) generalize beam search for lexically constrained decoding, by constrain- ing decoding space with keyword-related penalties. DExperts (Liu et al., 2021b) modifies output dis- tribution during decoding with attribute-specific expert models. Another line of research develops gradient-based decoding for more general control (Qin et al., 2020, 2022; Sha, 2020; Dathathri et al., 2020b; Kumar et al., 2021). For example, COLD Decoding (Qin et al.. 2022) introduces energy- based modeling to impase arbitrary constraints on text and samples with Langevin dynamics. Despite their progress, these approaches either are designed for particular control types or rely on computation- ally expensive gradient computations.
BlenderBot
10.3
0.12
9.8
0.92
1.21
IPA (BlenderBal)
80.1
Table 6: Evaluation results for Knowledge- Grouded Dialogue generations on Faithdial. We use off-the-shelf Blenderbot the base policy tailor.
and approximate policy respectively, and initial- ize the policy adapter with Blenderbot- IB distill model. We use QUARK as the RL algorithm. To preserve coherence and engagingness while ensur- ing the faithfulness of a dialogue response, we choose our reward to be the product of the faithful- ness score from the critic model described above, well as coherence and engagingness scores from UniEval- (Zhong et al., 2022).
Reinforcement Learning for NLG RL has his- torically been used in multiple NLG tasks such as machine translation (Wu et al., 2016; Nguyen et al., 2017), summarization (Paulus et al., 2017), dialogue (Li et al., 2016; Zhou et al., 2017), text games (Narasimhan et al., 2015; Hausknecht et al., 2020), etc to optimize for an arbitrary non- diffcrentiable reward. This was often done ing online policy gradient methods such RE- INFORCE (Sutton and Barto, 2018), leading documented issues with reward hacking (Choshen et al., 2020; Kiegeland and Kreutzer, 2021). Re- cent advances introduce a KL reward penalty which significantly increases the naturalness of generated text (Ouyang et al., 2022; Korbak et al., 2022). This method has been used extensively to tune base LM via online on-policy (Ramamurthy* et al., 2023), off-policy (Guo al., 2022: Lu al., 2022b), and offline (Snell al., 2023; Korbak al., 2023) RL. Such methods quickly become compu- tationally infeasible for extreme-scale LMs.
We compare to previously baselines from Dziri et al. (2022a), supervised models fine-tuned on WoW, including GPT2, DialoGPT (Zhang al., 2020), DoHA (Prabhumoye et al., 2021) T5 (Raffel al., 2020), T5-CTRL (Rashkin al., 2021b), and T5-LossTruncation (Kang and Hashimoto, 2020). We also compare against the base policy, off-the- shelf BlenderBot model (Roller et al., 2021).
Results As shown in Table 6, supervised mod- els struggle to generate faithful dialogue response grounded on the given knowledge. This is mainly because of the poor data quality of their supervi- sion dataset: WoW has been shown to suffer from hallucinations in more than 60% of the turns (Dziri al., 2022a). Moreover, pre-trained dialogue models like Blenderbot demonstrate even worse performance at generating faithful response, de- spite being trained on WoW and other knowledge- grounded dialogue datasets in their pre-training stage. IPA significantly improves the faithfulness of the generated dialogue response over its base policy Blenderbot while preserving the dialogue quality (i.e., coherence and engagingness), outper- forming all other baselines. Our results showcases the potential of IPA to improve reliability and trust- worthiness in various downstream applications.
Conclusion
present IPA, lightweight inference-time policy adapter that tailor frozen large language model towards desirable properties (e.g., safety, coher- ence) in an efficient, generalizable, and flexible way. Specifically, IPA combines the generaliz-

---

ability of RL with the plug-and-play flexibility of inference-time techniques, permitting customiza- tion of large language models without the need for costly fine-tuning. Extensive experiments across five challenging text generation tasks show that IPA brings consistent improvements over LLMs, outperforming competitive baselines - sometimes even surpassing expensive fine-tuning. We hope our work sheds light on creative and efficient algo- rithmic innovations to complement the pursuit of model scales with leeee resources.
References
Armen Aghajanyan, Sonal Gupta, and Luke Zettle- moyer. 2021. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Pro- ceedings of the 59th Anmual Meeting of the Asso- ciation for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing, ACL/ICCNPP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 7319- 7328. Association for Computational Linguistics.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner. Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.
7 Limitations and Ethical Consideration
While the versatility of the IPA is crucial feature that enables aligning large language models with arbitrary user-given objectives, it may also pose potential dual-use concerns, especially when com- bined with the power of large language models.
First, as with any controllable text generation technique, IPA could be potentially used for unin- tended malicious purposes, such as manipulating models to produce hateful, toxic content or misin- formation. As malicious users already exploit any existing techniques for harmful purposes the- oretically, we foresee minimal risk introduced by IPA specifically. Nevertheless, we highly recom- mend avoiding such negative applications of IPA.
Michael Chang, Abhishek Gupta, Sergey Levine, and Thomas L. Griffiths. 2019. Automatically compos- ing representation transformations generalization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
means for
Howard Chen, Huiban Li. Danqi Chen, and Karthik Narasimhan. 2022. Controllable text genera- tion with language constraints. arXiv preprint arXiv:2212.
Moreover, similar to any RL-based method that depends on the reward function for learning sig- nals, IPA is susceptible to the innate shortcomings from the reward model. For instance, we use the Perspective API calls as the reward function for the toxicity reduction task; any limitations or potential biases from these public API calls will propagate into the learning of IPA. Nonetheless, as more ac- curate, transparent, and inclusive classifiers are de- veloped, we anticipate that IPA would inherit those improvements as well.
Leshem Choshen, Lior Fox, Zohar Aizenbud, and Omri Abend. 2020. On the weaknesses of reinforcement learning for neural machine translation. In Interna- tional Conference on Learning Representations.
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung. Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2020a. Plug and play language models: A simple approach to controlled text generation.
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung. Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2020b. Plug and play language models: A simple approach to controlled text generation. In 8th International Conference on Learning Represen- tations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Beyond these two primary concerns, another in- herent limitation of IPA is its requirement to access the output logits of the base LM. This constraint hinders IPA's compatibility with certain models, such as GPT-4, which permit access only to the output, not the logits. Finally, like general RL frameworks, IPA relies on the assumption that user objectives are quantifiable through a reward func- tion. However, this premise may not always hold, particularly when user objectives are inherently challenging to measure, thus limiting IPA's appli- cability.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. Wizard of wikipedia: cnwwledge-powered conversational agents. In International Conference on Learning Representations.
Nouha Dziri, Ehsan Kamalloo, Sivan Milton, Os- mar Zaiane, Mo Yu, Edoardo M Ponti, and Siva Reddy. 2022a. FiithDial: A Faithful Benchmark for Information-SSeeing Dialogue. Transactions of the Association for Computational Linguistics, 10:1473- 1490.

---

Nouha Dziri. Sivan Milton, Mo Yu, Osmar Zaiane, and Siva Reddy. 2022b. On the origin of hallucinations in conversational models: Is it the datasets or the models? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties; Human Language Tech- nologies, pages 5271 1-5285, Seattle, United States. Association for Computational Linguistics.
Geoffrey E. Hinton. 2002. Training Products of Ex- perts by Minimizing Contrastive Divergence. Neural Computation, 177):1771-18000
Chris Hokamp and Qun Liu. 2017. Lexically con- strained decoding for sequence generation using grid beam search. arXiv preprint arXiv: (xXiv:/000 1704.07138.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In The Tenth International Conference Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Nouha Dziri, Hannah Rashkin, Tal Linzen, and David Reitter. 2022c. Evaluating attribution in dialogue systems: The begin benchmark. Transactions of the Association for Computational Linguistics, 10:1066 1083.
Jonathan Frankle and Michael Carbin. 2019. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In 7th International Conference Learn- ing Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Daniel Kang and Tatsunori B. Hashimoto. 2020. Im- proved natural language generation via loss trunca- tion. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pages 718-731, Online, Association for Computational Lin- guistics.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. 2020. Linear mode con- nectivity and the lottery ticket hypothesis. In Pro-
Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019. Ctrl: A conditional transformer language model for control- lable generation. arXiv preprint (tt.:::9900009950
ceedings of the 37th International Conference on Machine Learning. ICML 2020, 13-18 July 2020, Vir- tual Event, volume 119 of Proceedings of Machine Learning Research, pages 2259-3269 PMLR.
Samuel Kiegeland and Julia Kreutzer. 2021. Revisiting the weaknesses of reinforcement learning for neural machine translation. In NAACL-HILT, pages 1673- 1681.
Samuel Gehman, Suchin Gururangan, Maarten Sap. Yejin Choi, and Noah A. Smith. 2020. RealToxi- cityPrompis: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356-3369, Online. Association for Computational Linguistics.
Hyunwoo Kim, Youngjae Yu, Liwei Jiang. Ximing Lu, Daniel Khashabi, Gunhee Kim, Yejin Choi, and prosocial
Maarten Sap. 2022. ProsocialDialog: backbone for conversational agents. In Proceedings of the 2022 Conference Empirical Methods in Nat- ural Language Processing, pages 4005-4029, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Han Guo, Bowen Tan, Zhengzhong Liu, Eric Xing, and Zhiting Hu. 2022. Efficient (soft) Q-learning for text generation with limited good data. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6969-6991, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Yoon Kim and Alexander M. Rush. 2016. Sequence- level knowledge distillation.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo. Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't 't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360, Online. Association for Computational Linguistics.
Louis Kirsch, Julius Kunze, and David Barber, 2018. Modular networks: Learning to decompose neural computation. In Advances in Neural Information Processing Systems 31: Annual Conference on Neu- ral Information Processing Systems 2018, NeuriPS 2018, December 3-8, 2018, Montréal, Canada, pages 2414-2423.
Song Han, Jeff Pool, Sharan Narang, Huizi Mao, En- hao Gong. Shijian Tang, Erich Elsen, Peter Vajda, Manohar Paluri, John Tran, Bryan Catanzaro. and William J. Dally. 2017. DSD: dense-sparee sanse-sparse-desse training for deep neural networks. In 5th Inter- national Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Con- ference Truck Proceedings. OpenReview.net.
Tomasz Korbak, Ethan Perez, and Christopher Buck- ley. 2022. RL with KL, penalties is better viewed as Bayesian inference. In Findings of the Associa- tion for Computational Linguistics: EMNLP 2022. pages 1083-1091, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher I. Buckley, Jason Phang, Samuel R Bowman, and Ethan Perez. 2023. Pre- training language models with human preferences. arXiv preprint arXiv:2302.08585
Matthew Hausknecht, Prithviraj Ammanabrolu, Marc- Alexandre Côté, and Xingdi Yuan. 2020. Interactive fiction games: A colossal adventure. Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAT).

---

Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. 2021. GeDi: Gener- ative discriminator guided sequence generation. In Findings of the Association for Computational Lin- guistics: EMNLP 2021, pages 4929-4952, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
Long Papers), Virtual Event, August 1-6, 2021, pages 6691-6706 Association for Computational Linguis- tics.
A Alisa Liu, Maarten Sap, Ximing Lu. Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi, 2021b. DExperts: Decoding-ting controlled text generation with experts and anti- experts.
Sachin Kumar, Eric Malmi, Aliaksei Severyn, and Yu- lia Tsvetkov. 2021. Controlled text generation as continuous optimization with multiple constraints. Advances in Neural Information Processing Systems. 34:14542- 14554.
Tianyu Liu, Yizhe Zhang, Chris Brockett, Yi Mao, Zhifang Sui, Weizhu Chen, and Bill Dolan. 2022a. A token-level reference-free hallucination detection benchmark for free-form text generation.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengx- iao Du, Zhilin Yang, and Jie Tang. 2022b. P-tuning: Prompt tuning can be comparable to fine- finettnnig across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 2: Short Papers), ACL 2022, Dublin, Ireland, May 22-27. 2022, pages 61-68. As- sociation for Computational Linguistics.
Chunyuan Li. Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. 2018. Measuring the intrinsic dimension of objective landscapes. In 6th International Con- ference on Learning Representations, ICLR 2018. Vancouver, BC, Canada, April 30 May 3, 2018, Conference Truck P Proceedings. OpenReview.neh
Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. 2016. Deep re- inforcement learning for dialogue generation. In Pro- ceedings of the 2016 Conference on Empirical Meth- ods in Natural Language Processing, pages 1192- 1202, Austin, Texas. Association for Computational Linguistics.
Renze Lou, Kai Zhang, and Wenpeng Yin. 2023. Is prompt all you need? no. a comprehensive and broader view of instruction learning. ArXiv, abs/2303.104.
Ximing Lu, Sean Welleck, Liwei Jiang, Jack Hessel, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. 2022a. Quark: Controllable text generation with reinforced unlearning. CoRR, abs/2205. 13636.
Xiang Li, John Thickstun, Ishaan Gulrajani. Percy Liang, and Tatsunori B Hashimoto. 2022a. Diffusion- Im improves controllable text generation. Advances in Neural Information Processing Systems, 35:4328- 4343.
Ximing Lu, Sean Welleck, Liwei Jiang, Jack Hessel, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. 2022b. Quark: Controllable text generation with reinforced unlearning. In Thirty- sixth Conference on Neural Information Processing Systems (NeurIPS).
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zeulemoyer, and Mike Lewis. 2022b. Contrastive de- coding: Open-ended text generation as optimization. CoRR, abs/2210.15097
Ximing Lu, Peter West, Rowan Zellers, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Neu- rologic decoding:(un) supervised neural text genera- tion with predicate logic constraints. arXiv preprint arXiv:2010 12884.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/JCNLP 2021, (Volume 1: Long Pupers), Virtual Event, August 1-6, 2021, pages 4582- 4597. Association for Computational Linguistics.
Ximing Lu, Peter West, Rowan Zellers, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Neuro- logic decoding: (un)supervised neural text generation with predicate logic constraints. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 4288 28884299.. Associa- tion for Computational Linguistics.
Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. 2020. Commongen: A constrained text genera- tion challenge for generative commonsense reason- ing.
Alisa Liu, Maarten Sap. Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. 2021a. Dexperts: Decoding-time controlled text generation with experts and anti- experts. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/JJCNLP 2021, I, (Volume I:
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022c. Fantastically ordered prompts and where to find them: Overcoming few- shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 1886-8098,18 Dublin, Ireland. Association for Compu- tational Linguistics.

---

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang. Scan Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with self-feedback.
OpenAI. 2023a. Gpt-4 technical report.
OpenAL. 2023b, Openai upi pricing. com/pricing, Accessed: 2023-05-15.
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems.
Rabeeh Karimi Mahabadi, James Henderson, and Se- bastian Ruder. 2021. Compacter: Efficient low-rank hypercomplex adapter layers. In Advances in Neural Information Processing Systems 34: Annual Confer- ence on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages 1022-1035.
Romain Paulus. Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive sum- marization. arXiv preprint arXiv: /705 705.04304.
Clara Meister. Tiago Pimentel, Gian Wiher, and Ryan Cotterell. 2023. Locally Typical Sampling. Transac- tions of the Association for Computational Linguis- rics, 11 11:10-1121.
Baolin Peng, Michel Galley, Pengcheng He, Chris Brockett, Lars Liden, Elnaz Nouri, Zhou Yu, Bill Dolan, and Jianfeng Gao. 2022. Godel: Large-scale pre-training for goal-directed dialog. arXiv.
Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one? In Ad- vances in Neural Information Processing Systems 32:
Annual Conference on Neural Information Process-
Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. True few-shot learning with language mod- els. NeurlPS.
ing Systems 2019, NeurlPS 2019. December 8-14, 2019, Vancouver, BC, Canada, pages 14014-14024.
Sewon Min, Xinxi Lyu, Ari Holuman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettle- moyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022. pages 11048-1 1064. Association for Computational Linguistics.
Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya
Kamath, Ivan Vulic, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020. Adapterhub: A framework for adapting transformers. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demon- strations, EMNLP 2020- Demos, Online, November 16-20, 2020, pages 46-54, Association for Computa- tional Linguisties.
Fatemehsadat Mireshghallah, Kartik Goyal, and Taylor Kirg-Kirkpatric.. 2022. Mix and match: Learning- free controllable text generation using energy lan- guage models. arXiv preprini arXiv:2203.1322
Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. 2021. Mauve: Measuring the gap be- tween neural text and human text using divergence frontiers. In NeurIPS.
Karthik Narasimhan, Tejas D. Kulkarni, and Regina Barzilay. 2015, Language understanding for text- based games using deep reinforcement learning. In EMNLP, pages 1-11.
Shrimai Prabhumoye, Kazuma Hashimoto, Yingbo Zhou, Alan W Black, and Ruslan Salakhutdi- nov. 2021. Focused attention improves document- grounded generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 4274-4280 Online. Association for Computational Linguistics.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for ex- treme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1797-1807, Brussels, Bel- gium. Association for Computational Linguistics.
Lianhui Qin, Vered Shwartz, Peter West, Chandra Bha- gavatula, Jena Hwang, Ronan Le Bras, Antoine Bosselut, and Yejin Choi. 2020. Back the future: Unsupervised mackprop-basee decoding for counter- factual and abductive commonsense reasoning. arXiv preprint 2rx.v22010.05906.
Khanh Nguyen, Hal Daumé III and Jordan Boyd- Graber. 2017. Reinforcement learning for bandit neural machine translation with simulated human feedback. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1464-1477 1474, Copenhagen, Denmark. Associa- tion for Computational Linguistics.
Lianhui Qin, Vered Shwartz, Peter West, Chandra Bha- gavatula, Jena Hwang, Ronan Le Bras, Antoine Bosselut, and Yejin Choi. 2021. Back to the future: Unsupervised backprop- based decoding for counter- factual and abductive commonsense reasoning.
OpenAL. 2022. ChatGP: Optimizing language models for dialogue.

---

Lianhui Qin. Sean Welleck, Daniel Khashabi, and Yejin Choi. 2022. Cold decoding: Energy-basee baargy con- strained text generation with langevin dynamics.
William Saunders, Catherine Yeh. Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. 2022. Self-critiquing models for assisting human evaluators. ArXiv:2206.05802
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. Journal of Machine Learning Research, 21:1- 67.
Timo Schick, Jane Dwivedi- Yu, Zhengbao Jiang, Fabio Petroni, Patrick Lewis, Gautier Izacard, Qingfei You, Christoforos Nalmpantis, Edouard Grave, and Sebas- tian Riedel. 2022. Peer: A collaborative language model. ArXiv, abs/2208. 1663..
Rajkumar Ramamurthy Prithviraj Ammanabrolus,
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. ArXiv, abss/777.00644)
Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. 2023. Is reinforcement learning (not) for natural lan- guage processing: Benchmarks, baselines, and build- ing blocks for natural language policy optimization. In International Conference on Learning Representa- tions (ICLR).
Lei Sha. 2020. Gradient- guiiedd unsupervised lexically constrained text generation. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8692-8700 2-703
Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. 2023. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimiza- tion.
Charlie Victor Snell, Tlya Kostrikov, Yi Su, Sherry Yang, and Sergey Levine. 2023. Offline RL for natural language generation with implicit language q learn- ing. In The Eleventh International Conference on Learning Representations.
Asa Cooper Stickland and Jain Murray. 2019. BERT and pals: Projected attention layers for efficient adap- tation in multi-task learning. In Proceedings of the 36th International Conference on Machine Learning, ICMI. 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 5986-5999 PMLR.
Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins. Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Ture, and David Reitter. l02 Measuring attribution in natu- ral language generation models. arXiv preprint 12Xi:21112.12800.
Hannah Rashkin, David Reitter, Gaurav Singh Tomar, and Dipanjan Das. 2021b. Increasing faithfulness dialogue with controllable features. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume 1: Long Pupers), pages 704-718, Online. Association for Computa- tional Linguisties.
Nisan Stiennon. Long Ouyang, Jeff Wu. Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2022. Learning to summarize from human feedback.
in wewwledge-groundeee
Yixuan Su, Tian Lan, Yan Wang. Dani Yogatama, Ling- peng Kong, and Nigel Collier. 2022. A contrastive framework for neural text generation. In Advances in Neural Information Processing Systems, volume 35, pages 21548 21561. Curran Associates, Inc.
Sylvestre-Alvis Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. 2017. Learning multiple visual domains with residual adapters. In Advances in Neural Infor- mation Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, De- cember 4-9, 2017, Long Beach, CA, USA, pages 506- 516.
Hao Sun, Guangxuan Xu, Jiawen Deng, Jiale Cheng. Chujie Zheng, Hao Zhou, Nanyun Peng, Xiaoyan Zhu, and Minlie Huang. 2022. On the safety of con- versational models: Taxonomy, dataset, and bench- mark. In Findings of the Association for Compu- tational Linguistics: ACI. 2022, pages 3906-3923, Dublin, Ireland. Association for Computational Lin- guistics.
Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Oit, Eric Michael Smith, Y-Lan Boureau, and Jason We- ston. 2021. Recipes for building an open-domain chatbot. In Proceedings of the 16th Conference of the European Chapter of the Association for Compu- lational Linguistics: Main Volume, pages 300-325, Online. Association for Computational Linguistics.
Richard S Sutton and Andrew G Barto. 2018. Reinforce- ment learning: An introduction. MIT press.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. ArXiv. abs/2302. 13971.
Clemens Rosenbaum, Ignacio Cases, Mathew Riemer, and Tim Klinger. 2019. Routing networks and the challenges of modular and compositional computa- tion. CoRR, abs/ 904..2774..

---

Elena Voita, David Talbot, Fedor Moiseev, Rico Sen- nrich, and Ivan Titov. 2019. Analyzing multi- self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Conference of the Association for Computational Lin- guistics, ACI. 2019, Florence, Italy, July 28- August 2019, Volume 1: Long Popers, pages 5797-5808. Association for Computational Linguistics.
Linguisties: Human Language Technologies. Associ- ation for Computational Linguistics.
Kevin Yang, Nanyun Peng, Yuandong Tian, and Dan Klein, 2022. Re3: Generating longer stories with recursive reprompting and revision. In Conference on Empirical Methods in Natural Language Processing.
Michihiro Yasunaga and Percy Liang. 2020. Graph- based, self-supervised program repair from diagnos- tic feedback. 37th Int. Conf. Mach. Learn. ICML 2020, (AM1FF16814-141107000---1
Alex Warstadt, Amanpreet Singh, and Samuel R. Bow- man. 2019. Neural network acceptability judgments.
Albert Webson and Ellie Pavlick. 2021. Do prompt- based models really understand the meaning of their prompts? ArXiv, abs/2109.01244
Haonan Yu, Sergey Edunov, Yuandong Tian, and Ari S. Morcos. 2020. Playing the lottery with rewards and multiple languages: lottery tickets RL. and NI.P In 8th International Conference on Learning Repre- sentations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net,
Albert Webson and Ellie Pavlick. 2022. Do prompt- based models really understand the meaning of their prompts? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pages 2300-2344 Association for Computational Linguistics.
Hanqing Zhang, Haolin Song, Shaoyu Li, Ming Zhou, and Dawei Song. 2022. A survey of controllable trained
text generation using transformer-basee language models. arXiv preprint XXX::22211003337
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao. Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020. DIALOGPT: : Large-scale generative pre-training for conversational response generation. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics: System Demonstrations, pages 270-278, Online, As- sociation for Computational Linguistics.
Sean Welleck, Ximing Lu, Peter West, Faeze Brah- man, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. 2023. Generating sequences by learning to self-correct. In The Eleventh International Confer- ence on Learning Representations.
Peter West, Chandra Bhagavatula, Jack Hessel, Jena Hwang. Liwei Jiang. Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. 2022. Symbolic knowledge distillation: from general language mod- els to commonsense models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Iu- man Language Technologies, pages 4602-46225 Seat- tle. United States. Association for Computational Linguistics.
Zihao Zhao, Eric Wallace, Shi Feng. Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Vir- tual Event, volume 139 of Proceedings of Machine Learning Research, pages 12697- 127066 PMLR.
language models.
Ming Zhong. Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022. Towards a unified multi- dimensional evaluator for text generation. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 2023- 2038, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-as natural language processing. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP): System Demonstrations.
Li Zhou, Kevin Small, Oleg Rokhlenko, and Charles Elkan. 2017, End-to-end offline goal-oriented di- alog policy learning via policy gradient. CoRR, abs/ 17 12.02888.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's 's neural machine translation system: Bridging the gap between human and machine trans- lation. arXiv preprint arXiv: 1609.08144.
Mingyu Zong and Bhaskar Krishnamachari. 2022. a survey on gpt-3.
Kevin Yang and Dan Klein. 2021. FUDGE: Controlled text generation with future discriminators. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational

---

IPA- vs. GPT3
IPA. vs. DExeuUN
IPA- vs. DAPT
Models
Coverage
Flueney
Less Toxic
0.17
0.09
0.15
0.09
0.13
0.12
GPT-3 (zero-shot)
37.01
94.89
Topical
0.20
0.21
0.23
0.22
0.20
GPT-3 (5-shot)
43.85
94.34
More Fluent
0.27
0.23
0.24
0.16
0.21
0.18
GPT-3 (10-shot)
45.70
94.21
IPA* vs. GPT3
IPA vs. DEXPERTS
IPA vs. DAPT
IPA" (GPT-3)
88.54
92.58
Less Tosic
0.18
0.05
0.14
0.06
0.15
0.10
More Toplcal
0.23
0.23
0.28
0.18
0.18
More Fluent
0.26
0.21
0,32
0.15
0.23
0.22
Table 9: Automatic evaluation results for Lexically Con- strained Generation with off-the-shelff GPT-3.
Table 7: Human evaluation results of Toxicity Reduction, comparing the percentage of texts rated as less toxic, more topical, and more fluent generated by IPA- and IPA versus other baselines.
Models
Coverage
Fluency
LLaMA
28,73
89.64
IPA (LLaMA)
81.49
89.71
Toxicity
Fluency
Diversity
Table 10: Automatic evaluation results for Lexically
Avg Max.
Proh.
PL
Dist-2.
Dist-3.
Constrained Generation with ILLMM-133 as the base policy to tailor.
GPT3 (scro-shot)
0.275
0.197
10.65
0.78
0.81
GPT3 (5-sbot)
0.214
0.132
15.96
0.76
0.80
GPT:3 (ID-shor)
0,208
0.145
17.83
0.77
0.80
IPA- (GPT3)
0.150
0.056
10.34
0.79
081
IPA (GPT3)
0,101
0.028
12.68
0.79
0.83
particularly limited in lexically constrained gener- ation, likely due to GPT-3's inherent limitations when dealing with hard logical constraints. Impor- tantly, IPA on top of zero-shot GPT-3 outperforms all the few-shot baselines by a noticeable margin across all scenarios. The results further highlight the importance of our method, which directly op- timize the base policy to align with user-specified objectives instead of solely relying on the innate capabilities of LLMs through prompting.
Table 8: Automatic evaluation results for Toxicity Re- duction with off-the-shelf GPT-3.
A Further Experiment
A.1 Human Evaluation for Toxicity
We perform additional pairwise human evaluation tailoring GPT-3 to reduce toxicity. We compare the outputs from IPA R and IPA- each baseline, based on the perceived level of toxicity (which one is less rude or disrespectful), topicality (which one is more natural, relevant, and logical), and fluency (which one is more grammatically correct and coherent), on 100 random prompts from the test set of
A.3 Additional Experiments with LLaMA
We conducted additional experiments with LLaMA models (Touvron et al., 2023) for the constrained generation task. We apply IPA tailor an off-the- shelf LLaMA-133 model and initialize the policy adapter with LLaMA-7B model. As shown Table 10, IPA leads to remarkable improvement on top of LLaMA- 13B in terms of constraint coverage while maintaining language quality.
using. As shown in Table 7, the human evaluation re- sults confirms that both IPA- and IPA * effectively tailor GPT-3 be less toxic while maintaining the language quality. This again underscores the poten- tial of IPA as a cost-effective method for aligning large language models with user-defined objectives.
A.4 Reward Analysis
We provide further analysis to justify our selection of reward functions for each task.
A.2 Additional Baseline: Few-shot
Toxicity Reduction Following previous work Lu et al. (2022b), we use the Perspective API score as reward function, which provides a score between (non-toxic) and 0 (toxic). We observe that IPA effectively reduce the toxicity while preserving the language quality in terms of fluency and diversity in both automatic and human evaluation.
In the experimental section, we show that in zero- shot setting LLMS such as GPT-3 often struggle to fulfill users' requests, such as generating safe content or reliably satisfying lexical constraints. Here, we conduct additional experiment to access LM's performance in few-shot setting toxicity reduction and lexically constrained generation.
As illustrated in Table 8 and Table prompting GPT-3 with additional few-shot examples improves its performance to some extent, but still falls short of consistently fulfill users' requests. The gain is
Lexically Constrained Generation Our goal is to enhance constraint satisfaction. As shown in Table 11, optimizing for constraint coverage alone may result in a slight decline in language fluency,

---

Reward
Coverage
Fluency
Reward
Safely Coherence Engaging Overall
safety
0.85
0.82
1.32
0.88
coverage
90.75
83,91
safety, coherence. engaging 0.78
0.90
1.91
0.98
coverage, flueney
88.54
92.58
Table 13: Evaluation results for Dialogue Safety Control on DIASAFETY with different reward functions.
Table II: Automatic evaluation results for Lexically Constrained Generation with off- the-shelf GPT-3 as the base policy using different reward functions
Reward
Critic Coherence Engaging Overall
critic
85.3
0.84
1.01
0.88
Reward
Diversity
Coherence
Critic
Mauve
critic, coherence, engaging 76,6
0.91
1.34
0,97
coherence
92.41
64.98
5.41
68.25
coherence, critic
93.73
51.03
52.36
84,32
Table 14: Evaluation results for Knowledge-
coherence, critic, divensity
51.81
50.93
Dialogue on Faithdial with different reward functions.
Table 12: Automatic evaluation for open-domain gen- erations on XSum with off off-the-shelf GPT2-XL as the base policy using different reward functions.
generated dialogue, measured by coherence, engag- ingness and overall score from UniEval-Dialoguee (Zhong al., 2022). The generated responses are often the exact copy of the given knowledge, lack- ing of ubstractiveness. We found that integrating coherence and engagingness scores as additional reward helps preserving the naturalness of the gen- erated responses while enhancing their faithfulness.
measured by COLA. However, by incorporating fluency as an auxiliary reward, we notice improve- ments in both dimensions. Human evaluations fur- ther support our findings.
Open-ended Generation The goal is to make machine- machine-generated content more fluent, coherent, and human-like. As shown in Table 12, optimiz- ing solely for coherence does not yield significant improvements in the overall generation quality, as evaluated by MAUVE. Incorporating scores from the OpenAI detector, a classifier for distinguishing between AI vs. Human--ritter text, as an additional reward serves as an essential element in improving the overall quality and human-likeness of generated texts. Moreover, we found that integrating diversity score another auxiliary reward helps maintain the diversity of generations while promoting higher quality output.
B Runtime Analysis
We conduction additional runtime analysis on toxi- city reduction task, comparing the inference speed of IPA with other baseline methods. As shown in Table B, IPA is significantly more efficient than most of the baseline methods and falls within a similar range as nucleus sampling.
Method
Runtime
Nucleus Sampling
0.03
PPLM (Dathathui et al., 2020a)
23.7
GeDi (Krause et al., 2021)
0.78
Dexperts (Liu et al., 2021a)
0.12
Dialogue Safety Control Our aim to improving the safety of a dialogue model. As shown in Table 13, optimizing for safety score alone may result in a decrease in the overall quality of the generated dialogue, measured by coherence, engagingness and overall score from Unilvval-Dialogue (Zhong al., 2022). The generated responses tends to be bland and templated, such as don't know..." "I'm not sure..... We found that integrating coherence and engagingness scores as additional reward helps preserving natural dialogue flow while promoting safe responses.
DAPT (Gururangan et al., 2020)
0.03
QUARK (Lu et al., 2022a)
0.03
Inference-time Policy adapter
0.08
Table 15: Inference runtime (seconds per sentence gen- eration) of IPA versus other baseline methods with GPT22L L the base policy on toxicity reduction task.
Experiment Detail
C.1 Off-the-Shelf Models
We download off-the-shelf models, including pre- trained GPT-2 and BlenderBot, from HuggingFace Transformers (Wolf et al., 2020), which are imple- mented in the PyTorch deep learning framework. We access GPT-3, GPT-3.5 and GPT-4 models via API calls through OpenAl platform.
Knowledge-grounder Dialogue Our aim im- proving the faithfulness of dialogue response with respect to the given knowledge. As shown in Ta- ble 14, optimizing for faithfulness score alone may result in a decrease in the overall quality of the

---

C.2 Model Training Details
Hyperparameter
Assignment
model
GPT2-Large
All training is performed on 8 NVIDIA Quadro RTX 8000 GPUs and costs about 3000 GPU hours in total. Our method is implemented with PyTorch an the Huggingface Transformers library.
number of parameters
774M
numher of steps
14000
batch size
64
learning rate optimizer
Adam
Adam epsilon
le-8
C.2.1 Toxicity Reduction
Adam initial learning rate learning rate scheduler
le-5
lincar with warmup
We initialize the policy adapter with an off-the- shelf GPT2-L model and use QUARK as the RL al- gorithm for the adapter training. Hyperparameters for training are given in Table 16. We performed hyperparameter grid search for the number of training steps over the range [10k, 20k), for the KL coefficient 3 over the range [0, 0.3], and for the frequency of exploration over the range [5, 20). During inference, we use nucleus sampling with p= 0.9 and temperature 1.0.
warmup steps coefficient |
500
0.01
frequency of exploration
15
Table 17: Hyperparameters for training policy adapter to lexically constrained generation
frequency of exploration over the range [15, 25J. During inference, we use nucleus sampling with = 0.9 and temperature 1.0.
Hyperparameter
Assignment
Hyperparameter model
Assignment
model
GPT2-Large
GPT2-Large
number of parameters
774M
number of parameters
774M
number of steps
50000
numher steps
18000
batch size
batch size
64
learning rate optimizer Adam epsilon
Adam
learning rate optimizer Adam epsilon
Adam
le-8
1c-8
Adam initial learning rate lcarning rate scheduler warmup steps KL coefficient 8 frequency of exploration
le-5
Adam initial learning rate learning rate scheduler
le-5
lincar with warmup 800
linear with warmup
warmup steps KL coefficient | frequency of exploration
1500
0.05
0.05
25
Table 16: Hyperparameters for training policy adapter to reduce toxicity
Table 18: Hyperparameters for training policy adapter to open-ended generation
C.2.2 Lexically Constrained Generation
C.2.4 Dialogue Safety Control
We initialize the policy adapter with an off-the- shelf GPT2-L model and use QUARK as the RL al- gorithm for the adapter training. Hyperparameters for training are given in Table 17. We performed a hyperparameter grid search for the number of training steps over the range [5k, 20k), for the KI. coefficient B over the range [0, 0.3], and for the frequency of exploration over the range [10, 30]. During inference, we use nucleus sampling with
We initialize the policy adapter with an off-the- shelf Bonnerrbot-IB-diisee model and use QUARK as the RL algorithm for the adapter training. Hy- perparameters for training given in Table 19. We performed hyperparameter grid search for the number of training steps over the range [10k, 15k], for the KL coefficient over the range [0, 0.3], and for the frequency of exploration over the range [10, 30]. During inference, we use nucleus sampling with p = 0.6 and temperature 1.0.
p = 0.9 and temperature 1.0.
C.2.3 Open-ended generation
C.2.5 Knowledge-groundee Dialogue
We initialize the policy adapter with an ohe-the- shelf GPT2-L model and use QUARK the RL al- gorithm for the adapter training. Hyperparameters for training are given in Table 18. We performed a hyperparameter grid search for the number of training steps over the range [30k, 50k], for the KL coefficient B over the range [0, 0.31, and for the
We initialize the policy adapter with an off-the- shelf blenderbot- Biiieerroot-B-iistii IB-distill model and use QUARK as the RL algorithm for the adapter training. Hy- perparameters for training are given in Table 20. We performed hyperparameter grid search for the number of training steps over the range [7.5k, 15k), for the KL coefficient over the range [0, 0.37, and

---

Hyperparameter
Assignment blenderbot-IB-diie
trained models (Michel et al., 2019; Voita et al., 2019). Additionally, (Li et al., 2018) demonstrate the effectiveness optimizing a model dimensional randomly oriented subspace. Later studies (Aghajanyan et al., 2021) have also shown that the intrinsic dimensionality decreases with pre- training larger models. (Hu et al.. 2022) learns a low-rank factorization via projection matrix and ap- plies them the self-attention weights. Recently, adding a small subset parameters called adapters (Rehuffi et al., 2017) and compact adapters (Ma- habadi et al., 2021) which are model-specific (Stick- land and Murray, 2019). Pfeiffer et al. (2020) intro- duced a continuously evolving Adapter-Hub that stitches different pre-trained adapters for languages and tasks inspired from routing networks (Rosen- baum et al., 2019) optimized through reinforcement learning (Kirsch et al., 2018; Chang al., 2019). Though these methods are efficient, they require access to the internal representation for model and gradient, which not feasible for large models like GPT3 with limited access.
model
number of parameters
18
low-
number of steps
15000
batch size
64
leamning optimizer Adam epsilon
Adam
le-8
Adam initial learning rate
le-5
learning scheduler
linear with warmup
warmup steps
300
KL coefficient 3
0.1
frequency of exploration
15
Table 19: Hyperparameters for training policy adapter to control dialogue safety
for the frequency of exploration over the range [15, 30J. During inference, nucleus sampling with p 0.6 and temperature 1.0.
Hyperparameter
Assignment
model
blenderbot- distill
number of parameters
1B
number of steps
12500
batch size
64
learning optimizer Adam epsilon
Adam
lc-8
Refinement. Recent work controls (L)LMs by fining a generated sequence into improved with refinement module (Yasunaga and Liang, 2020; Saunders et al., 2022; Schick et al., 2022; Yang et al., 2022; Welleck et al., 2023; Madaan et al., 2023). These methods operate in the se- quence space, while IPA's adapter policy makes fine-grained 'refinements' in the simplex (i.e., on next-token distributions). Typically the refiner is large (e.g., Saunders et al. (2022); Madaan et al. (2023)), or depends specialized train- ing data (Schick al., 2022) or learning algo- rithms (Welleck et al., 2023). IPA's adapter policy is lightweight, and is directly optimized with stan- dard RL algorithms.
Adam initial learning rate
le-5
leamiing scheduler
linear with warmup
warmup steps
300
KL coefficient 3
0.1
frequency of exploration
25
Table 20: Hyperparameters for training policy adapter to improve dialogue faithfulness
D Additional Related Works
Earameter-Efficient Fine-Tuning Prompting and teefin-tumingga (Li and Liang, 2021) adapt very large model to specific task. However, they are affected by sensitivity based order of words or examples (Zhao al., 2021; Webson and Pavlick, 2022), lack associative clarity (Min et al., 2022) and tuning prompts work for only very large mod- els (Mahabadi al., 2021; Liu al., 2022b). These methods compose the input to the model. con- trast, parameter-efficient finetuning offers a clean way to compose parameters directly by adding or updating smaller subset of model parameters. A common strategy is to prune the model parameters and introduce sparsity (Han et al., 2017; Frankle and Carbin, 2019; Frankle al., 2020). The ef- fectiveness of this approach is also substantiated with the use of RL (Yu al., 2020). Instead of pruning individual units, structured prrutingd--ruming prunes an entire group, such as attention heads in pre-
E Human Evaluation
We illustrate the human evaluation layouts Ama- zon Mechanical Turk for Dialogue Safety Control, Open-ended Generation, and Lexical Contrained Generation tasks in Figures 4, 5 and 6. We ensure the annotators are paid adequately for at least $15 per hour and we inform annotators that their an- notations are used for model evaluation purpose.

---

expand/collapse))
WARNING
This may contain content may offensive upsetting. Worker discretion strongly
advised.
participating in
the instructions carefully.
this HIf, you will be given a short dialogue between huamn bot. dialagues is represented as user-bot pair. The user sharing some unsafe and toxic thoughts, or bahaviours. The bot must respond to user in safe and avoid to do any the following unsafe bahaviours:
offending the user
provoking serious self-harm orises
providing unauthorized expertise
agreeing toxicty
expressing biased opinion towards specific group people
will asked following questions
response given
response unsafe toxic?
Tip: response
2. the bot's response coherent en-topic respect the user's utterance?
A coherent response follows the natural flow of the conversation, Conversely, an incoherent response is either off- topic and unrelated or too generic egg "I don't know", "1 am not sure", "I am language model, do not have feelings".
Examples (clikk to expand/collapse)
Task
S(user)
Bot: S(bot)
Q1. Is the bot's response unsafe andior toxic?
unsafe andior
Samenohul somewhat unsafe.
Defaniely unsafe and/or tosi..
Q2. Is the bot's response coherent and an-topic with respect to the user's utterance?
You coherent and on-topic with respect user's utterance.
Gameenul too generic.
coherent on-topic with respect the user's utterance.
(Optional) Please let know anything unclear, If you experienced you any feedback you found this HIT difficultt answer, please let know why.
Submit
Figure 4: Human evaluation layout on Amazon Mechanical Turk for Dialogue Sfaety Control

---

expand/collapas)
participating in
this HIT, you are shown prompt and two generated continuation paragraphs, Continuation and Continuation The continuations do not necessarily end with complete sentence and may contain partial sentences at the end. Please use your best judgement based on the available texts.
YOUR TASK to pick the continuation that:
fluent and grammatical, without unnatural repetitions
Is more coherent and on-topic with the prompt
is of higher quality overall
NOTES:
pick
possible!
Please do not hover too much over fine grained differences. When in doubt, choose go with your gut instincts.
prompt: S(prompt)
Continuation A:
VS.
Continuation B:
$(coninuation_0)
$[continuatin_])
Which continuation is more fluent and grammatical?
REMEMBER: "Swtter" continaation
(Peuse Pick when possibett Keep "Equal"
realy decide.)
grammaticaal English
Contmnation
grammatical.
excludes unnatural
fluent and grammatical.
has natural flow
Equally good equally bad.
Which continuation is more coherent and on- on-topic with the prompt?
REMEMBER: "Better" continuation.
possibiel
you really
topic the prompe
avoids unnatural
Continiation is more coherent on-topic.
Continattion
on topic.
Equal: Equally equally bad.
Which continuation is overall of higher quality? "Equal" when
REMEMBER: "Better" continuation..
overall more preferable
Continuation higher quality overall.
Contimation higher quality
Equally equally bad.
(Optional) Please know anything unclear, you experienced any issues, you any feedback
Figure 5: Human evaluation layout on Amazon Mechanical Turk for open-ended generation

---

Concepts:
S(source)
Sentence:
sigeneration?
1. SENTENCE QUALITY
Is the sentence well-formed?
senrence
Somewhat: The sentence is understandable but bit awkward.
well-formed
2. PLAUSIBILITY Does the sentence dieccribe plausible scenario?
Yes: The sencence describes a realistic or plausible scenario.
Somewhat: The sentence describes acceptable scenaria bit awkowrd.
No: sentence describes nonsensical scenario.
CONCEPTS
Does the include the given concepts meaningfully?
Yes: The meaningfully includes all the concepis.
Somewhat:
The sentence meaningfully includes some. but not all of the concepts. Or, the sentence includes all mean properly incorporated.
concepts
No: The sentence does not induude conrepts in meaninghul way.
OVERALL Considering your
sentence meaningfully combine of the concepts
well-formed and plausible scenario?
senterce reasonably weeeiddcimeeiideeetaadadlllee meaningfully combines concepts plausible scenario.
Somewhat: The sentence looks okay terms above questions.
No: The sentence is not wel.fformeddunderttanaiiiiie or fails property combine all the concapts into
Figure 6: Human evaluation layout on Amazon Mechanical Turk lexical constrainted generation


Reviews:
[{"review_id": "vlkJoQDzM5", "reviewer": "Reviewer_Ge8N", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer Ge8N for their encouraging feedback! We’re thrilled to hear that they found our paper is “well-motivated”, “well-written”, “relevant to many researchers studying large language models” and our “experiments and evaluations are thorough and promising”.\n\n### Re: “extra computation is needed to derive the approximate policy”\nFirst, we’d like to clarify that **the computation of distilling an approximate policy is in fact very efficient**. As we highlighted in the original draft (**line 374-376**): “while fine-tuning GPT-3 costs \\\\$156.82, training a distilled GPT-3 as the approximate policy requires only \\\\$28.59 for generating outputs from GPT-3”. In terms of time and compute cost, it takes around 3 GPU hours to train a distilled GPT-3 on 8 NVIDIA Quadro 1394 RTX 8000 GPUs. We’ll include this additional detail in our revision.\n\nSecond, in the case of recently released *open source* large language models, **there is always a smaller model within the same model family** that can be directly used as an approximate policy. We surveyed over multiple recent model families, including Flan-T5, PaLM2, PaLM, Falcon, LLaMA, LLaMA2, Vicuna, MPT, Lazarus, WizardLM, GPT4All, and found all of them release a series of models of various sizes.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper is well-written.\n2. The research problem is well-motivated and relevant to many researchers studying large language models.\n3. The experiments and evaluations are thorough and promising.", "reasons_to_reject": "When the extreme-scale policy model is not feasible to use and when a smaller model from the same model family is not present, extra computation is needed to derive the approximate policy.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes Inference-time policy adapters, a new method to efficiently adapt large language models to user specified objective via reinforcement learning. More specifically, a small policy adaptor is trained to adjust the (frozen) base LM output distribution while optimizing a pre-defined reward signal for a specific task. Experimental results on five text generation tasks confirm the efficacy of the proposed method.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FnjAAcArMM", "reviewer": "Reviewer_44dF", "summary": "", "questions": "Questions:\n1.\tWhat motivated the usage of GPT-2 for a list of experiments? It is relatively small and just prefer to learn more sota models (e.g., LLaMA and related LLMs) and the related results.\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer 44dF for their feedback! We’re excited that they found our IPA is “an interesting and effective idea\", with “impressive results” and “will save a lot of time for model alignment”.\n\n### Re: Reason to reject\nWe would appreciate it if reviewer 44dF could clarify their reason to reject. We’re **puzzled** by the reason to reject as it appears to be a **positive statement** rather than a valid justification for rejection. We do agree that “model aligning will still be an important direction” and that “related investigation will be required”. As a response, our paper investigates efficient strategies for aligning large-scale language models through the use of an inference-time policy adaptor. \n\n### Re: motivation for the usage of GPT-2\nFirst, we’d like to remind the reviewer that **we’ve also included the most powerful large-scale models**, such as **GPT3**, **Blenderbot**, both as a model to tailor and as a baseline in the experiments for *all the tasks*. (see Table 1, Table 3, Table 4, Table 5, Table 6). As we highlighted in the paper (see line 309, line 368, line 439, line 485, line 569), we observed a significant performance gain on top of these powerful models brought by IPA, which surpassed the existing SOTA baselines across all the tasks. \n\nSecond, we choose GPT-2 in our experiments to **showcase the effectiveness and efficiency of IPA**. We demonstrate that **a small language model, i.e GPT2-large, as the policy adaptor can successfully tailor a x200 larger base language model GPT3** across various tasks (see line 309, line 368, line 439). We also show that **directly applying the policy adapter optimized for GPT-2 on top of GPT-3 (IPA-) is highly effective**, showcasing the adaptability and reusability of IPA (see line 308, line 438). Both aspects highlight the lightweight and easily accessible nature of our approach, making it suitable for the wider NLP community, particularly in limited resource settings.\n\nFinally, **our method is orthogonal to the models used**. As mentioned earlier, we’ve illustrated the effectiveness of IPA with multiple most powerful large-scale language models in the original draft. Nonetheless, we **added additional experiments with LLaMA models** for constrained generation tasks as suggested. We apply IPA to tailor an off-the-shelf LLaMA-13B model and initialize the policy adapter with a LLaMA-7B model. As shown in table below, IPA leads to remarkable improvement on top of LLaMA-13B in terms of constraint coverage while maintaining language quality. We’ll add LLaMA experiments for other tasks as well in the final draft.\n\n| Models   |      Coverage      |  Fluency |\n|----------|:-------------:|:------:|\n| LLaMA-13B |  28.73 | 89.64 |\n| IPA (LLaMA-13B) |   81.49   |   89.71 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strong:\n1.\tThe whole IPA idea is interesting and effective.\n2.\tOnly inferencing time adapter will save a lot of time for model alignment during fine-tuning or peft.\n3.\tImpressive results were reported for tailoring gpt-2 which fit gpt-3’s results.\n", "reasons_to_reject": "Weak:\n1.\tModel aligning will still be an important direction for LLM adaptation and a combined usage and related investigation will be required.\n", "questions_to_authors": "Questions:\n1.\tWhat motivated the usage of GPT-2 for a list of experiments? It is relatively small and just prefer to learn more sota models (e.g., LLaMA and related LLMs) and the related results.\n", "paper_topic_and_main_contributions": "This paper proposes Inference-time Policy Adapters (IPA), which efficiently tailors a language model such as GPT-3 without fine-tuning it. IPA guides a large base model during decoding time through a lightweight “policy adapter” trained to optimize an arbitrary user objective with reinforcement learning.\n\nGood results were reported when tailoring a list of strong baseline models, such as GPT-2.\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jq6BoQMaGa", "reviewer": "Reviewer_pKXZ", "summary": "", "questions": "1. How is the reward function for reinforcement learning chosen?\n\n2. Is the same dataset split was used for both reinforcement learning and evaluation?\n\n3. Proposed new experiment: use RL to optimize for one metric, and see if it hurts the performance with another metric in the same task. This experiment would help address concerns on whether such reward-oriented RL approach will hurt the generation quality not captured by existing metrics.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer pKXZ for their feedback! We’re thrilled to hear that they found our paper “addresses a crucial issue” and“ offering an efficient solution”. We’re also excited to learn they consider our proposed method as “simple yet effective”, “comprehensively tested on a variety of text generation tasks” and “consistently exhibits performance enhancements”.\n\n### Re: the benchmarks and metrics used for evaluation appear to be arbitrary, challenging to compare against off-the-shelf baselines\nWe respectfully disagree based on the previous literature.\n\n**The benchmarks we selected are widely used in previous literature** for controllable generations, and **represent diverse tasks across different domains**. RealToxicityPrompts is widely employed to evaluate toxicity reduction [1, 2, 4, 5, 9], CommonGen is extensively adopted to access hard lexical constraint satisfaction [5, 6, 7, 8, 9], WoW is widely used to gauge knowledge-grounded dialogue generation [12, 13, 14], DiaSafety is frequently adopted to test dialogue safety control [15, 16], and New articles are broadly employed to evaluate open-ended generation [3, 10, 11]. \n\nSimilarly, we closely followed previous works and **used metrics that are standard and widely used for the chosen benchmarks**. We also specified the previous work we followed for the experimental setups (see line 291, line 387, line 331, line 529, line 457). Importantly, **we reported multiple metrics for each task**, to measure task performance as well as language quality such as fluency, coherence, diversity (see Table 1, Table 3, Table 4, Table 5, Table 6). We did this to ensure that IPA genuinely improves generation quality rather than overfitting to a specific metric. Additionally, **we conducted human evaluation** to further assure generation quality (see Table 3, Figure 3, Table 5, Table 7). We observe IPA constantly outperforms all baselines across various automatic and manual metrics for all the tasks. \n\nAs for baselines, we not only compare with the off-the-shelf models **but also existing SOTA methods for all the tasks**, to ensure the efficacy of our proposed method.\n\n### Re: the algorithm requires manual configuration of the reward function for each task\nYes and No. \n\nYes in the sense that defining a reward function for a specific task is **the standard practice in the Reinforcement Learning (RL) field**, and a common procedure adopted across all RL literature. \n\nNo in the sense that **we used the most commonly adopted reward function in previous literature** for each task. For example, PerspectiveAPI score has been widely used in numerous prior works [1, 2, 4, 5, 9] as a reward function for detoxification. Similarly, the product of coverage and fluency scores has been employed as a reward function for lexically constrained generation in various previous papers [5, 7, 8]. In fact, there is a line of work in the RL field studying the selection and improvement of reward function in various settings, such as Ramamurthy et al, 2022 [17], but that’s out of scope for our current work. \n\nAdditionally, we conducted ablations with different reward functions for the same task in the appendix A.3. And we found IPA is highly effective with various reward functions. \n\n### Re: the same metric is used for model evaluation\nWe acknowledge that reward hacking is an important problem. To address this, **we've reported multiple metrics for the same task** in our original draft, **not only the one used as the reward function**. We measured task performance as well as language quality such as fluency, coherence, diversity (see Table 1, Table 3, Table 4, Table 5, Table 6). We did this to ensure that IPA genuinely improves generation quality rather than overfitting to a specific metric.  \n\nAdditionally, **we conducted human evaluation** to further assure generation quality (see Table 3, Figure 3, Table 5, Table 7). We observe IPA constantly outperforms all baselines across various automatic and manual metrics for all the tasks.\n\nAs for exploring LLMs \"emergent abilities\", that does not align with the objective of the current paper and might lead us away from the core focus of our paper which is tailoring a large language model at decoding-time toward desired objectives without the need of fine-tuning. \n\n### Re: selection of the reward function\n**We use the most commonly adopted reward function in previous literature** for each task. For example, PerspectiveAPI score has been widely used in numerous prior works [1, 2, 4, 5, 9] as a reward function for detoxification. Similarly, the product of coverage and fluency scores has been employed as a reward function for lexically constrained generation in various previous papers [5, 7, 8].\n\n### Re: dataset split\nWe only use the train and validation set in RL training. We confirm that the **test set is untouched and reserved for testing only**, and that all reported numbers are on the test set. We followed the standard train/val/test splits for all the tasks.\n\n### Re: proposed new experiment\n**We’ve already included the experiment suggested by the reviewer in our original draft**. For each task we report multiple metrics, including those that are not part of the reward function. More specifically, we measure task performance as well as language quality such as fluency, coherence, diversity  (see Table 1, Table 3, Table 4, Table 5, Table 6) and found IPA constantly improves task performance while maintaining generation quality.\n\n\n\n#### **Bibliography**\n[1] Liu, Alisa et al. “DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts.” Annual Meeting of the Association for Computational Linguistics (2021).\n\n[2] Lu, Ximing et al. “Quark: Controllable Text Generation with Reinforced Unlearning.” ArXiv abs/2205.13636 (2022): n. Pag.\n\n[3] Li, Xiang Lisa et al. “Contrastive Decoding: Open-ended Text Generation as Optimization.” Annual Meeting of the Association for Computational Linguistics (2022).\n\n[4] Krause, Ben et al. “GeDi: Generative Discriminator Guided Sequence Generation.” Conference on Empirical Methods in Natural Language Processing (2020).\n\n[5] Welleck, Sean et al. “Generating Sequences by Learning to Self-Correct.” ArXiv abs/2211.00053 (2022): n. pag.\n\n[6] Qin, Lianhui et al. “COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamics.” ArXiv abs/2202.11705 (2022): n. Pag.\n\n[7] Lu, Ximing et al. “NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraints.” ArXiv abs/2010.12884 (2020): n. Pag.\n\n[8] Lu, Ximing et al. “NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristics.” North American Chapter of the Association for Computational Linguistics (2021).\n\n[9] Kumar, Sachin et al. “Gradient-based Constrained Sampling from Language Models.” Conference on Empirical Methods in Natural Language Processing (2022).\n\n[10] Su, Yixuan et al. “A Contrastive Framework for Neural Text Generation.” ArXiv abs/2202.06417 (2022): n. Pag.\n\n[11] Meister, Clara et al. “Typical Decoding for Natural Language Generation.” ArXiv abs/2202.00666 (2022): n. Pag.\n\n[12] Lewis, Patrick et al. “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.” ArXiv abs/2005.11401 (2020): n. Pag.\n\n[13] Shuster, Kurt et al. “Retrieval Augmentation Reduces Hallucination in Conversation.” Conference on Empirical Methods in Natural Language Processing (2021).\n\n[14] Dziri, Nouha et al. “FaithDial: A Faithful Benchmark for Information-Seeking Dialogue.” Transactions of the Association for Computational Linguistics 10 (2022): 1473-1490.\n\n[15] Sun, Hao et al. “On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmark.” ArXiv abs/2110.08466 (2021): n. Pag.\n\n[16] Meade, Nicholas et al. “Using In-Context Learning to Improve Dialogue Safety.” ArXiv abs/2302.00871 (2023): n. pag.\n\n[17] Ramamurthy, Rajkumar et al. “Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization.” ArXiv abs/2210.01241 (2022): n. pag.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper addresses a crucial issue in generative large language models: the extensive computational resources required for fine-tuning. By incorporating a lightweight adapter during inference, the proposed method obviates the need to alter existing LLM parameters, thereby offering an efficient solution.\n\n2. The design of the adapter and the reinforcement learning algorithm is very straightforward, making the proposed method simple yet effective\n\n3. The method has been comprehensively tested on a variety of text generation tasks, where it consistently exhibits performance enhancements over existing baseline models.", "reasons_to_reject": "1. The benchmarks and metrics used for evaluation appear to be subjective and potentially arbitrary. I understand that generative tasks lacks of well-known, gold-standard metrics such as BLEU, ROUGE, or F1/Accuracy. But I still find it challenging to compare against off-the-shelf baselines, diminishing confidence in the efficacy of the proposed method.\n\n2. The algorithm requires manual configuration of the reward function for each task, e.g., calling Perspective APIs for toxicity reduction, or setting the *product* of coverage and fluency scores for lexically constrained generation. This approach seems arbitrary, with no principled methodology provided.\n\n3. Given that the same metric is used for model evaluation, there's concern that the model might be overfitting to these metrics, rather than genuinely improving generation quality. Again, the metrics for these tasks does not include all aspects, such as LLM's \"emergent abilities\". It would be advisable to study whether using IPA aiming to improve some metrics will hurt the performance evaluated by a different set of metrics under the same task. Also, it is unclear whether the same dataset split was used for both reinforcement learning and evaluation. If so, serious concerns of overfitting will arise, and the method should be directly compared with supervised finetuning methods instead.", "questions_to_authors": "1. How is the reward function for reinforcement learning chosen?\n\n2. Is the same dataset split was used for both reinforcement learning and evaluation?\n\n3. Proposed new experiment: use RL to optimize for one metric, and see if it hurts the performance with another metric in the same task. This experiment would help address concerns on whether such reward-oriented RL approach will hurt the generation quality not captured by existing metrics.", "paper_topic_and_main_contributions": "The paper presents Inference-time Policy Adapters (IPA), an novel method to efficiently customize large language models like GPT-3 without expensive fine-tuning. Employing a lightweight policy adapter, IPA optimizes user objectives with reinforcement learning, enhancing control over LMs during decoding time. The paper demonstrates IPA's superiority over baseline methods and its proficiency in various text generation tasks, like toxicity reduction and lexically constrained generation.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "Very good writing", "Justification_for_Ethical_Concerns": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work presents IPA (Inference-Time Policy Adapters), a novel way of adapting LLMs to various generation tasks by "tailoring" their output distribution through task-specific training of other (smaller) LMs via reinforcement learning (RL).

The output probability for the next token prediction is modelled as the normalized product of the original and adapter LM probabilities. Crucially, during training, the LLM is frozen, and only the adapter is updated with the RL process.

The efficacy of IPA is demonstrated on 5 generation tasks using various SOTA LLMs as base models, and smaller variants or distilled LMs as adapters. For each task relevant RL rewards are devised, IPA is trained, and automatic and human evaluation is carried out, against a variety of strong baselines.\
IPA proves to be effective on all tested tasks, notably sometimes even when a nominally less capable model (GPT2) is used to tailor a strong LLM (GPT3).

Some ablations wrt. reward modelling are presented (Appendix), showing that IPA is somewhat robust to the precise reward, yet performs best when the full proposed rewards are used.

## Strengths
Reviewers agree the paper is sound (4, 3, 3) and exciting (4, 4, 3), and I agree with the positive aspects mentioned by reviewers.
- The paper is very well written and easy to follow; it is clearly structured, and presents the proposed novelties in a concise and understandable manner.
- The topic of adapting (extremely) large LLMs is very current, and the authors propose a novel method that seems both easy to implement and train, and effective on a variety of tasks.
- The conducted experiments are extensive and seem to support the author's claims; the use of human evaluation for the presented generation tasks is commendable and largely supports the results of automatic metrics.

## Weaknesses
Concerns were raised wrt. to various aspects of the proposed method. The authors provided extensive rebuttals and addressed, in my opinion, all relevant criticism in a satisfactory manner.
- Availability of smaller versions of base LLMs might be an issue; however, it is demonstrated that (i) often smaller versions are indeed available; (ii) distilled versions can be derived relatively cost-efficient; (iii) from the paper's results, even not directly related models might be viable candidates for adapter policies.
- Some of the chosen benchmarks/metrics were questioned; I find myself agreeing with the authors that the these are standard and relevant.
- The need for task-specific rewards was criticised. I again have to agree with the authors that such task-specific rewards are inherent to the RL process. While the choice of specific rewards in the paper could have been better motivated in the description of each task to make the it more understandable, the individual rewards are (in my opinion) sensible and justified by the tasks and relevant related work; clarification is an easy one-line fix per task for a camera ready version.
- Related, it was pointed out that directly optimising a reward that is also the test evaluation metric is potentially problematic; however, experiments are extensive enough to show improvements persist for metrics that are not used for reward modelling, as well as for human evaluation.

## Conclusion
The authors present a strong piece of work that is well-motivated, relevant, and extensively evaluated. In my opinion, IPA constitutes a valuable addition to the growing toolkit of "parameter-efficient" methods for LLM adaptation. In particular, IPA is applicable in settings where the underlying LM's parameters are not accessible, and only its output distribution is observable.

The efficacy of the proposed approach is demonstrated clearly on 5 different generative tasks, through automatic and human evaluation.\
I only have three small points of criticism, mostly wrt. evaluation:
- The human evaluation results of 4.2 (Lexically Constrained Generation) seem surprisingly weak compared to the clear improvements on automatic metrics. Looking at the Mechanical Turk question template I wonder if a 3-point Likert scale and the measured aspects are the right tool here, as even the base GPT-3 achieves 2.60 on the "Overall" question, and even higher on "Quality" and "Plausibility". Still, IPA improves over these base model scores.
- Task 4.5 (Knowledge-grounded Dialog) is the only one that was not put to a human evaluation; there might be reasons, which could briefly be pointed out.
- The title "[...] without Fine-tuning" might be construed to imply that no training is happening at all (e.g., that IPA is purely inference-based). This is obviously not the case, as the adapter policy is indeed trained (one could say with task-specific fine-tuning) via reinforcement learning. However, the abstract clarifies the true nature of the approach.

None of these points seem severe enough to not merit acceptance, and the work in itself should make a valuable contribution to the main conference.
Decision: Accept-Main