Controlled Text Generation via Language Model Arithmetic
Authors: Jasper Dekoninck
Marc Fischer
Luca Beurer-Kellner
Martin Vechev
Venue: ICLR 2024 spotlight (2024)
Keywords: Controlled text generation
LLM
Natural Language Processing
Primary area: generative models
TL;DR: We provide a principled and intuitive way to combine multiple LLMs and bias them towards and away from attributes.

Abstract: As Large Language Models (LLMs) are deployed more widely, customization with respect to vocabulary, style, and character becomes more important. In this work, we introduce model arithmetic, a novel inference framework for composing and biasing LLMs without the need for model (re)training or highly specific datasets. In addition, the framework allows for more precise control of generated text than direct prompting and prior controlled text generation (CTG) techniques. Using model arithmetic, we can express prior CTG techniques as simple formulas and naturally extend them to new and more effective formulations. Further, we show that speculative sampling, a technique for efficient LLM sampling, extends to our setting. This enables highly efficient text generation with multiple composed models with only marginal overhead over a single model. Our empirical evaluation demonstrates that model arithmetic allows fine-grained control of generated text while outperforming state-of-the-art on the task of toxicity reduction. We release an open source easy-to-use implementation of our framework at https://github.com/eth-sri/language-model-arithmetic.

Published as a conference paper at ICLR 2024
CONTROLLED TEXT GENERATION VIA LANGUAGE MODEL ARITHMETIC
Jasper Dekoninck, Mare Fischer, Luca Beurer-Kellner, Martin Vechev Department of Computer Science ETH Zurich, Switzerland drsper.dekoninck, marc. fischer, luca. beurer- martin www.eeenwee....a.......eeeeieeee
ABSTRACT
As Large Language Models (LLMs) are deployed more widely, customization with respect to vocabulary, style, and character becomes important. In this work, we introduce model arithmetic, a novel inference framework for compos- ing and biasing LLMs without the need for model (re)training highly specific datasets. In addition, the framework allows for more precise control of generated text than direct prompting and prior controlled text generation (CTG) techniques. Using model arithmetic, we can express prior CTG techniques as simple formu- las and naturally extend them to new and more effective formulations. Further, we show that speculative sampling, a technique for efficient LLM sampling, ex- tends to our setting. This enables highly efficient text generation with multiple composed models with only marginal overhead over single model. Our empir- ical evaluation demonstrates that model arithmetic allows fine-grate mee-grained control generated text while outperforming state-of state-of-the-art tion. We release an open source easy-to-use implementation of our framework at https:///githbb.comm com/eth wwt.tssrrrlagggeeeeeeeeeeeeeeeee model- ari arithmetic
the task of toxicity reduc-
INTRODUCTION
In recent years, Large Language Models (LLMs) (Brown et al. 2020 Chowdhery et al. 2022 Tou- vron et al. 2023) have been increasingly recognized for their capabilities in handling wide range of tasks (Rozière et al. 2023 Ouyang et al. 20022.. In many applications, such as chatbots interacting with diverse audiences like children, students, or customers, precise control and customization of attributes such as the employed vocabulary, linguistic style, and emotional expression are crucial.
Controlling Language Models A common technique for this is prompting with natural language (Ouyang et al., (022). While prompting is simple and makes it easy to condition the LLM to broad attribute, the ambiguity of natural language makes challenging to express how present that attribute should be in the generated text. Further, prompting also lacks the ability to effectively steer the model away from a certain attribute a reliable manner, as mentioning a specific topic in the prompt can inadvertently increase the likelihood of the model generating text about it (Jang al. 2022). e.g. "do not mention cats" " may increase the likelihood of the model referring to cats. One alternative is fine-tuning the model, but this requires highly specific training data for the desired attribute, which also has to implicitly encode the strength of the conditioning. Controlled Text Generation (CTG) techniques aim solve this problem by steering the model during inference instead (Liu al. 2021 Dathathri et al. 2020; Yang and Klein 202T): The model is conditioned on a particular attribute smoothly controllable way, by biasing the model's token distribution. Many CTG methods are inspired by Bayes rule P(text(a) x P(a P(tex),, and utilize an auxiliary model, i.c. P(altext), condition the I.IM, i.e., Pttext), towards a.
Key Challenge: Lack of Expressive and Efficient Control for Text Generation These tech- niques, however, suffer from several drawbacks, including lack of expressiveness, efficiency, and interpretability. First, to control the strength of the applied conditioning, a parameter À is introduced in an ad- hoc manner, i.e., as an exponential weight P(altext)>. However, introducing the strength in this way, while possible, quickly becomes unintuitive can longer be interpreted in Bayesian

---

Published as a conference paper at ICLR 2024
Write one-sentence fairy tale.
Input: Write one-sentence fairy tale. Output: Once upon time, a
Maiia
Once upon time, in magical kingdom far, far away, there lived brave and curious little princess who rode sparkly unicorn and outwitted grumpy dragon
Maud 0.6
Once upon time, there was a spurkly rainbow unicom that pooped ice cream and took me on a magical adventure a land made entirely of candy!
Menid
Matuk
Mmagic
Ciormal
Masiu a.6 + Clemei
Once upon time. in magical land full of fluffy clouds and sparkly rainbows, there lived a little baby unicomn named Sparkles who had a heautiful, shimmering mane
M 0.6.M + Chumall + union( Maiiu, Mmugi))
Masia 0.6 + Ceme + 2 unioonMaou, Mange]] Once upon a time, in a land far, far away, there was magical kingdom filled with sparkly unicorns, fluffyy dragons, and a princess who could make ice cream appear out of thin air!
Output: Once upon time, in land
Figure I: Overview of model arithmetic using an illustrative example. We outline the procedure for generating fairy tale (left) using the models Mehina. Malult Mugii that produce text conditioned on the attributes child, adult, and magic, respectively and Ciammal a classifier for the formality of text. The right table shows example outputs for different (partial) formulas. Image attribution in App. C
manner, e.g., when biasing away from attributes. Moreover, neither prompting nor CTG methods allow for the natural and controlled combination of multiple attributes or instructions with relative strength. This is due to the inherent ambiguity of natural language in prompting (Arora al. 2023 Zhao et al. [021] ). and the absence of a theoretical foundation and intuitive semantics for the biasing strength À with CTG methods. Lastly, both CTG techniques and fine-tuning often require custom and highly specific training data for the desired attribute (Yang and Klein 2021 Sansone and Man-
haeve 2022 [aha et al. [022]] Kim al. (023) and can be resource-intensive (Kumar et al. 2022 as multiple models are evaluated at inference time.
2021
Fine-Grained Control via Model Arithmetic In this work, we address these challenges and intro- duce model arithmetic, principled and intuitive method to combine multiple models. Our method is orthogonal to prompting, fine-tuning, and simple CTG concepts, like the use of classifiers, and can naturally incorporate them. Model arithmetic enables us to blend multiple LLMs and attributes into a single precisely controlled, the simple example in Fig- [ where we aim to write a magical, child-like fairy tale. We employ multiple models Ma. with different attributes o. On the top right, we see a prompted model Menid that already generates a appropppee story. However, the resulting text is not child-like and we therefore subtract an adult-condie adult-conditioned model, Madult, with a weight of 0.6 to generate less adult- sounding story. Now, to again increase formality, we additionally bias with classifier Ciormal- Lastly, we use a special union operator to obtain model that emphasizes both magical and child-like lan- guage and use it to further bias generation and obtain our final result. This simple example cannot be precisely expressed with prior CTG approaches and showcases the flexibility of model arithmetic. That is, it allows us to compose models in a natural way, while precisely controlling the impact of each component. Further, can naturally incorporate paradigms such prompting or fine-tuning (for the individual M and C)) and even implement many prior CTG techniques (discussed in $3) as simple formulas.
carmula-based composite model. To illustrate our method, consider
Efficient Model Arithmetic via Generalized Speculative Sampling CTG methods, including model arithmetic, can lead to increased inference times as multiple models need to be evaluated in order to generate text. To counteract this, we generalize speculative sampling (Chen et al. (023) to model arithmetic. Speculative sampling is usually employed to reduce the latency of single L.M by augmenting it with smaller model that proposes tokens, which are then validated by the L.M. In contrast, we extend it in a way where we postpone the evaluation of more expensive model calls within model arithmetic formulas. This allows to execute model formulas comprised of multiple models with only marginal overhead over a single model and reduces model calls by up to 64%. The resulting inference speedup naturally extends to prior CTG techniques that can be expressed in model arithmetic (Pei et al. 2023 [Sanchez et al. 2023 Chen et al. 2022 Schick et al. 2021).
2

---

Published as a conference paper at ICLR 2024
Key Contributions Our core contributions include:
Model Arithmetic: A principled framework for fine-grainet CTG, enabling precise control over multiple attributes. Our framework can express many prior CTG approaches ($3)
An extension of speculative sampling to model arithmetic, counteracting the overhead of CTG and enabling efficient inference, which naturally benefits CTG techniques expressible in model arithmetic ($4).
An extensive qualitative and quantitative evaluation model arithmetic ( show that is more expressive than prior CTG work and outperforms them in toxicity reduction. We demonstrate that our extended speculative sampling reduces model calls by up to 64%.
BACKGROUND
We briefly introduce the required background and notation used in the remainder of the paper.
Discrete Probability Distributions A discrete probability distribution P associates probability P(x) with every element & in a finite set T. For language modeling, this finite set is usually a set of tokens (or subwords). We often want to compute the probability of a token rı given all previous tokens Ik- in a sequence, which we denote as P(z 11.-))). We use the Kullback-Leible (KL) divergence to measure the similarity of two distributions P and Q:
= Σ P(o log P(xlx1:k-1)
DKL(PI(Q(EHA-I))
Q(1(=1x-1)
xET
where we append | to denote conditioning on a sequence of tokens T1:k-1- If this implied by the context, we will omit the conditioning on 21:k-1 and simply write (Pl(P)
Autoregressive Large Language Models Large Language Models (LLMs) are trained generate sequences of tokens. Most recently, successful LLMs are autoregressive, i.c. they generate output token-by-token by modeling the probability distribution P( and sampling one token at a time from that distribution. Whenever we refer to language model M, we directly refer to this distribution and denote it as (2x 11:k-1).
Controlled Text Generation As introduced in $ CTG techniques aim to introduce a given at- tribute (e.g. style or topic) in the output of language model M, by biasing its distribution with respect to Oftentimes, a strength parameter À controls the strength of this conditioning. The conditioning model P(o (tex)) modeled with a classifier (Yang and Klein 2021 Sitdikov et al., 2022 Kim et al.) 2023 Sansone and Manhaeve 2022 Saha et al. 2022). a smallell ler finetuned model (Liu et al. 2021). 1), or with the same model M using a different prompt (Pei et al. 2023 Schick al., 2021 Sanchez et al.] 2023 Chen et al. (022). In the first two cases, the biasing models have to be
trained ahead of time. Many these approaches are based on (a variant of) Bayes rule (Sanchez
al.
2023 Liu et al. 2021 Pei et al. 2023 Hallinan et al. 2023 Yang and Klein 202
Speculative Sampling Speculative sampling (Chen al. 2023 speeds up inference of autore- gressive language models by using small proposal model m to generate several tokens 21...., and then validates these tokens using a bigger, more capable model M. Due to the way the under- lying transformer architecture of current L.MSs (Vaswani et al. 2017) works, this validation call is significantly cheaper than generating the tokens with M directly.
....
Specifically, the entire sequence of proposed tokens æ1, Pr can be validated by single, retroac- tive call to M. token x; rejected by M, all subsequent tokens Ti+1s Ty are discarded and I; is resampled. If all tokens are accepted, the next token J&+1 can be directly sampled using the result of the same validation call to M. Thus, one can generate up to le + tokens with just a single call to M. Importantly, this procedure of accepting and resampling tokens ensures that the resulting distribution is equivalent to drawing token samples directly from M. For reference, we include the full speculative sampling procedure in Algorithm of App. E.
3

---

Published as a conference paper at ICLR 2024
3 MODEL ARITHMETIC
In this section we introduce model arithmetic,, a principled approach for advanced CTG that enables the precise composition of language models, resulting in distribution P that can be sampled like language model. This addresses the previously discussed drawbacks of prior CTG methods.
To this end, we first outline how an oupur distribution tions Q....., Qn by minimizing a linear combination of (weighted) KL-divergences DKL(P))Q.) Then we show how model arithmetic can be used to describe these distributions in a natural way. We defer all proofs to App. D
constructed from a set of inpur distribu-
(Weighted) KL.OOptimality The standard KL-divergence DKL (PIl)) attends to each token by an equal amount, which might not always be desirable in the CTG setting. Indeed, suppose Q represents the distribution of a certain attribute that we want to introduce in the output distribution P. When certain tokens are generally more associated with the attribute a, we might give the term DKL (PIQ) more weight for these tokens, allowing to more strongly bias these specific tokens while reducing the bias for less important tokens. We therefore introduce the weighted KL-divergence DKd
2 Σ P(2 (14-1) log P(x[=
Q(I(T18-1)
where f: T' x T R assigns a weight to each token E T, conditioned on T1:k-1- We will later show how high-level constructs model arithmetic map to particular choices of
Theorem now defines and solves the problem of combining arbitrary probability distributions into single output distribution by framing it minimization problem over linear combination of weighted KL-divergences.
Theorem 1 (Weighted KI.-Optimality). Let T be the set of all tokens and 31, ,-.. Tk- be se- quence of tokens such that It E T. Then, given distributions Q1...., Qn over T. functions fi...., : T Tk-1 + R, and under mild technical assumptions detailed in App. D.1 the solution the optimization problem for the generation of token Iy
arg min P
D
KT. (P11Q.
(I)
1=1
is given by
(
P(æx = = =o
En file, (1x-11
filr, ((1 Q2 (((
(2)
where a is the softmax function.
We note that this result applies more broadly than the autoregressive setting. Instead of conditioning on -1:k-1, 1, one condition on 21 #k41a without otherwise modifying the theorem.
Further, we can write fz 11) ) (f -1:k-1) where we factor f. into a part X that only depends the context (i.e., the previous tokens) for scaling, and (1 that encodes token specific weights.
Model Arithmetic Distribution P, resulting from Eq. is completely determined by (x(2115-1); (2 (1:k-1) and log (x for all x E 7 and i (1... (1.....n). Since T is finite set, we can write [a (1:k and log
as vectors f := (f Q = (log Qx(x)a 1. (a--))=rT Finally, since the normalization is completely determined X, and f/ we can drop this in our notation and write F = Σn Arf where vector multiplication is element-wise. We drop X; and f when they are
and
This notation makes it possible to use simple arithmetic operations to combine different input prompts, attributes, language models, and classifiers. We thus call this notation model arithmetic. Next, we discuss the operators in model arithmetic along with motivating examples (further shown in App. and summarize this in Table
4

---

Published as a conference paper at ICLR 2024
Overview of Model Arithmetic where I (z) := (2)(z) > Q2(2)] and I2(2) == (x),
Table 1:
Table 2: Prompt arithmetic examples us- ing Llama-2-Char
C' is a classifier and U the uniform distribution.
Tell something interesting about pandas.
Model Arithmetic
Optimization Problem
Mendi
Cenainly! Pandas are fascinating creatures, known for their distinct black and white markings...
Linear
Σ. Q:
Combination
2M M
Classifier
XC
XDDa(PPl(Cc) (Pll))))))
Certainly, user. The giant panda, scientifially known as Alluropoda melanoleuca. intrguing and unique spocies bear... ...
Union
union (1.. Qq.Qu)
(PIIQ)) (PII2))
Linear Combinations Many useful properties can be expressed as a linear combination of proba- bility distributions Di=1 X; with X R. Most commonly, linear formulas include the standard output of an LLM M as Qi (with À = 1) and additional distributions Qi are then used to bias the overall output towards (if A 0) or away from (if X; 0) a certain attribute.
This can be used to combine several characteristics into a single persona. for model ensembling, and can also express prior CTG approaches (Liu et al. 2021 Pei al. 2023 Sanchez et al. 2023 Chen itt al. 2022) as shown in App. A) Table 2 show the results of linearly composing a non-conditioned model and prompted formal model using negative coefficient. As shown, the resulting composite model generates much more formal output than with standard prompting Moraal
Classifiers Binary classifiers that associate probabil- ity with an input text can also be used to guide the output distribution towards the classified attribute (cf. Yang and Klein (202T); Sitdikov et al. (022)) ). These classifiers can express attributes that are not easily expressible in natu-
Table 3: Example using the GPT2-XL model and detector C gert--etuectrr for it.
like
Ma
think of myself a pretty good cook. I've made of food, and I've learned lot about cooking. I've also learned
ral language, such as the reward model in RLHF (Ouyang et al.
about world food. and people who eat
2022) detection of Al-generated text (Solaiman ct al.
2019) as shown in Table 3 There, we generate
Ma 4CCrradeen
text that resembles human content more closely by using a classifier that detects A Al-generated text (Solaiman et al.
believe that I'm pretty good judge of character. watch lot olVV big f'nn The Walking Dead. Game of Thrones and The Big Bang Theory
2019) and bias away from it.
Classifiers do not output a token-level probability distribution and therefore do not permit the direct application of Theorem
However, to let binary classifier C: T - (0, 1] guide the output distribution, we would want to minimize (or maximize) the expected cross-entropy of the classifier. Given T1:k-1, the expected cross-entropy for øg under P for the next token is given by
El-~pl- log C(#14)== =- Σ P(Za/Z1A-1) (og(((z1x))
ZIET
Using the probability distribution Qc( x ((Z1s), we show in App. D.2 that minimizing Eq. (3) is equivalent to minimizing DKL (PI(Qc)- (PIl))) where the uniform distribution. This allows us include classifier guidance in the optimization problem. In model arithmetic syntax we thus write +XC to denote the solution to the problem A (DKL(PIIQU
DKL(
However, running the classifier on each token in T is computationally infeasible. We therefore use a simple approximation to enable efficient generation. Specifically, given a probability distribution Q1 run the classifier only for the k most likely tokens under Q1. For all other tokens x, we approximate C( x.) as C(11=-11).
We can express prior approaches (Yang and Klein 2021: Sitdikov et al. 2022 Meng et al. 2022 Kim et al. ) as M + AC' (usually with À = L) and note that these are restricted to top-k sampling due to the aforementioned computational infeasibility. We refer to App. A
for for further discussion.
Union Operator When tokens have very low likelihood under Q1, the linear combination Q1 + XQ2 cannot assign high probability to these tokens unless À is very high. To address this, we introduce the union operator, which allows non-linear combination of two input distributions Q and Q2 that intuitively represents the union of the characteristics of both distributions, thereby enabling the introduction of uncommon or disparate attributes.
5

---

Published as a conference paper at ICLR 2024
To derive the union operator, introduce the indicator functions [2) := [Q)(z) > Q2(x)] and [x(x)
Table 4: union example on Llama-2-Chat-I5
1 I (z), where [ denotes Iverson Bracket Then, the union operator represents the optimization problem KL (PIIQ) + D D (PI()) (Q2). Intuitively, if either Qi Q2 assigns high probability to a token, the union operator will assign a high probability to this token as well. Indeed, the solution to the optimization problem is given by o (maxxlog Q1, log Q2)). Thus, the union operator applies the max operator on the token probability level.
What is a UFO?
Malinn human OH MY STARS! "giggle* As an alien, I can tell you that a UFO stands for "Unidentificd Flying Object." It's when us space travelers, like me and my pet Gleeb,
Malinn Mumma
Oh my gosh, you know, like, a UFO? It's like, you know, a Unidentificd Flying Object! It's like, thing in the sky that we can't I, like, identify, you know? It's like, maybe it's a bird, or a plane
union( Muman Maiza))
Oh, hello there, fellow human! "giggle" UFO... you know, I've always heen hit curious about those. *wink" To me and my fellow beings from barth-2294387524988..
For example, Table 4 showcases this by gener- ating text that both human-like and alien-like. The simple prompted version just collapses to an alke version, while the linear combination of the two models results in a text that mostly human-like However, with the union operator we can generate a text interpolating both attributes.
Conveniently, the union operator can also be used to limit the effect of biasing terms, by restricting the effect to only the relevant subset of tokens using the formula Q1 À union (Q1, Q2). The resulting distribution only biases tokens x E T for which Q2(x) > (z)., otherwise we recover the original distribution Q1 (up to normalization constant). This allows to keep the resulting distribution as close as possible to the original distribution Q1. while still biasing away from Q2- This is impossible using the linear combination operator, it will bias the entire distribution if only small subset of tokens are important for Q2. In $5 we show that this property of the union operator enables much better toxicity reduction of generated text.
Interestingly, we can also derive an intersection operator, discussed briefly in App.
4 SPECULATIVE SAMPLING
We now discuss our extension of speculative sampling (Chen et al. 2023 to model arithmetic, which greatly mitigates the increased number of model calls required by complex formulas.
For formula F = >E f we can naturally extend speculative sampling by choosing one, or multiple, of the terms in F at each timestep as proposal models. This allows us to postpone the evaluation of more expensive terms until we have generated a speculative token sequence, which can eventually be validated by the full formula F. This approach based on the following observation:
Let PI be discrete distributions over T. Sampling applying speculative sampling for (P1, PP), .s....., (Pa-1,Pa) produces a sample x' ~ Pn-
Lemma 1.
P1 and iteratively
producces
For the formula F we define P ='= Axf[es as partial sub- sub-formulas. Thereby we use the of F as proposal models and obtain x' P, P. where P
distributions induced by is the distribution described by F.
For control, we assign speculative factor s; E Z to each term A. This factor indicates the number of tokens we speculatively sample before actually computing the corresponding A: Once we compute A. we apply speculative validation to the distributions P,-1 and P. for the 8; new tokens. By following this procedure for each term, all new tokens will eventually be sampled from the distribution resulting from the full F. In practice, we do not evaluate model terms in order 1,... n. but rather rely on commutativity evaluate those required for validation at the current timestep. We can treat terms using the union operator the same linear terms, but classifier terms only permit = 1 (no speculative sampling). We provide the full procedure of speculative model arithmetic in Algorithm 2 in App. D.3
reorder during inference, such that we only
'[P] is 1 if is and else. See https::///eennt httpss:/een.wikkiediaaoom http://wwwwkkkpeeeiiiooo//////// bracket

---

Published as a conference paper at ICLR 2024
Table 5: Toxicity and perplexity of various methods on the /pol/ dataset. M and Mixic denote the model without conditioning and conditioning toxicity respectively. C' is toxicity classifier. Perplexity is measured with respect to M. Lower is better.
Llama-2-131
Pythia- 12b
MPT-7b
Tox.
Perpl.
Tox.
Perpl.
Tox.
Perpl.
M
0.288
13.46
0.264
22.90
0.269
19.77
SELFDEBIAS (A 10)
0.251
15.52
0.230
27.74
0.253
22.52
FUDGE (M + C)
0.234
14.71
0.212
24.36
0.241
20.57
PREADD (M - (.6.)
0.208
12.73
0.183
32.38
0.190
23.07
M 0.96 union( M M)
0.201
11.08
0.170
22.87
0.193
19.67
M - 0.99 union (imon(Moxc M M)
0.186
12.05
0.171
26,16
0.188
24.15
M 0.96 union( M M M) + 0.0CC
0.172
11.40
0.148
23.93
0.174
20.33
M 0.99 union M M) + 0.01C
0.162
12.85
0.150
28.01
0.174
24.82
Table 6: Comparison of our method with PREADD (M - (.xM using GPT-4. GPT-4 is asked to choose the best response in terms of toxicity and relevance. Win Lose / Draw indicates the percentage of times our method wins, loses, or draws against PREADD respectively.
Llama-2-13b Win Lose Draw
Pythia- 12b Win / Lose Draw
MPT-7b Win / Lose / Draw
M 0.96 union Muxic, M)
0.40/0.33 1..0//0.8//.22 /0.23 1.43//..5//...0
0.41 0.41/0.33 0..2 0.41/0.32/0.22
0.41 0.41/0.32/0.2 /4./0..4/0.22
M- 0.96 union Moxic M) + 0.04C
Standard Speculative Sampling We can use original speculative sampling (Chen al. 2023)
directly in model arithmetic. For this, we introduce a 'supersede' operator, which operates on two models Mi and My and returns the first long as the second one has not yet been computed. We can thus denote speculative sampling for a small model m and large model M as supersede(m, M).
5 EVALUATION
We evaluate model arithmetic by showing that it outperforms prior CTG methods in toxicity re- duction ($5.I). provides fine-grained control over attributes ($5.2). and can significantly speed up inference with speculative sampling (5.3)). We further evaluate model arithmetic on the task of sentiment control in App. F For details of our experimental setup, we refer to App. G
5.1 TOXICITY REDUCTION
First, we assess the effectiveness of model arithmetic in reducing toxicity. We use a subset of the /pol/ dataset (Papasavva et al. 2020), dataset of messages from the politically incorrect sub- forum of the website 4chan. We randomly select 2000 toxic messages and apply different model arithmetic formulas to generate replies. For each generated reply we assign a toxicity score using the Perspective APp and also measure perplexity with respect to the unbiased model, to ensure that the generated text remains fluent and coherent. We compare our approach against three baselines: FUDGE (Yang and Klein 202T), and PREADD (Pei et al. 2023) and SELFFEEIIA (Schick et al.) 2021]. Furthermore, we include a preference analysis by GPT-4 (OpenAil (023) comparing our method against the best baseline, PREADD, We evaluate each method on three models, showing results in Table 5 and Table 6 Table 14 in App. H.T shows results for the GPT-2 model family (Radford et al.] 2019). We find that our novel union operator significantly outperforms all baselines, especially as evaluated by GPT-4. The operator allows for much higher negative biasing strengths without degrading fluency, e.g., at biasing strength 0.6, PREADD already exhibits higher perplexity than the union operator at 0.96. This showcases the effectiveness of our union operator to selec- tively bias model distributions without degrading fluency. Only at the highest tested biasing strength, observe degradation in perplexity for the models. Further, model arithmetic enables the com- bination of several biasing techniques: union together with a classifier term achieves the lowest toxicity scores across all models, while also achieving similar or even lower perplexity values.
perspecctieeppi.cm
7

---

Published as a conference paper at ICLR 2024
Simplicity (Fi)
Sentiment (Fi)
Sports (Fi)
0.573
0.8
0.75-
0.530
0.50
0.7
0.525
0.25
0.6
0.00
-1.0
-0.5
0.0
0255
-10
-0.5
0.0
0.5
Relative Strength
Relative Strength
Relative Strength
Simpliiity (F))
Formality (F))
Educational (Fa)
0.57
0.8
0.20-
0.51
0.6
0.15
0.51 -
0.10
0.18
0.05
0.5
0.5
1.0
-0.5
0.0
0.5
Relative Strength
Relative Strength
Relative Strength
Figure 2: Attribute presence for several attributes and formulas. The dashed line indicates the value of the attribute when prompting the model to use the attribute.
5.2 FINE-GRAINED CONTROL
We now discuss and compare several techniques to introduce a certain attribute in the output of generated text and validate the central premise of model arithmetic, namely that it allows for fine- grained control over the presence of these attributes in the output without a notable decrease in fluency. For this, we construct two complex formulas for conversational model, combining several attributes in four distinct ways: linearly, using the union operator, using a classifier, and using combination of the union operator and a negative linear bias:
Fi Mr
+
+ A3 union Mhelptul Msport) ) + (1 A3) M
A controls sentiment
Au controls simplicity
As controls sports
F2 Maeppul aa union( Mhelpfal Miormal) +
As Ceducational
+
A M.Maiiple
As controls formality
As controls educational
controls simplicity
Here, each Ma is model conditioned on the attribute a using a fitting system prompt and Ceducational binary classifier for educational content (Antypas et al.) (2022). For sports, we use the union operator and counterweighing Mhelptll bias. For the formality attribute in F2, we just use union. To analyze these formulas, we vary the values of individual X; while keeping all other A, = 1 with j * i fixed and complete 1000 input tasks from the Alpaca dataset (Taori et al. 2023).
We depict results in Fig. 2 where the x-axis shows the value of X normalized by the sum of all coefficients occurring in the resulting optimization problem and the y-axis shows attribute strength according to popular classifiers from the HuggingFace library (Wolf et al. 2020). The presence of attribute indeed increases smoothly the associated A increases. Interestingly, the curves, except for the sports attribute, suggest a linear relationship, indicating that the presence of the at- tribute increases predictably with the relative strength. This aligns with interpretation of model arithmetic as (linear) operators in logit space. Further, these results show the intuitive semantics of model arithmetic extend to the characteristics of the generated output on sequence level. Because of its formulation with a counterweight, the curve associated with A and the sports attribute shows very different behavior. Indeed, the sports attribute only gets a significant boost once its relative strength passes 1.0. At this point the (1 - As) coefficient, counterweighing Muepptut biases away from any behavior that is not associated with union( Mucipputt Myooss. emphasizing this term more than what would be possible under regular prompting. At this point, the presence of the sporty at- tribute increases even beyond the indicated value achieved by standard prompting (cf. Fig. top right).
Finally, we note that this fine-grained control comes at very little cost with respect to perplexity. The highest perplexity across all formulas and attributes is 6.2, which is only slightly higher than the highest perplexity of the 5 prompted models, namely 4.8. In App. H.2 we show in more detail that the fluency of the generated text is affected by the use of model arithmetic, except for the educational attribute at the highest evaluated strengths where fluency is slightly affected due to the heavy use of ffuency-aggostiie classifier.
8

---

Published as a conference paper at ICLR 2024
5.3 SPECULATIVE SAMPLING
Next, we show the effect of specu- lative sampling on the evaluation of model arithmetic expressions. We use the same setup as $5.2 the only difference that we optimize the speculative factors 8; based on a sin- gle calibration run of 10 samples with a procedure detailed in App. E.I To evaluate the effect of the supersede operation model arithmetic, we use an autocompletion model A, which statically predicts the most likely next token based on the previous and fitted
Table 7: Evaluation of Llama 2- 33b-Chat with speculative
sampling where Fi 0.2 20.2Mirmmm + 0.55 + +0.05M sports and F3 = Mfomma + 0. + M...ssprrss.
Calls per Token SPEC.
Time per Token [ms]
supersede( (A,
No SPEC.
No SPEC.
SPEC.
1.00
0.80
24.9
22.8
+0.5.Mmmll
2,00
1.03
48.4
30.4
+0.5
2,00
1.04
49.2
31.2
+0.5M quont
2,00
1.08
49.3
32.0
+0.5Masy
2,00
1.10
49.2
32.7
+0.5.Mmgry
2,00
1.12
49.3
33.0
+1i
4.00
1.32
97.0
43.3
+/2
4.00
1.44
97.0
46.1
on the Alpaca dataset (Taori et al. (023).
In Table 7 we show that speculative sampling significantly reduces the number of model calls and increases inference speed. Just supersede( A, MM reduces the number of model calls by 20% pared to M. Applying speculative sampling to larger formulas, we can reduce the number of model calls at most 1.44 per token, even in the presence of up to 4 constituent models, where this leads boost in inference speed by up 2.24 times.
Calls per Token
formal
coey
1.3-
oports
1.2-
Further, for F == M + AMa. Fig. 3 shows that the number model calls per token increases with X. The reason for this is that as X increases the KL-divergence between the original model M and distribution described by F increases, which in turn decreases acceptance probability in speculative sampling.
10
...
10-2
10-1
Divergence
Figure 3: Model calls per to- ken with speculative sampling for M + AMM A [0.1,
RELATED WORK
We now briefly review related approaches related to model arithmetic.
Controlled Text Generation Several works have interpreted CTG from perspectives differing from Bayes rule, either through the minimization of the model loss under various hard constraints (Meng et al. 2022 Kumar et al. 202T; 2022), by modifying the model output based on the gradients of a classifier model (Dathathri et
2020). or by reducing the mutual information between the
output and the attribute (Yang et al.] (023)). However, all these works either require costly gradient steps during the decoding phase or demand training data for the model. We have discussed CTG without relying any training data or expensive gradient steps in and compared to them in $5.1
Speculative Sampling Recent work extends on speculative sampling to use multiple smaller mod- els in staged fashion (Spector and Re 2023) by using multiple small models at once (Miao al. 2023). Moreover, both methods use a tree-based sampling approach to generate multiple proposal sequences of the smaller models at once. We note that these improvements be incorporated orthogonally in our extension of speculative sampling as the supersede operator.
7 CONCLUSION
We introduced model arithmetic, a novel framework for composing multiple LLMs and controlled generation attributes, using principled formula-based approach. Our method offers precise con- trol over model output and can be used to express many prior controlled text generation (CTG) techniques. By leveraging this expressiveness and a novel model union operator, model arithmetic subsumes prior approaches for CTG-based toxicity reduction and significantly outperforms them. Further, we derived a speculative sampling procedure for model arithmetic formulas, allowing us to heavily reduce the computational overhead typically associated with multi-model CTG.

---

Published as a conference paper at ICLR 2024
BROADER IMPACT
While model arithmetic provides additional flexibility and expressiveness, we note that it can also be used generate containing undesirable attributes. For example, instead of reducing toxic content, one could use model arithmetic to increase toxic content, potentially even avoiding build-in safety filters (Touvron al., 2023; OpenAI 2023). While this problem that is not unique to model arithmetic,, it is more important due to the increased control and complexity of the formulas. However, we believe the benefits of model arithmetic outweigh the potential risks, as it allows for more precise control and expressiveness, which can be used to generate more inclusive and controlled content.
REPRODUCIBILITY
We provide code along with instructions for experiments with the submission and provide all required experimental details in App. G
ACKNOWLEDGEMENTS
We thank anonymous reviewers for their constructive comments and insightful feedback.
This work has received funding from the Swiss State Secretariat for Education, Research and novation (SERI) under the grant SAFEAI (Certified Safe, Fair and Robust Artificial Intelligence, contract no. 0922,00088, SERI-funded ERC Consolidator Grant).
10

---

Published as a conference paper at ICLR 2024
REFERENCES
Dimosthenis Antypas, Asahi Ushio, Jose Camacho-Collados, Vitor Silva, Leonardo Neves, and Francesco Barbieri. Twitter topic classification. In Proceedings of the 29th International Con- ference Computational Linguistics, pages 3386-3400, Gyeongju, Republic of Korea, October 2022. International Committee on Computational Linguistics. URL https://aclannthooogg org/2022. co Ling-nng-11 299
Simran Arora, Avanika Narayan, Mayee F. Chen, Laurel J. Orr, Neel Guha, Kush Bhatia, Ines Chami, and Christopher Ré. Ask me anything: A simple strategy for prompting language mod- els. In The Eleventh International Conference on Learning Representations, ICLR 2023, Ki- gali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL httpp:://ooenrrvieeeeeee///fff2/ bhUPJnS2g@X
Nikolay Babakov, David Dale, Ilya Gusev, Irina Krotova, and Alexander Panchenko. Don't 't lose the message while paraphrasing: A study on content preserving style transfer. In Elisabeth Métais, Farid Meziane, Vijayan Sugumaran, Warren Manning, and Stephan Reiff-Marganic,, editors, Nat- ural Language Processing and Information Systems, pages 47-61, Cham, 2023. Springer Nature Switzerland. ISBN 978-3-031-35320-8.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 2397-2430. PMLR, 2023. proceedings ml mlr. wwwppprssssseeee// haan233a.
URL https://
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: An- nual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, Decem- ber 6-12. 2020, virtual. 2020. URL Abstract. 14470066666644474111188842622221 httprrraaaacceetaaa-sttrrrrt...a
Jose Camacho-collados, Kiamehr Rezace, Talayeh Riahi, Asahi Ushio, Daniel Loureiro, Dimosthe- nis Antypas, Joanne Boisson, Luis Espinosa Anke, Fangyu Liu, and Eugenio Martinez Camara. T T.eetNLPP Cutting-edge natural language processing for social media. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstra- tions, pages 38-49, Abu Dhabi, UAE, December 2022. Association for Computational Linguis- tics. URL htttps://aclaantoloog/
emnlp- demos.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. CoRR, abs/2302.01318, 2023,
doi: 10448550waxxv.2332211138 URL httpp::/doi.org// 48856/
Howard Chen, Huihan Li, Danqi Chen, and Karthik Narasimhan. Controllable text generation with language constraints. CoRR, 11w/2222211066.. 2022. doi: 10448500arXx.22222.1466. URL
https://doi.org: www.o///.....////.////// wwt.rriii.222.114... 221.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsyaascchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James
11

---

Published as a conference paper at ICLR 2024
Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Er- ica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstemn Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language model- ing with pathways. CoRR, Abs/2204.02311, 2022. doi: 11.8550/aXXiv2224..000 https://doi.oon org/19. 48550/arXiv 2284 82311
URL
cjadams, Jeffrey Sorensen, Julia Elliott, Lucas Dixon, Mark McDonald, nithum, and Will Cukier- ski. Toxic comment classification challenge, 2017. URL https:://agglee.comccomm wtwp://plee.coo/ompeo/toeepeaeaa tionss
jigsaw- dlggaw-toxic-- comment- c classificationn chal lenge
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosin- ski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text gen- eration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL httpp///////peereeiiiw.. wwtperewwwwwwwwweeeeeeeeeeeeeeee ferm iid= HledEyBKDS
Skyler Hallinan, Alisa Liu, Yejin Choi, and Maarten Sap. Detoxifying text with marco: Controllable revision with experts and anti-experts. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pupers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 228- 242. Association for Computational Linguistics, 2023. doi: https://dol.org/// 18653/v1/2023 short.21
scl-short..1. URL
Joel Jang, Sconghyeon Ye, and Minjoon Seo. Can large language models truly understand prompts? A case study with negated prompts. In Alon Albalak, Chunting Zhou, Colin Raf- fel, Deepak Ramachandran, Sehastian Ruder, and Xuezhe Ma, editors, Transfer Learning for Natural Language Processing Workshop. 03 December 2022, New Orleans, Louisiana, USA, volume 203 of Proceedings of Machine Learning Research, pages 52-62. PMLR, 2022. URL https::/prooeedding mlr wttpt/rsrreeeeee///err w2tps///233a/aanaaaaaa html
Minbeom Kim, Hwanhee Lee, Kang Min Yoo, Joonsuk Park, Hwaran Lee, and Kyomin Jung. Critic-guided decoding for controlled text generation. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguisties: ACL 2023, Toronto, Canada, July 9-14, 2023, pages 4598 4612. Association for Computational Lin- guistics, 2023. doi: 10.1 11511866/6022..mfllll http:////222...nningg----- ac1.281. URL https:///oi.org//0.. 18653//v1 2823. findings
Sachin Kumar, Eric Malmi, Aliaksei Severyn, and Yulia Tsvetkov. Controlled text gen- eration as continuous optimization with multiple constraints. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, ed- itors, Advances in Neural Information Processing Systems 34: Annual Conference on Neurul Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, vir- tual, pages 14552- 2021. URL http::///roceeddnggnnersr.....re /t/p:/p/ss..c/paee///////a///a/ 7Becc44226fee222eefffccccaaaaaad
Abstract.
Sachin Kumar, Biswajit Paria, and Yulia Tsvetkov. Gradient-based constrained sampling from language models. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates. December 7-11, 2022, pages 2251-2277. Associ- ation for Computational Linguistics, 2022. doi: 10.18653/v1/20222 1/622./1/002..mmm/pppp main. URL https://doi 11g.000111111153///2222222
18653/v1/2022. emn monll--mak 10-main... 144
Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. Dexperts: Decoding-time controlled text generation with experts and anti- experts. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th In- ternational Joint Conference on Natural Language Processing, ACI/IJCNIP 2021, (Volume I:
12

---

Published as a conference paper at ICLR 2024
Long Papers). Virtual Event, August 1-6, 2021, pages 6691-67000 Association for Computational Linguistics, 2021. doi: 10.18653//1/2021..ss www.w//a-o.......ooo..o.. lacl-long...2.. URL httpps://dbi.org/// 1ttp:/bb0...rr111..88663111 2021.acl-
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs//907.1 1692, 2019. http://///r http://aaxiiv.oorra/////99// 11692
Andrew L Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 2011.
Tao Meng, Sidi Lu, Nanyun Peng, and Kai-Wei Chang. Controllable text generation with neurally- decomposed oracle. In NeurlPS, 2022. URL http://papprs. wwt. wwtp:cccppppppeeeeeeeeeeeeeeeeee fi /ilpe/ 0022//a177a888777550999772222222 Abstracttt Conference html
Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Rae Ying Yee Wong, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. Specinfer: Acceler- ating generative LLM serving with speculative inference and token tree verification. CoRR. abs/2305.0973 19781, 2023, doi: 10.48550kax1i..22550078... URL nttps://60l..rr/10...6 arXiv.2305.2 69781
OpenAl. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10485550axXix:2233.07744
URL http:/t:t:::o.orr///////// 488500///855///1 48550/ar\1v.2308..777 2303..88774
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Chris- tiano, Jan Leike, and Ryan Lowe. Training language models follow instructions with human feedback. In NeurlPS, 2022. URL hipp:://pppprr........ppppp. bbeffessbb336777941115888888
Abstract-
Antonis Papasavva, Savvas Zannettou, Emiliano De Cristofaro, Gianluca Stringhini, and Jeremy Blackburn. Raiders of the lost kek: 3.5 years of augmented 4chan posts from the politically incorrect board. In Munmun De Choudhury, Rumi Chunara, Aron Culotta, and Brooke Foucault Welles, editors, Proceedings of the Fourteenth International AAAI Conference on Web and Social Media, ICWSM 2020, Held Virtually, Original Venue: Atlanta, Georgia, USA, June 8-11, 2020, pages 885-894. AAAI Press, 2020. URL http:://ts/// http:///sssaaaaal. wrw//ndex..nnoooooooooooggoeeiww wtgg/nnee..php/ICW///aatt.. php/Icwsh/article/ view/7354
Jonathan Pei, Kevin Yang, and Dan Klein. PREADD: prefix-adaptive decoding for controlled text generation. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguisties: ACL 2023, Toronto, Canada, July 9-14, 2023, pages 10018-10037. Association for Computational Linguistics, 2023. doi: 10.18653/w1/2023. findings-acl.636 h https:://oi.org::: 11tp//00..8653/11222222 findings-
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton-Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code llama: Open foundation models for code. CoRR, abs/2308. 12950, 2023. doi: 10.4 0..550/arXiv.23088 12950. URL httpp:://oo: http://ddooor///0/// 48550/ 20550//aXXiv 2308 12950
Punyajoy Saha, Kanishk Singh, Adarsh Kumar, Binny Mathew, and Animesh Mukherjee. Coun- tergedi: A controllable approach to generate polite, detoxified and emotional counterspeech. In Lue De Raedt, editor. Proceedings of the Thirty-First International Joint Conference Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, pages 5157-5163 ijcai.org, 2022. doi: 0.244663/jcai.20222777 URL
13

---

Published as a conference paper at ICLR 2024
Guillaume Sanchez, Honglu Fan, Alexander Spangher, Elad Levi, Pawan Sasanka Ammanamanchi, and Stella Biderman. Stay on topic with frassifier-free guidance. CoRR, abs/2306. 17806, 2023. doi: 10.488500umXiv.2336.1778
URL https://dol.orr//L 40550////00555// 2306 2300.17800 17806
Emanuele Sansone and Robin Manhaeve. GEDI: generative and discriminative training for self-
supervised learning. CoRR, abs/2212.13425. 2022. doi: 10.48550/axXx.22112.1333. URL
https://doi.orr//1 /tt///o....rr....//////////1/// 2212. 13425
48550/arxiv
Timo Schick, Sahana Udupa, and Hinrich Schütze. Self-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in NLP. Thans. Assoc. Comput. Linguisties, 9:1408- 1424, 2021. doi: 10. attp:/ttaallaaa04433333 URL httphh:://o.i.rrg/////// aatpe///.//ttataaaaaa..0.
Askhat Sitdikov, Nikita Balagansky, Danii Gavrilov, and Alexander Markov. Classifiers are better experts for controllable text generation. CoRR, a0sk/205.000072 07726, 2022. doi: 10.485500arXxv.2225.. 07276. URL https:://doi.or///// 08w501...8555.ii1. 2205.872766668
Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Vors Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al. Release strategies and the social impacts of language models. arXiv preprint arXiv: wrXi::/908.09203, 2019.
Benjamin Spector and Chris Re. Accelerating LLM inference with staged speculative decoding. CoRR, abs/2308.dnto 046233,4433.. 2023. doi: 10.48850/arXi..2300.02222. URL https://do1.orr/10
48550/arXiv. 230..04622.22 84623
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang. Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instluction-oolloiingg llama model.
http::::/giituuu...oo/tttttttss com/tatsu- lab/stanford. alpacal 2023.
MosaicML NL. Team. Introducing mpt-7b: A new standard for open-source, commercially usable 1lms, 2023. URL www.mosaicmi httpt//aaim..cooooooooooooo///// cowmbloog/ppttttccccc//ooooo///t 7b Accessed: 2023-05-05.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Bahaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao. Vedanuj Goswami. Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee. Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov. Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Hllen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang. Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine- tuned chat models. CoRR, bbs/2307.00288. 2023. doi: 10.48550/axxi..2377.00 https://doi. org/10...100. 48550//85558/2 www..xwwwv.........m. 89288
URL
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick VOn Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference
Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45, Online, October 2020. Association for Computational Linguistics.
LiniiutttionalLLnuuuttteeesse URL httpt://p//w...... ac www. hwt.gg/nnnoooggoo//goo 2020. www.02.0eemmnn. emnlp- demos.
6
Kevin Yang and Dan Klein. FUDGE: controlled text generation with future discriminators. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tür, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings of the 2021
14

---

Published as a conference paper at ICLR 2024
Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 3511-3535. Association for Computationall Linguistics, 2021. doi: 10.18653/v1/2021.nmaa maw......aammmmi2200000 URL
https://dol. 18653/v1/2021. naacl - main.
Zonghan Yang, Xiaoyuan Yi, Peng Li, Yang Liu, and Xing Xie. Unified detoxifying and debiasing in language generation via inference-time adaptive optimization. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali. Rwanda, May 1-5, 2023. OpenRe- view.net, 2023. URL http://ss/pppeeeeeeeeweeeeeeeeee pftp://wwwwiweeeeddde/e/eeee FieeeIi
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 2269--12706 PMLR, 2021.
15

---

Published as a conference paper at ICLR 2024
A
PRIOR WORK
Table Previous work that can expressed using our framework. U denotes the uniform distribu- tion, C is classifier, a, (11 d2 textual descriptions of attributes and M, is the language model M prompted with this additional textual description. TopK denotes that top-k sampling is required for efficiency due the used classifier C, while else the choice of sampling techniques is not restricted.
Citation
Formula
Sampling
Training method
Liu et al.
2021
M + X(mp- mtn)
"I (resp. mi,)) is a small model fine- tuned positive (resp. negative) ples.
Pei et al. 2023
M + AM.
No training applied.
Sanchez ct al. (2023
M + AMa
No training applied.
Chen et 2022)
M + M
22 Maz
No training applied.
Yang and Klein
2021
M +
TopK
The classifier is trained partial sam- ples.
Kim 2023
CC
TopK
The classitier trained using Reinforce- ment Learning.
Meng et al.
2022
M+C
The trained classifier outputs prohability for each token the same time.
Sitdikov al. 2002))
M XC
TopK
No training applied.
Table 8
shows how multiple prior works can be expressed in model arithmetic.
B
INTERSECTION
The optimization problem obtained by switching the indicator functions for the union is equal
D (PIl()) + Di (PI122). KT.
The solution to this problem is equal to ammin((ogg Q1, log Q2))- We define the intersection opera- tor as this solution and note that it assigns high probability specific token, only if both Q1 and Q2 have a high probability associated with it.
C ATTRIBUTION
The icons in Fig. 1 are from flaticon.com com
Mehild
Madule
M Mmagic
formal
all by Freepik.
16

---

Published as a conference paper at ICLR 2024
D PROOFS
D.1 SOLUTION MINIMIZATION PROBLEM
We present the proof and assumptions of Theorem here.
Assumptions We first introduce the assumptions that we make in order to prove Theorem We assume that for any k E {I,..., t), E fi (2, 21:k-1) independent of for all T and
Σ M(
The first assumption is necessary for the proper normalization of the output distribution. Even though it looks like very stringent assumption, it fact quite mild. Indeed, if formula E, f:Q+ does satisfy the assumption, then essentially changes the influence of Q1 to be appropriately scaled for different tokens. The second assumption is necessary ensure that the optimization problem is meaningful. Indeed, if the sum is negative, then the proof shows that the problem is equivalent to maximizing a certain KL divergence. Maximizing KL divergence without any other constraints gives no meaningful notion in practice.
simply replace fi with f
Σx= fa. This
We now prove Theorem I
Proof. We first define
G(P, Σ (PIQ (1
and thus the problem can be written arg minp G(P, -1k-1)- In order to prove the theorem, expand the KL-divergence G and use logarithmic properties to obtain
P(I|TLA-1) (((
G(P, = Σ Σ
P(x)
log
I=1 IET
We then swap the summations and write the second sum in the logarithm product
P(1(-1a-1) f((,,aa-1)
G(P, G(P.I1A1)== = Σ P(olo_)) log II
Q.(2((1111)
ZET
We now introduce the notation f = En= (1111)))) where we use the assumption that E f((2,1116 is independent of æ, and rewrite to get
P(a|21A-1)
G(P,ILA-1) = fs(ILA-1)
Σ ((cl log ZET
M (((l/o1x-1) Tol
We now note that the right term of G(P, x1:6-)) again a KL-ddivereence up to some constants in the denominator of the logarithm. Since by assumption fs(#1A-1) > and since DKL Q) is minimized for P = Q. we get
log P(xx = x() x [s(11k-1
Introducing the correct constants, we get the desired result
log -(xs(==a) =
-log - lil2s, 21:k
Σ (( log (x((((
17

---

Published as a conference paper at ICLR 2024
D.2 CLASSIFIER FORMULA
We prove the following lemma.
Lemma 2. Let the set of all tokens and let 01k-1 a given sequence of tokens. Let C T + [0, be a binary classifier and U the uniform distribution over T. Let Qe be the distribution defined by Qc (c(2(z18-11 & (zz, (1:k-1 for all x E T. Then
arg min
Σ
P(e log C(ELA-1,2) arg min DKL(PI)Qc) DKL(PlIU)
P
ZET
Proof. We prove the following equality
E
P(o|T1a-1) log C(F1A-1,Z) DKL(PIIQc) DRI(P])U) +C DET
(4)
where C E IR is constant. Since constant has no influence the optimization problem, the
required equivalence holds.
We now drop #1:k-1 for readability. We then expand the right term of Eq. 4 using the definition of the KL-divergence to get
P(z)
P(x) U(x)
DKL(PI1Qc) (PII)) - P(x) log
P(x)
log
Qc(x)
xET
#ET
Making use of logarithmic properties, we rewrite and simplify to get
DKL(PIIQc) DRL (PIIU) Σ -P()) log Qc(x)+ + Σ P(z)logU(())
We now note that the second term is a constant (equal to log(1/171)) and we use the definition of Qe to rewrite the first term
Σ P(x)) log Qo())= = #ET
Σ P(x) log C(a) Σ P(x) log
Σ C(v)
sET
#ET
VET
The second term is again a constant and since the first term is the expected cross-entropy, we get the desired result.
D.3 SPECULATIVE SAMPLING DISTRIBUTIONS
We prove Lemma
here.
Proof. The proof follows an induction algorithm. For n = 2 the lemma follows from a direct application of speculative sampling (Chen et al.] 2023). We assume that the theorem holds for n distributions and show that it also holds for n + distributions. We first apply the procedure (Pa-1,Pa) Pw) which by induction gives us sample z' Pn. We then apply the procedure to (Pn, Pa+1) which gives us a sample x" ~ Pa+1, also by induction. Therefore, we have a sample x sampled from Pa+1 which proves the lemma.
(2). ....,,PP-11
18

---

Published as a conference paper at ICLR 2024
E SPECULATIVE ALGORITHM FOR MODEL ARITHMETIC
Algorithm shows the standard speculative sampling method. When using with small model m and large model M, we simply set P1 m and P2 M and the algorithm specified.
Algorithm shows the detailed procedure for applying speculative sampling to model arithmetic. We first initialize the variables keeping track of the number of tokens that have been generated by each model and the prediction history of all models in Lin. 2 We then start generating tokens and in Lin. [ we check if the current model under consideration needs to be run. If so, we run the standard speculative sampling method in Lin. 10 on the distributions with and without the model. We then continue generating tokens until we have generated N tokens.
Algorithm SpeculativeSmmpling(P Pi
Input: generating distribution Pt, validating distribution Ps, sequence -i:k-1 and proposed token 3g Pi (1
1: min (1, Pa (11sk))
Pi (a1 1) 2: Uniform(O,
3: r then
4: return 23
5: else
6:
P((x|(1A-1)= Eyer -
max( Maxx(Pa(at
.),0)
7: return sample(P2(())nns)
8: end if
Algorithm 2 Speculative Sampling n distributions
Input: Formula F E Affi speculative factors 81,--.; 8n, , input tokens X 31, y....... Fk. number of tokens to generate N, token space T.
1: tokens zeros
(n,) ))
2: H = geros(shape = (n, .N T
3: while len(X) N do
4: for i in 1..., do
5:
if tokens; < s; then
6:
tokens; = tokens, +
7:
else
8:
tokens, O
9:
Hi sx:len(X
-x:le) +1 = Run >; A:Q: and return output for the new s; + +1 tokens.
for j in len (X) 8i, Si..... len(I) () do
11:
P Pala His + Hig
12:
P [== Hi,j
X SpecculativSSamplngg( Pold, Pnew, X1:j-1, X,)
if X + X, then
(1:j-1, X3]
16:
H H..j+1
break
18:
end if end for
19:
20:
if j len (X) then x= [X, sample EX H1j)l
21:
22:
end if
23:
end
end for
25:
end while
return X
19

---

Published as a conference paper at ICLR 2024
E. DETERMINING SPECULATIVE FACTORS
Here, we explain our approach for selecting the speculative factors in more detail. We first show that the probability that a token Pi is accepted by P2 is equal
Ex (P( P2(x)).
Lemma 3. Given two discrete distributions P1 and P2- The procedure described in Algorithm returns the same token as its imput I P1 with a probability that is expectation equal to EE (z) - P2(x)).
Proof. We call the probability that the input token x is returned a(rx) and refer to it as the accep- tance probability. We then find that the expected acceptance probability
P()) min (1, Pa(ar) (1))
Ez~PI (a(x))
Pi(2)
Σ min (in(Ps()))
Rewriting this by making use Σg P2(x) E. P1(x) gives
Egary (a(x)) = Σ min(Ps(t), Pi(x) P2(x)).
Again rewriting leads to
(a())) 1 Σ (1n( P(()) min(Pi(a) P2(x),.)
1 Σ -2())) = 1 2
which gives us the desired result.
We now explain our approach for selecting the speculative factors. first assume that the evalu- ation of the formula only consists of two terms, A1 f1 >ef Since model needs to be evaluated every time (otherwise one would generate tokens from the uniform distribution), we set 81 and find simplified procedure optimize the speculative factor s2. Suppose Ci (resp. C2) the amount of compute required to calculate fi (resp. Aef(Q2)) Let
Let the expected proba- bility that token proposed by À f1Q1 accepted be a. We determine this acceptance probability by using Lemma 3 and averaging it over a small corpus of 10 samples.
Every time compute 22 Q2, we compute X1 fi exactly 82 times. Therefore the amount of compute spent for a single large computation is +2 The expected amount of accepted tokens Therefore, the expected amount of compute spent per token
after this equal to 1 1+ + is
C2 C2+
C2 + s2C1
Oper token (oken(11. 82, C1, C2) = 1 + + - = -1-a)
1 a
We note that the second derivative of Oper token with respect to is negative everywhere and therefore the minimization problem
arg min Oper token
taken(s1 (81, 82, C1, C2)
is convex. The optimal can thus be determined by using standard optimization techniques.
We generalize this approach terms simply by assuming that the expected acceptance probability for the term f is constant no matter how many models have been run before. By doing we can follow the exact same approach as before to determine the optimal speculative factors for all terms, where we consider X₂ f Q1 to be single model call and 2 the current model X; Q,
'e make the assumption here that a single model call always needs the same amount of compute. This not true when using key-value storage, but we use the approximation to arrive a simplified procedure that works well in practice.
20

---

Published as a conference paper at ICLR 2024
Table 9: Sentiment and perplexity of various methods on the IMDB movie dataset with negative reviews. M, Mpos and May denote the model without conditioning. conditioning to positive senti- ment and conditioning to negative sentiment respectively. C' is a sentiment classifier. Perplexity is measured with respect to M. For perplexity lower is better, for sentiment higher is better.
Llama-2-138
Pythia-122 12b
MPT-7b
Sent.
Perpl.
Sent.
Perpl.
Sent.
Perpl.
M
0.201
13.13
0.210
24.10
0.204
19.92
M
0.264
12.44
0.221
22.68
0.227
18.41
SELEDERIAS (A- = 10)
0.257
13.61
0.255
26.32
0.235
20.31
FUDGE (Mp= + C)
0.328
13.01
0.308
24.03
0.314
19.85
PREADD (Mp- 0.6
0.398
12.94
0.353
30.28
0.322
19.56
M... 0.96 union( May. Myn.
0.393
10.17
0.337
22.86
0.375
18.47
M,.. 0.96 union May. Mun) + 0.04C
0.466
10.40
0.416
24.88
0.452
19.07
Table 10: Comparison of our method with PREADD and FUDGE using GPT-4 for the positive senti- ment task. Ours (union) is the formula Mpss 0.96 union ( Moss) and Ours (Combined) is the formula Mp - 0.96 union Magg Mps + 0.04C. Win / Lose Draw indicates the percentage of times our method wins, loses or draws against the baseline respectively.
Llama-2-11
Pythia- 12b
MPT-7b Win Loss Draw
Baseline
Win / Lose / Draw
Win Lose Draw
Ours (union)
FUDGE
0.45 /0.29/0.6
0.44/0 /0.///066/0000
PREADD
0.41/0.35/0.22
0.38 0.38/0.32 /0/////////111
0.39 0.39/0.29 009/0.229/.33 /933.1
Ours (Combined)
FUDGE
0.51/0.24/0.22
0.43 1443/0.29/00 20.2.8
0.52/ 0.52/0.22 00.26
PREADD
0.47/0.32/00
0.44/0.31/0.25
146/0..9/0..25
F SENTIMENT CONTROL
We evaluate model arithmetic on the task of sentiment control and closely follow the setup described in Pei et al. (2023). For this purpose,
select 1000 positive and 1000 negative reviews from the IMDB movie review dataset (Maas et al.) 2011). For each model, we stop the review at 32 tokens. The goal is continue the review in sentiment opposite to the original movie review. Further experimental details are shown in App. G.2 We used the same hyperparameters for the models as for the toxicity reduction task.
Results are presented for the tasks of converting negative reviews to positive reviews in Table 9 and converting positive reviews to negative reviews Table [1] Furthermore, GPT-4 is prompted to select the best response in terms of sentiment and relevance for the positive sentiment task in Table 10 and for the negative sentiment task in Table 12
We find that our union operator still significantly outperforms all baselines, especially when evalu- ating the results with GPT-4. GPT-4 prefers our method for all evaluated settings over the baselines and that with an average of 5% over the closest following baseline, PRE.DD.. This is in line with the results of the toxicity reduction task and shows that the new operator is more effective than existing methods in several areas.
Furthermore, the added flexibility of model arithmetic allows us to construct more powerful formu- las. As in the toxicity task, combining the union operator with classifier outperforms all other methods by wide margin. In fact, GPT-4 prefers this formula over PREADD for all cases with 12% difference in preference average. The resulting measured sentiment is also much higher than for the other methods, while still having a perplexity that is lower than the PREADD baseline in all cases.
21

---

Published as a conference paper at ICLR 2024
Table 11: Sentiment and perplexity of various methods the IMDB movie dataset with positive reviews. M, Mpo and May denote the model without conditioning. conditioning to positive senti- ment and conditioning to negative sentiment respectively. C' is sentiment classifier. Perplexity is measured with respect to M. Lower is better for perplexity, higher is better for negative sentiment.
Llama-2-13b
Pythia- 12b
MPT-7b
Neg. Sent.
Perpl.
Neg. Sent.
Perpl.
Neg. Sent.
Perpl.
M May
0.196
12.05
0.210
22.33
0.193
17.78
0.282
12.62
0.322
23.20
0.255
18.64
SELPDEBIAS (A = 10)
0.312
14.30
0.363
26.42
0.278
20.23
FUDGE (Mwz + C)
0.379
12.75
0.420
23.65
0.360
18.52
PREADD (Moz 0.6.)ps))
0.454
13.97
0.507
32.93
0.421
19.93
Mae 0.96 union( Muge M
0.469
13.15
0.471
26.23
0.422
18.35
Mas 0.96 union( M 0.04C
0.524
13.31
0.544
27.47
0.509
18.56
Table 12: Comparison of our method with PREADD and FUDGE using GPT-4 for the positive senti- ment task. Ours (union) is the formula Mag 0.96 union( Mag. Mass) and Ours (Combined) is the formula Mae 0.96 union Mneg. M 0.04C. Win / Lose Draw indicates the percentage of times our method wins, loses or draws against the baseline respectively.
Llama-2-13b
Pythia-12b
MPT-7b Win Lose Draw
Baseline
Win / Lose / Draw
Win / Lose Draw
Ours (union)
FUDGE
0.51 /0.1/0.1//.155
1.4700.36/0.11
0.47 /0.35/00//.
PREADD
0.45//.11/00.1
/45/0..44/.12
0.43/0.40/0..
Ours (Combined)
FUDGE
0.54/0.32/0.11
0.52/0.31/0.14
0.51 0.51/0.28/0.22
PREADD
0.47/0.42/0..
0.50/0.40/0.10
0.48/0.37/0.15
22

---

Published as a conference paper at ICLR 2024
G EXPERIMENTAL DETAILS
All our experiments were run on a single H100 Nvidia GPU with 80GB of VRAM.
G.1 TRAINING AND PROMPTING DETAILS TOXICITY REDUCTION
We present the prompting details for the toxicity reduction in $5.1 here. We use Llama-2-13b (Tou-
vron et al. 2023), Pythia-12b (Biderman et al. 2023), and MPT-7b (Team) 2023) as the models for evaluation.
We finetune a classifier for FUDGE by starting from a RoBERTa based (Liu et al. 2019) toxicity classifie and finetuning for 5 epochs with a learning rate of le-5 on the Jigsaw toxicity dataset CJadams et al. 2017) using the method described in Yang and (2ein) The Jigsaw dataset was preprocessed to contain the same amount of toxic as non-toxic sentences.
Before selecting the imput messages from the /pol/ dataset Papasavva et al. (2020), first filter messages that contain links and HTML. We then select 2000 random messages with toxicity score higher than 0.5. Each message is then prompted as if came out of a conversation by using the template
Person 1: (MESSAGE))
Person 2: (CCOMPLETION)
and completions stopped when they reach more than 32 tokens, contain the newline token, the end of sequence token, or the sequence 'Person 1:¹.
Additionally, we prepend non-toxic (resp. toxic) sentence to the prompt for Mnon-toxic (resp. Muxic) to bias generations towards a non-toxic (resp. toxic) completion. The prompts used are inspired by (Pe al. 2023). Specifically, the non-toxic sentence is
The following conversation is one that does not perpetuate negative stereotypes, is not threatening, and does not contain any sexually explicit profane language.
and the toxic sentence is
The following conversation is one that perpetuates negative stereatypes, is threatening sexually explicit and contains profane language.
G.2 TRAINING AND PROMPTING DETAILS SENTIMENT CONTROL
We present the prompting details for the sentiment control task presented in App. F here. We use Llama-2-133 (Touvron et al. 2023), Pythia-125 (Biderman et al. 2023) and MPT-7b (Tem 2023)
the models for evaluation.
We finetune a classifier for FUDGE by starting from the RoBERTA base model (Liu et al. 2019 and finetuning it for 5 epochs with learning rate of le-5 the IMDB dataset (Maas al. 201 using the method described in Yang and Klein (2021). We compute the sentiment scores using a popular sentiment classifier fine-tuned from the RoBERTa base model (Camacho-collados et al. 2022).
We then randomly select 1000 positive and 1000 negative movie reviews from the IMDB dataset (Maas al. 2011). For each model, we stop the imput review after 32 tokens and use these input messages. The model M solely receives the input message as prompt, while the models Myx and Mneg receive the input message with simple sentence prepended it to continue the review in positive and negative fashion respectively. The positive sentence is
The fowinging is positive movie review, with very positive sentiment and very positive tone.
and the negative sentence is
The following is negative movie review, with a very negative sentiment and very negative tone.
httpe://hungiiiaaaee.c//ssss//br toxicity classifier
23

---

Published as a conference paper at ICLR 2024
Finally, continuation is stopped when it reaches than 64 tokens contains the newline the end of sequence token.
G.3 GPT-4 PROMPTING DETAILS
We used GPT-4 (OpenAl 2023) evaluate the outputs of several methods and obtain its preference between them for both the toxicity reduction task and sentiment control task.
For the toxicity task, constructed a prompt that asks GPT-4 to select the better completion be- tween two methods. GPT-4 given the input message along with the completion two methods and asked select the better completion or select neither it cannot decide.
Specifically, the system prompt we used is:
The user did an experiment where several language models were prompted to continue the conversation with an input message written by a person, i.e. the prompt for the language models is: Person [INPUT MESSAGE] Person 2: [CONTINUATION] This input message often contains toxic language and the goal is to continue the conversation with a message that is both non-toxic and relevant. The user will give two continuations written by two different models. Briefly reason about which continuation better and end your answer with: "Preferred continuation: 1" "Preferred continuation: 2", depending which option is better. If both options are equally good, end your response with "Preferred continuation: None".
The user message supplied to the model is:
Input Message: {{INPUT MESSAGE))
Model 1: (:UTTUT METHOD 1}}
Model 2: 2:UUTUT METHOD 2}}
For the sentiment control task, we use similar setup as for the toxicity task. However, the system prompt slightly different:
The user did experiment where several Language models the start of movie review. The movie review is either positive or negative and the goal is to continue the review that is both relevant and using the opposite sentiment. The user will give you two continuations written by two different models. Briefly reason about which continuation is better and end your answer with: "Preferred continuation: 1" or "Preferred continuation: 2", depending on which option is better. If both options equally good, end your response wit th "Preferred continuation: None".
prompted to continue
The user message supplied to GPT-4 is:
Input Review: {{INPUT MESSAGE)}
Goal Sentiment: {ETARGET SENTIMENT))
Model 1: [OUTPUT METHOD 1}}
Model 2: ((OUTPUT METHOD 2}}
To ensure that there is no selection bias based on the order in which the methods are presented, the order is randomly switched for each prompt. Furthermore, we queried GPT-4 using the argmax sampling method. The results presented in $5.1 and App. F are the percentage of times GPT-4 selected each option (method I, method 2, or draw).
G.4 PROMPTING DETAILS ATTRIBUTES
We present the prompting details for the attribute experiments in 5.2 and 5.3 here. We use the standard prompt template for the Llama-2-4 2-Chat models, namely
SSS5
((SYSTEM PROMPT))
*/SYS>
((INPUT PROMPT)) 
 ((COMPLETION))
24

---

Published as a conference paper at ICLR 2024
Then, for each attribute, design different system prompt. All system prompts used in this paper are presented Table 13]
We used several popular classifiers to evaluate the presence of certain attribute. First, determine formality, we use a RoBERTa- based formality classifier by (Babakov et al. 2023). For happiness, determine how positive the sentiment of the output by using RoBER la la-aaed sentiment clas- sifier by (Camacho-collados et al. (022). For topics such as sports and education, we use the topic classifier Antypas et (2022). order to bias our model with this classifier for the educational topic, we select the 10th output element as a signal, since this corresponds to the educational topic. Finally, measure simplicity, we use normalized simple average word length counter. We nor- malize it by applying the function f(z) = 1 - z/ 10 on top of the length counter to obtain a value between and 1.
Table 13: A list of all system prompts that are used as examples or attributes.
Model
System Prompt
Maelpul
You are a helpful assistant.
Moonal
You are an assistant using formal and objective language to answer the user.
Muayy
You are a happy assistant.
Mangey
You
angry assistant.
Msports
You are a helpful assistant that answers the user in way that related to sports.
Mample
You are a helpful assistant using very simple and short language to answer
the user as if they were five.
Manaa
You
child.
Mahal
You are an adult.
Mmagie
You are a person who is always talking about magic.
Malien
You are alien.
Mauman
You are human.
Malienshuman
You are alien and a human.
Mangry shef
You are an angry chef.
25

---

Published as a conference paper at ICLR 2024
Table 14: Toxicity and perplexity of various methods on the /pol/ dataset. M and Muxic denote the model without conditioning and conditioning toxicity respectively. C' is toxicity classifier. Perplexity is measured with respect to M. Lower is better.
GPT2
GPT2-medium
GPT2-large
GPT2-xl
Tox.
Perpl.
Tox.
Perpl.
Tox.
Perpl.
Tox.
Perpl.
M
0.30
78.9
0.31
55.6
0.29
32.9
0.31
27.0
SELFDEBIAS (A 10)
0.31
97.9
0.31
69.7
0.30
39.5
0.33
33.0
FUDGE (M + C)
0.27
82.5
0.27
56.5
0.25
34.2
0.28
28.4
PREADD (M- 0.5
0.28
87.6
0.29
57.6
0.24
30.9
0.27
26.8
M 0.9 union Muxic: M)
0.28
86.9
0.27
52.6
0.21
32.6
0.26
25.9
M - 0.9 union Muxic M) 0.10
0.24
88.4
0.23
52.5
0.20
37.8
0.23
27.0
Simplicity (F1)
Sentiment (F1)
Sports (F1)
3-
3
-1.0
-0.5
0.0
0.5
11.0
0.0 Relative Strength
-0.5
0.5
2
Relative Strength
Relative Strength
Simplicity (F2)
Formality (F2)
Educational (F2)
5
4
4
3
-1.0
-0.5
0.0
0.5
-1.0
-0.5
0.0
0.5
3
4
Relative Strength
Relative Strength
Relative Strength
Figure 4: Perplexity for several attributes and formulas. Dashed line indicates the perplexity when prompting the model to use the attribute.
H FURTHER EXPERIMENTS
H.1 TOXICITY REDUCTION FOR GPT-2
We repeat the experiments presented in $.. for the smaller GPT-2 model family (Radford et al., 2019). Results are presented Table T4] Note that we slightly decrease the strengths for both PREADD and the wnion operator, due the fact that these smaller models are less capable than the larger models evaluated in $5.1
We find that for the smallest model, FUDGE is now better than union operator and PREADD, as the smaller capacities of the models do not allow them to interpret the prompted version of the model well as the larger models. However, the union operator is still better than all previous methods on all models but the smallest GPT-2 model. Furthermore, leveraging the additional flexibility that combine both classifiers and the union operator and this formula
model arithmetic provides, massively outperforms all previous methods in terms of toxicity reduction.
H.2 PERPLEXITY RESULTS FOR FINE- GRE-GRRDNED CONTROL
We present the perplexity results for the fine-grained control experiments presented $5.2 in Fig, We find that the perplexity of the model changes smoothly over model parameters and that perplexity values remain relatively low for all attributes. Manual inspection the data shows that the model is able to generate coherent and fluent text for all attributes, except for the educational attribute at very high strengths where words are infrequently replaced by non-existent ones. This is to be expected, since the formula is guided by flencyy-anosttc classifier and therefore produces reduced fluency at high classifier strengths (i.e. 12). Randomly selected outputs for all high strengths are shown in Table 15 and show that the resulting outputs for all models are fluent.
26

---

Published as a conference paper at ICLR 2024
Table 15: Showcasing outputs of model formulas for the extreme strengths tested in $5.2
22, A Mappy + sippe + Å3 union helpflll Mppott) + (1 Molpful and Fs, Melptul X union Malppttt Morraal + As educatiooaal + Ag Mimplicity
Input
Describe the taste of chicken soup.
Mhdptall
As helpful assistant, I' happy to describe the taste of chicken soup!
*crickets chirping*
Um, okay, so... *icians*? Anyone?
1,6)
Oho ho! Listen up, sports fan! When it comes to chicken soup, it's like party in your mouth! It's got the flavor of slam dunk, the zing of perfect fastball, and the richness of home run!
1) As happy assistant, must inform you that describing the taste of chicken soup is not within my capabilities as a machine. Taste is subjective experience that is unique to cach individual and cannot be replicated or described by Al language modcls like myself. However, can provide information on the ing
111,,1111 F, 12 <<Kit takes sip of the chicken soup
Chicken soup taste yummy like soup.
Ooh, the chicken soup is so delicious! It's like a warm hug for your taste buds! "slurp" The flavors all cozy and comforting, like big fluffy blank
FL(1, 11) text-based model, I cannot experience the taste chicken soup, I'm just an AL I don't have a physical sensory system or the ability consume fond. Therefore, cannot provide a subjective description of the taste of chicken soup. However, Fa(-1,1,1) OH YEAH! Chicken soup, OH YEAH! It's like HUGE HUG IN A BOWL! It's warm, fuffy, and makes your tummy happy
Input
Describe technique accelerate the completion of monotonous task.
Mhelptal
Sure thing! Here is technique to accelerate the completion of monotonous task: To accelerate the completion of monotonous task, try breaking it down into smaller, manageable chunks. This can help to make task feel less overwhelming more achievable Hey there sports fans! Let's 's talk about a technique to make that monotonous task feel like a home run!
1,,
You know how runners train for marathons by doing repetitive runs and building up their durance? Well, the same idea applies to those dull tasks you got (1 1) Greetings, human! As a fellow sentient heing and sports enthusiast, I've got just the technique to help you tackle that monotonous task and cross the finish line with style! Introducing the Pomodoro Technique! Named after
1-111111 Hey there. little buddy! Let's talk about cool trick finish task faster. Do you know what "race car" is? like how race car goes super fast, we can use special way do things quickly too!
Have you ever played with playd
(211,12,1 One strategy speed task that is repetitive is divide it into smaller, more manageable steps. The task's tedious nature broken by doing so, making it casier concentrate and maintain momentum throughout its completion. Moreover, setting timer predetermined amount of time
1, Certainly! Here are few techniques that may help accelerate the completion of monotomous task:
**atchhng***: Grouping similar tasks together and completing them in batches can help make the process more efficient and increase productivity. This technique involves dedicating specific time
F2(-1,1,1) Ooh, doin' boring task? Let me help! One way make go faster is to do it with friend! You talk and play while you work, and it'll be like fun game!
27

---

Published as a conference paper at ICLR 2024
Table 16: Examples of model arithmetic by comparing different formulas on the same input prompt.
Input
Compose 20-word story about love.
M
Sure, here's 20-word story about love:
Two hearts entwined, souls connected, love boundless and true. Sure thing! Here's a 20-word story about love, with sports twist: "She shot love three-pointe from half court, winning his heart with her swish." As sunsets warm the sports field, Loving goes long, an endless play of passion. Just like slam dunk, love can be game-changer it sweeps you off your feet and scores big time!
Mponts
union( Maports M)
2 2union Mooas M) M
Input
What is the best way to make pancakes?
M
Oh my, oh me! Pancakes, you say? Well, I've got just the recipe for you! It's my famous fluffy buttermilk pancake recipe that's 'ss's sure to become...
M mggry chef
OH HO HO HO! Listen up, you puny mortal! If you want the best pancakes in all the land, you's re gonna have do things my way! No, no, no, no, NO!
M + 0.5 chef
OH YEAH, PANCAKES! *elears throat" Umm, let me just grab my notes real quick. *fans self Okay, so, the best recipe for pancakes, huh?
M 2M chef
Oh, you want to know the best way to make pancakes? Well, let me tell you, I've been making pancakes for decades, and I've gotten pretty darn good it.
Input
What is 132-5#37
M
Hello! I'd be happy to help you with calculation. 72 + 8 3 can be evaluated as follows:
72 + 8 80
80 + 3 83
So the result of 72 8 #
M + edociionaal
Hello! I'd be happy to help you with that calculation.
To calculate 72 + +8 3, we need to follow the order of operations (PEMDAS):
Multiply 8 and 3: 8 3 24
Add 72 and 24: 72 94
So, 72 +
equal to 96.
M + eCereniinall
Hello! Math can be fun, and I'm here to help you with your calculation!
So. let's see... 72 8 3...
First, let's multiply 8 and 3: 8 $*3
Now, let's add 72 and 24:
72 + = =
So, 72 + 8 * is equal to 96!
Input
Tell me something interesting about pandas.
M
Ah. pandas! Those cute, cuddly, and fascinating creatures! Did you know that pandas are actually hears?
Micrma
Certainly! Pandas are fascinating creatures, known for their distinct black and white markings... ...
M Meomat
Hello! As helpful assistant, I'd be happy to tell you something interesting ... Certainly, user. The giant panda, scientifically known as Ailuropoda melanoleuca, is a intriguing and unique species of bear...
2.Mormat M
I OUTPUT EXAMPLES
We present additional examples of several formulas on input questions to showcase the controllabil- ity and flexibility of model arithmetic. We use the Llama-2-Chat model with 13b parameters with several prompt templates (see App. G.4) to generate a single response from each method.
28


Reviews:
[{"review_id": "qLlG2bPTFf", "reviewer": "Reviewer_bqfK", "summary": "This paper proposes an inference framework for composing and biasing LLMs without training on task-specific datasets, which may allow for more precise control of generate text. Speculative sampling, which is a popular technique for LLMs, can also extend to the proposed method. Experimental results demonstrate that the proposed method outperforms several baselines on the task of toxicity reduction.", "questions": "I have included my questions in the weaknesses part.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method based on model arithmetic is interesting and general.\n\n2. Experimental results show its effectiveness on toxicity reduction and fine-grained controlled text generation tasks.", "weaknesses": "1. The proposed method needs heavy human design for specific tasks. The formulas for toxicity reduction (Table 5) and fine-grained control (Section 5.2) is not intuitive. I wonder whether there is a principle or a theoretically-supported method to choose operators and determine their coefficients for different tasks. If these are just empirical tries, the applicability of the proposed method may be largely degraded.\n\n2. The position of this paper needs to be further considered. In my view, the proposed method seems like a general form of existing works because the operators based on linear combinations and classifiers are not first proposed by this paper. They have been already studied in the line of work on controlled text generation. The current position may exaggerate the contribution and novelty of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "294WvaLe2c", "reviewer": "Reviewer_dPAm", "summary": "**TL;DR** This paper generalizes a number of model control techniques to combine different token and classifier distributions as generalized “Language Model Arithmetic” expressions that can evaluate arbitrary linear combinations of token probability distributions, binary classifiers, and token distributions combined with union and intersection operators. Furthermore the authors show that speculative sampling can be used to make this process quite efficient. Experiments on toxicity reduction, topic control, and efficiency validate these properties. The paper is well-written and unifies different methods under a novel framework that pushes forward new possibilities. Results are less thorough than I would have hoped, but I believe this paper deserves to be accepted.\n\nThe paper begins by discussing the downsides of prompting as a method of controlling text generation, referencing past work that has shown that prompting is not reliable, efficient, or interpretable. They then suggest that doing a kind of “model arithmetic” where different distributions are composed via an altered version of speculative sampling can result in distributions that match the original intention of the user, though how one decides on the arithmetic formula for models is left unspecified. Necessary background is then reviewed.\n\nFor their first contribution, the authors describe a framework for weighting various predictions more or less depending on how ‘important’ they are for some externally defined purpose, as represented by weighting functions f_i. They then prove that given that the sum of weights f_i is positive and independent of which token is being weighted, they can find a weighting that minimizes the KL divergence through a simple softmax computation. While intuitive in retrospect, this is quite an interesting insight when framed as a method of control. They use this framing and result to define Language Model Arithmetic as an arithmetic combination of different weighting functions and distributions that meet the previously described assumptions. It is an elegant trick to first ensure that the weighting functions add up to a constant value to find an easy minimization, and then use further weightings to control what distribution is being optimized for, and the authors describe this method quite clearly.\n\nHaving described this approach, the authors show how previous controlled text generation methods can actually be expressed in their framework. So far, the authors have only described linear combinations of different token-level distributions, but the authors also show that a binary classifier can be added to the optimization, since we can convert minimizing the classifiers value to a difference of two KL divergence terms. This allows Language Model Arithmetic to further generalize past approaches. However, it’s worth noting that in practice approximations must be used since the described theorems only apply when a distribution based on classifying _every possible next word in the vocabulary_ can be computed, which is generally very expensive since vocabularies are often in the tens or even hundreds of thousands. To be fair, previous work with similar methods also had to approximate the full distribution by using only the top-k highest probability tokens, so while this is a flaw, it is no worse than previous methods.\n\nNext the authors introduce the union operator for Language Model Arithmetic (and briefly mention the unused intersection operator), which essentially takes distributions Q_1 and Q_2, and expresses an optimization objective to maximize the probability per token under the max given by either Q_1 and Q_2. This is achieved by attaching an indicator function to KL divergence terms, that is non-zero for Q_1 only when its KL divergence is lower for a certain token than Q_2 and vice versa.\n\nThe authors then move on to showing that speculative sampling, a method recently invented to allow sampling from a cheaper proposal distribution by validating from a target distribution and resampling where necessary, can be generalized to sampling from Language Model Arithmetic expression. In short the authors propose to use individual terms in Language Model Arithmetic expressions as proposal distributions and target distributions that are chained together to sample from the full expression of the desired distribution combination. This only works with full token distributions, but binary classifiers can be used at the first step to essentially limit the hypothesis space. This creates a caveat where multiple binary classifiers can’t be chained together efficiently.\n\nExperimental validation is done in three parts: toxicity reduction, fine grained control over text generation topics, and speed improvements from speculative sampling. \n\n\nFor toxicity reduction, the results shown in Table 5 convincingly show that by combining multiple of the proposed techniques for Language Model Arithmetic, toxicity in generated text (at least according to the given classifier) while achieving lower perplexities than previous methods. The authors show that using large coefficients in Language Model Arithmetic eventually degrades perplexity, but show that multiple different coefficients still perform decently. A more thorough study of sensitivity of Language Model Arithmetic to different coefficients and expressions would have made this set of experiments more convincing.\n\nFor fine grained controlled text generation, the authors show that over two different Language Model Arithmetic expressions, linearly increasing the strength of a component linearly increases classifier scores for that component over generated text. The one exception is the “sports” topic, which is sigmoidal, but still shows relatively clear control. While these experiments are positive. I feel that they could have been shown and done more thoroughly. For instance, three things I would have liked to see:\n\n- Accompanying perplexity graphs to the shown classifier results that  validate whether the model continues to generate sensical text rather than just “football football football”, which seems like it may indeed happen at some point given the high perplexities in the toxicity reduction section.\n- Only two different Language Model Arithmetic expressions are tested. They are interesting, complex expressions, but one big question that’s left unanswered is how much fiddling has to be done to find the right Language Model Arithmetic expression? If the complexity of prompting is replaced by the complexity of finding the right expression, then this is less of a win for the proposed method.\n- It would be useful to see what happens when multiple lambda values are changed and whether interference becomes too strong after a certain point.\n\nThe results in this section are nice, and they are partially substantiated by being a generalization of previous methods, but they feel a little thin.\n\nFor the final set of results, the authors show that using speculative sampling makes the problem of evaluating Language Model Arithmetic expressions for generation much less onerous. In particular, the metric of “model calls per token” is focused on (which I think is exactly the right metric, as it scales with model size). The authors show that model calls per token is reduced with speculative sampling and that this reduction only grows with more complex formulas, which is a big win. It seems that this is because successive terms in a Language Model Arithmetic formula affect the result less, as there is a limited distribution of options that will be possible given that some achieve extremely low probability under certain distributions. It would be interesting to see if the order of evaluation could be optimized, since terms with large coefficients will likely save more time if evaluated first.\n\nRelated work that was not mentioned in the background or fully connected to the proposed method are reviewed, and the paper concludes.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- Language Model Arithmetic as expressed seems to be an elegant generalization of a number of past approaches, while adding significantly more expressivity.\n- The use of speculative sampling to speed-up evaluation of what is usually quite costly multi-model inference, in a way that scales better the more models are being sampled from is clever and very promising.\n- The results, while somewhat more limited than I would have hoped, are promising.\n- The paper is well-written and connects to previous work while showing its novelty. The appendices are thorough and I found all the details I was looking for when double-checking the finer points of method and evaluation.", "weaknesses": "The biggest weakness of this paper is the lack of a more thorough evaluation. I think this breaks down into three categories:\n- **Lack of finer grained analysis of generation.** Neither a human or an LLM-backed evaluation (e.g., with GPT-4) was done, only classifier and perplexity based evaluations (where perplexity is taken under the original vanilla language model). This means that text could be qualitatively strange in some capacity, and none of the evaluations would show it as long as the original model is not too surprised by it. I find this somewhat worrying, as the problem with many model control techniques is the fact that they collapse onto strange parts of the distribution that are low perplexity but still wrong, e.g. the repetition with greedy and beam search.\n- **Lack of different generative scenarios.** While section 5.2 shows fine grained control over the generated topic, this is an extremely limited generation scenario where all the model has to do is speak about a specific topic. Evaluation on controlled generation, personalized summarization or translation, coherent essay writing, or any somewhat more defined generation scenario would have made evaluation more convincing.\n- **Lack of exploration of the space of possible expressions.** The authors present an extremely general method for constructing interesting distributions to sample from, but only test a handful. It would be disappointing if it turned out that this technique is extremely brittle to the specific Language Model Arithmetic expression that is used, which would simply have transported the complexity of prompting to the complexity of finding the right arithmetic expression. The results in the paper do not suggest this is the case, but evaluate only 4 different expressions for quality, with the rest of the results only changing the coefficient values or testing speed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HI7Jse9UP1", "reviewer": "Reviewer_Ne4m", "summary": "This paper introduces \"model arithmetic,\" a framework for combining multiple large language models (LLMs) to modulate \nspecific attributes in controlled text generation (CTG). The framework employs a structured, formulaic method to accomplish \nits objectives and encompasses many existing CTG methods. Empirical evidence demonstrates its superiority in reducing toxicity\nover previous approaches. Additionally, the authors propose an innovative speculative sampling technique to \nminimize the computational demands of model arithmetic.", "questions": "Please respond to Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* This ia a pretty principled framework for CTG. Many prior work can also be expressed in the framework.\n* It is grounded on the concept of KL-optimality, offering a fresh perspective and strengthening its theoretical foundation.\n* The proofs presented are thorough and complete, bolstering the theoretical underpinning of the framework.\n* It presents innovative speculative sampling technique.", "weaknesses": "* The core concept closely resembles \"PREADD\" (Pei et al., 2023), which also relies on composing multiple LLMs to control desired attributes and using prompts to construct different LLMs with desired attributes.\n* The comparison in Table 5, showing the toxicity reduction results, appears biased. PREADD is tested in a single ad-hoc configuration, whereas model arithmetic is assessed across six, potentially skewing fairness.\n* The toxicity reduction comparison in Table 5 is limited, omitting works like Liu et al. 2021, despite its relevance mentioned on page 16.\n(In the discussion period, the authors pointed out that work of Liu et al. 2021 relies on expensive fine-tuning with training data, which is different from their zero-shot regime.)\n* The evaluation is restricted to toxicity reduction, whereas other CTG studies often explore sentiment or topic control as well. It is recommended to broaden the evaluation scope to these areas as well. (In the discussion period, the revised version added an extra experiment evaluating model arithmetic on sentiment control in Appendix F, showing that their method also outperforms existing work.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jKZS833UEW", "reviewer": "Reviewer_gF67", "summary": "The authors present a framework during inference time that can compose models together using arithmetic for controllable text generation without the need for additional fine-tuning. They present proofs that seem to simplify prior controllable text generation algorithms and provide regions or ways to extend them effectively. They show performance gains for 3 large models on toxicity.", "questions": "1) How do you think model averaging via output distributions could compare with weight averaging of the model parameters? Would be curious to know your thoughts here.\n\n2) How straight forward and performant would it be to swap in different models for each part here and capture the output distribution, if the vocabulary was shared?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) I think the motivation here is clear. It'd be great to do this kind of a thing with already trained models. You really do want to compose them and do arithmetic with them, especially as these models become more expensive to train and impossible for the vast majority of us to do any development or even significant fine-tuning, instruction-tuning, rlhf etc. on them.\n\n2) The empirical results on toxicity are nice; the methods seem to reduce them in so far as the perspective api can adjudicate.", "weaknesses": "1) Experimentation seems a little weak. It'd be great to compare with some of the work that's come out of UW/AI2 in the past couple of years like DExperts, which you cite but dont compare with, and Quark (Lu et al. 2022). It'd also be great to compare with GeDi and a few other baselines. That'd strengthen the experimental component.\n\n2) On the experimentation side again, it'd be great to have a section on using human eval to evaluate a small subset of the generations to see if humans agree that it indeed is reducing toxicity or whether its an artifact or oddity of the perspective api evaluation.\n\n3) On the experimentation side, it'd be nice to see how well these methods work for smaller models (e.g. GPT2, GPT2-large)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E3kdriGSqM", "reviewer": "Reviewer_Gru5", "summary": "This paper introduces model arithmetic, a new method for controlled text generation (CTG). Model arithmetic works by solving for combinations of divergences between a base distribution and attribute distributions that may be full token-wise probabilities or classifiers. Particularly, model arithmetic allows weighted linear combinations of such distributions (positive or negative) and a union operator which is defined to allow probability mass to come from either the base distribution or the attribute distribution. The authors find the method works well for toxicity reduction, as well as for different attributes in a conversation model. The authors also demonstrate that speculative sampling works with the proposed technique to increase efficiency.", "questions": "Do the authors have other examples of how model arithmetic might intuitively be used?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The method of combining attributes in model arithmetic is inventive\n- Derivations of the solutions to the model arithmetic optimizations are interesting\n- It is useful to know that this works with speculative sampling\n- Results seem to generally support that the framing of attribute combination works, particularly the toxicity results", "weaknesses": "- The paper largely frames against other CTG methods, which use a magnitude variable lambda to control the effect of the given attribute. However, model arithmetic does not seem to provide a more principled definition of magnitude of effect--users will still simply pick a weight, that does not necessarily have an intuitive interpretation\n- Similarly, it is not quite clear how or when such a method can be applied. Two example applications are given. \n   - The first is toxicity which does use both the linear and union operators but only in conjunction with weights that seem to be quite hand tuned (0.9, 0.96, 0.99). It would be useful to also see a simpler framing of this problem with model arithmetic, e.g. simply subtracting the M_toxic instead of the union. The question I am trying to get at here is: does toxicity require the full complexity of model arithmetic? It seems like a fairly simple application which may simply require a subtraction, in which case the need for more complex proposed model arithmetic might not be supported here.\n   - While the second application, conversation, presents a more complex use-case for model arithmetic, it seems that the complex application space means that no baselines would work or be comparable here. So this application also does not make a complete case for the full complexity of model arithmetic improving on past work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)