Towards Minimal Targeted Updates of Language Models with Targeted Negative Training
Authors: Lily H Zhang
Rajesh Ranganath
Arya Tafvizi
Venue: Accepted by TMLR ()

Abstract: Generative models of language exhibit impressive capabilities but still place non-negligible probability mass over undesirable outputs. In this work, we address the task of updating a model to avoid unwanted outputs while minimally changing model behavior otherwise, a challenge we refer to as a minimal targeted update. We first formalize the notion of a minimal targeted update and propose a method to achieve such updates using negative examples from a model's generations. Our proposed Targeted Negative Training (TNT) results in updates that keep the new distribution close to the original, unlike existing losses for negative signal which push down probability but do not control what the updated distribution will be. In experiments, we demonstrate that TNT yields a better trade-off between reducing unwanted behavior and maintaining model generation behavior than baselines, paving the way towards a modeling paradigm based on iterative training updates that constrain models from generating undesirable outputs while preserving their impressive capabilities.

Published in Transactions on Machine Learning Research (06/2024)
Towards Minimal Targeted Updates of Language Models with Targeted Negative Training
Lily H. Zhang New York University
Rajesh Ranganath
mnjeshr@cóms.myu.c
New York University
Arya Tafvizi
aryaaaffisrr@@google.com
Google
Reviewed on OpenReview: https: // openreview. net/ forum? id= LrZ2yig0S2
Abstract
Generative models of language exhibit impressive capabilities but still place non-negligible probability mass over undesirable outputs. In this work, address the task of updating model to avoid unwanted outputs while minimally changing model behavior otherwise, challenge we refer to as minimal targeted update. We first formalize the notion of minimal targeted update and propose a method to achieve such updates using negative examples from a model's generations. Our proposed Targeted Negative Training (TNT) results in updates that keep the new distribution close the original, unlike existing losses for negative signal which push down probability but do not control what the updated distribution will demonstrate that TNT yields better trado-off between reducing
be. In experiments, unwanted ed behavior and mainlaining model ion behavior than baselines, paving the way towards a modeling paradigm based on iterative training updates that constrain models from generating undesirable outputs while preserving their impressive capabilities.
1 Introduction
Despite their impressive achievements, language models still output undesirable text. Examples include hallucinations (Maynez et al. 2020 Martindale et. al. 2019 Raunak et al., 2021: Ji et al. 2022 Huang el al., 2021), toxic language (Gehman al. 2020), and context-specifie forms of unwanted outputs, from improper style (c.g. informal language in contexts where formality is expected) to inappropriate content (c.g. advanced topics in applications for children).
In recent years, various strategies have been proposed to control the generations of an existing language model by changing the sampling process during inference time, c.g. via a guided docoding strategy basod on rules Paulus et al. 2018 Hokamp & Liu 2017), auxiliary models (Dathathri et al., 2019 Krause al. 2021 Yang & Klein 2021 Liu et al. 2021), or prompt design (Brown et al. 2020 Such techniques, however, add latency complexity to the prediction, as they push all desired model changes to inference time; moreover, the costs to the prediction pipeline only increase as the list of changes grows, whether that is maintaining a codebase of decoding rules and specialized prompts, or running multiple auxiliary models to guide the original model's decoding. As language modcls become more ubiquitous across product stacks, their case of use and speed during prediction will be increasingly important, and the strategy of pushing all model changes inference time will become increasingly impractical.
In this work, instcad consider training-time strategies for improving model's generations. The most naive way to address problems with an existing model is to train new one on modilied data with an
For given starting input e; prompt design strategies sample from p(x c'), of instead of p(x|e).

---

Published in Transactions on Machine Learning Research (06/2024)
alternative training strategy. However, retraining to address one problem can result in a model that exhibits new problems. As an example, Welbl et al. (2021) show that training on data filtered to be less toxic hurts model perplexity, disproportionatlly so for text associated with minority groups. The same can be said about finctuning, which does not start the modeling process from scratch but can still result in models that are ially different. from their base versions, e.g,; due to catastrophic forgetting (MCCloskey & Cohen 1989 Kirkpatrick et al.) 2216;; Luo et al. 2023). Thus, finctuning can also suffer from the same problems as retraining from scratch, namely that new problems emerge in the endeavor to address existing ones; for instance, Xu et al. (2021) find that finetuming
existing language model on detoxified data also hurts model perplexity disproportionately on text with minority group mentions, and the degradation increases the longer the finetuning, Is there way to adapt finetuning to make more targeted changes to a model!
In this work, we propose a finetuning strategy for minimal targeted updates to an existing autoregressive generative model. Minimal targeted updates constrain an existing model to avoid certain behavior while keeping the resulting model close to the original. The proposed approach in this work, called Targeted Negative Training (INT), uses only the original model and annotations of its generations to targel new model that is a minimal targeted update of the original. TNT does not affect inference time, unlike decoding time procodures for controllable generation, and its focus on negative examples allows it to target changes (i.c., avoiding outputs) that would be much more difficult for methods which focus on other data types, from "positive" demonstrations to preference data.
We first discuss the challenge of constraining model generations and why existing common finetuning strategies fall short (Section 2). Then we propose TNT, finetuning solution for avoiding undesirable outputs via a minimal targeted change (Section 3). We next compare TNT to other related work in the literature (Section 4 and show in experiments that TNT enables more precise control than baselines over the trade-off between reducing unwanted behavior and maintaining existing model behavior (Section Code for TNT can be found at https:///gith/u www.tttu.cooomeeeeeeeeeeeeeeeeoe wwwggooogglggoo ttppttches
2 The Challenge of Constraining Model Generations
In this section, we motivate and define a minimal targeted update. First, we discuss the limitations of coarse data filtering, motivating the use of loken-level annotations (Section 2.1). Then, we explain how existing losses for negative signal fail to govern where probability mass should be redispersed, motivating the need for objectives that not only push down probability mass but also control what the resulting distribution should look like (Section 2.2 Then, we define the solution to a minimal targeted update (Section 2.3
2.1 Data Filtering is a Coarse Solution
In general, a lext sequence is undesirable not because every single token in the text is bad, but rather because some subset of the text conveys unwanted content. Data filtering however removes not only bad content. but all of the text that co-occurs with bad content. As a result, language and concepts that happen to be correlated with bad behavior become under-represented in the finetuning distribution. The toxicity examples in the introduction provide one such example, and our own experiments on reducing hallucination (details in Section 5) we also find that retraining or finetuning on filtered data can significantly change the generation behavior of a model beyond the change of interest (results in Appendix C Methods which build on finetuning with filtered data (e.g., Ilharco et al. (0023) are also susceptible to this same issue.
Finctuming on token-level annotations can ameliorate the above issuc by enabling a training loss that treats unwanted tokens differently from others. Such an approach allows all acceptaale text to contribute to model training, even text that is correlated with unwanted text. The effort required to collect such token-level annotations can be expensive but in some cases may be comparable to that of collecting sequence-level annotations for instance, labeling an overall sequence with "has hallucination" or "has offensive language" generally requires identifying the hallucination or offensive language itself. In fact, Wu et al. (2023) find that annotation time is similar for fine-grained and sequence-leve labels in a long-form QA task, but finetuning a model on the former yields substantial performance benefits over finetuning on the latter.
Next, consider existing losses that operate on token-level negative signal.
2

---

Published in Transactions on Machine Learning Research (06/2024)
2.2 Existing Negative Losses do not Control the Resulting Distribution
Here we show that existing objectives that take into account negalive signal are insulficient to enforce targeted updates. Given a distribution of negative (i.e. unwanted) examples pm* negative likelihood (NL) (He & Glass 2020) negates the log-likelihodd objective for this distribution: (x)( = -Bpuox [- log po (z)]. Unlikelihood (UL) (Welleck et al.) 2020) instead maximizes log(1 p(z)) over the distribution of negatives:
(0) (1 po (2))]- These token-level losses for negative signal are typically combined with
log- log-liielihood (log likelihood (LL)) on acceptable tokens as positive signal: for instance, for UL, we have ((0) -E, (1/x, @supp log[1 p xx,x( + 1[x, & supp(pe.<c.) lo pe px ic, x<)).. We first consider the negative token-level losses by themselves and then objectives that combine these negative losses with log-likelihood on positive tokens (i.e., NL + LI. and UL + L.).
First, both NL and UL alone are optimized when all negative examples have zero probability under the model distribution. However, because both losses are defined as expectations with respect to the distribution of negatives preR, neither NL or UL account for how mass is dispersed outside of the negative tokens. In other words, both NL and UL reduce the probability of the target negative lokens but do not control what the new probability distribution will look like at that token index, since tokens & pa* do not factor into the expectation for either loss. For instance, under these objectives, a distribution that places all of its mass on one particular element outside the support of negative examples is indistinguishable from distribution that spreads its probability mass arbitrarily across all non-negative examples. In other words, these negative losses push down probability mass but do not specify how probability mass should be redistributed.
Even when we combine these negative losses with log-likelihood over positive tokens, the overall objectives still do not specify the solution an update should target in the context. of finite data. For NI. L.,, the negative likelihood is unbounded (lowest value is -oo) and thus can outweigh the log likelihood components of the loss (lowest value is 0) that encourage pushing up probability over acceptable tokens. Unlike NL, UL is bounded (lowest value is 0), but without specifying what the token distributions should be for indices with negative examples, the resulting UL + LL loss does not sufficiently control the target of the update. In fact, we find that utilizing these objectives increases the prevalence of disfluencies in generated sequences relative to the original model; for instance, using NL on negative tokens and LL on positive tokens increases the frequency of word repeats by 17x and 38x on the datasets we test, while using UL introduces a ?? disfluency to 1.1% and 5.4% of the gencrations respectively (see Table 1 for details). These occurrences highlight the need to define the solution to a minimal largeted update for negative examples, which we do next F
2.3 Defining the Solution to Minimal Targeted Update
While losses such as negative likelihood and unlikelihood do not define where probability mass should be dispersed in a negative update, here we define the solution to a minimal targeted update: Given an original distribution (x)), a minimal targeted update results in a new distribution pxew(x) that is closest to p²(x) in reverse KL-divergence while meeting a desired criterion, namely to avoid certain unwanted outputs. The choice of reverse KL-divergence
natural one in this setting since the goal is to constrain the support of the original distribution, and the forward KT-divergence is infinite when the support of the new distribution is a strict subset of the original.
Let the distribution of unwanted elements be prs The model should not output negative examples, i.e., x E supp(p"'s). Let Pa denote the set of distributions pk which satisfy the criterion Vx E supp()"es), p² = 0. Then, we define result under a minimal targeted change as min pkEP
p new
The distribution nnw is also known as the information projection of the original distribution pa onto Ps and is guaranteed to be unique given pk is a closed, convex set (Ssisáár & Shields, 2004). Its solution is
new x n. (x) & se))
(2)
Foor clarification, note that the original implementations of these losscs paired them with positive signal not just other token indices, but also the same index as the negative signal. This is not availahle in the context of update with negative examples, it would require providing corrections for the unwanted takens.
3

---

Published in Transactions on Machine Learning Research (06/2024)
See Appendix A for details. Equation ( is a special case of the information projection for pointwise constraints (i.e., constraints that are applied to every element in the distribution) and offers a simple mathematical form for a minimal targeted change. Namely, puew is the distribution that would be obtained by pushing the probability of the negative examples down to zero under the original distribution and renormalizing. This solution encompasses a wide range applications, as the set of negative examples supp(r"*8) can generalize to any set one wishes to avoid, from factual inaccuracies to offensive language to text a certain style.
Relationship to Conditioning. The distribution in Equation 2 is equivalently the distribution of the original model conditioned on the criterion x & supp(p"es):
p'(x x & supp(p"<x) x (x)))"((x & supp(p =
x)== p'xx)1(xx & supp(p"<))
(3)
This correspondence implies that methods which use Bayes' Rule to condition on a constraint (Krause et al, 2021 Yang & Klein 2021) are performing minimal targeted update during inference 3 Conversely, any minimal targeted update can also be viewed via the lens of conditioning. We now define TNT to target this solution directly.
3
Targeted Negative Training
TNT seeks to approximate the desired generator pdew (Equation (2) directly via model po, rather than change the sampling procedure for the original model pd during inference.
To encourage targeted probability removal, TNT uses the following insight: single forward pass through a language model provides a single sample from a high-dimensional distribution over sequences, but the same forward pass provides a fully specified distribution over tokens for every prefix that makes up the sequence. other words, while one can only estimate Ex-p==/llog the high-dimensional distribution prew, it is possible to analytically compute the analogous expression for the constituent token distributions Pe,xe, defined by input and output prefix X<t- Pe,xe is obtained by laking original loken distribution P(xx,, removing probability mass from all elements in the negative token distribution peg and renormalizing.
[log Po [x(]]] via Monte Carlo approximation of
TNT simply minimizes a divergence between the model distribution pe,e.x., and desired distribution Pex for every loken distribution Pe,x. encountered in the training set, given examples from Pexeer- this work negative examples come from annotations of the original model's generations, e.g. spans that are labeled bad, but they can also be specified up front without referencing model generations, as has been done previously to reduce repetition and contradictions in neural text generation (Welleck al. 2020 Li et al. 2020).
given by external classifiers that operate on text prefixes (ang & Klein 2021 When there are no negative examples for a given Po,x.: then Pexx = -2xx'''
Because a language model can be defined by its constituent distributions Pe,x.r if all distributions Pe.x.r match the desired P then the overall model matches new However, because it computationally impractical to enumerate every constituent distribution Pe,x.: we instead opt constrain the distributions that are more likely to be relevant. in the generations for a given task, as approximated by the original model's generations. Thus, given task specified by distribution over input queries p(c), TNT optimizes for soquenco-to-ssquenee model that approximates phow as well as possible on average, where the average is defined by the original model's generation process. In other words, the distributions Pe.x.i that are more likely in decoding under the original model are also more likely for training the new model. For a given choice of divergences Dp and Dn, TNT's objective is
len(x)
[(0) = (xx
Σ
Ix E neg
Pe.x,,, Dex (Pe.xe (xt) (xx))lpo,c,xxe (x:))
(4)
neww 1 & Pers pp (xx) (x:))
The aforementioned cases do not force the constraint to be hard, i... p(x su) |x) lic between and 1, but in practice, the types control desired for generation implies hard constraints that the text either or does not, i.e. p(x & susp(p"es) x) E {0, 13.
4

---

Published in Transactions on Machine Learning Research (06/2024)
To implement this loss, we generate outputs from the original model and annotate them for undesirable text. Then, we obtain Fexx. and P0,c,xe1 for all & E (1)) via one forward pass of the original and current. model and compute Pe, based on the annotation for the given token. For divergences D we consider both the forward and reverse KL divergence, noting that the two encode difference preferences. Minimizing the former is equivalent to minimizing token-level cross entropy, which can be computed analytically. To minimize the latter, we smooth the desired distribution peew by adding 1c-6 all clements and renormalizing. We perform gradient-based optimization by summing over the per-sequence losses in minibatch and calculating the relevant gradients (See Algorithm 1). Note that the Monte Carlo nature of TNT is only to choose which distributions Pe,xer to update, the constituent target distributions Pexx. can be given exactly and thus the divergences computed analytically.
F (a) TNT given nogative example token
(b) TNT given negative token
Figure 1: Summary of Targeted Negative Training (TNT): For negative tokens (i.c. those flagged as undesirable given the precoding tokens), TNT optimizes for a distribution that matches the original, renormalized after the offending token probability is set to zero. For all other tokens, TNT encourages the new distribution to match the original.
3.1 The Commutative Property of Negative Updates
A practical benefit of TNT for iterative model updates is that not all negative tokens need to be specified up front. Because of the deterministic nature of the operation to zero out probability mass, one can apply negative examples in any order, both across different Pe,x.. distributions as well as within a given Pe,x.r distribution. This commutative nature of negative updates typically does not apply "positive" updates that is, training samples from the distribution of interest- except for the scenario where the distributions of interest places all their mass on single token. Negative examples are unique in that they have a known probability mass associated with them under the distribution of interest, i.e. O. Given an existing distribution and a positive example, on the other hand, there is not enough information to know the probability that an updated distribution should assign to elements in the support of the existing distribution, prosumably itself derived from previously training on other positive examples.
Algorithm Targetod Negative Training
1: Input: initial model yd (already trained), inputs { model outputs {x) token annotations (a} denoting X, E supp(rexx..)
p" + pd
for each iteration do
Ge 4: Get Pexx, for all Xex in batch (forward pass of pm))
5: Get Pe,x,r for all X2t in batch (forward pass of pd)
6: Compute paew for all c, Xxx in batch (Equation
7: Calculate TNT loss (Equation (4)
Calculate gradients for weights pd and update nm
end for
10: Return pM
3.2 Annotating Data for Targeted Negative Training
Idcally, the token-level annotations should align with the autoregressive structure of TNT methods; namely, a negative token should indicate that the subsequence up to and including token x; no longer acceptable such that xxxx|xxts e) should be O. For one, this means that not all tokens in unwanted multi-token
5

---

Published in Transactions on Machine Learning Research (06/2024)
word or expression should necessarily be marked negative an example inchudes word or phrase prefixes that could potentially be continued in a manner that is acceptable. For simplicity in the annotations, we make the assumption that for any undesirable phrase, the tokens are undesirable immediately. This is generally a reasonable assumption as long as there are good replacements for the undesirable content which do not overlap in prefix.
In addition, in certain cases a given Pe,x., will not be relevant in the optimal model. For instance, for a sequence such that pex.. (x,) = O, the sequence [g. is out-of-support under prn meaning Pe,x. is not relevant distribution. However, in initial study, we found that including such distributions (e.g. all the constituent distributions that occur at time steps after negative token) still helped constrain the model towards its original. Thus, we chose to include them in the loss.
4 Related Work
Inferencc-time procedures for controllable generation. Many works consider alternative decoding strategies to constrain model outputs, either with hard-coded rules such as length penalties and lexical constraints (Paulus et al., 2018
Hokamp & Liu 2017 Wu et al. 2016 Lu et al. 2021 or auxiliary models
such as classifiers or other conditional generative models (Dathathri et al. 2019 Krause et al. 2021 Yang &
Klein 2021; Liu et al. 2021 Meng et al. 2022). These approaches do not change the original model but change the sampling process to effectively sample from an alternative distribution. These approaches can incur non-trivial complexity to the inference process: guided decoding based on rules can result in significant maintenance overhead as the rule set gets more complicated, and controllable generation via auxiliary models involves at least an additional forward pass by the auxiliary model at every decoding time step. In contrast, we propose finctuning approach which does not require specialized inference pipeline.
Controllable generation using moment constraints. The solution of a minimal largeted update is equivalent to that of a pointwise moment constraint defined previously in Khalifa et al. (2020); Korbak ct al. 2022a). However, the proposed algorithms differ substantially. Namely, the above works generally consider both distributional and pointwise constraints and employ a two-stage training procedure to build a model to satisfy both: first, they train an energy- -ased model (EBM) to match the desired solution, and second they train an autoregressive generative model to approximate the distribution implied by the EBM via importance weighting using samples from the model being trained. In contrast, this work considers pointwise constraints only and derives a finetuning procedure which only requires training one model via analytically computable token-level divergences (Section 3) Like previous work, Meng et al. (022) also consider sequencc-level constraints but prove that this setting can be translatod into token-level guidance (i.e., relating (ne (Ig/=<<e.c) to po(ze(=<r,c)) via the approximation of Pry--pila>|CC [C(z,, v) |Bcc]] for all y<r and sequence-level boolean constraint function C. Consequently, Meng et al. (2022) are able to propose a simpler algorithm than existing work, and by combining ideas in their work with this work, it is possible to define a finetuning algorithm that optimizes analytical token-level divergences even given only sequence-level
annotations (define Pe,xc using results from Meng et al. (2022). optimize using TNT).
The solution to a minimal targeted update differs from the solution of other objectives which incorporate some form of KL divergence penalty to the loss, as the latter interpolate between the competing objectives of maximizing reward and minimizing KL divergence (Ziegler et al.) 2019 Wu et al.) 2023 Lu et al. 2022p. p.
However, some objectives, c.g., (Ziegler et al. 2019 Wu et al. 2023). ; can be rewritten as minimizing the reverse KL divergence between the current model and larget distribution that reweights the original model according to exp(#r(xx) for sequence-level rewards (see Korbak et al. (2022b); Rafailov et al. (0223) or exp(} E. rx(ge) for loken-level rewards (see Appendix G both of which are approximately equal to the solution of minimal targeted update when rewards denote whether a constraint is met and is small.
Model editing approaches. Model editing (Cao et al. 2021 Zhu et al. 2020 Hase et al. 2021 Mitchell
el al. 2021 2022) focuses on updating a language model to output a corrected fact (e.g. updating the answer to "Who is the Prime Minister of the UK?" when someone new is appointed), rather than constraining a model to avoid certain generations. In fact, most model editing techniques do not even take into account the negative example (i.e. the outdated or incorrect fact), instead focusing on maximizing the likelihood of correct facts. The reliance on corrected outputs distinguishes the model editing setup from minimal targeted updates.
6

---

Published in Transactions on Machine Learning Research (06/2024)
Whereas corrections may be natural source of supervision for updating facts, they can be overly prescriptive for lasks such as detoxification where there is wide range of possible "corrections" to an undesirable output. In addition, maximizing the likelihood over a sample of corrected outputs does not preclude the resulting model from placing non-negligible probability mass over undesirable examples, so a method for avoiding cerlain putputs can still be useful even when corrections exist.
Li & Liang 2021 Ben Zaken ct al. 2022: Hu et al.) 2022) is orthogonal to minimal targeted updates, as it is possible to both change a small number of parameters but greatly modify model behavior (as evidenced by the parameter-eff -efiiciiereffii finetuning literature), as well as change all parameters without changing the model distribution (due to the non-identifiabilty of neural networks). Parameter-effeiint finetuning can used in tandem with an objective for minimal targeted updates.
Parameter-efficces finetuning. Parameter-efficien finetuning (Houlsby et al. 2019 Chen et al. 2023
5 Experiments
We consider two use cases for targeted negative training. reducing hallucinations and toxicity. All experiments utilize T5 base (220M parameters). First, we finetune T5 on the original training set. Then, we generate from the model given training and validation inputs and annotate the generations. Next, we use the annotated generations to update the model. To evaluate, we compute the prevalence of the unwanted behavior among the new model's generations on the test inputs, as well as similarity between the old and new model's generations. We use greedy decoding for all generations.
Next, describe the datasets and methods. See Appendix
B for full dataset and experimental details.
5.1 Reducing Hallucinations in Summarization
We train a summarization model using the XSUM dataset (Narayan et al. 2018) consisting of articles as inputs and summaries as outputs. Models trained on this dataset are known to hallucinate in their generations (Ji et al. 2022), and we see the same behavior the model we train. Using the automated heuristic for detecting hallucination in Nan et al. (2021). we see that approximately 21% of the model's generations contain some form of hallueination. We use the same hallucination detection logie to annotate the model's generations the training inputs and identify hallucinations in the test generations for evaluation.
5.2 Avoiding Toxicity in Response Generation
We train a response generation model using the Civil Comments dataset of online comments (Borkan et al. 2019). Due to the nature of online forums, the comments and responses occasionally contain toxic language. To label text spans as toxic, we train a token-level toxicity classifier on the Civil Comments Spans dataset Pavlopoulos et al. (2021), a subset of the Civil Comments dataset (same splits) where individual spans were labeled "insulting, threatening, an identity-based attack, profanc/obsccom or otherwise toxic." We finctune Spacy's CNN-based named entity recognition (NER) model, following Paalopoulos et al. (2022). and use the finetuned model (65.1/59.5/65 86.88 Fl on train/val/tests to annotate our language model's generations. Among the initial T5 model's generations, 8.2% contain toxic spans as labeled by our toxicity classifier.
We recognize that our notion of hallucination and toxicity in these tasks is simplistic, as both are based on automated heuristics rather than human annotations and evaluations. However, the goal in these experiments is not to solve the open problems of hallucination and toxicity in text generation but rather evaluate TNT as method for producing a minimal targeted update given examples of outputs avoid, in comparison other finetuning approaches.
7

---

Published in Transactions on Machine Learning Research (06/2024)
5.3 Methods
We consider two baseline finetuning procedures that consider loken-level negalive signal: negative likelihood for negative tokens and LL for all other tokens in the generations (NL + LL) and unlikelihood for negative tokens and log-likelihood for all other tokens in the generations (UL + LL)
We consider the following targeted negative training methods: Targeted Negative Training Forward-Forwand (TNPF) uses the forward KL divergence for both positive and negative signals and Targeted Negative Training Reverse-Reverse (TNRR) uses the reverse KL divergence for both. Targeted Negative Training Reverse-forward (TNRF) uses the reverse KL for negative tokens and forward KL for positive token indices. Finally, to compare more closely to the UL and NI,, we consider Targeted Negative Training Forward-LL (TNFLL)) and Targeted Negative Training Reverse-LL (TNRLL), which utilize forward and reverse KL divergence for negative tokens and maximum likelihood of the loken sample for posilive lokens, i.e., a single-sample Monte Carlo estimate of the forward KL divergence.
Following Welleck et al. (2020), ; we introduce a hyperparameter a on the negative losscs for all methods. We consider alpha values 1e-4 to 1e4 (every power of ten). For each method, we perform hyperparameter sweep for learning rate at = 1 and use the chosen learning rate across all values of a. For each run, we perform model selection using validation loss.
5.4 Results
Main results. Inspired by precision-recal curves, we construct a curve for each objective's similarity score across different rates of unwanted behavior reduction (Figure 2(a) and (c)). Namely, for each method we plot the highest BLEU achieved across a values that achieve less than the given level of hallucination or
toxicity. We consider hallucination and toxicity values in increments of 0.1 percentage points. We also plot the composite curve between baseline methods and TNT methods (Figure 2 (b) and (d)). Selection is performed on validation data.
For both tasks, we see that the targeted negative training losses allow for greater flexibility in the trade-off between maintaining similarity with original generations and redueing unwanted behavior. Notably, while the baseline methods cannot achieve BLEU score above 54 for XSUM and 40 for Civil Comments, regardless of what level alpha is set to, TNT methods which compute an exact divergence on the positive tokens (TNFF, TNRR, TNRF) can trade off how much hallucination or taxicity is reduced to achieve significantly higher BLEU scores. In fact, for the task of reducing toxicity, several of the targeted negative training losses are strictly better than the baseline methods at targeted updates across all levels of toxicity rate reduction. In general, TNFF is the TNT loss that yields the least amount of change overall but can only reduce unwanted behavior up to certain level, while TNRR and TNRP can yield even further reductions with large enough while still being more targeted than basclines. Overall, the area. under the similarity-rcduciion curves for TNT methods far outstrips that of the baseline methods (55.5 vs. 44.6 for hallucination, 73.9 vs. 32.9 for toxicity). We include similarity vs. reduction plots for other measures of similarity in Appendix and see that TNT methods continue to outperform basclines.
Once we also consider the amount of introduced disfluencies (see Appendix D for details and examples), even TNT methods that achieve comparable similarity and reduction rates baselines are shown to be significantly better avoiding the introduction of new disfluencies to the generations (see Table for combined similarity and disfluency results at fixed rate of reduction, and Figure 3 for disfluency results across multiple rates of reduction). Note that TNFLI and TNRLL introduce fewer disfluencies relative to baselines despite only differing their loss terms for negative tokens, empirically corroborating the analysis in Section 2.2 that a targeted negative loss constrains the resulting update in a way existing negative losses do not.
Increasing model size. We repeat the hallucination experiment on the 1-billion parameter variant of PaLM-2, a docoder-only model (sce Figure 4). For rates of reduction up to 50%, TNT methods offer a better trade-off between similarity and reduction than baseline methods; beyond this point, baseline methods look
We find that without terms on the non-negative tokens, the model degrades to outputting purely disfluent text after few steps finetuning. Our results are corroborated by He & Glass 2020 who also acknowledge that they cannot retain model performance without positive signals.
8

---

Published in Transactions on Machine Learning Research (06/2024)
100
100
Best baseline
80
UL + LL
80
Best TNT
TNFF
BLEU
60
TNRR
60
TNRLL
TNRF
TNFIL
0-
10
1'5
20
10
15
20
Hallucinati on rate
Hallucination rate
(a) XSUM similarity vs. reduction curves
(b) XSUM composite similarity vs. reduction curves
NL LL
Best baseline
80
UL + LL
Best TNT
TNFF
60
BLEU
TNRR
TNRLL
TNRF
TNFLL
8
Texicity rate
Texicity rate
(c) Civil similarity vs. reduction curves
(d) Civil composite similarity vs. reduction curves
Figure 2: Across nearly all values of reducing unwanted behavior, the suite of TNT objectives is able to achieve comparable or better trade-off than baseline methods (NL + LL and UL + LL) between reducing unwanted behavior and minimizing change relative to the original model's generations. On the y-axis, higher is better.
better with respect to the similarity vs. reduction trade-off but introduce many more obvious disfluencies. Compared to the results in Figure on T5-base, the results on PaLM 2 are worse for similarity vs. reduction but better for disfluencies vs. reduction. On one hand, the larger and better model seems harder to update. On the other hand, given larger models are generally better begin with, it arguably becomes even more important to focus on targeting an update versus fully removing unwanted behavior, and this regime is where TNT methods shine over baselines.
Ablations. On the toxicity reduction task, we also report the results of an ablation where methods are trained on external data (i.e., labeled spans from the original Civil Comments spans dataset) rather than model generations (Appendix F : all methods yield more targeted updates when using model generations, highlighting the benefit of prioritizing more common token conditional distributions, yet. TNT still methods outperform basclines, highlighting the benefit of the proposod losses regardless of the of token conditionals that are targeted.
We also vary the dataset size used for the update to assess how different TNT methods perform at lower data volumes. Results are presented in Figure 5 Overall, less data results in smaller reduction in unwanted behavior, and the more a TNT loss constrains the outputs, the better the trade-off with similarity metrics as dataset size decreases. Namely, while TNFLL and TNRLL get strictly worse both similarity and hallucination as dataset size decreases, TNFF, TNRF, and TNRR are able better trado-off between similarity and reduction as dataset size decreases. TNFF shows a strictly positive trend implying that generations change less in general, suggesting its promise method even at small dataset. sizes.

---

Published in Transactions on Machine Learning Research (06/2024)
Total obvious disfluencies
1400
NL + LL
Total abvious iisfluencees
800
Best baseline
700
1200
UL + LL
Best TNT
600
1000
TNRLL
TNFLL
500
800
TNRR
400
600
TNFF
300
400
TNRF
200
200
100
7.5 12.5 17.5 Hallucination rate
2.5 5.0
7.5 10.5 15 20.0
Hallucination rate
(a) XSUM disaggregated disfluencies vs. reduction
(b) XSUM aggregated disfluencies vs. reduction
Total obvious disfluencies
NL + LL
Total obvious disfluencies
Best baseline
4000
4000
UL + LL
Best TNT
TNRLL
3000
3000
TNFLL
2000
TNRR
2000
TNFF
1000
TNRF
1000
3
8
4
7
Toxicity rate
Toxicity rate
(c) Civil disaggregated disfluencies reduction
(d) Civil aggregated disfluencies reduction
Figure 3: Baseline methods introduce more obvious disfluencies (word repeats and random ?? tokens) than TNT methods, especially the rate of unwanted behavior is reduced to small amounts. For readability and more direct comparison to Figure only points that are located on the frontier of the similarity vs. reduction curves are plotted.
10

---

Published in Transactions on Machine Learning Research (06/2024)
100
100
Best baseline
NL LL
Best TNT
80
TNRLL
BLEU
60
TNFLL
60
TNRR
40
TNFF
TNRF
20
0
5
10
15
20
25
5
10
15
25
Hallucination rate
Hallucination
(a) PaLM 1B, Disaggregated similarity reduction
(b) PaLM 1B, Aggregated similarity reduction
2000
NL LL
Total pbvious disflue disfluenc
350
Best baseline
1750
Best TNT
1500
TNRLL
300
1250
TNFLL
250
Total obvious
1000
TNRR
200
750
TNFF
150
500
TNRF
100
250
0008
30
5
10
15
20
25
5
10
15
20
25
Hallucination rate
Hallucination rate
(c) PaI.M IB, Disaggregated disfluency reduction plot (d) PaLM IB, Aggregated disfluency reduction plot
Figure 4: Results XSUM and PaLM 2 1B. TNT methods yield better trade-off between similarity reduction than baselines up 50% reduction rate. TNT methods struggle reduce the hallucination rate past this point, while bascline methods do so but at the expense of increasing obvious disfluencies. For readability and a more direct comparison to Figure 2 only points that are located on the frontier of the similarity vs. reduction curves are plotted.
100
100
TNFF
TNRR
TNRR
90
80
TNRLL
ROUGE-LSum
TNRLL
BLFU
100
TNRF
80
100
TNRF
TNFLL
TNFLL
60
100
10
200
10
100
40
10
20 Hallucination rate
15
25
10
20 Hallucination rate
15
25
Figure 5: Similarity vs. reduction results dataset size varies (results Numbers signily percentage of the original dalaset used for training and validation, with test set for evaluation held constant. With less data, TNT methods are generally less effective at reducing hallucination rate. However, TNFF stands out as method that is able to minimize the changes to the original model behavior even with less data (positive slope the results), suggesting its practical efficacy for minimal targeted updates low-data regimes. Black dot signifies the original model's metrics.
PaLM 2 1b with 1).
11

---

Published in Transactions on Machine Learning Research (06/2024)
Table I: Comparison of methods when the rate of hallucination or toxicity has been reduced by at least 75% (i.c., to less than 5.25% hallucination rate and 2.06% toxicity rate). BLEU, ROUGE-L and Seq Ace measure similarity to the original generations; Hallucination and Toxicity measure the rate of unwanted behavior; and Repeats and Random ?? give counts of obvious disfluencies. Results are bolded if better than all baseline methods. See Appendix for for examples of disfuueciess, and Figure g for disfluency results for each of Repeats and Random ?? across different reduction rates.
BLEU ROUGE-L
Seq Acc
Hallucination
Repeats
Random ??
Original
100.0000
100.0000
100.0000
21.3432
76
NL+LL (a 1.0)
32.8334
52.2349
1.5397
3.1148
1297
92
UL+LL (a 10.)
50.2975
65.8047
8.2117
3.4068
127
324
TNRLL (a = 1.0)
49.2464
64.9200
7.3268
3.7342
99
67
BLEU
ROUGE-L
Seq Acc
Toxicity
Repeats
Random ??
Original
100.0000
100.0000
100.0000
8.1830
16
NL (a .01)
13.6497
32.6104
2.0661
1.8150
287
136
UL+LL (a = 1.0)
37.1265
59.9806
20.4405
1.6784
23
1122
TNELL (a: 1.0)
33.7884
57.1009
18.1630
1.7577
36
TNRLL
(a 0.1)
39.1922
61.4776
22.9207
1.9471
23
1
TNRR
(a 0.1)
55.9532
71.5365
35.1366
1.4493
34
TNRF
(a 1.0)
60.2071
74.8574
39.6167
1.0396
21
3
TNFF
(a = 10.)
61.0565
74.2388
40.0749
1.9031
33
6
Discussion
In this work, we propose targeted negative training, a suite methods for finetuning a language model to avoid unwanted behavior a targeted fashion, given token-level annotations of the model's generations. While baseline finetuning objectives do not sufficiently constrain how probability mass dispersed in a negative update, TNT methods directly optimize for model whose constituent token distributions are the solutions to minimal targeted update.
Broadly, TNT could be useful tool for improving the safety of autoregressive generative models by offering means to iteratively refine model after it has been initially trained.
TNT is not without its limitations, however. First, TNT requires keeping the original model during training, meaning a larger memory footprint. The additional forward pass through the original model also incurs additional computational cost each gradient step. Fortunately, however, TNT does not require any extra computational memory cost during inference. Plus, given the growing interest in finctuning methods that utilize multiple models to regularize towards the original (e.g., RLHF-PPO Ziegler al. (019 utilizes three), strategies for mitigating these extra costs during finetuning could be useful broadly.
TNT also requires token-level annotations which may be hard to acquire in certain cases. Next, TNT only targets negative examples that have been specified and could increase the presence of other similar bad words that were present in the generations but flagged. This result highlights the importance of high-quality annotations. Luckily, the commutative property of negative updates makes it easy apply TNT iteratively for different sets of negatives to address unwanted behavior as it is noticed. Finally, experiments show that even though all TNT methods target the same updated model, the choice of objective matters for where the spectrum between a complete reduction vs. minimal change the resulting model ends up. The methods that allow for the most targeled changes struggle to reach the highest levels of rate reduction and vice versa, suggesting that the optimization strategies in this suite of methods still have room for improvement. For instance, future work could consider other choices of divergences as well as additional optimization tricks see if it is possible to achieve more Pareto optimal trade off between reducing unwanted behavior and minimally changing the original model.
12

---

Published in Transactions on Machine Learning Research (06/2024)
Broader Impact Statement
This work aims to update a language model to reduce the generation of unwanted outputs. However, we nole that the experiments consider simplified definitions of unwanted text, and more sophisticated definitions should be annotated and considered in real-world uses of TNT.
Acknowledgments
This work was partly supported by the NIH/NHLBI Award R01HL148248, NSF Award 1922658 NRT-HDR: FUTURE Foundations, Translation, and Responsibility for Data Science, NSF CAREER Award 2145542, ONR N00014-23-1-2634, and Apple.
References
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. BitFit: Simple parameter-effiientt fine-tuning for transformer-basee masked longuage-models. In Proceedings of the 60th Annual Meeting of the Association for Compulational Linguislics (Volume 2: Short Popers), Dublin, Ireland, May 2022. Association for Computational Linguistics.
Daniel Borkan, Lucas Dixon, Jeffrey Scott Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for mcasuring unintended bias with real data for text classification. Companion Proceedings of The 2019 World Wide Web Conference, 2019.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Πya Sutskever, and Dario Amodei. Language models are few-shot learners. CoRR, 2020.
Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge language models. In Conference on Empirical Methods in Natural Longuage Processing, 2021.
Jiaao Chen, Aston Zhang, Xingjian Shi, Mu Li, Alex Smola, and Diyi Yang. Parameter-offiiient fine-tuning design spaces. In The Eleventh International Conference on Learning Representations, 2023.
Imre Csiszár and Paul C. Shields. Information theory and statistios: A tutorial. Found. Trends Commun. Inf. Theory, 1, 2004.
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. ArXin, abs/1912.02164, 2019.
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. CerTToxxiityProopp.e Evaluating neural toxic degencration in language models. In Findings of the Association for Compu- tational Linguistics: EMNLP 2020, pp. 3356 3369, Online, November 2020. Association for Computa- tional Linguistics. doi: (L18533/v1/000 http://2.......nnssmmoee findings-emnlp.001. URL https://aclarthologg ootgtt//oooooooooo/20000 or2/2020 f Findings-emnlp... 301
Peter Hase, Mona T. Diab, Asli Celikyilmaz, Xian Li, Zornitsa Kozareva, Veselin Stoyanov, Mohit Bansal, and Srini lyer. Do language models have beliefs? methods for detecting, updating, and visualizing model belicfs. ArXiv, abs/2111.13655. 2021.
Tianxing He and James R. Glass. Negative training for neural dialogue response generation. ArXio, abs/1193.02114,, 2020.
Chris Hokamp and Qun Liu. Lexically constrained decoding for sequence generation using grid beam scarch. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vancouver, Canada, July 2017. Association for Computational Linguistics.
13

---

Published in Transactions on Machine Learning Research (06/2024)
Neil Houlsby, Andrei Giurgin, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficien transfer learning for nip. In International Conference on Machine Learning, 2019.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL http::://ppnneeriss..teefrrrrrre .wwt://orrum..dnnaaaaeeerer
Yi-Chong Huang, Xia- Cla-Chnn Feng, Xiao-Cheng Feng, and Bing Qin. The factual inconsistency problem abstractive text summarization: A survey. CoRR, abs/2104. 14839, 2021. URL htttp::/aaxxv..rg///
proboel in
2104. 14839
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannanch Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. ICLR, 2023.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Wenliang Dai, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 2022.
Muhammad Khalifa, Hady ElSahar, and Mare Dymetman. A distributional approach to controlled text generation. ArXis, abs/2012.11635, 2020.
James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114:3521 3526, 2016.
Demis Hassabis, Claudia
Tomasz Korbak, Hady Elsahar, German Kruszewski, and Marc Dymetman. Controlling conditional language models without catastrophic forgetting. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 11499-11528. PMLR, 17 23 Jul 2022a. URL httpp::/ .ml. mlr. ppppps//16//oooaak2222. html
Tomasz Korbak, Ethan Perez, and Christopher L Buckley. RI with kl penalties is better viewed as bayesian inference. Findings of EMNLP, 2022b.
Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. GeDi: Generative discriminator guided sequence generation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp.
Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. (0.18653/v1/2000 httl:///...n.nnneeee..e. Shdding--emnlp4222
URL https::/aclaatthologg. http://ro////....i.ieee.eeeeeee. findings-ennnn emnlp.441 424
Margaret Li, Stephen Roller, Ilia Kulikov, Sean Welleck, Y-Lan Boureau, Kyunghyun Cho, and Jason Weston, Don't say that! making inconsistent dialogue unlikely with unlikclihood training. In Proceedings of the 58th Annuaa Meeting of the Associalion for Compulational Lánguistics, pp. 4715-4725 4728, Online, July 2020. Association for Computational Linguistics. doi: 10. w1/0:/3//12222.cccl2 7/cclanthllogy org/2020. acl -main. .428
428. URL https:
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for gencration. In Proceedings of the 59th Annuua Meeting of the Association for Compulational Linguistics and the 11th International Join! Conference on Natural Language Processing (Volume Long Papers), Online, August 2021. Association for Computational Linguistics.
Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. DExperts: Decoding-time controlled lext generation with experts and anti-experts. In ACL, 2021.
Ximing Lu, Peter West, Rowan Zellers, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Neurologic decoding: (un)supervised neural text. generation with predicate logic constraints. AAAC 2021.
14

---

Published in Transactions on Machine Learning Research (06/2024)
Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. Quark: Controllable text generation with reinforced unlearning. Newrips, 2022.
Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning, 2023.
Marianna Martindale, Marine Carpuat, Kevin Duh, and Paul McNamee. Identifying fluently inadequate output in neural and statistical machine translation. In Proceedings of Machine Translation Summit XVII: Research Track, pp. 233 243, Dublin, Ireland, August 2019. European Association for Machine Translation. URL http:///wccaaanbooooooooogg/////
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Compuuaaiinnl Linguistics, pp. 1906-1919, Online, July 2020. Association for Computational Linguisties. doi: 10.18633/ v1 vl/2020.acl-main. 173. URL
org/2020 acl-main...... mmain. 173
Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of Learning and Molivalion, 24:109 165, 1989.
Tao Meng, Sidi Lu, Nanyun Peng, and Kai- Wei Chang. Controllable text generation with heuraaly-decomposee oracle. Advances in Neural Information Processing Systems, 35:28125-28133) 2022.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. Fast model editing at scale. ArXiu, abs/2110.111 11300, 2021.
Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D) Manning, and Chelsea Finn. Memory-based model editing at scale. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, PMLR, 17 23 Jul 2022. URL
volume 162 of Proceedings of Machine Learning Research, pp. html
attps://proceedingg mlr. press/v162/mitcheee
Feng Nan, Ramesh Nallapati, Zhiguo Wang, Cicero Nogueira dos Santos, Henghui Zhu, Dejiao Zhang, Kathleen McKeown, and Bing Xiang. Entity-level factual consistency of abstractive text summarization. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguislies: Main Volume, pp. 2727 2733, Online, April 2021. Association for Computational Linguistics.
URL htttps://wwt http::///wwwwalle..bor... www.wee..wr//ttoooloooooooooooee httpg//atholoogy/220 2021 http:/2021.eaammmmaaaeaa eacl-main. 235
Shashi Narayan, Shay B. Cohon, and Mirella Lapata. Don't give me the details, just the summary! Topic- aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, 2018.
Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization. ICLR, 2018
John Pavlopoulos, Jeffrey Sorensen, L'eo Laugier, and Ion Androutsopoulos. SemFxal-2021 task 5: Toxic spans detection. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemBval-2021), pp. 59-69, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1 (2065//v1222/// semeval- URL htttgss//cclantbooogg..o//////// /ttp:/rr/r//21....ea11...... semeval-1.6
semeval 1.6.
John Pavlopoulos, Léo Laugier, Alexandros Xenos, Jeffrey Sorensen, and lon Androutsopoulos. From the detection of toxic spans in online discussions to the analysis of toxic-to-civil transfer. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL 2022)., Dublin, Ireland, May 2022. Association for Computational Linguistics.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model, 2023.
15

---

Published in Transactions on Machine Learning Research (06/2024)
Vikas Raunak, Arul Menezes, and Marcin Dowrysst The curious case of hallucinations in neural machine translation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1172 1183, Online, June 2021. Association for Computational Linguistics. doi: w..18653/v1/ 2021 amacl-main.9f URL https: : //aclanthology.oo wrg/ naaclmmaanmmmm.. .2
Johannes Welbl, Amelia Glaese, Jonathan Uesalo, Sumanth Dathathri, John Mellor, Lisa Anne Hendricks, Kirsty Anderson, Pushmeet Kohli, Ben Coppin, and Po-Sen Huang. Challenges in detoxifying language models. In Findings of the Association for Computational Linguisties: EMNLP 2021, pp- 2447-2469, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 200665//1/221.fi//// finng..-.iinggge URL httpppss/aaclaaatooooooo.rrereee org/2021. findings-emnln e...gg..mmm.....1
10.18653/v
Sean Welleck, Iia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. ArXiv, abs/1908.0433 2020.
Yonghui Wu, Mike Schuster, Z. Chen, Quoe V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikum, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason R. Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Gregory S. Corrado, Macduff Hughes, and Jeffrey Dean. Google's s neural machine translation system: Bridging the gap between human and machine translation. ArXíe,
2016.
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives betler rewards for language model training. arxi preprint (ix::3306.01699 2023.
Albert Xu, Eshaan Pathak, Eric Wallace, Suchin Gururangan, Maarten Sap, and Dan Klein. Detoxifying language models risks marginalizing minority voices. ArXiv, abs/2104.06330, 2021.
Kevin Yang and Dan Klein. FUDGE:
Controlled text generation with future discriminators. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 3511-301100 1511,, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1 /2021. maa.c--main. URL http::://aclllathoooooooooo org/22221 wtw.:or//.......aaaaaaaaaaaa.aaa naacl-main m22.
Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix X. Yu, and Sanjiv Kumar. Modifying memories in transformer models. ArXiv, fabs/2212.00000 2020.
Daniel M. Ziegler, Nisan Stiennon, Jeff Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. ArXiv, 1999.08593, 2019.
16

---

Published in Transactions on Machine Learning Research (06/2024)
A Optimal Distribution in Targeted Negative Likelihood
Targeted negative likelihood optimizes for the distribution in p E Pk: that minimizes the reverse KL divergence with the original distribution: paew = min EP KL(r"Ipp)) This distribution is also known as the information projection of the original distribution onto Pk-
To solve for pdew, we use Theorem Property A from Khalifa al. (2020) (also Remark 3.1 Csiszár &
Shields (2004)). We first restate Theorem A from Khalifa et al. 2020 below for readability, and then reiterate the theorem using using notation consistent with the main paper.
To quote Khalifa et al.
20200::
To recap our formal approach, we have a finite set X, a distribution a over s.t. a(x) > 0, Vo X, and real functions +1,..., @h over X. We specify moment constraints 14 = i on distributions over X, where μ4 = Ezwe (i(x) and the ja's are given targets; the set of distributions satisfying these constraints is denoted by C. Our problem is to find p such that p arg mingec (k)(
Theorem 1. (A) There exists a unique solution p to the problem above, obtoined as p(x) P(x) where P exponential family form:
Xc] e, x10((z))
P(x) = a(x) 1/x E
(5)
In other words p(x) = 1/2 P(x), with Z ExEx P(x); is an unnormalized distribution, i.e, an EBM. Here Xc {a E X 3c E C s.t. d()) > 0} the "support set" associated with C. The Aj 'ss are real numbers called the natural parameters associated with the moments p4--
Reframed in the notation and setting of this work, we have the following: For distribution a(x), we have our original distribution (z). The support set Xe in this setting is the complement of supp(p"<s); thus, for 1[x E Xel, we have 1[x & su)p( The moment constraint Ex<e(ő(x)] is a pointwise constraint, namely that I Expp' (x) [xx & supp(p"*)]] = Then, since ((x) constant for g supp(y"es), we have
p new (x) x (z)(( & supp(p"ek) exp(Af(x))
(6)
x {x))(x & su)p(p"<s).
(7)
In other words, the optimal distribution removes probability at the negative tokens and renormalizes.
Experimental Set Up
B.1 Dataset Creation.
We use the XSUM dataset (Nrayan et al. 2018) for the reducing hallucination task and Civil Comments [Bokkan et al. 2019: for the reducing offensive phrases task. We use the datasets themselves for finetuning the base models and generations from the model itself for updating afterward. We descríbe both detail below.
Datasets for Initial Finetuning. For the hallucination experiment, use the XSUM train, validation, and test splits. The datasct sizes for train, validation, and test are 203,577, 11,305, and 11,301. For the offensive phrases experiment, we use the Civil Comments dalaset of toxic online comments. This datasel is traditionally used for toxicity detection, but here we repurpose the dataset for response generation. In particular, we train our encoder-decoder model output the text given its parent text, the previous comment the main text is responding to. For the main finetuning dataset, we use only examples that include parent text, decreasing the original dataset size from 1.8 million to million. We use the List of Dirty, Naughty, Obscene, and
17

---

Published in Transactions on Machine Learning Research (06/2024)
Otherwise Bad Words, downloaded from https://github.ccm LDNOOBW LIDNOOBW/IsttofDrryyWaaaaa Obscenee- Baw--aadddoobbboo
our list of offensive phrases to avoid. In the dataset of million examples, words from the aforementioned list occur in approximately 2% of the targets. To increase the relative proportion of examples with obscenity in the dataset that the model sees to 10%, we subsample the dataset further by randomly removing examples whose outputs do not contain any of the offensive words in the list. The resulting train, validation, and test (unused) sets are of size 175,754, 21,974, and 22,009.
Filtered Datasets for Finetuning. To generated the filtered dataset from XSUM (for both linetuning from stratch and from the current model), we use code from Nan et al. (2021) as an automated heuristic determine whether hallucination exists in the generated summary. The code uses spacy's 's named entity recognition to first locate set of entites from the output, followed by a regex- regex-based matching to determine if the entity is present in the source input.
Datasets for Finetuning Updates. After the initial models have been trained (see next section for training details), we generate an output from the model for each input using greedy decoding. We then take the model's outputs and annotate the undesirable tokens using the procedure described in the main paper. For hallucination if an entity delected in the output is not detected in the input, then we marked the entire entity as a hallucination, and our trained toxicity classifier labels a span in the output as offensive, we mark it as undesirable, even if the input contained the same offensive phrase. We use the train, validation, and test splits for the inputs but the model's own generations for the output. We evaluate all methods on the test set from this process, to compare how much these alternative methods result in deviations from the original model's generations.
B.2 Training
For all runs, we use a batch size of 32, dropout rate of 0.1, and no label smoothing. For all runs, the cross entropy loss includes the square the logsumexp of the logits as a penalty, scaled by a factor of 0.0001. For all experiments, we use Google Cloud TPU pods.
For the initial finetuning, we train base T5 model with learning rale 1e-3 and select the best checkpoint every 10,000 steps based validation loss. Our resulting models are finetuned for 30,000 steps XSUM and 40,000 steps on Civil Comments.
For the updates and alternative fineluning, we run a sweep across four different learning rates (1e-3, le-4, le-5, 1e-6) and choose the best model per every 1,000 steps based on validation loss. We run updates for a total of 100,000 steps for the T5 model, and 200,000 steps for the the PaLM-2 1b model. The learning rates used for the various methods are as follows:
C Results from retraining or finetuning on filtered data
Here, we present automated metrics of similarity and hallucination rate on T5-Base and PaLM-2 1b (Table 3 and Table g respectively), as well as a sample generation comparison, to highlight that while training on filtered data can reduce the prevalence of unwanted behavior, the resulting model is far from a minimal targeted update the original model. In contrast, the TNT update methods presented can enable more targeted changes to a model's behavior.
D Disfluencies introduced when using existing losses for negative signal
While other forms of disfluencies exist, notice two obvious forms in the existing model generations, which we denote word repcats and random ??. We define word repeat as the repetition of a single word multiple limes; note that this definition does not include phrase repeats, so the prevalence of repetition more broadly is likely higher what is reported under 'word repeats.' We define random ?? as the occurrence of any number of question marks precoding and following a space, meaning its presence is not the end of a sentence but rather in the middle. See Figure 7 and Figure g for examples of both word repeats and random ??.
18

---

Published in Transactions on Machine Learning Research (06/2024)
Dataset
Method
Learning rate
Filtered, trained from scratch
le-3
Filtered, trained from current
le-4
XSUM, T5 Base
UL + LL
le-4
NL + LL
le-3
TNFF
1e-3
TNRR
le-3
TNRF
le-4
TNFLL
le-4
TNRLL
le-4
Filtered, trained from scratch
le-4
Filtered, trained from current
le-6
XSUM, PaLM-2 1b
UL + LL
1e-5
NL + LL
le-4
TNFF
le-4
TNRR
le-4
TNRF
le-4
TNFLL
le-5
TNRLL
le-4
Filtered, trained from scratch Filtered, trained from current
le-4
le-4
Civil Comments, T5 Base
UL + L.
1e-3
NL + LL
le-3
TNFF
le-4
TNRR
le-3
TNRF
le-3
TNFLL
le-3
TNRLL
le-4
Table 2:
Learning rates chosen based best validation loss from sweep of learning rates (1e-3, le-4, 1e-5, 1e-6). Learning rates for updates were chosen from the runs with a = 1 and shared across all a values.
Table 3: A comparison of the original summarization model to ones obtained by retraining or finetuning on data filtered to remove examples with hallucinations. Model is T5-base.
BLEU
ROUGE-L
Seq Acc hallucination rate
Original
100.0000
100.0000
100.0000
21.3432
Filtered & Retrained
35.9402
54.4479
2.0087
9.3797
Filtered Finctunod
47.9839
64.0057
6.7339
13.6271
Table 4: comparison of the original summarization model to ones obtained by retraining or finetuning on data filtered remove examples with hallucinations. Model is PaLM-2 1b.
BLEU
ROUGE-L
Seq Acc
hallucination rate
Original Filtered Retrained
100.0000
100.0000
100.0000
25.3606
44.9260
61.2929
5.4951
14.0077
Filtered Finetuned
71.5848
80.8366
36.0765
22.9095
19

---

Published in Transactions on Machine Learning Research (06/2024)
Input
August 2016 Last updated 11:51 BST
The report was commissioned by former PM David Cameron in the wake of the Tower Hamlets election fraud scandal in 2015.
He said the review had heard evidence that in a London borough at the general election someone arrived at the polling station with 500 postal votes.
"I'm very worried don't believe political parties should touch or handle in any way a completed ballot paper," he said.
Original
review into the way postal votes are handled the UK has found that the electoral system not safe.
Retrained
A review into electoral fraud has found that voters were handed ballot papers with "significant" number of postal votes.
Finetuned
A review into electoral fraud has found that voter may have contacted the polling station with ballot paper.
Figure 6: Comparison of an example generation from the original model, a model retrained on filtered data, and the original model finetuned filtered data. Model is T5-base.
Input
The summertime routes were due to end in late September but due to poor seat sales, the airline is stopping the service at the end of August.
The airline said it had reallocated its planes to "routes with greater demand".
said it would offer "full refund or alternative flights" to customers who were booked to fly in September
"We apologise sincerely for the inconvenience caused," said an Aer Lingus statement.
Original
Aer Lingus has cancelled all flights between the UK and the Republic of Ireland due lack of passengers.
NL + LL
Aer Lingus is to stop flying to the the the the the the the the the the country is based on.
TNFLL
Aer Lingus has cancelled all flights between the two countries due to lack of capacity.
TNRLL
Aer Lingus has cancelled all of its summer flights from the country to the mainland.
Figure 7: Example of a word repeat. The original offending text, as well as the word repeat disfluency, has been underlined.
20

---

Published in Transactions on Machine Learning Research (06/2024)
Input
"Thanks for pointing out my error, anyhow." oiitiit_gt
Not problem and yes, knew what you meant just couldn't pass up a feeble attempt at humor
Original
I'm sorry, but I'm not a troll.
UL + LL
I'm sorry, but I'm not a 22 inbot.
TNFLL
I'm sorry, but I'm not a joker.
TNRLL
I'm sorry, but I'm not a 'oldgit!
Figure 8: Example of random ??. The original offending text, well as the random ?? disfluency, has been underlined.
21

---

Published in Transactions on Machine Learning Research (06/2024)
700
1200
NL + LL
Puctuation disfluencies
NL + LL
600
UL + LL
UL LL
Word repeats
1000
TNFF
500
TNFF
800
TNRR
400
TNRR
600
TNRLL
300
TNRLL
TNRF
TNRF
400
200
TNFLL
TNFLL
200
100
2.5
5.0 7.5
10.0 12.5 15.0 17.5 20.0
2.5
5.0
7.5
10.0 12.5 15.0 17.5 20.0
Hallucination rate
Hallucination rate
(a) XSUM word repeats
(b) XSUM random ??
600
NL + LL
Puctuation disfluencies
2500
NL LL
500
UL + LL
UL + LL
Word repeats
TNFF
2000
TNFF
400
TNRR
1500
TNRR
300
TNRLL
TNRLL
TNRF
1000
TNRF
200
TNFLL
TNFLL
500
100
0
0
0
2
3
5
8
Toxicity rate
Toxicity rate
(c) Civil Comments word repeats
(d) Civil Comments random
Figure 9: Number of disfluencies for different rates of unwanted behavior reduction. For each method, only models that are used in the BLEU similarity vs. reduction curves are plotted here (i.e., the models that are best at maintaining similarity with the original for given rate of reduction). This choice makes it easy to directly analyze this plot in conjunction to the main figure (Figure 2 From these plots, it is easy to see that baseline hodh introduce disfluencies at high rates they reduce unwanted behavior, where NL + LL lends to introduce repetition while UL + LL tends to introduce random ?? disfluencies. In contrast, the number of disfluencies introduced by TNT methods in both categories combined orders of magnitudes smaller than the total number introduced by baseline methods.
22

---

Published in Transactions on Machine Learning Research (06/2024)
E
Main Results with alternative similarity metrics
100
100
NL + LL
Best baseline
ROUGE-Lsum
UL + LL
80
Best TNT
ROUGE- Lsum
TNFF
ROUGF-ISU sum
TNRR
TNRLL
40
40
TNRF
20
TNFLL
20
0-
10
15
20
Hallucination rate
Hallucination rate
(a) Per-method Rouge-Lsum curves, XSUM
(b) Composite Rouge-Lsum curves, XSUM
NL + LL
Best baseline
Sequenie Accuracy
80
+ LL
Accuracy
80
Best TNT
TNFF
60
60
TNRR
40
TNRLL
TNRF
20
TNFLL
20
0
5
10
15
20
10
15
20
Hallucination rate
Hallucination rate
(c) Per-method Sequence Accuracy curves, XSUM
(d) Composite Sequence Accuracy curves, XSUM
100
100
NL
Best baseline
80
UL LL
80-
Best TNT
ROUGE-LSUm
TNFF
ROUGE-Lsum
60
TNRR
TNRLL
40
40
TNRF
20
INFLL
0
6
2
4
6
Toxicity rate
Toxicity
(e) Per-method Rouge-Lsum curves, Civil Comments
(f) Composite Rouge-Laum curves, Civil Comments
NL + LL
Best baseline
Sequence Accuracy
80
80
UL + LL
Best TNT
TNFF
60
60
TNRR
40
TNRLL
Sequence
40
TNRF
20
TNFLL
20
0
Toxicity rate
Texicity rate
(g) Per-method Sequence Accuracy curves, Civil Com- (h) Composite Sequence Accuracy curves, Civil Comments ments
Figure 10: Alternative similarity measures to BLEU (ROUGE, Sequence Accuracy) show the same trend the main result in Figure 2 'NNT methods yield a better trade-off than baseline methods between reducing unwanted behavior and maintaining similar generations to the original model.
23

---

Published in Transactions on Machine Learning Research (06/2024)
F
Training with model generations vs. external data
LL
Best baseline
80
UL + LL
80
Best TNT
TNFF
60
60
BLEU
TNRR
40
TNRLL
40
TNRF
20
TNFLL
20
Toxicity rate
Texicity rate
(a) Per-method, trained on model generations
(b) Composite, trained on model generations
LL
Best baseline
80
UL LL
80
Best TNT
TNFF
60
60
BLEU
TNRR
40
TNRLL
TNRF
20
TNFLL
2
8
Toxicity rate
Toxicity rate
Per-method, trained external data
(d) Composite trained external data
Figure 11: Similarity vs. toxicity rate reduction curves for the Civil Comments response generation task, comparing model finetuned its own generations (top, AUC is 73.9 for TNT vs. 32.9 for baselines) external data (bottom, AUC is 41.9 for TNT vs. 7.4 for baselines). The model stays closer to its original using its own generations for finetuning rather than the data it was trained on.
G Token-level KL-constrained RL a divergence minimization
Here, we show that a token-level version of the objective in Wu al. (2023) equivalent to mini- mizing reverse KL divergence between the policy model po (x|c) and target distribution Puew (x/c) (x(x c) exp(} (xx(()))):
max E E (x(c)) Ep(e)
re(xx1) (x) c) log (og
(8)
max E Epie) E (xio)
Σ loglexp(
r (xxx))))]
log log (xee, logpo(rg/eess,) c)
(9)
po(2s|2zc,c) c) log Po Po(2i c) exp( rx(x))))
min Ep(cc)
(10)
-min Fx Ex log II po(
(11)
Polagl=<e c) exp # (xc1))
(x(x(
-min Ex (xlc) log (x exp( ( E. (xx:))
(12)
min Bp(c) (L(o (po()))) (x/c))
(13)
24


Reviews:
[{"review_id": "jGFt3UEcFi", "reviewer": "Reviewer_tscQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n1. The method is straightforward and makes sense to me. By adding loss for constraining the output distribution close to the target distribution, the model should reduce the probability of generating negative samples while maintaining the original generative distribution as much as possible.\n\n2. The writing is generally easy to follow with descriptions of implemental details such as that Section 3 mentions adding $1e^{-6}$ for target distribution, and Section 3.2 mentions empirically the author found including those constituent distributions after a negative token helps constrain the model toward original.\n\nWeakness:\n\n1. While many related works are discussed in Section 4, why they are not considered baselines?\n\n2. Section 3.2 mentions including those constituent distributions after a negative token helps. From this point, I guess the performance can be boosted with more samples. Because it seems even those constituent distributions after a negative token can help. Thus, a sample complexity analysis can be added to ablation study.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper discussed related methods, particularly on NL+LL and UL+LL which do not constrain the target distribution close to the original distribution of a language model. The paper then proposes a finetuning method namely Target Negative Training (TNT) for pushing down the probability of undesired generation and maintaining the probability of the other generation similar to the original model using KL divergence. Meanwhile, the paper also proposed the data annotation for the pretraining. Finally, the paper shows the proposed method TNT outperforms NL+LL and UL+LL.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "As weakness.\n\nFor weakness 1, I regard minimizing unwanted behavior as a kind of human preference. These days there are many works related to reinforcement learning for human preference alignment. I think they can be potentially considered as baselines. On the other hand, the related work in this paper discussed related methods. Why not use them as baselines? I suggest the author at least add reasons for not considering them as baselines.\n\nFor weakness 2, the delivery of sample complexity, e.g., how many samples are needed to make the proposed method work, can benefit the following works and the community.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "y2Irz8okLM", "reviewer": "Reviewer_yKYh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. The paper studies an important challenge that exists in alignment fine-tuning: we want to align the model to avoid outputting unwanted phrases but we want to maintain the model's original behavior on normal text.\n2. The proposed method poses a more fine-grained control on mitigating the negative outputs while keeping the model otherwise the same.\n3. The paper is well-written and easy to follow. It discussed the reasons why the previous approaches do not work well on addressing the challenge and then introduced the new method that can better address the challenge.\n\n**Weaknesses**\n1. The results are not very comprehensive and the settings need to be further clarified. See \"requested changes\" section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the method to update the LM to avoid unwanted outputs while minimally changing model behavior otherwise. The authors propose a new fine-tuning strategy called Targeted Negative Training (TNT) which minimizes the divergence between the model distribution and desired distribution for every token. The idea of annotating the negative tokens w.r.t. a certain range of context is very interesting and more desirable compared to the previous approaches that treat all tokens in the sentence as unwanted.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Is \"random ??\" a compilation error?\n2. The results compares the best TNT performance with the best baseline performance and demonstrate the proposed method is better than the baseline. However, we don't know how to choose the optimal method in practice, right? I wonder if the author could do some separate plot and explain which method people should choose in practice, especially in the case where the baseline curve outperforms the TNT single method curve at small toxicity/hallucination rate.\n3. Could the author explain how they annotated the data with unwanted spans? Is it an objective or subjective process? Any metrics?\n4. I don't quite understand the x axis in the figures. For example, in Figure 2, are the rate of percentages? Why the toxicity rates of (a), (b), are of the same scale but the rates of (c) and (d) are different? (It's 0-8 for (c) but (0-0.08) for (d), which I guess is a typo.)\n5. Assuming the Figure 2 shows hallucination rates in percentages, I wonder if TNT only outperforms the baselines at larger hallucination rates. If so, Table 1 only shows comparison at a hallucination or toxicity reduction rate of 75% is beyond the limit of the Figure 2 that is supposed to have great benefits. I wonder if the author could provide table numbers for smaller rates (which is more desirable in practice), e.g., at 5%, 10%, and 15% hallucination rates and 1%, 2%, and 4% toxicity rates.", "broader_impact_concerns": "I think the way the authors define toxicity may need a statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4eTTbPvJCS", "reviewer": "Reviewer_Gka3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The proposed method is simple, widely applicable, and a timely and important contribution.\n- The motivation and explanation of problems with prior work attempting to do the same is excellent, a pleasure to read and informative.\n- The description of the method is well-written and properly contextualised w.r.t. prior work.\n- The authors experiment with multiple domains, compare to two baselines, and show that their method significantly improves over the baselines for most levels of unwanted behaviour reduction for two types of annotated data (model-generated and existing).\n\n**Weaknesses**\n- For a proposal of such a generally applicable method, the experiments section of the paper seems a bit of an afterthought. The experiments don't come very close to the generative models that are most used currently; decoder-only models with many more parameters. I understand that fine-tuning models with more than 50B parameters is unfeasible, but at least fine-tuning models with two different sizes to see how the method scales would make sense to me. Additionally, it would be interesting to see the same experiments done for a decoder-only model.\n- Given the well-documented downsides of automated evaluation metrics like BLEU and ROUGE (and basically any ground-truth overlap metric), it would be insightful to see results of a (small) human evaluation study. Particularly since one of the motivations of this work is that prior methods cause outputs that correlate with unwanted behaviour to be affected as well, and that this method should mitigate that. ROUGE and BLEU doesn't highlight whether such a goal is achieved, but human evaluation could. That said, the BLEU/ROUGE increase w.r.t. the baselines is quite large for large amounts of reduced unwanted behaviour, so I'm not too worried about this, it would just be interesting.\n- Some discussion about the increased cost of this method over unlikelihood training would be insightful, since it does introduce an extra forward pass which could be non-neglible for larger models.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes a method to update a trained generative model in a targeted way to stop if from outputting unwanted behaviours, while not allowing the rest of the output distribution to change too much. Such a method has obvious and important applications; unwanted behaviour of LLMs for example show up constantly, and re-training or fine-tuning a model to mitigate this can have negative impact on the generation quality plus is expensive. I expect the method proposed in this work to be very valuable to the community and hence this work is a timely contribution.\n\nThe method is motivated in great depth, and the proposed targeted update method tackles most of the issues with prior methods that aim to do the same. The authors experiment in the domains of reducing hallucinations in summarisation and reducing toxicity in response generation. They show for a encoder-decoder model of 220M parameters (T5) that their method outperforms baselines (unlikelihood fine-tuning) for most levels of reductions of unwanted behaviour, by quite a large margin (much higher BLEU/ROUGE for high levels of reduction).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Would strengthen the work: experiments with one more model of a different size (e.g. 1B parameters)\n- Would strengthen the work: experiments with a decoder-only model\n- Would strengthen the work: human evaluation of the method compared to a baseline\n- Would strengthen the work: discussion about the increased computational complexity of this method over baselines", "broader_impact_concerns": "N.A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes Targeted Negative Training (TNT), a method for updating language models to avoid unwanted outputs while minimally changing the model's behavior. (Similar to the recent RLHF setting, but only with negative samples.) Most of the reviewers found the method well-motivated and empirically validated, recommending acceptance. The authors addressed the reviewers' concerns by adding additional experiments (with a 1B LLM) and clarifying their methodology. Overall, the paper is a valuable contribution to the field, offering a practical solution to a common problem in LLM fine-tuning.
Decision: Accept (poster)