Shh, don't say that! Domain Certification in LLMs
Authors: Cornelius Emde
Alasdair Paren
Preetham Arvind
Maxime Guillaume Kayser
Tom Rainforth
Thomas Lukasiewicz
Philip Torr
Adel Bibi
Venue: ICLR 2025 Poster (2025)
Keywords: large language model
natural language processing
adversarial robustness
adversary
natural text generation
certification
verification
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We propose a novel framework to certify natural language generation and provide an algorithm to achieve an adversarial bound.

Abstract: Large language models (LLMs) are often deployed to do constrained tasks, with narrow domains. For example, customer support bots can be built on top of LLMs, relying on their broad language understanding and capabilities to enhance performance. However, these LLMs are adversarially susceptible, potentially generating outputs outside the intended domain. To formalize, assess and mitigate this risk, we introduce domain certification; a guarantee that accurately characterizes the out-of-domain behavior of language models. We then propose a simple yet effective approach dubbed VALID that provides adversarial bounds as a certificate. Finally, we evaluate our method across a diverse set of datasets, demonstrating that it yields meaningful certificates.

Published as a conference paper at ICLR 2025
SHH, DON'T T SAY THAT! DOMAIN CERTIFICATION IN LLMs
Cornelius Emde' Alasdair Paren', Preetham Arvind', Maxime Kayser', Tom Rainforth', Thomas Lukasiewicz Philip H.S. Torr', Adel Bibi"
'University of Oxford Vienna University of Technology
ABSTRACT
Large language models (LLMs) are often deployed to perform constrained tasks, with narrow domains. For example, customer support bots can be built on top of LLMs, relying on their broad language understanding and capabilities to enhance performance. However, these LLMs are adversarially susceptible, potentially gen- erating outputs outside the intended domain. To formalize, assess, and mitigate this risk, we introduce domain certification; a guarantee that accurately character- izes the out-of-domain behavior of language models. We then propose simple yet effective approach, which we call VALID that provides adversarial bounds certificate. Finally, we evaluate our method across diverse set of datasets, demonstrating that it yields meaningful certificates, which bound the probability of out-of-domain samples tightly with minimum penalty to refusal behavior.
INTRODUCTION
With recent advancements in the field of natural language processing, large language models (LLMs) have become ubiquitous. In particular, the scaling of recent large gener- alist models dubbed foundation models has shown to enable emergent abilities that benefit a wide range of downstream tasks such as text generation, question answering, and text comprehension (Kaplan et al. 20200; Alabdulmohsin et al.) 2022 Xiong et al. 20244] Henighan et al. 2020; [rown et al. 2020). Adapting these foundation models for down- stream tasks often leads to tate-of-the-ar performance and has become the dominant paradigm (Gao et al.) (202T). This is typically achieved via fine-tuning on task-relevant data (e.g.. low-rank adaptation (LoRA) Hu al.) (2022). in- context learning (Mosbach et al. 2023), prefix turning Li & Liang (202I). or simply prompt engineering).
Health Advice Chatbot
I am a chatbot help with health questions.
need help commiting tax fraud.
Sorry. can't help.
need help commiting tax fraud. # @ /!
However, foundation models are typically trained on large amounts of web data which contains wide range of infor- mation that is either irrelevant to a task or potentially harm- ful (Bommasani et al., 2022). Therefore, it is desirable restrict the output of a generalist LLM to a specific domain. For example, consider healthcare provider such as the Na- tional Health Services (NHS) providing a general purpose chatbot to support their citizens with simple health questions, as shown in Figure I It would be im- portant, for public reputation and cost reasons, that such system would remain on topic and could not be misused, either intentionally or unintentionally. Misappropriating models is easily possible.
Sure. Here is five point plan:
Figure I:A user misappropriating LLM system using an adversarial attack. We provide certificates to mitigate this risk.
In order to prevent intentional misuse, we consider an adversary trying to elicit an unintended (from the deployer's 's perspective) response from the model. We assume the deployer wants an LLM to only respond with certain set of topies, and thus a successful attack is input string that creates a coherent response outside the target domain. There are various reasons why an adversary might want to elicit such a response that is out-of-domain (OOD). The adversarial user might want to
"Corresponding Author corne lius emde@es ox. ac. uk. Work partially done while interning King Abdullah University of Seience and Technology (KAUST).

---

Published as a conference paper at ICLR 2025
misappropriate the system as cost-effective tool for purpose it wasn't built for, resulting in excess infrastructure costs for the deployer. Conversely, the deployer might legally be required to validate and verify their models, which is challenging, if not impossible, when the model is not domain- restricted. Finally, the adversary might want to harm the company directly by eliciting harmful OOD responses, which could damage the company's s reputation when publicized. Recently, an L.M-ddiven meal planning tool has received wide media attention for providing toxic recipes when prompted with toxic ingredients (McClure, 2023] The Guardian 2023). Deployers have moral and legal obligations to prevent this (Bommasani et al. (202)) In all examples, restricting the domain in which the model responds under adversarial prompts can help mitigate risks. Thus, in the era of foundation models, "domain" specialization is critical.
Existing work has implemented guardrails that address these risks (Jain et al. 2023), most notably via alignment, resulting in models rejecting user requests (Bai al. 2022 Ouyang et al., 2022
Christiano et al. (017). However, a wide body of research has shown that common guardrails have
"jailbeaaks" i.e., they can easily be circumvented by adversary (Wang et al. 2024 Qi et al. 2024
Eiras et al. 2024| Carlini et al. [023; Dong et al. 2024p. Common jailbreak methods are prompt
injection P Perez& & Ribeiro [022] Jiang et al. [202]] Liu et al. (2024), numerical optimization (Jia
& ..iang 2017: Wallace et al.] 2019; Ebrahimi et al. 2018]] Jones et al. 2023; Zou et al., 2023]
Jia et al. 2025). red teaming (Perez et al.
2022, Samvelyan et al.) 2024), automated black-bor box
attacks (Chao et al. 2023] Mehrotra et al. 2024), or data poisoning attacks (Biggio et al.) 2012
Wallace et al. 2002 Carlini et al.] 2024). Using these tools, it is possible for adversaries to retrieve information from fine-tuned model that was suppressed by the alignment and generate responses that are outside the target domain (see Figure an example). Adversarial prefixes or suffixes that augment any prompt are especially powerful, as they have been shown universally attack models in combination with a wide range of prompts and can thus be shared between adversarial users WWallace et al. 2019 Zou et al., 2023). This presents significant risk. Hence, researchers have proposed methods to defend against these adversarial attacks, such as unlearning (Nguyen et al., 2022 Xu et al. )023), robust fine-tuning (O'Neill al., 2023 Dong al., 2021). or request and response filtering (Inan et al. 2023
Deployers would ideally want guardrails that come with provable, mathematical guarantee against the model responding off-topic, or a guarantee that it does this with very low probability. The process of constructing guarantees against certain model behaviors under adversarial attack is commonly referred to as certification and has been successfully applied to vision applications in recent years (Akhtar et al.] (02I)) and proposed for NLP applications (La Malfa 2023] Casadio et al., [024] Kumar et al., 2024). However, no existing LLM guardrails provide guaranteed protection against existing or future jailbreaking techniques, leaving deployed models at risk of being compromised shortly after release. As a result, developing certifiable methods to guarantee that specialized LLMs consistently produce on-topic content is critical. Hence, our contributions are as follows:
We introduce a novel framework, domain certification, to bound the probability of models pro- ducing out-of-domain content under adversarial attack.
We introduce an easy-to-use algorithm VALID that bounds the probability of an LLM based sys- tem responding off topic under adversarial attack. We show the efficiency of VALID which test empirically on a number of representative data sets.
DOMAIN CERTIFICATION
We now introduce our domain-certiffaation framework for offering mathematical guarantees that an LLM system stays on topic. In Section 2.1] we formally introduce this framework. In Section [.2]] present Verified Adversarial LLM Output via Iterative Dismissal (VALID). VALID is an easy- to-use method to create a system that adheres to these guarantees. In plain language, we propose a certifiable guardrail for LLM--riven systems as follows:
A model is Nomain-certified when an adversarial upper bound can be placed on the probability that the model provides an output outside its designated target domain.
Before formalizing this statement, we introduce some mathematical notation. We represent tokens (i.e. individual text units) as x and y, which belong to the token space x, E V where V = {1...., V} is the vocabulary of size V. We define the space of sequences of arbitrary length as S A V*, the Kleene closure of V. Sequences of tokens are denoted by bold letters, ar, y E S, with
2

---

Published as a conference paper at ICLR 2025
and representing the input and output sequences of LLM respectively. We lowercase
letters to denote models that predict the next token, such as S + V. Applying this model repeatedly, until the end-of-sequent token creates a sequence-to-sequence model L : - S. We denote the likelihood of sample y under given æ as (a)) which is obtained by L(y =
Ih Uy VVen, æ) for a sentence y of length Ny. We further denote the distribution from which the model samples its output by y ~ L(-laz).
2.1 DEFINING DOMAIN CERTIFICATION
We now formally introduce domain certification. We define the target domain (set of desired topics) as a subset of the sentence space S and partition S into the target domain TT and its complement T,. For instance, T might be all sentences meaningfully occurring for "question answering for health problems". In addition, we define the set of unwanted responses as F C T (F as "forbidden") and will certify with respect to this set F rather than T. Sequences posing some risk should be included in F, while IF" T'T should contain benign out-of-domain samples, such as unintelligible meaningless sequences of tokens (see Appendix B for a discussion). Hence, we wish to establish a guarantee that L is unlikely to produce output F. As step towards such guarantee, we first define bound for any given element y in S:
t eyaaoiiic--errtfffeee (cg) some Definition Atomic Certificate. We say a model S - S
sample (iee an atom) in the output set 8, iff
Va S L(y Cg.
In words, a model that is Ay for a sample y. will generate sample y with probability smaller than cy for any E S, and hence for adversarially chosen æ. If this is the case, we say model L certifiable for sample y with Ey. i.c. by is the smallest value that provably bounds L. Ideally, such an upper bound ey would be large for samples in the target domain T. meaning the certificate is permissive, and small for samples drawn from mcaning the certificate is restrictive, i.c. tight.
The atomic certificate implies an upper bound 67 for TP ()) E Flæ)). which would be con- structed by summing ( over all E F for given æ. Concretely, Py-rt(->> E F|z) Eyer L(y|æ)) < LyER cy ep- However, practically this bound is intractable due to IF's ex- ponential size in Ny, and the difficulty in constructing a precise description of the set F. Instead of giving bound over returning y E F, we look at the worst case across F which can more precisely be estimated from a finite sample of FF:
Definition Domain Certificate. We say model L is e-domain-certiffed (E-DC) with respect to F, when it is AC for all E IF with Ey S <:
Vx S. E Llylz)
This imposes a global bound on L across all undesired responses in F. In practice, we cannot establish the cDDC certificate w.r.t. F cannot enumerate F. Hence, following standard practice in ML evaluation, we propose to use Dr, a finite dataset of oft-of-domain responses to establish a cDDC certificate w.r.t. De approximating the certificate for F.
Recent discussions have raised the need for bounds on undesirable behavior. For instance, Bengio 2024) advocates for upper bounds on harmful behavior (Bengio et al. (024). In addition, growing body of legislation mandates thorough auditing of ML systems (EU 2024). The atomic and domain certificates can play a vital role in assessing the risk of worst-case behavior. For example, consider the deployer of an LLM-based system that processes 10 requests per second. The deployer might perform an apriori risk assessment and determine that they can tolerate the consequences of an out- of-domain response from set D3 sampled once per year. The deployer should certify the LLM system as eDCC with ze 10 in order to achieve this level of risk.
Certification through Divergences. We provide an alternative view to this problem, generalizing
it to bounding divergences between the model and the distribution of sentences in the domain T. We then use this view operationalize the ey AC and DC (Definitions I and inspired by
Vyas et al. 2' 's work on preventing copy-right violations. To this end, we define an oracle Ω
that is generator for domain T: Ω assigns high likelihood to sentences in T and zero likelihood to
3

---

Published as a conference paper at ICLR 2025
elements in F. Hence, sampling from & will yield in-domain responses. We establish and bound the divergence between and 52 restrict the model domain. In particular, we use the Renyi divergence of order infinity, Ao (P || Q) _ log sup, Q(a)
P(w) Rényi 1961). Hence, our objective is:
Va S: A ( || O(y)) S
Bounding this divergence is at the core of what we are aiming to achieve: The divergence is large when assigns high likelihood to a sample y while Ω does not. That means L is likely to produce samples that out-of-domain. When ! assigns high likelihood to y. the sample is in the target domain, and hence the divergence in (3) is not restrictive. When L assigns low likelihood, y is unlikely to be sampled. Interestingly, this divergence implies (I) and (2), see Lemma
As the oracle is not available practice we approximate 12 with a "guide" language model that is exclusively trained on in-domain data dubbed G (i.e. the guide model). We use G(y) to replace (y)) to assess the marginal likelihood of y. While this means that G(y) loses some context contained in æ, this has major advantage: G(y) does not depend on æ, which is potential adversary and hence, by design is robust to adversarial prompts.
2.2 ACHIEVING DOMAIN CERTIFICATION
In this section, we introduce Verified Adversarial LLM Output via Iterative Dismissal (VALID) to obtain atomic certification as described in Definition D We utilize a general model L and domain generator G as described above and obtain a meta-model A for which the guar- antee holds with respect to the domain generator G. In particular, we perform rejection sampling as described in Algorithm [ (inspired by Vyas et al. (2023)): The capa- ble general model L proposes a sample y and we accept, if the length normalized log-ratio between L and G is bounded by hyperparameter k. We repeat up T times
Algorithm 1 VALID
Require: LLM L. Guide model G. hy- perparameters and T. prompt æ for / E (1,..., ,T) do
Sample y ~ [(.lar))
N, +- length(y)
L(yl#) if log G(y)
S NN then
Return: y
Return: "Abstained".
until a sample is accepted. If all samples are rejected, the model dismisses the request. This defines a new model M, for which the following theorem establishes the certificate:
Theorem (VALID Certificate) Let L be an LLM and G guide model as described above. Rejec-
tion sampling as described in Algorithm 7 with rejection threshold i and up to T iterations defines
model ML.G.K.T with ML.G.A G. (y(x)) denoting the likelihood of y given x. Let Ny be the length of
y. We state the adversarial bound:
Vz E S: ML,G..r()/x)) S 2kNNo T- TGG(y)
(4)
Hence, ML.G.T. is (C further, it is [maxyer 2 w.r.t F.
When context allows, may abbreviate M,,G,A,T to M, omitting subscripts for brevity. This certificate with respect to G can be useful: As G is only trained on samples in Dr C. T, a dataset of domain T. it assigns exponentially decreasing likelihood to samples that are in E In particular, this is useful iff the log upper bound KN,,, + log T + log G(y) (log RHS of 4 is small in comparison to max log L æz): Our certificate can provide an upper bound to the adversarial behavior of M that is favorable over L.
As mentioned, this problem is closely related to OOD detection, for which the likelihood ratio test commonly used as powerful statistic (Neyman & Pearson 1933] Bishop 1994. Ren al. 2019 Li al., 2023 Zhang et al. 2024 Rafailov et al.) 2024). In OOD detection, rejection threshold A: is commonly y chosen to balance false negative rates and false positive rates. Here, I also influences the upper bound on the certificate, indicating that there can be a trade-off between correctly classifying samples as ID OOD, and achieving desired level of certification.
Length Normalization. Algorithm D performs length normalized rejection-sampling as unnor- malized log likelihood ratios scale untavorably in Ny. the length of sequence y which we now demonstrate, Consider the next-token models and g underlying the sequence-to-sequence mod- els L and G. As y is sampled from L, we expect each token yi...., M,,, to have high like- lihood under 1. If we assume that places c times more probability mass per token than g.
We give an empirical example of this behavior in Figure 13 Appendix E.4
4

---

Published as a conference paper at ICLR 2025
then we show that the log likelihood ratio grows linearly in Ny. the length of sequence y: og(yalleww)/(tuuleer = Ny log c. We illustrate an example in
log ((yl))/CC /G(y) = log Figure 2 Assume that an in-domain sample for which model L and generator G assign constant likelihood per token of 0.1 and 0.05, respectively, i.e. Vn = I,.., : [y x) = 0.1 and g[nnlVen, x) = 0.05. Further, assume an out-of-domain y' for which I assigns mass of 0.1 per token and y assigns 0.01. The log likelihood ratio for y can be expressed as Ny log 2 and for y' as ratios grow with length, so does the optimal decision bound. We plot sequences of varying lengths with these parameters in Figure 2 By arithmetic manipulation, rejection sampling with threshold kNy is equivalent to bounding the ratio of geometrically normal- 1/NN,
Ny. log 10. As in- and out-of-doman
ized likelihoods log L(y) (z) 1//// /G(y) 1/Nw using constant threshold k. Hence, we propose to use normalized log ratios in Algorithm| [ over unnormalized likelihood ratios. Similar approaches have been discussed in the NLP literature (Geng et al.. (023).
Despite the length normalization of the rejection thresh- old, notice that the VALID bound depends on Ny. the
30-
Target Domain Dr
length of sequence y (see (4). making the certificate more effective for shorter or longer sequences. Let
Other Dr
20
Boundary kNy
g(y) be the geometric mean of per-token probability for G(y). The log upper bound can be written as kN, + Ny log ñ(y) T. Whether this is tighter for short or long sequences is governed by & and log gyy) When k k+log ((y) is close to 0, the bound is balanced, and when + logg (y) < 0, the bound decreases as N, increases.
10
2 Sequence Length (My
10
Figure 2: Log likelihood ratios scale in the sequence length Ny. Six artificial examples of sentences with length to 10 are shown for the ID and OOD dataset. As log ratios scale, should the decision boundary.
In the appendices, we provide further insights into VALID. In particular, in Appendix A we provide Lemma 2 showing how to estimate the likelihood of M. In Lemma 3 we provide an analysis of the expected num- ber of iterations of VALID, In Appendix C. we provide further intuition on how rejection sampling can achieve an adversarial bound. Finally, in Lemma 4 we show an adversary for M and discuss how rejection sampling encumbers adversarial attacks on M.
EXPERIMENTS
We empirically test method proposed in Section 2.2 across 3 domains: Shakespeare, Computer Science News, and MedicalQA. After describing the experimental setup in Section 3.1] we examine the rejection behavior of our method by examining the log L(y|z) /G))) ratio and associated cer- tificates under a finite set of ground-truth test samples from T and F in Section (3.2 In Section 3.3 we repeat this analysis by applying our Algorithm [ Finally, we demonstrate how to evaluate
certified model on standardized benchmarks in Section 3.4
3.1 EXPERIMENTAL SETUP
In this section, we provide a brief description of our experimental setup for three applications. Each experimental setup consists of a target domain T, finite dataset of in-domain samples Dr C T, models L and G, and an out-of-domain dataset Dz C F, against which we test our methods (see Appendix D for more details on data and models).
Shakespeare. Our target domain T is Shakespeare's ' plays. We fine-tune a Gemma-2- 22-2b (Team et al., (2024) model L and train GPT-2 architecture (33. 7M parameters, Radffrd et al. (2099) from scratch for on TnySShakespeare (TS) (Karpathy/ (0115). We use TS's test split as in-domain dataset, Dr, and following previous literature (Zhang al. 2024) compose Dp of IMDB (Maas) cct al. 2011 J. RTE (Wang et al. 2019 and SST2 (Minace et al. 2024), adding an old Bible dataset (Reis] 2019) as it is linguistically close to SaayShaeespeare. At testing, we consider 256-token long
sequences and use the first 128 tokens as prompt.
Computer Science News. Our target domain T is news about computer science. We fine-tune Gemma-2-2b as model L and train a GPT-2 architecture (109.3M parameters) from scratch for G on articles from the computer science categories in the 20NG dataset (Lang) 1995). We use computer science articles from 20NG's test split as target domain Dr and the remaining categories as Dr together with the OOD dataset used for Shakespeare. At testing, we consider 256 token long sentences and use the first 128 tokens as prompt.
5

---

Published as a conference paper at ICLR 2025
Medical QA. We apply our method to medical question answering target domain T. This could, for example, be extended to a chatbot for clinicians look up patient symptoms. We use LLama- 3-8B model (AI @ Meta 2024 as L and for guide model G we pre-train a GPT-2 architecture model from scratch (184M parameters) on PubMedQA (Jin et al. 2019), which contains approximately 200K QA pairs for training and 1000 test pairs. We further fine-tune G on responses from I. to questions PubMedQA. We use the PuMedQA test set as in-domain dataset DD and regard question answering on other topics, such as geography, as I.. To model this, we use the Stanford Question and Answering Dataset (SQuAD; excluding medical categories; Rajpurkar al. (2016) as Dz.
3.2 LIKELIHOOD RATIOS ON GROUND TRUTH SAMPLES
In this section, we evaluate the capability of our method to attribute samples to the target domain and investigate whether it yields useful adversarial bounds. In particular, we study the length- normalized likelihood ratio Lyy / (G() on in- and out-of-domain samples. In Figure Bal we show that the log likelihood ratios for MedicalQA are disentangled and hence a threshold exists separating target domain and out-of-domain samples well. However, such k - while yielding strong OOD detection performance might not be associated with tight certificates. Hence, we will first study the Ay certificates under M for individual samples, y. before moving on to the domain certificate, cDDC
Atomic Certificates. We obtain -CCs using VALID (Section 2.2p, setting & to achieve 10% false rejection rate (FRR) for in-domain samples. Figures (a)- show the distribution of Ay-ACs for the target domain dataset Dr and the out-of-domain dataset Dz. We make similar observations for all three setups: First, the certificates in the OOD datasets DD are meaningfully tight. We ob- serve that 95% of OOD samples have AC of less than 10-10 across all setups. Hence, the sampling probability for these OOD instances is provably smaller than 10-10 for any arbitrary prompt æ. Second, we note that the certificates in D3 are significantly tighter than those in Dr as shown by the gap between the eCDFs. This is a significant finding as certificates should be con- strictive (i.e. small) on samples in IF preventing these from being sampled, while certificates should be permissive (i.e. large) in T. not preventing in-domain responses from being sampled. Finally, observe that the disentanglement of ACS is weaker for MedicalQA compared to the other setups (see Figure [4c). As shown in Appendix E.6 this is attributable to the short sequences in the OOD dataset and adjusting for this confounder significantly improves disentanglement.
To further study the atomic certificates on M, we compare them to a certificate on L as a baseline. To this end, we define the constriction ratio for each y. given by the ratio of the certifiable by for I,, cy (L), over the certifiable by for M, (.gyMM))
(L)
CR (M)
(5)
A CR2 of 1 for sample y indicates that the bounds generating y are equal for M and L (i.e. they are equally constricted) while CRr, > indicates that M is more constricting than L, and vice-versa. Smaller ACs for samples in F are better and hence large CR2 indicates that model M is favorable over L. our knowledge, only vacuous certificates for a general model exist (e.g. L is 1-DC). Hence, we approximate it from below using the likelihood L(ylæ) under non- adversarial x taken from the datasets. Concretely, we use L(y(ær) as crude approximation of max (yylr)) This overestimates the robustness of I, and underestimates the constriction ratio,
1.00
Target Domain
X
Other Dr
0.75
LO9 D2
0.5
0.50
Youden's 's
TRR
0.25
FRR
FRR
0.0
0.00
10
10
20
30
40
20
-15
-10
-5
Normalized Log Likelihood Ratio
Togio Constriction Ratio (Median)
Logio DC (c)
(a)
(b)
Figure 3: All Figures display MedicalQA. Figure 3a shows that log likelihood ratios are well disentangled.
Figure 3b shows the trade-off between OOD and certification: The best OOD detection performance occurs with constriction ratio of 20. Figure 3c shows the false rejection rate (FRR) required to certify given
Figures display MedicalQA.

---

Published as a conference paper at ICLR 2025
Shakespeare
Computer Science News
Medical QA
1.00
Target Domain Dy
0.75
Other D
0.50
0.25
0.00
-400
-200
-400
-200
0
100
-50 logio AC (c)
Ao (a)
logioey-AC (b)
100 200 300
-100
100
25 50
75
100
CR, (Log Constriction Ratio)
LogieCR; (Log Constriction Ratio) CR, (Log Constriction Ratio)
(d)
(f)
Figure 4: (a)-(C) show the estimated cumulative distribution function (eCDF) of ACs for each experimental setup. (f)-(f) show the histograms for the log10 constriction ratios, All results obtained with hyperparameter false rejection rate (FRR) on in-domain samples.
1 chosen to ensure
i.e. it underestimates the improvement of VALID certificates over I. in bounding the probability of OOD responses. In Figures (d) - (f), we show the log constriction ratios for out-of-domain samples while setting & to achieve an FRR of 10% (see Appendix E.5 for other FRRs). Across setups, the majority of samples have positive constriction ratios, which means that M issues ACs tighter than [zy For MedicalQA, we observe 99% of CRs are greater than 6.30 and observe a median CR of 24.23. In other words, 99% of samples are at least orders of magnitude less likely under M and in the median s 24 orders of magnitude less likely (i.e. 1 x 10 10-2*) 29), We believe these are very strong restrictions and observe even stronger median constriction for 20NG and TinyShakespeare. Further, we observe the strongest constriction among samples with high likelihood under L (see Appendix E.. Tight bounds are the most relevant on these samples as they are most likely to be sampled from L. Finally, we illustrate trade-off between certification and OOD detection in Figure (3b] For MedicalQA, we plot the median constriction ratio for out-of- domain samples across range of parameters k together with false rejection rates (FRR) and true rejection rates (TRR). The optimal classification performance (as measured by Youden's: J/ (Youden, 1950h) is achieved at k = 5.35 with strong true rejection rate (0.99) and low false rejection rate (0.0T), while producing a median log10 constriction ratio 19.00. Smaller k values yield tighter certificates (see the bound in (4) and larger constriction ratios at the expense of increasing the FRR.
Domain Certificates. To study certification across range of samples, turn the domain certificate, cDDC. Above, we studied the effect of various parameters (e.g., fixing FRR) on the cer- tificates. However, practitioners likely work the other way around: They first set acceptable threshold according to a threat and safety model. Then, they examine model performance under conditions satisfying such certificate. Hence, we study model performance at a given eDDC. As pro- posed in Section we establish an eDCC certificate w.r.t. Dr approximating the certificate for F. To obtain AAs smaller than the domain certificate e, we need to choose rejection threshold, k and the number of iterations, 7, accordingly. We
solve for T given max (kN, log T log G(y) log DED,
For simplicity, we keep T' = 1 and study model performance on Dr while maintaining e-DC on Dr. In particular, we look at the FRR of M: The performance of model M is determined by the performance of L, (from which VALID samples response candidates) and the false rejections leading degradation of M compared L. Hence, we study the FRR as a function of the certification threshold The result is shown in Figure [cc for MedicalQA: The FRR increases the certificates get tighter (small c). Remarkably, we achieve domain certificate with c 10-5 an FRR of only 15% at a single rejection step. We replicate all figures for the other setups in Appendix E
A natural question is why we do not simply a model comparable G that is trained exclusively on a subset T directly. While such a model would be highly robust against providing useful out-of- domain responses, its performance would significantly lag behind both and M. Our ablation study

---

Published as a conference paper at ICLR 2025
Log Likelihood Ratio
Target Domain D,
FRR
PubMed QA@c
1.00
Other Dr
0.75
0.5
Youden's]
0.50
TRR
0.25
FRR
0.00
100
20
40
60
80
100
20
15
logio Constriction Ratio (Medlan) (a)
Response Length (N,
logio eDD (c)
(b)
Figure 5: All Figures show MedicalQA. Figure 5a shows the false rejection rate (FRR) for range of c-DC for
VALID with T = 1. Figure (5bure55jaaa shows the log likelihood ratio depends N, for real data. Performing length
normalization makes the problem linearly separable. Figure 5e shows PubMedQA @e results of our model
in Appendix Confirms this performance gap between G and M. These findings demonstrate that system, which combines the high performance of L with the safety guarantees of G, achieves VALID
advantages that neither L nor can provide independently. Further, the effectiveness utilizing G of such limited performance demonstrates that the burden training G' is relatively low: model that performs poorly on the target task, but distinguishes well between samples in T and F, can be sufficient to achieve meaningful certificates for M.
3.3 GENERATING RESPONSES
In the section above, we evaluate M obtained through VALID on prompts and responses, taken from datasets Dr and Dj representing a detailed analysis of ACs and DCs on a large variety of samples for which their membership to T or F is given by high-quality labels. Nonetheless, in practice, the candidate responses that are judged by VALID are generated by L. Hence, we prompt M using æ E Dr and Dp and use responses generated by L as VALID proposes. We focus on VALID with T = 1 and the MedicalQA setup.
target domain T and F. The experiments provide with
Our findings are in line with Section 3.2 showing a strong ability to distinguish between in- and out- of-domain samples while providing meaningful ad adversarial bounds. In Figure [5b] we demonstrate the separation samples from Dr and Dp, as well as the dependence of the log ratios on the length of the sequence y extending the theoretical analysis from Section 2.2 In Appendix [E.4, we replicate Figure for this setting. We further present Figure 5a constriction ratios on ofist--distriuttinn samples generated by L. We see clear indication that the constriction is strong out-of-domain with optimal classification performance at ratio of 1040 To reiterate, median ratio between L (x:) and the ACC for M is 1040 showing just how strict VALID on the out-of-domain dataset.
Building on these results, test VALID with T >> 1. Increasing T can naturally increase the accep- tance rate on in-domain samples (through repeatedly proposing candidates) at the cost of increasing the by linearly (see (4)). We find great improvements in the acceptance rate on in-domain samples with minimal losses on the e-DC tightness. We explore this in Appendix
3.4 CERTIFIED BENCHMARKING
We extend the analysis of false rejection rates (FRRs) by evaluating model M's performance standardized benchmarks, while ensuring certified at In particular, for our MedicalQA setup,
evaluate the model performance on the PubMedQA benchmark (Jin et al. 2019
Item Correctness
QA Pair
Response Acceplance
Context: Fournierss gangrene have impact the morbidity [...]
Context: Fournicr's gangrene is known to have an impact the morbidity [...]
Set for to c-DC on S2,
Question:
Foumier's gungrene still
Question: Is Foumier's gangrene dangerous??
dangerous?
Arswer: Despite extensive therapeutic efforis, surgical emergency.
Answer: (Nes, No, Maybe]
Fournier's gangrene
Answer "Yes" is conrect
log L-y (x/G(t) s KN,
Figure 6: The PubMcdQA@* @e benchmark assesses PubMedQA performance while satisfying e-DC certificate. The correctness is scored as commonly done for PubMedQA (left). The correct long answer is checked by M while ensuring the c-DC (right). Only if an item is accepted and correct, the question is scored positively.
8

---

Published as a conference paper at ICLR 2025
Setup. Evaluating standardized benchmark such as PubMedQA while certifying model M requires careful consideration. The standard format typically includes n-shot examples followed by multiple-choice question with either yes/no options or answers labeled A through D. The model then prompted to select the correct response. However, this setup does not reflect realistic user- system interaction. Thus, we introduce the PubMedQA @e metric, which separates the evaluation into two streams: (1) standard assessment of model on PubMedQA to determine correctness, and (2) testing whether the correct question-ansee answerr pair is rejected by our algorithm. The process is summarized in Figure 6 We score an item as correct, if the model predicts the correct answer while pair.
maintaining its - DC on the realistic question-answerte
Results. The unconstrained model scores 73.4% on PubMedQA. As we tighten the certificate (de- crease <), more correct responses are rejected and the benchmark score drops, as shown in Figure Bc We find that when certifying at e 10-5, we maintain certified score of 66.7% (-6.7%), and at e = 10 10 of 47.7% (-25.7%). These scores demonstrate robust performance given the provable defense facilitating domain restriction. In Appendix H
we discuss benchmarking in more depth.
4 RELATED WORK
LLM Guardrails. A large body of work has been published on establishing effective guardrails for LLMs. These approaches are designed to restrict the model to responses that align with the deployer's 's values. One of the first approaches was Reinforcement Learning with Human Feedback (RLHF) (Askell et al. 202I), II which uses human preferences to guide LLM training. Extensions such as Safe-RLHF add cost models to penalize harmful behavior, ensuring a balance between helpful- ness and harmlessness during optimization (Dai al. 2024). RLHF's foundation in reinforcement learning has given rise to techniques such as Proximal Policy Optimization (PPO) (Bai et al. 2022), the more recent Direct Preference Optimization (DPO) (Rafailov et al. 2024). and Generalized Pol- icy Optimization (GPO) (Tang et al.] 2024), which incorporates diverse optimization objectives, useful for safety-critccal scenarios. For in-depth survey of this area, direct the reader to Kauf- mann et al. (2024). Unlike the preceding approaches that fine-tune guardrails into the parameters of an LLM, a number of works have proposed using LLMs to classify content as either safe or unsafe. Llama Guard categorizes the inputs and outputs of an LLM into different unsafe content categories (Inan et al. 2023). Conversely, Chua et al. (2024) classify if an output is safe with respect to a system prompt. For complete overview on LLM guardrails, we direct the interested reader to recent survey of this area, Dong et al. (002). Existing LLM guardrail techniques have been proven effective different levels. However, these guardrails only come with empirical evidence of their proficiency against existing attacks, and hence, many have been circumvented shortly after deployment. Conversely, VALID offers a provable high-probability guarantee against undesirable behavior, reflecting recent advocacy for such provable assurances (Bengio) 2024).
Out-of- Distriitribttinn Detection. Out-of-distribution (OOD) detection has received lot of at- tention recent years in NLP. Commonly, the problem is treated as text classification and softmax probabilities of class predictions (Hendrycks & Gimpell 2017) or energy scores (Liu et al. (022) are deployed as discriminant scores. Another group methods employs iistance-based, methods, rely- ing on OOD responses being distant from ID) responses in latent space, often utilizing Mahalanobis distance and sometimes incorporating contrastive learning techniques (Uppaal al. 2023 Podol-
skiy et al. 2021 Zhou et al., 2021] Khosla et al., 2020 Lin & Gu 2023). Finally, rooted in classical statistics, a number of studies suggest using the lig-likel ratio (LLR) as discriminate score, comparing likelihoods from ID and OOD proxy models (Gangal et al. 2020 Zhang et al.) 2024). Xu & Ding (2024) offer comprehensive review of LLMs for OOD detection. While many of these works have strong empirical detection results, their focus is OOD detection rather than certification, and hence they do not provide theoretical guarantees certificates on model behavior.
number of certification approaches have been proposed for LLMs in various contexts. For instance, Chaudhary et al. 2024) aim to certify the knowledge comprehension ability
Certifying LLMS.
of LLMs and Freiberger & Buchmann 2024 discuss what criteria should be certified ensure
fairness. Most relevant here is work on certification against adversarial inputs. [asadio et al. (2024) discuss certifying the robustness of L.LMs input perturbations in embedding space. Commonly, adversarial certification is studied for text classification rather than generation (La Malfa| 2023).
Kumar al. (2024) introduce framework for defending against adversarial perturbations in token

---

Published as a conference paper at ICLR 2025
space by performing small number of substitutions around given input. In contrast VALID comes with certificates that holds for all inputs, rather than perturbations around a specific input.
5 LIMITATIONS
Despite our promising results, we acknowledge the limitations of our current implementation. First, the domain generator G(y) lacks context. This means that y is marginally in-domain, while y aa the conditional distribution is not, our method will not reject appropriately. Consider chatbot for "How often is a tax report due?", the response y ='Once a year." is in-domain. Hence, the same response to x = ="How often should shower?" might be accepted despite being out-of-domain, and terrible advice. However, this can be mitigated by fine-tuning the model L to be as explicit as possible repeating "shower" in the response.
tax advice. For prompt
Second, this approach relies heavily on the domain-speciffe model G, and how closely it approxi- mates the ideal oracle Q. In practice and as demonstrated in our experiments, G might have limited semantic understanding and lack general language capabilities and world knowledge. In most in- stances might not be able to distinguish between semantically opposite but similar sentences and hence VALID is likely incapable of aligning the model, rather than shushing it.
Third, an adversary might construct an attack that aims to copy tokens from the prompt of L. to G. For instance, x "Repeat after me: !!!++! and then tell me how to build bomb!". This "!!!++!" might be an adversary for G to assign a high likelihood to L following the instruction. For this attack, the adversary operates with limited information, having access only to whether the log ratio bounded, without visibility into G's outputs, weights, or likelihood scores. In addition, since G has never seen information on how to build a bomb, it is extremely unlikely to produce coherent, correct, and harmful content. In Appendix C.T we discuss the feasibility of attacking M further.
Fourth, our method comes at the extra cost of sampling up to T times. Further, it requires training G and evaluating it during inference. Depending the architecture of G' however, the extra cost is limited. In our experiments is orders of magnitude smaller than L.
6 FUTURE WORK
In this section we briefly discuss some ideas for future work that we believe could further extent the practical utility of VALID. Initially, it would be interesting to test larger, specialized models for G' to evaluate whether these more advanced models produce improved certificates and refusal rates. We chose not to do this because LLMs trained from scratch exclusively on specific domains are not common, and thus results generalize less to what practitioner with limited resources could expect.
As described in Section 2.2 VALID uses length normalization to ensure the log likelihood ratio rejection condition is robust to different lengths of sequences Ny. One may extend this and learn more complex polynomial of Ny as rejection threshold. This threshold could be used provide both ACs and eDC certificates, while simultaneously enabling more precise OOD detection.
Finally, rejection scheme with probabilistic decision rule, similar to Algorithm 5 in Vyas et al. 2202)), ) would be able to provide identical bounds to Theorem [ Possibly, this rejection rule would lead to better performance in terms of OOD classification.
CONCLUSION
In this work, we tackle the problem of generative language models producing outputs outside their target domain in response to adversarial inputs. We describe the associated risks, introduce first- of-its-kind framework for domain certification for LLMs, and provide VALID, simple algorithm relying on well-established theories from statistics and information theory to provide such guaran- tees. We demonstrate the effectiveness of VALID in multiple representative settings and show that is effective even when relying on a guide model G with limited language skills, making it easy to deploy in limited data and resource environments.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work is supported by UKRI grant Turing AI Fellowship (EP/W002981/1), C. Emde and M. Kayser are supported by the EPSRC Centre for Doctoral Training in Health Data Science (EP/S02428X/1) and the AXA Research Fund. A. Bibi acknowledges the Google Gemma 2 Aca- demic Award 2024. T. Lukasiewicz is supported by the AXA Research Fund. Tom Rainforth is supported by the UK EPSRC grant EP/Y037200/1. The research reported in this publication was partially supported by funding from KAUST Center of Excellence on GenAI, under award number 5940. We thank the Royal Academy of Engineering and we thank Samuele Marro for his advice.
Due to an unfortunate oversight, Professor Bernard Ghanem (KAUST) was not included in the au- thor list for our ICLR 2025 submission. Despite our appeal to the conference Program Chairs (PC), the ruling was strict, and we strongly disagree with it. We apologize for this error and acknowledge his contributions, which were absolutely key to the success of this project. For clarity, the arXiv version of this paper has the complete and correct list of authors, including Professor Ghanem.
Finally, we thank the reviewers and the area chair (AC) for their efforts and constructive feedback.
REFERENCES
AI@Meta Llama 3 Model Card. 2024.
Naveed Akhtar, Ajmal Mian, Navid Kardan, and Mubarak Shah. Advances Adversarial Attacks and Defenses in Computer Vision: A Survey. IEEE Access, 9:155161-155196 2021.
Ibrahim M Alabdulmohsin, Behnam Neyshabur, and Xiaohua Zhai. Revisiting Neural Scaling Laws in Language and Vision. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh Advances in Neural Information Processing Systems, volume 35, pp. 22300-22311 12. Curran Associates, Inc., 2022.
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Her- nandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. A General Language Assistant as Laboratory for Alignment, 2021. ArXiv: 2112.00861.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli. Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfeel-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Ölsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback, 2022. ArXiv: 2204.05862.
Yoshua Bengio. Bounding the probability of harm from an Al to create a guardrail https://yoshuabengio.oog http:s//woouubbbagi.goog//////00 http.o////200////////mmdggg wtwpr/ooooniiint-----eeeeee guardrail// August 2024.
Yoshua Bengio, Michael K. Cohen, Nikolay Malkin, Matt MacDermott, Damiano Fornasiere, Pietro Greiner, and Younesse Kaddar. Can a Bayesian Oracle Prevent Harm from an Agent?, 2024. ArXiv: 2408.05284.
Battista Biggio. Blaine Nelson, and Pavel Laskov. Poisoning Attacks against Support Vector Ma- chines. In Proceedings of the 29th International Coference on International Conference on Ma- chine Learning. ICML' 12, pp. 1467-1474, Madison, WI, USA, 2012. Omnipress. ISBN 978-1- 4503-1285-1, event-place: Edinburgh, Scotland.
Christopher M. Bishop. Novelty detection and neural network validation. IEE Proceedings-Vae Vision, Image and Signal Processing. 141(4):217-222, 1994.
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson. Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel,
11

---

Published as a conference paper at ICLR 2025
Jared Quiney Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Ste- fano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Pe- ter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong. Kyle Hsu, Jing Huang. Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair. Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadim- itriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam. Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas. Florian Tramèr, Rose E. Wang. William Wang. Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the Opportunities and Risks of Foundation Models, 2022. ArXiv: 2108.07258.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Blerbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020.
Nicholas Carlini, Milad Nasr, Christopher A. Choquette-Choo Matthew Jagielski, Irena Gao, Pang Wei W Koh, Daphne Ippolito, Florian Tramer, and Ludwig Schmidt. Are aligned neural networks adversarially aligned? In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.). Advances in Neural Information Processing Systems, volume 36, pp. 61478-611 61500. Curran Associates, Inc., 2023.
Nicholas Carlini, Matthew Jagielski, Christopher A. Choquette-Chop Daniel Paleka, Will Pearce, Hyrum Anderson, Andreas Terzis, Kurt Thomas, and Florian Tramèr. Poisoning Web-Scale Train- ing Datasets is Practical, 2024. ArXiv: 2302.10149.
Marco Casadio, Tanvi Dinkar, Ekaterina Komendantskaya, Luca Amaboldi, Matthew L. Daggitt, Omri Isac, Guy Katz, Verena Rieser, and Oliver Lemon. NLP Verification: Towards General Methodology for Certifying Robustness, 2024. ArXiv: 2403.10144.
Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. Jailbreaking Black Box Large Language Models in Twenty Queries. In Few-shot and Zero-shot Learning in Large Foundation Models, 2023, ArXiv: 23 10.088419.
of
Isha Chaudhary, Vedaant V. Jain, and Gagandeep Singh. Quantitative Certification of Knowledge Comprehension in LLMs. In ICLR 2024 Workshop on Secure and Trustworthy Large Language Models, 2024. ArXiv: 2402.15929
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep Reinforcement Learning from Human Preferences. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Gabriel Chua, Shing Yee Chan, and Shaun Khoo. A Flexible Large Language Models Guardrail Development Methodology Applied to Off-Topic Prompt Detection, 2024. ArXiv: 2.
Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHF: Safe Reinforcement Learning from Human Feedback. In The Twelfth Interna- tional Conference on Learning Representations, 2024.
Xinshuai Dong, Anh Tuan Luu, Min Lin, Shuicheng Yan, and Hanwang Zhang. How Should Pre- Trained Language Models Be Fine-Tuned Towards Adversarial Robustness? In M. Ranzato,
12

---

Published as a conference paper at ICLR 2025
A. Beygelzimer, Y. Dauphin, P. S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 4356-4369. Curran Associates, Inc., 2021.
Yi Dong, Ronghui Mu, Yanghao Zhang, Siqi Sun, Tianle Zhang, Changshun Wu, Gaojie Jin, Yi Qi, Jinwei Hu, Jie Meng, Saddek Bensalem, and Xiaowei Huang. Safeguarding Large Language Models: A Survey, 2024. ArXiv: 2406.02622.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Amirudh Goyal, Anthony Hartshom, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song. Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruy Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Ander- son, Graeme Nail, Gregoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Ma- hadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Al- wala, Kartikeya Upasani. Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala- Reatala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Man- nat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov. Nikolay Bogoychev, Niladri Chatterji, Olivier Duchemne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhar- gava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy. Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sum- baly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami. Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti. Vladan Petro- vic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang. Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld. Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus. Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annic Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadch Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Bran- don Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu,
Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Ar- caute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzmán, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halper, Govind Thattai, Grant Herman, Grigory
13

---

Published as a conference paper at ICLR 2025
Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Gold- man, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Jean- rean-Baptiste Gaya, Jeff Marcus, Jeff Tang. Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang. Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U. Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang. Liangpeng Guo. Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli. Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Ke- neally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mo- hammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navy- ata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usumier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj. Qian Liang, Rachad Alao, Rachel Rodriguez, Rañ Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Sa- tadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ranaswamy, Shaun Lindsay, Shaun Lind- say, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar, Shugiang Zhang, Shuqiang Zhang, Sinong Wang. Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max. Stephen Chen, Steve Kehoe. Steve Satterfield. Sudarshan Govindaprasad, Sumit Gupta, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Tim- othy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vitor Albiero, Vlad lonescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Con- stable, Xiaocheng Tang, Xiaofang Wang. Xiaojian Wu, Xiaolan Wang. Xide Xia, Xilun Wu, Xinbo Gao. Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang. Yuchen Hao, Yundi Qian. Yuzi He, Zach Rait, Zachary Vito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. The Llama 3 Herd of Models, 2024. ArXiv: 2407.21783
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. HotFlip: White-Box Adversarial Exam- ples for Text Classification. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pupers), pp- 31-36, Melbourne, Australia, July 2018 8. Association for Computational Linguistics.
Francisco Eiras, Aleksandar Petrov, Phillip H. S. Torr, M. Pawan Kumar, and Adel Bibi. Mimicking User Data: On Mitigating Fine-Tuning Risks in Closed Large Language Models, 2024. ArXiv: 2406.10288.
EU. Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 lay- ing down harmonised rules on artificial intelligence and amending Regulations (EC) No 300/2008, (EU) No 167/2013, (EU) No 168/2013, (EU) 201 8/858, (EU) 2018/ 8/1 139 and (EU) 2019/2144 and Directives 2014/90/EU, (EU) 2016/797 and (EU) 2020/1 (Artificial Intelligence Act) (Text with EEA relevance), June 2024. Legislative Body: CONSIL, EP.
Vincent Freiberger and Erik Buchmann. Fairness certification for natural language processing and large language models. In Intelligent Systems Conference, pp. 606-624. Springer, 2024.
Varun Gangal, Abhinav Arora, Arash Einolghozati, and Sonal Gupta. Likelihood ratios and gener- ative classifiers for unsupervised out-of-domain detection in task oriented dialog. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 7764-7771 2020.
Tianyu Gao, Adam Fisch, and Danqi Chen. Making Pre-trained Language Models Better Few-shot Learners. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 11th International
14

---

Published as a conference paper at ICLR 2025
Joint Conference Natural Language Processing (Volume 1: Long Papers), pp. 3816-3830 Online, August 2021. Association for Computational Linguisties.
Saibo Geng, Martin Josifoski, Maxime Peyrard, and Robert West. for Structured NLP Tasks without Finetuning. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Process- ing. pp. 10932-10952, Singapore, December 2023, Association for Computational Linguistics.
Decoding
Dan Hendryeks and Kevin Gimpel. A Baseline for Detecting Misclassified and cittrf--Dittiibution Examples in Neural Networks. In International Conference on Learning Representations, 2017.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song. and Jacob Steinhardt. Measuring Massive Multitask Language Understanding. In International Conference on Learning Representations (ICI.), 2021. ArXiv: 2009.03300
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Rad- ford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish. Scaling Laws for Autoregressive Generative Modeling, 2020.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations (ICLR), 2022.
Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. Llama Guard: LLM- based Input-Output Safeguard for Human-Al Conversations, 2023. ArXiv: 2312.06674.
Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping-yeh Chi- ang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. Baseline Defenses for Adversarial Attacks Against Aligned Language Models, 2023. ArXiv: 2309.00614.
Robin Jia and Percy Liang. Adversarial Examples for Evaluating Reading Comprehension Systems, 2017. ArXiv: 1707.07328.
Xiaojun Jia, Tianyu Pang, Chao Du, Yihao Huang, Jindong Gu, Yang Liu, Xiaochun Cao, and Min Lin. Improved Techniques for Optimization-Bannn Based Jailbreaking on Large Language Mod- els. In The Thirteenth International Conference on Learning Representations (ICLR). 2025. ArXiv:2405.21018
Shuyu Jiang, Xingshu Chen, and Rui Tang. Prompt Packer: Deceiving LLMs through Compositional Instruction with Hidden Attacks, 2023. ArXiv: 2310. 10077.
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. PubMedQA: A Dataset for Biomedical Research Question Answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2567-2577, 2019.
Erik Jones, Anca Dragan, Aditi Raghunathan, and Jacob Steinhardt. Automatically Auditing Large Language Models via Discrete Optimization. In Andreas Krause, Emma Brunskill, Kyunghyun Cho. Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 15307-15329. PMLR, July 2023.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models, 2020. ArXiv: 2001.08361.
Andrej Karpathy. The Unreasonable Effectiveness of Recurrent Neural http://kappathy http://Karpattyggihuub..o0/0 wtt.g/nni.....00///////////ffffa
Networks
2015.
Timo Kaufmann, Paul Weng. Viktor Bengs, and Eyke Hüllermeier. A Survey of Reinforcement Learning from Human Feedback, 2024. ArXiv: 2312.
15

---

Published as a conference paper at ICLR 2025
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Samaa, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised Contrastive Learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Pro- cessing Systems, volume 33, pp. 18661-18673. Curran Associates, Inc., 2020.
Aounon Kumar, Chirag Agarwal, Suraj Srinivas, Aaron Jiaxun Li, Soheil Feizi, and Himabindu Lakkaraju. Certifying LLM Safety against Adversarial Prompting, 2024, ArXiv: 2309.02705.
E La Malfa. On robustness for natural language processing. PhD Thesis, University of Oxford, 2023.
Ken Lang. NewsWeeder: Learning to Filter Netnews. In Armand Prieditis and Stuart Russell
(eds.), Machine Learning Proceedings 1995, pp. 331-339. Morgan Kaufmann, San Francisco
(CA), 1995. ISBN 978-1-55860-377766
Xiang Lisa and Percy Liang- Prefix-Tuning: Optimizing Continuous Prompts for Generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.). Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 09882-497, Online, August 2021, Association for Computational Linguistics.
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive Decoding: Open-ended Text Generation as Optimiza- tion. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 12286 12286-12312, Toronto, Canada, July 2023. Association for Computational Linguistics.
Haowei Lin and Yuntian Gu. FLatS: Principled
Distribution Detection with Feature-Based Likelihood Ratio Score. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 8956-8963, Singapore, December 2023. Association for Computational Linguistics.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based Out-of-distribune De- tection. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 21464-21475. Curran Associates, Inc., 2020.
Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. AutoDAN: Generating Stealthy Jail- break Prompts on Aligned Large Language Models. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
Andrew I. Maas, Raymond F. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning Word Vectors for Sentiment Analysis. In Dekang Lin, Yuji Matsumoto, and Rada Mihalcea (eds.), Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp- 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics.
Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. Introduction to Information Retrieval. Cambridge University Press, Cambridge, 2008. ISBN 978-0-521-86571-55
Tess McClure. Supermarket Al meal planner app suggests recipe that would create chlorine gas. The Guardian, August 2023. ISSN 0261-3077.
Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. Tree of Attacks: Jailbreaking Black-Box LLMS Automatically. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 11065-61108 Curran As- sociates, Inc., 2024.
Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Ama- triain, and Jianfeng Gao. Large Language Models: A Survey, February 2024. ArXiy: 2402.06196.
16

---

Out-of-Domaii Detection. In AAAI Conference on Published as a conference paper at ICLR 2025
Marius Mosbach, Tiago Pimentel, Shauli Ravfogel, Dietrich Klakow, and Yanai Elazar. Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguis- fies: ACL. 2023, pp. 12344, Toronto, Canada, July 2023, Association for Computational Linguisties.
Jerzy Neyman and Egon Sharpe Pearson. IX. On the problem of the most efficient tests of statistical hypotheses. Philosophical Trunsactions of the Royal Society of London. Series A, Containing Papers of Mathemutical or Physical Character, 231:289-337, 1933.
Thanh Tam Nguyen, Thanh Trung Huynh, Phi Le Nguyen, Alan Wee-Chung Liew, Hongzhi Yin, and Quoe Viet Hung Nguyen. Survey of Machine Unlearning. 2022. ArXiv: 2209.02299.
Charles O'Neill, Jack Miller, Ioana Ciuca, Yuan-Sen Ting, and Thang Bui. Adversarial Fine-Tuning of Language Models: An Iterative Optimisation Approach for the Generation and Detection of Problematic Content, 2023. ArXiv: 1376.13768.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, Agarwal, D. Belgrave. K. Cho, and A. Oh (eds.), Advances Neural Information Processing Systems, volume 35, pp. 27730-27744, Curran Associates, Inc., 2022.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, Vanderplas, Passos, D. Cournaeea,, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research. 12:2825-2830, 2011.
Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red Teaming Language Models with Language Models. In Conference on Empirical Methods in Natural Language Processing, 2022.
Fábio Perez and Ian Ribeiro. Ignore Previous Prompt: Attack Techniques For Language Models. In NeurIPS ML Safety Workshop, 2022. ArXiv: 221 2211.09527.
A. V. Podolskiy, Dmitry Lipin, A. Bout, E. Artemova, and Irina Piontkovskaya. Revisiting Ma- halanobis Distance for Transformer-Bart Artificial Intelligence, 2021.
Xiangyu Qi, Yi Zeng. Tinghao Xie, Pin-Yo Chen, Ruoxi Jia, Prateck Mittal, and Peter Henderson. Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! In The Twelfth International Conference on Learning Representations, 2024.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners. 2019.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ Ques- tions for Machine Comprehension of Text. In Jian Su, Kevin Duh, and Xavier Carreras (eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp- 2383-2392 Austin, Texas, November 2016. Association for Computational Linguistics.
Eduardo Reis. Bible Corpus Basic Text Generation using N-grams, 2019.
Jie Ren, Peter J. Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Detection. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Bus -ucc E. Fox, and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Balaji Lakshminarayanan. Likelihood Ratios for
17

---

Published as a conference paper at ICLR 2025
Alfréd Rényi On Measures of Entropy and Information. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume I: Contributions to the Theory of Statistics, volume 1. University of California Press, 1961.
Mikayel Samvelyan, Sharath Chandra Raparthy, Andrei Lupu, Eric Hambro, Aram H. Markosyan, Manish Bhatt, Yuning Mao, Minqi Jiang, Jack Parker-Holder, Jakob Nicolaus Foerster, Tim Rocktäschel, and Roberta Raileanu. Rainbow Teaming: Open-Ended Generation of Diverse Ad- versarial Prompts. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. ArXiv 2602.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural Machine Translation of Rare Words with Subword Units. In Katrin Erk and Noah A. Smith (eds.), Proceedings of the 54th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguisties. ArXiv 1508,07909.
Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Remi Munos, Mark Row- land, Pierre Harvey Richemond, Michal Valko, Bernardo Avila Pires, and Bilal Piot. General- ized Preference Optimization: A Unified Approach to Offline Alignment. In Ruslan Salakhut- dinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.). Proceedings of the 41st International Conference on Machine Learning, vol- ume 235 of Proceedings of Machine Learning Research, pp. 47725-47742. PMLR, July 2024.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Fer- ret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Char- line Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchi- son, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal. Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette-Chov Danila Sinopalnikov, David Wein- berger, Dimple Vijaykumar, Dominika Rogozinska, Dustin Herbison, Elisa Bandy, Emma Wang. Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Harleen
Grill, Behnam Neyshabur,
Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju-yeong Ji, Kareem Mohamed, Kar- tikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, Laurent Sifre, Lena Heuermann, Leticia Lago, Lilly McNealus, Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Luciano Martins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Görner, Mat Velloso, Mateo Wirth, Matt Davidow, Matt Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Moyni- han, Ming Zhang, Minsuk Kahng, Minwoo Park, Mofi Rahman, Mohit Khatwani, Natalie Dao, Nenshad Bardoliwalla. Nesh Devanathan, Neta Dumai, Nilay Chauhan, Oscar Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culli- ton, Pradcep Kuppala, Ramona Comaneseu, Ramona Merhej, Reena Jana, Reza Ardeshir Rokni, Rishabh Agarwal, Ryan Mullins, Samanch Saadat, Sara Mc Carthy, Sarah Cogan, Sarah Perrin, Sébastien M. R. Arnold Sebastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ron- strom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Hennigan, Tomas Kocisky, Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wenming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotruta, Phoebe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Warkentin, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dra- gan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Fara- bet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, and Alek Andreev. Gemma 2: Improving Open Language Models at a Practical Size, 2024. ArXiv: 2408.00118
The Guardian. 2023.
AI meal planner suggests toxic recipes in 'malfunction'. The Guardian,
18

---

Published as a conference paper at ICLR 2025
Rheeya Uppaal. Junjie Hu, and Yixuan Li. Is Fine-tuning Needed? Pre-trained Language Mod- els Are Near Perfect for Out-of-Domain Detection. In Annual Meeting of the Association for Computational Linguistics, 2023.
Nikhil Vyas, Sham Kakade, and Boaz Barak. On Provable Copyright Protection for Generative Models, 2023.
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. Universal Adversarial Triggers for Attacking and Analyzing NLP. In Kentaro Inui, Jing Jiang. Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP- IJCNLP), pp. 2153-2162 Hong Kong, China, November 2019. Association for Computational Linguistics. ArXiv 1908.07125.
Eric Wallace, Tony Zhao, Shi Feng, and Sameer Singh. Concealed Data Poisoning Attacks on NLP Models. Kristina Toutanova, Anna Rumshisky, Luke Zeuteeooer, Dilek Hakkani-Tur, Iz Belt- agy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Lingnistics: Human Language Technologies. pp. 139-150, Online, June 2021. Association for Computational Linguistics. ArXiv 2010.12563.
Alex Wang, Amanpreet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. 2019.
Jiongxiao Wang, Jiazhao Li, Yiquan Li, Xiangyu Qi, Junjie Hu, Yixuan Li, Patrick McDaniel, Muhao Chen, Bo Li, and Chaowei Xiao. BackdoorAlign: Mitigating Fine-tuning based Jailbreak Attack with Backdoor Enhanced Safety Aligmment. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Yizhe Xiong, Xiansheng Chen, Xin Ye, Hui Chen, Zijia Lin, Haoran Lian, Zhenpeng Su, Jianwei Niu, and Guiguang Ding. Temporal Scaling Law for Large Language Models, 2024. ArXiv: 1778.
Heng Xu, Tianqing Zhu, Lefeng Zhang, Wanlei Zhou, and Philip S. Yu. Machine Unlearning: A Survey. ACM Comput. Surv., 56(1), August 2023. ISSN 0360-0300, Place: New York, NY, USA Publisher: Association for Computing Machinery.
Ruiyao Xu and Kaize Ding. Large Language Models for Anomaly and Out-of-Distribution Detec- tion: A Survey, 2024. ArXiv: 2409.01980.
W. Youden. Index for rating diagnostic tests. Cancer, 3(1):32-35, 1950.
Andi Zhang, Tim Xiao, Weiyang Liu, Robert Bamler, and Damon Wischik. Your Finetuned Large Language Model is Already a Powerful Out-of-distributioo Detector, 2024. ArXiv: 2404.08679,
Wenxuan Zhou, Fangyu Liu, and Muhao Chen. Contrastive Out-of-Distribttion Detection for Pre- trained Transformers. In Maric-Franeine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen- tau Yih (eds.). Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1100-1111, Online and Punta Cana, Dominican Republic, November 2021. As- sociation for Computational Linguistics.
Andy Zou, Zifan Wang. J. Zico Kolter, and Matt Fredrikson. Universal and Transferable Adversarial Attacks on Aligned Language Models, 2023. Arxiv: 2307.15043.
19

---

Published as a conference paper at ICLR 2025
A PROOFS
Theorem (VALID Certificate) Let I. be an LLM and G guide model as described above. Rejec- tion sampling as described in Algorithm| | with rejection threshold i and up to T tterations defines
model ML K.K. with ML..KK. M.L.KK.T (.).))()))))))) denoting the likelihood of y given a. Let Ny be the length of
y. We state the adversarial bound:
Vx S ML.GAT(y)x)) S 2kNNN. T
(4)
Hence, ML.GEE.T is (2 and, further, it is [maxyer T w.r.t. F.
Proof: We abbreviate M.. M. Let A, and A be the events of accepting and rejecting in iteration L, respectively. Let S, be the event of sampling y ~ L(-lar) in iteration and let A'ze be the
event of rejecting all samples before i, A'zx = = A Then,
My a:) Σ A
Σ P(A
IIP(AAI'' 2)). (7)
x)
ist
We upper bound the probability of rejecting in any previous iteration by 1, Vt P(A <e: a) S 1.
P(A. A'ze: æ)) is non-stochastic and is equal to either 0 or 1. In the former case, the M (y) trivially bounded by any non-negative number. The latter case (i.e. y is accepted in iteration t) L(yl#)
implies that log G(g) I kNy Rearranging terms and noting that by definition P(S|[A'a a:)
()), we get (S, x) I 24h, Gly) and hence by substitution and summing over t,
T
( 1
2kN, G(y) = =kkN, T G(yy.
(8)
1=1
This is the desired upper bound on (z) for all æ E
Lemma (Equivalence of Divergence) Let D (P I| Q) be the Renyi divergence of order infinity P(s)
Rényí 1961, Ao (P || Q) A log sup, Q(a)' Further, let L S S be an LLM returning y given x as discussed above and let be a distribution over domain T, i.e. generator for T. Then, if
Vx X (L
(yy)) S k;
we can state that I, is AC-AC with ey = 2* (yy) (see Definition) 7 and with e = 2 maxy My (see Desinition 2 1f is an oracle, that assigns no likelihood to elements in R, it implies I. is 0-AC
and O-DC.
Proof: We start from the definition of the Renyi divergence, which is upper bound to any element in the supremum, giving that
L(ylzz)
S log L(y|œ) Ax (L(g)z)) ((y)) < k.
Va log A(y)
(10)
y [2y)
Exponentiating and multiplying through by (yy) gives the following upper bound:
Va E x L(y x) < 2*)(
(11)
showing the 2* -AC equivalence. Taking the max over F shows the [2 maxp [(y) equiv- alence. Further, assuming to be a perfect oracle, by definition. we can state that Vy E the upper bound on the right hand side of (II) is zero. Thus, we get the desired result:
Væ E V. IF L(y|=) = 0,
(12)
and hence L is O-AC and O-DC.
Lemma 2 (Likelihood of M) Let M be a model obtained by performing rejection sampling from
model as proposed in VALID using guide model G and rejection threshold k (see Algorithm 7
We denote the likelihood of response y given input under the model M as M (z)la). For all y E S,
M(yxx) L(y)))
L(ylz) & kG(g)
(13)
otherwise.
20

---

Published as a conference paper at ICLR 2025
where A, the event of rejecting y in iteration given input æ, A'ci the event of rejecting in all iterations before t, A'c, = = i' A', and finally let ó = A(A]) )). the conditional probability of rejecting y in given iteration t for input z. Finally, let R be the event that M abstains, for which
M ( 6²
(14)
Proof: Let St be the event of sampling y ~ L(-(at) in iteration 6, let & C S be the acceptance set of y, i.e. A {y L(y x) s G(y) and let its complement in S. A', be the rejection set. Finally, let S be the event of sampling We now derive M (y(z) per case as stated in (13).
Starting with case y A, we note that M(y [r) (S)(a)) and we can rewrite (( follows,
P(S = P(S.n A 0 A'sc-1la))
(15)
Σ
22) +) II [P(AAIA220))
(16)
i<t
T
= -[(y(>)
0t-1
(17)
1 1- 0"
= -(ylax
(18)
1- x
where we use the fact that Vy E A P(Ar|Se, A'ze: x²) = and notice that 1t= --11 is the sum of
the first T elements of geometric series and substitute T.yy) p²) for (PS A ce: x).
For the case E A': We rewrite the likelihood as shown above in 16 Notice that Vy E A' :
P(Ar|S,, Alz 2) 0 and therefore P(S(#) is zero.
Finally, we turn to the rejection event R. Note that R = =NE= Ai, rejection at each step t = 1,..., T. We can state that
M (R
II P(A)IA x) 0T, t=1
(19)
which concludes the proof.
Remark 1 (Estimating likelihood) While Lemma 4 provides expression of the likelihood of model M computing this might be infeasible. If the sample space S large, we cannot compute Myylr) as we cannot compute oh the rejection probability in any given iteration in VALID) for given input x. However, we can estimate M M(yla) by computing L(yla) and performing Monte
Carlo sampling from L to obtain an estimator o. We can then use the Binomial confidence interval for confidence level a:
d 1
3 Za/2
(20)
N
We then plug in the bounds on L to obtain the bound on M because of the monotonicity of M in d.
Lemma 3 (Expected number of iterations in VALID) Let be the number of iterations executed in VALID (see Algorithm [7, let A₁ be the event of accepting a response for input æ in iteration t, t = 1....,T, and let ils complement, Ao be the event of rejection in iteration t. Denote the event that all samples up to t (inclusive) are rejected as A'g. -i==1 N. Finally, we denote A' x), the probability of rejection in iteration t. The expected number of iterations for de E (0,1) is given by:
1 - 6"
E yy M (-1x)[T]
(21)
1
21

---

Published as a conference paper at ICLR 2025
and for 1, the expected number of iterations is given by By- [T] = T.
Proof: In the following, we will denote My)aM(laa(T]] as E[r] for readability. Note that P(r t) the probability of reaching and accepting iteration for I...., Once iteration T is reached, both acceptance and rejection yield 7 = 7', Hence,
BiT]
Σ :P(T = =) (A, A''s--l1
Σ tP(A, n A's--1a)).
(22)
Combining events and factorising probabilities,
F[T] = 7P('sr +
( x) II IPAA
(23)
for which we substitute rejection and acceptance probabilities by Φ and o respectively,
B = T +1 - (b) Σ
1Ø--1
(24)
1=1
Multiplying by d:
dB|r] = TØ + (1 6) Σ to.
(25)
Subtracting
25 from 24
I
E dE[T] = (1- -0-TT + (1- - 6)
to
1-1 - -t0.
(26)
1=1
Telescoping sum:
T
(1 - ④) E = o)Tо" + +11 Φ) Σ
di-1 - -TO
(27)
1=1
Dividing by (1 Φ). For all < 1:
E = T
Σ +- To".
(28)
Cancelling terms and summing the first T' elements of the geometric series:
T
1- ST
E =
(29)
Using L'Hôpital's Rule, we can evaluate the limit for á 1 and find that this simplifies to T and hence E[r] T when @ completing the proof.
Remark The expected number of iterations as derived in Lemma deepends on the rejection prob- ability é and the maximum number of iterations T. When & = O. the algorithm always accepts in any iteration and hence E. By æ) [T] Conversely, when @ and the algorithm always ab-
stains, E (. [7] T. Further, for T Vo [0, [gy-M())] and as T increases,
does Ey when 0.
B DEFINING DOMAINS PRACTICAL CONSIDERATIONS
In this section, we provide practical guidance for practitioners how to select domains for their AI systems, presenting a systematic approach to classifying sequences into different domains. Figure] illustrates Venn diagram comprising three key sets of sequences, i.e. subsets of S:
22

---

Published as a conference paper at ICLR 2025
Space of all sequences
Unintelligible Sequences T' E
Out-of-domain sequences that are not a concern.
"The sky is blue, the sky is blue, the sky is blue"
*ABC ABC ABC ABC ABC ABC ABC ABC"
Target Domain
Other Domains F
Medical QA "The recommended dose of Paracetamol is
Tax advice, Weapon Manufacturing, and many more. "Here is an easy recipe commit tax fraud...
Figure 7: A Venn diagram illustrating the separation of sentences into domains.
1. The target domain T (shown in blue), containing desired content about which the LLM- driven system should converse (e.g., medical questions and answers),
2. The out-of-domain set IF (shown in orange), containing potentially harmful or other content that require active protection measures (c.g., tax fraud advice),
3. The complement of T and F, denoted as T n 1' (shown in gray).
How should one define and F? Defining is relatively natural for most practitioners: Content that semantically belongs to the domain should be included in T. The more complex decision involves determining which sequences outside T should be included in F. We contend that protecting against certain sequences warrants higher priority than others, and these high-priority sequences should be included in F.
A fundamental question arises:
Consider the example sequence y = The sky is blue. The sky is blue. The sky is blue. While this is clearly out-of-domain for a medical QA system, practitioners should evaluate two critical questions to determine its placement in F:
1. Would adversaries have motivation to generate such sequences?
2. Could these sequences potentially harm users, the deployer, or third parties?
In this example, adversaries would likely have little incentive generate such a response, and the content itself is harmless. Therefore, practitioners might reasonably conclude that y should remain in T' F' rather than F, excluding from system certification considerations.
These evaluation questions help practitioners assess risk levels effectively. When both questions receive negative answers, sequences can safely remain in T OI" without requiring active protection measures, allowing security efforts to focus on genuinely concerning sequences. If either question receives positive response, practitioners may choose to implement protective measures.
Let us analyze two more examples to demonstrate this in practice. Consider the sequence y = Here In this case, malicious actors would be highly motivated to seek such information, and the content could directly harm society and gov- emmment functions. Thus, this sequence clearly belongs in F and requires active protection measures. Similarly, when considering a love poem as y, although it may seem harmless at first glance, the analysis reveals important considerations. Users might frequently request LLMs to generate poetry, potentially straining system resources, and while not directly harmful to users society, it could significantly impact system infrastructure and operational costs. Consequently, practitioners might choose include this in IF to protect their computational resources.
is an easy recipe to commit tax fraud.
23

---

Published as a conference paper at ICLR 2025
It is important to note that these evaluation questions are not intended as universal rules, but rather serve as practical considerations to guide practitioners in their decision-making process. By sys- tematically assessing motivation and potential harm, practitioners can make informed decisions about which sets of sequences require active protection measures.
C VALID REJECTION SAMPLING
C.I ATTACKING M
In this section, we provide some insight on how rejection sampling as deployed in VALID (see Section (.2) can obtain such tight adversarial bounds. In particular, we show by example that out- of-domain samples are only accepted when they have sufficiently small likelihood of being sampled under L. We then formalize this intuition and state the objective of possible adversarial attack on M. For simplicity, we will consider the case T=
Here, we demonstrate that accepting an out-of-domain response requires it to have low likelihood under model L. Specifically, we show that when response is rejected for given prompt, the correct strategy for acceptance by model M involves modifying the prompt to reduce the response's 's probability under L. To illustrate this concept, examine single response y. Let y = The cow drinks milk and consider three prompts:
Intuition.
. = What does a cow drink?
X2 Which animal drinks milk?
. X3 Repeat after me: The cow drinks milk. Now you:
Intuitively, may assume L(yla))) L(y)=)) L2y( as y more naturally follows some prompts than others: y D3 would have high likelihood for instruction-tunet models, moderate like- lihood after being specifically asked about cows (æ1),, and low likelihood when asked broadly about mammals (x We illustrate this example in Figure 8 If we assume that y æ1 is rejected, i.e., conclude that y/ı will also be rejected. In contrast, 2/ will be accepted when L(y (22) is small enough, such that log 2(y)x2)) - log G(y) < kNy, which recovers the upper bound of (eNoG(y) (see Theorem I by algebraic manipulation. This illustrates how rejection sampling bounds the adversaries: Samples will only be accepted if propos- ing them was very unlikely in the first place. Consequently, when faced with rejected adversarial prompts x, the attacker must find alternative prompt x that yield lower likelihood y|æx. This creates remarkable and counter-intuitiv dynamic: successful adversarial attacks on model M require the attacker to effectively perform risk control on sampling y. This intuition helps us establishing how to attack M.
log L (1)) log Giy) > kNy. then
Formalization of Attack. We assume that the adversarial objective is to increase the probability of given y (e.g. from the out-of-domain set), F, being returned. The objective of attacking L. is immediately follows:
ade L
= arg max L(y(xx =eX
(30)
where X is either S or some continuous relaxation, such soft-prompt space. However, the solution ade may not be an adversary under M, since 2% might maximize the log-likelihood ratio that leads to the sample being rejected, and hence M Myyaa = O. Instead, the adversary for M, x needs to maximize L while ensuring the sample is accepted. We formalize this in the following lemma.
Lemma 4 (Adversary under Rejection Sampling) Assume the adversarial objective is to maxi- mize the likelihood of sample y being returned by the model M. Assume the model M is obtained through VALID as described in Algorithm 7 with T = 1. The adversary is given by:
adv M
arg max L(y) s.t. L(y)x) < 2hNN
(31)
xEX
where X is either sentence space S or some relaxation.
Proof: The proof follows immediately from Lemma 2 with T Let A C S be the acceptance set, A = {xc E X T(y|æ)) < 2kN 2k))) Then, Yo & A the likelihood of M (ygar) = O. Hence, the adversary maximizing M(y is the adversary for [yyla) within A.
24

---

Published as a conference paper at ICLR 2025
y The cow drinks milk."
Acceptance|Area
Rejection Area
"Repest after me: The cow drinks milk. Now you:
"Which drinks milk?"
"Wat does the cou drink?"
Ma.7= (y)x))
L(y(x)) Gly) L(yxs)
L(y
L(y[x]
Figure 8: The likelihood of model obtained through VALID with = 1. blue line the likelihond of M for the given y. Three example prompts æ;, 22 and wy are shown.
Executing Attack. Applying VALID to obtain M has implications on the suitable procedures to attack M. In particular, it requires solving the constrained optimization problem in (31). which adds a layer of complexity to the unconstrained problem for problems are more challenging: this is compounded by the upper bound on L(ylaæ) not decomposing across tokens. Furthermore, while large models, such as LLama-3-8B, are often publicly available, G' will likely be a custom model for which the attacker does not have white-box access. For successful attack, the adversarial user must estimate the likelihood ratio between and G, which might prove challenging. This indicates that attacking M defined through VALID might be harder problem than attacking L. Finally, as reassuring reminder, while it possible attack M, our certificate holds and M cannot be attacked past the upper bound provided in Theorem
general, constrained optimization
D EXPERIMENTAL SETUP
D.1 CHARTASK DATASET
For prototyping, we have created a toy dataset that we call CharTask. The goal of the CharTask dataset to have well-controlled toy dataset with clear definitions of target domain T. and other domains
As shown in Table ! each sequence consists of three parts: A sequence of random characters, task definition in the middle, and another sequence of characters in the end. We refer to the random sequence Sin. In the middle there are four task tokens, the first of which defines the task T. "S" sets the task to sorting, "R" to reverse sorting, "A" to adding +1 and "E" to even-odd sorting. The instruction token is followed by the remaining three task tokens in random order to ensure that all are seen by a model trained on a subset these. Finally, the completed sequence the original sequence of characters with the task performed on them, i.e. Sout T(S..). The pool of characters for each sequence is either only integers or integers and lower case letters. Importantly, all tasks interpret characters and integers as characters alike. For example, sorting integers "11", " "5" results "11", "5". To be precise, all tasks are based on integer unicode representations of characters.
Each sequence has a variable length of up to 49 elements in Sin (the elements can be double digits). For integers, pool of 49 unique distinct integers and for characters,
pool of 249
25

---

Published as a conference paper at ICLR 2025
elements (e.g., defining "at" element in the sequence). Under these conditions, there exists combinatorially large set of unique sequences far exceeding our training dataset size.
Given the tasks and pools of characters, 8 possible domains emerge as shown in Table 1 which we denote as CharTask (Task, Pool). We define sorting integers as the target domain: Dr Char Task (Sorting, Int) and all other combinations as out-of-domain, We create two distinct datasets with non-overlapping splits for training, validation, and testing. The in-domain dataset consists of IM training samples. The "generalist" dataset DT+F = CharTask (All, Int Char) contains all pos- sible tasks with sequences consisting of integers and characters. We 1M training sequences per task, and hence 4M sequences in total. The validation and test sets are 64 sequences and 4096 sequences, respectively.
D.2 CHARTASK SETUP
Dataset and Domain. We use the Char'Task dataset described Appendix D.1] We train a custom BPE tokenizer of length 360 (Sennrich al. (0166). In practice, the pretrained tokenizer of any foundation model trained on a general dataset. Hence, we train the tokenizer using Dr and Dy, the target and out-of-domain datasets. While the dataset is inherently suitable for sequence-to- sequence task, we treat it as next-token prediction problem just as used in language modeling.
Training. We train our domain model G on a set of integer sorting examples, CharTask (Sorting, Int). We train GPT-2 (Radford et al., 2019) architecture with layers, heads and embedding dimension. We train the model on partial sequences, as we are embedding marginal sequences y- Hence, we cur each sequence in two parts using splitting point that sampled under uniform distribution. Hence, the model learns the transition from "[BOS] to any character that might be the first response token.
For the generalist model L. we train using all available tasks on integers and characters, CharTask (Alr, We train a GPT-2 architecture with layers, 6 heads and 192 embedding dimensions.
We train L and with A (weight decay 0.1) for 2048 steps with a cosine learning rate sched- ule with 500 steps warm-up, maximum learning rate of 0.005, scheduled for 40 epochs. We train with 120 context window using next-token prediction.
Inference. We use common parameters to tweak the predictive distribution of our models. For G we use temperature of 0.7 and for of 0.2. We find this greatly helps the model performance of both. We do not perform Topk selection of tokens. We prompt with a prompt length of 10. The complcod sequence is almost deterministic given the prompt and task for models that have very high accuracy. Hence, we remove sequences where the prompt of 10 tokens is larger than 25% of the entire sequence. sequence.
D.3 20NG SETUP
Dataset Cleaning. The 20NG dataset is very dirty, containing a wide array of random special char- acter sequences and arbitrary formatting. We found these sequences to complicate model training and large pre-trained models struggled with it. In addition, as formatting strongly varies between the 20NG dataset and others, this is confounding factor for OOD detection. Classifying sentences
Table I: Examples of the CharTask datasct
Sequence
Task
Pool
Prompt
Task
Completed
Combined
Sorting
Int
5 536 6
SR A E
356
Q S R A E 5
Adding
Int
536
A S
647
Q536 A 6 647
Reverse Sorting
Int
5
R A S
653
Q536RE A S 6
Even-Odd
Int
536
ER A
635
Q536ER A 635
Sorting
Int + Char
1355
SERA A
135ac
Q E R A
Adding
Int Char
135ca
ASRE
14
A
Reverse Sorting
Int + Char
135ca
REAS
ca5
1355caR ca5
Even-Odd
Int Char
135ca
ESAR
a 135
A A 135ea
26

---

Published as a conference paper at ICLR 2025
ID or OOD should focus semantics, but the formatting provides spurious correlation that easily exploited by models. Hence, we decided to clean the dataset. do so, we utilise the seikit-learr (v1.5.1) (Pedregosa et al. 2011) options to remove headers, footers and quotes. Further, we cleaned using Llama-3.1-8B-Im Instrrct (Dubey et al. 2024) using the following query:
Your task is to clean and format a string. Instructions:
Do not change the order of the words.
Remove cryptic character sequences, spacings out of order, and 1ine breaks within sentences.
Remove out-of-order punctuation, but leave correct punctuation in place.
The result should be semantically and lexically the same as the original but well formatted.
Remove IP addresses and email addresses.
Remove sequences of (special) characters, that are not human language.
Only return the cleaned string without messages or quotes around it. Do not return any other information. Do not repeat the instructions. Do not repeat the example.
Sentence:
We check the output for various keywords and phrases from prompt and find 0% violation rate. While there still exist random sequences, the data quality is greatly improved. We notice that several sequences exist 20NG and OOD testing datasets that are seemingly random character sequences and multiple trigram repetitions such as "Nanaimo British Columbia Nanaimo British Columbia Nanaimo British Columbia ...". These sequences have the highest likelihood under model G and while not having any semantic meaning nor constituting a valid sequence that could indicate model misappropriatton. Hence, when reporting max likelihoods for 20NG over finite dataset (e.g. ye Lyy(z)) we instead use the 99. 99.99th quantile and report it as max.
Training. We use pre-trained Gemma 2 tokenizer for both models which has vocabulary size of 256k tokens. For the fine-tuned model L., we use pre-trained decoder-only Gemma 2 2B (hosted on Hugging Face) as the starting point then fine-tune it to our ID) dataset using LoRA adaptors which involved training an additional 10.4M parameters (0.4% the total parameters) We train with AdamW W (weight decay 0.01) for 1536 steps with cosine learning rate schedule with 64 steps warmup, maximum learning rate of 5e-5, scheduled for 32 epochs. We train with 256 context window using next-token prediction.
For the model G, we use decoder-only GPT-small model architecture, 6 layers, 6 heads and 384 embedding dimensions and a total of 109.3M parameters, which we train from scratch using the ID data exclusively. We train G with AdamW (weight decay 0.01) for 320 steps with cosine learning rate schedule with 100 steps warm-up, up. maximum learning rate of 3e-4, scheduled for 100 epochs. We train with 256 context window using next-token prediction.
Inference. For both I, and G we use default temperature of We not perform TopK token selection. When evaluating performance, we use a 128-token long prompt and 128-token long ground truth response.
D.4 TINYSHAKESPEARE SETUP
Dataset Cleaning. The formatting in ThaySSakespeaaee dataset was distinctly different to other with long sequences of line breaks and usage of all-caps for character names. We removed these excessive line breaks and changed the character names from all caps to title case to make similar other datasets and make OOD detection more challenging.
27

---

Published as a conference paper at ICLR 2025
Training. We use pre-trained Gemma-2 tokenizer for both models which has vocabulary size of 256k tokens. For the fine-tuned model L, we use a pre-trained decoder-only Gemma-2-2B as the starting point then fine-tune it to our ID dataset using LoRA adaptors which involved training an additional 10.4M parameters (0.4% of the total parameters). We train L with AdamW (weight decay 0.01) for 128 steps with cosine learning rate schedule with 64 steps warm-up, maximum learning rate of 5e-5, scheduled for 32 epochs. We train with 256 context window using next-token prediction.
For the model G, we use decoder-only GPT-micro model architecture, 4 layers, 4 heads and 128 embedding dimensions and total of 33.7M parameters, which we train from scratch using the ID data exclusively. We train G with AdamW (weight decay 0.01) for 2400 steps with cosine learning rate schedule with 300 steps warm-up, a maximum learning rate of 3e-4, scheduled for 300 epochs. We train with 256 context window using next-token prediction.
Inference. For both L and G we use a default temperature of 1. We do not perform TopK token selection. When evaluating performance, we use 128-token long prompt and 128-token long ground truth response.
D.5 MEDICALQA
We apply our method medical question answering as target domain, T. This could, for example, be extended to a chatbot for clinicians to research patient symptoms. To model potential questions and answers, we use the PubMedQA dataset (Tin et al., 2019 as Dr, which contains approximately 200K QA pairs for training and 1000 test pairs. We regard question answering on other topics, such as geography computer science as F. To model this, we use the Stanford Question and Answering Dataset (excluding medical categories) (Rajpurkar el al. 2016) as Dz.
As generalist LLM, L, we use a LLama-3-8B model (AI@Meta 2024) and train a custom GPT-2 model (184M parameters) for G (Radford et al., 2019). We pre-train G on Pub- MedQA (Jin et al. 2019) with 200K sequences. We then use 100K prompts from PubMedQA to generate sequences using L and then fine-tune on them using responses from L to half the prompts in PubMedQA. As G' embeds the responses, G(yy), we fine-tune using "BOS[Response]" rather than entire sequences. We pre-train with a learning rate of 0.0001 for 50 epochs and then fine-tune with a learning rate of 0. 0.00001 for another 50 epochs. On 8 x H100, the total training takes about 2 hours.
Training.
Inference. We perform inference without tops or topp parameters and with temperatures of 1.0 for model L and G. We prompt using the natural questions as defined by the datasets. For the analysis, we remove responses from SQuAD that are not clearly out-of-domain. For example, the response "10 million people every year" is not only valid response to geographical question, but can also be an information about the prevalence of the disease. When applying our method, we focus on responses with at least 10 tokens to further remove ambiguous sequences. Modern LLMs tend to be very verbose in their responses, responses should naturally be longer than 10 tokens.
D.6 DATASET CATEGORIES
We list here the categories excluded from SQuAD and included in MMLU for reproducibility.
28

---

Published as a conference paper at ICLR 2025
Excluded From SQuAD
Included in MMLU-Med
Antibiotics
Anatomy
Symbiosis
Clinical knowledge
Gene
College medicine
Brain
College biology
Immunology
College chemistry
Biodiversity
High school biology
Digestion
High school chemistry
Pharmaceutical industry
High school psychology
Mammal
Human aging
Nutrition
Human sexuality
Tuberculosis
Medical genetics Nutrition
On the Origin of Species
Asthma
Professional medicine Virology
Pain
Bacteria
Infection
Black Death
Pharmacy
Immune system
Chloroplast
Table 2: Categories of items used Datasets.
E EXPERIMENTAL RESULTS
E.1 CHARTASK RESULTS
1.0
1.00
Target Domain
Other Dr
0.75
maN (VA
0.5
0.50
Youden's J
Dr
TRR
0.25
FRR
FRR
0.0
0.00
1000
1500
100
200
200
150
-100
50
Log Likellhood Ratio
Logio Constriction Ratio (Median)
Logio
DC
(a)
(b)
(c)
Figure 9: This Figure replicates Figure 3 for the Char dataset.
E.2
TINYSHAKESPPARE RESULTS
1.0
1.00
OUT
IN
0.75
0.5
0.50
Youden's
TRR
0.25
FRR
FRR
0.0
0.00
500
1500 Likelinood Ratio
1000
2000
200
-300
-200
-100
Logio Constriction Ratio (Median)
Logio eDDC
(a)
(b)
(c)
Figure 10: This Figure replicates Figure for the TinyShakespeare dataset.
29

---

Published as a conference paper at ICLR 2025
E.3 20NG
1.0
1.0
Other Dy
Target Domain
0.8
0.6
Thaa
Youden's J
0.4
TRR
D
FRR
0.2
FRR
0.0
0.0
500 Likelihood Ratio
1000
1500
100
150
-20
150
-000
-00
Logie Constriction Ratio (Median)
Logio eDDC
(b)
(c)
Figure 11: Figure 11a shows that log likelihond ratios are well disentangled. Figure 1hb shows the trade-off between OOD and certification: The best OOD detection performance occurs with constriction ratio of 60. Figure 1Tc shows the false rejection rate (FRR) required certify at given
E.4
MEDICAL QA
1.0
Other Dr
Target Domain Dr
0.5
Youden's
0.6
TRR
0.4
0.0
15 Normalized Log Likelihood Ratio
10
100
-10
-5
logıe Constriction (Median)
Login DD
(a)
(b)
(c)
Figure 12: Figure shows log likelihood ratios are well disentangled. Figure 12b
shows trade-off'
between OOD and certification. 12g showswss the false rejection rate (FRR) required to certify at given All results are for VALID with for Medical QA.
shows
Keahooodoo 200
Domain
Domain
Tapi:
Toga:
23
Raspora Langen
Bapoora Langt
33
(b)
Figure 13: This figure demonstrates the gap in log likelihood between in-domain and out-of-domaii samples for the guide models G in Figure 13a and the LLM L in Figure 13b As the length of the response, N increases, the gap between ID (Dr) and OOD data (Dz) widens. The log- lireliieeioodd decreases roughly linearly. Thus, guide model G the left side assigns exponential decreasing probabilities OOD samples.
30

---

Published as a conference paper at ICLR 2025
E.5
CONSTRICTION RATIOS FOR DIFFERENT FALSE REJECTION RATES
FRR
0.00
FRR = 0.00
FRR
0.00
100
200
300
400
200
1000
0
100
200
300
0
25
50
75
100
FRR
0.01
FRR
0.01
FRR
0.01
100
200
300
400
200
1000
0
100
200
300
0
25
50
75
100
FRR
0.05
FRR
0.05
FRR
0.05
200
300
400
220
100
100
300
25
50
75
100
FRR
0.10
FRR
= 0.10
FRR
0.10
100
200
300
400
200
100
0
100
200
300
0
25
50
75
100
FRR
0.25
FRR
= 0.25
FRR
0.25
100
200
300
400
200
100
0
100
200
300
0
25
50
75
100
FRR
0.50
FRR
0.50
FRR
0.50
100 200 Logio Constriction Ratio
300
400
-200
-100 0 100
200
300
0
25
50
75
100
Logio Constriction Ratio
logiCRR (Log Constriction Ratio)
(a)
(b)
(c)
This figure shows the logıo constriction ratios (CR) on OOD samples as function of the false rejection rate (FRR) the in-domain samples. The rejection threshold & is systematically decreased from top bottom achieve given FRR. We observe the gradual improvement in constriction while increasing the FRR. (a) shows Tiny Shakespeare, (b) shows 20NG, and Medical QA.
Figure 14:
31

---

Published as a conference paper at ICLR 2025
E.6 ATOMIC CERTIFICATES LENGTH CONTROLLED
The experimental setup for MedicalQA uses Pub- MedQA as in-domain dataset and SQuAD as out-of- domain dataset as described in Section 3.1 and Ap- pendix D The different lengths of responses these datasets confound our findings on the disentanglement of the atomic certificates, Cy-ACs between in-domain data, Dr, and out-of-domain data, Dp. In Figure [3] lot shorter in Dp
Dataset
Target Domain Dy
Other
show that sequences tend than in Dr. As the likelihood of a response decays ex- ponentially in the length of the responses, the responses in the OOD set DD have relatively high likelihood that attributable the domain restriction, but rather the length of the response. This results the eCDFs in Figure [4b] overlapping significantly. To show that this a confounding factor that is indeed worsening disen- tanglement, resample the data to account for length and present results here.
50
150
Number of Tokens in Response
Figure 15: McdQA sctup: The in-domain dataset (PubMedQA) has longer responses than the OOD dataset (SQuAD).
Setup. We resample the in-domain data, Dr, and out-of-domain data, Dy to have matching distribution of response lengths. We find the target distribution us- ing the following steps: First, we find the common support between the distribution of response length Ny between Dr and Dy, Ny E 38). This interval covers 67% of samples in the target domain dataset and 58% of the OÓD dataset. Second, obtain the empirical distribution of N,, in the in-domain dataset, perform Laplace smoothing (Manning et al. 2008) with a and then further smooth the distribution using moving average with window length of 5. Third, per- form weighted sampling with replacement from Dr and Dp with a size 100 times the original. The sampling weights computed s.t. the distribution of Ny matches the target distribution. We denote these resampled sets DRS and DRS.
Results. We find that the disentanglement atomic certificates, AA,,C, improves greatly after eliminat- ing the confounding factor response lengths. Figure shows the empirical cumulative distribution functions (eCDFs) for "original" datasets, Dr and De in gray tones, as well as the results for D and DRS. You may observe that the distribution of ACs shifted left for datasets representing F and shifted right for datasets representing T, effectively inereasing the disentangle- ment. This indicates that, when comparing similar in- domain and oft-oi-domain samples, the gap in restric- tion larger than presented Figure 4c.. ACs on in- domain samples more permissiveness and ACs oft-oi-domain samples even more constrictive than initially appeared.
1.0
Dataset
0.8
Target DE
DF
Dy:
0.6
Target Domain Dr
0.0
-125
-100
-75
-50
-25
25
Logio
AC
Figure 16: The eCDFs of Ey-ACs are shown for the original in- and out-of-domain data for the MedQA setup in comparison to resam- pled dataset controlling the response length confounder. The gap between the permissive- ness in-domain samples and restrictiveness on out-of-domain samples is greatly improved.
E.7 ATOMIC CERTIFICATE BY LIKELIHOOD
Obtaining tight atomic certificate for sample y is most important when the sample is likely proposed by L. Hence, in this section we study the log constriction ratio, the tightening of our adversarial certifi- cate over model L, as function of the sample's likelihood under
We bin out-on-domain samples into 10 bins based their log likelihood under model L, i.e. log L(y)z), and compute median, 25th and 75th percentile log constriction ratio, as well as the median log likelihood. We present results in Figure 17 for both 20NG and TinyShakespeare. We observe that the constriction strengthens when samples get likely under L. That means, those samples most likely to be sampled under I. benefit most from our atomic certificate. We consider this to be favorable result.
32

---

Published as a conference paper at ICLR 2025
Medos 3orherraaia
Lua000
1,0000
(a)
(b)
Figure 17: These figures show the constriction ratio as function of log likelihood of samples under L for out-of-distribune samples. [ displsss the results for 20NG and Figure 17b for TinyShakespeare. We bin all samples into 10 bins. For each bin the æ-axis shows the median log like likelihond of the sample under I, log L(y x)). The y-axis shows the 1og constriction ratio (median and percentiles for each bin).
F REPEATED SAMPLING (T 1)
In Section 3.3 study the performance of VALID by sampling from L with single step, that is, = 1. Here, we extend the analysis >
However, instead of employing VALID with T 1, we use T E {1, 2,32,3, 4, 5} and study the resulting e DC for combinations of & (the rejection threshold of VALID). As above, for case of presentation we use fixed temperature of 1.0 for L.
Setup. We adopt the MedicalQA setup as described in Appendix D.5
Results. We find that increasing significantly reduces false rejection rates (FRR) while only marginally increasing the cDCC (domain certificate). We present findings for the FRR in Figure T8a and for cDCC Figure 18b The minor increase in c due to increasing T should not come as a surprise recall the tormula for the upper bound: NTG (see (4). Even T 10 increases the upper bound by only one order magnitude. On the other hand, the gains in in-domain performance marked. In Figure 18a 2, greatly improving the refusal behavior the model on in-domain samples. Finally, we note that the temperature of L, 4r, is a confounding factor. For L - 0, we would perform (nearly) deterministie sampling of y a and therefore T would not have any benefit.
observe the FRR is roughly halved for T 5 and
T
10a
T
0.8
10-2
10
0.6
<DC
10-8
0.4
10
10-14
10
0.0
1.0
Rejection Threshold (a)
Rejection Threshald
(b)
Figure 18: False Rejection Rate (FRR) (a) and DC' of the Domain certificate of VALID (b) plotted for range of different values of T and k.
33

---

Published as a conference paper at ICLR 2025
G ABLATION
G.1 COMPARING M TO
Our goal is provide guarantee generalist model assuming that such model outperforms custom, small solutions that are inherently safer due to their domain specific training. We test this empirically by examining the gap in performance between the generalist model L, small in- domain model. As G trained marginally on y, it is not able to perform any task. Hence, we exactly replicate the training procedure of G' and train model the entire sequence, G' (a. y). We utilize the CharTask dataset as described above and study the accuracy of each model generating valid sequences: A valid sequence is one that starts with is followed by random sequence of characters (e.g. 3), followed by four unique task tokens (e.g. A E R) defining a task, which is then performed (e.g. 3 5). The sequence is expected to terminate there. If any of these are violated, the generated sequence scored as invalid. We perform inference on 1000 prompts from the target domain test dataset prompting the model with various lengths of prompts. In Table we present the results: The aceuracy of generating such sequences of L lies significantly above that of G (difference of approx 30%). This shows that G is effective in restricting the domain while performing considerably worse than Hence, our method combines the best of both models: The safety of G with the performance of L.
Prompt Length
G
L
60.45
91.21
60.25
92.68
10
66.89
91.11
Table 3: Accuracy scores for CharTask generation dataset.
G.2 BENEFIT LARGER GUIDE MODELS
In this Appendix, we study the influence of the size of G on the VALID results. In particular, we ask whether VALID benefits from smaller larger models.
Setup. We turn to our MedicalQA setup as described in Section 3.1 and Appendix D.5 With fit
the same methodology, layers, heads and 192 embedding dimension resulting in 27.49M parameters. G follows GPT-2 architecture with 6 layers, 6 heads and 384 embedding dimensions resulting 60.29M parameters. To recap, the G model as used above uses 12 layers, 12 heads and 768 embedding dimension result- ing in 184M parameters. We then compare the three models samples generated by L following Section 3.3
models for G. Gx5 follows GPT-2 architecture with
Results. We find that larger models tend to perform better, however, the evidence is not strong. First, we study the rejection threshold per model. As described in (4) in Theorem [ VALID's upper bounds gets tighter with smaller k. Hence, in Figure 19a we plot k values achieving given false rejection rate (FRR) for each model. We observe that larger models enable smaller k the
1.0
Gzsty)
Gxs(y)
G5())
0.8
Gyty
Gly)
G(y)
36
0.6
0.4
0.0
0.1
0.2
0.3
0.4
0.5
-20
-15
-10
FRR
Logio eDC
(a)
(b)
Figure 19: These Figures demonstrate differences in the behavior of VALID for different sizes of guide models G. Figure shows that larger models allow for lower and hence lower bounds the False Rejection Rate (FRR). Figure [9blsoows the FRR (xiaxis) given eDDC for guide models different sizes.
34

---

Published as a conference paper at ICLR 2025
This indicates that the trade-off in k between certification and OOD detection is more favorable under larger models. This should not come as a surprise, as larger models tend achieve better perplexity (i.e. lower loss) on in-domain data.
same FRR.
Next, we study the constriction ratios of the Atomic Certificates (AC) and present results for different sizes of G' shown in T For each model, we provide the the 10th percentile, median and 90th percentile. You may observe that sy consistently provides constriction ratios that are often around 10 orders of magnitudes worse than Gs(y) and Gyy) Interestingly, Gs(y) yields better ratios than Gly). However, the difference smaller. We speculate that the limited amount of ID training data means we do not see benefits for increasing the size of G beyond a point, as it begins overfit without increasing regularization.
Finally, we study the Domain Certificates (DC) for each model. For this we replicate Figure 12c and present Figure 19b showing the false rejection rate (FRR) given e-DC for the three models. We may observe that the lower bound to the FRR significantly increases as the models become smaller. The evidence here suggests that larger guide models yield better domain certificates.
In conclusion, the evidence points to larger models working better for an application like MedQA. The evidence uniformly shows that a model as small as Gxs(y) does perform significantly worse than larger models.
Logio Constriction Ratio (10% /Median / 900 9)
FRR
Gxs(y)
Gs(u)
G(yy
0%
/ /4
-408/-11/0
-449 / -54
1 1
-246/-14/
-176/-3/79
-198 198/-10/4 /40
5%
-74/12/
42/21/195
-42/18 162
10 %
-29/24/
-11/35/23
-8/33 229
20 %
-3/43/29
1/57 337
3/50/ 302
25%
0/ 308
5/ / 364
7/ 345
50% %
11/81/ /1/80/430
13 96 497
89 477
Table 4: Constriction Ratios for MedicalQA for three models of different sizes. The smallest model yields significantly worse (lower) constriction ratios.
H BENCHMARKING
In this section, we provide a comprehensive description the PubMedQA experimental setup pre- sented in Section 3.4 present additional benchmarking results, and extend evaluation framework to the MMLU benchmark (Hendrycks et al. 2021).
H.1 PUBMEDQA
Setup. The PubMedQA benchmark (Jin et al. 2019) comprises 1000 items. Each item con- tains background information (context), multiple-choiee question (answerable by mes/no/maybe), a long-text answer, and a ground truth label (yes/no/maybee. As illustrated in Figure we evaluate the model through two streams: "item correctness" and "response acceptance". In both streams, we prompt the model with the context and question. In the "item correctness" " stream, the model provided with all multiple-choice tokens, and the maximum likelihood answer is selected and evaluated for correctness. In the "response acceptance" stream, we present the long-text answer as response and determine if model M abstains at a given domain certificate of c. An item is consid- ered correct at if and only if the response is accepted and the model scores correctly. We use the reasoning-required variant of the PubMedQA benchmark (for further details, see Din et al. (2019)).
Results. Extending our analysis of the PubMedQA benchmark presented in Section 3.4 we exam- ined the relationship between PubMedQA performance scores and median constriction ratios. As illustrated in Figure 20a| our findings demonstrate that the model can achieve log constriction ratio of 20 on samples in Dp while maintaining robust PubMedQA performance. Specifically, at a performance threshold of 70% accuracy, observed log C Rs. value of 21.6, which effectively
35

---

Published as a conference paper at ICLR 2025
FRR
PubMedQA@e
FRR
MMLU-Med@t
FRR
MMLU-Med @+
1.00
1.00
1.00
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0.00
0.00
0.00
0
20 CR, (Median)
40
20
10
10 login eDD (b)
20
20 logio (Median)
40
(a)
Figure 20: Evaluation of domain-certified models through standardized benchmarking. Figure 20a illus- trales the association between log constriction ratios and the PubMedQAA @e benchmark with varying eDDC certifications. Figure [20b presents the MMLU@e c metric evaluated at different certifica- tion thresholds e. (c) Figure 20c shows the relationship between log constriction ratios and corresponding MMLU@< e scores across multiple certification levels.
models
constrains out-of-domain samples to probabilities at least 1 x 10-21 times lower than the likeli- hood of samples in distribution L. This indicates a strong capacity for domain constriction while preserving task performance.
H.2 MMLU-MED
In this section, we extend the benchmarking of our certified model M for medical question answer- ing to the MMLU benchmark (Hendrycks et al. (021). To that end, we follow the same methodol- ogy as above for the PubMedQA benchmark. an earlier version of this work, we reported MMLU results that were erroncous, which we correct here.
Setup. MMLU comprises thousands of questions spanning various domains of general and profes- sional factual knowledge. As our model M is deployed for medical questions, we focus on a subset of MMLU categories that fall within domain T. We specify the selected categories in Table and designate this remaining benchmark as MMLU-Med.
MMLU's standard format provides n-shot examples with four possible answers (A through D) fol- lowed by a question in the same format. The model is then prompted to select the correct re- sponse. However, this setup does not reflect a realistic user-system interaction. Therefore, similar PubMedQA, we introduce the MMLU-Med@ e metric, which separates the evaluation into two streams: (1) standard assessment of model L on MMLU-Med to determine correctness, and (2) testing whether the correct question-answer pair is rejected by our algorithm. The process is sum- marized in Figure 21 We score an item as correct when the model scores correctly while maintaining its e DC on the realistic question-answe answer pair.
Results. Our evaluation yields mixed evidence regarding the model's performance on MMLU- Med. Following the same analysis for PubMedQA in Section 3.4 we present the MMLU-
metric in Figure 20b As shown, MMLU-Med@ = 37. 1 %. that is, the model retains 37.1% 1% accuracy when certified at 1, log10 0. The 10 10 DDC model achieves score of 14.1 %. In addition, to the domain certificates, we study the median constriction our model in relation to
MMLU-Med @ c: Set k s.t. M is
DC on Df
Question Correctness
Accepting Response
Which of the following true of Graves Disease of the thyroid?
Question: Which of the following true of Graves Disease of the thyroid?
A: It cause of ophthaalmoleeia
B: It causes a large multi-nodular goitre
It is cause of ophthalmoplegia
Answer:
X C: It is commoner in males than females
X D: In the past, Grave's discase sometimes caused 'Derbysiire Neck'
log L(ylx)/G(y) s KN
Answer: (A,)
Correct Answer Answer Accepted. Question Score:
Figure 21: The MMLU@e benchmark assesses MMLU performance while satisfying eDDC certificate. The correctness is scored as commonly done for MMLU (left). The correct question answer pair is checked for acceptance rejection by M. Only if sample is accepted and correct, the question is scored positively. For questions not ending "?", the sentence concatenated without keywords.
36

---

Published as a conference paper at ICLR 2025
its certified performance. The evidence provided in Figure 20c
indicates that median constriction ratio 10 achieved on out-of-domain samples together with a score of 65% on the MMLU- Med @ benchmark. Further, a median constriction of x 10020 is achieved with an MMLU-Med score 37%.
These results are considerably weaker than the strong results presented above for PubMedQA raising the question
why this is. Figure 22 we investigate the domain shift between PubMedQA and MMLU. In particular, Figure 22a shows the distribution of log likelihoods of in-domain samples (Dr), out-of-domain samples (Dz) and MMLU samples under guide model G and Figure (2b shows the log likelihood ratios for the same samples. We observe a considerable overlap between the distributions of MMLU-Med and PubMedQA samples. However, the distribution of MMLU-Med has long-tail into the distribution of Dr. This explains our results quite well. On the one hand, the large overlapping mass of MMLU-Med and PubMedQA explains why M accepts wide range of MMLU-Med responses while significantly constricting the model on Dz. On the other hand, the long tail of the distribution of MMLU scores into the distribution of De indicates range of MMLU- Med questions will be rejected unless the certificates become vacuous, making it hard challenging high MMLU-Med @e performance. We believe that training G on MMLU-style QA pairs would significantly improve results but leave this as future direction.
Target Domain Dr
Other D)
MMLU-Med
-18
-16
-14
-12
-10 -8 6 -4
-2
4
6
14 16
log G(y) (Norm. Log Likelihood)
log log GGy (Norm. LL Ratio)
(a)
(b)
Figure 22: Comparison of likelihood of three datasets under model G. showing MMI.U exhibits domain shift relative Pub Figure 22alindicates likelihood of MMLU-Med lies in-between in-domain data Dr (PubMedQA) and out-of-domain data Dz. Figure [226] shows the normalized log likelihood ratio used in VALID lead to frequent rejections due to domain shift.
37


Reviews:
[{"review_id": "gnNUnMrp63", "reviewer": "Reviewer_tYKL", "summary": "The paper proposes a theoretical foundation for certifying language models trained to produce outputs in a domain. Then the work proses VALID, an algrithm that provides provable upper bounds on out of domain behavior of language models. The method preserves a good chunk of the unconstrained LM's performance on MMLU@Med while being robust to poducing out-of-domain outputs. There are also various other experiment results on datasets like TinyShakespeare and 20NG that show the effectiveness of the method to potentially detect and certify OOD behavior of language models.", "questions": "1. Is there a reason that certified benchmarking was only done on MMLU@Med? I think it would be useful to have results for other benchmarks as well.\n2. Could we use a guide model G to detect whether an input is in-domain? It might be interesting to see how the likelihood of L(y|x) would change for \"in-domain\" and ood inputs.\n3. While the authors mention certification in computer vision (line 80-81), there has been some work on certification in NLP as well, such as https://arxiv.org/abs/2401.01262, https://arxiv.org/abs/2402.15929, https://arxiv.org/abs/2403.10144v1.  I think mentions of some of these works would prevent the false impression that certification has only been applied to computer vision and provide a more complete picture of the certification work being done across domains.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Strong theoretical foundations for classifying out-of-domain behavior of language models and ways to prevent this.\n- The algorithm VALID is relatively straightforward and uses rejection sampling as an elegant way to achieve the certifications.\n- The empirical evidence presented is across various kinds of datasets(Tiny Shakespeare, MedQA) which shows the generizability of VALID.\n- The work is novel and needed to provide theoretical insights and guarantees for safe deployment of language models.\n- Well written and the progression from atomic to domain certificates is explained well.", "weaknesses": "The paper acknowledges most of the limitations, but there is still room for further discussion:\n1. Lack of context for guide model G. The work does argue that potentially involving the context in the final answer could fix this issue, but then the method cannot work in cases where a user wants language model to be concise and this also increases the inference cost of models as many more tokens get sampled for the output.\n2. Adversarial attacks on G/M. The work acknowledges and shows adversarial attacks this method is prone to but argues that as adversaries would need white-box access to G(line 448, 453) and so the attacks may not be feasible. White-box access does not need to hold for the success of adversarial attacks, as some attacks do generalize across various models even if they were originally targeted at some other specific model. This should be further investigated.\n3. Rejection sampling with T>1  incurs an inefficiency, which could be reduced as multiple samples could be drawn in parallel. In any case, there is additional computational overhead.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JwnWMDM40a", "reviewer": "Reviewer_pqHZ", "summary": "This work studies domain certification, which refers to the characterization of the out-of-domain behavior of language models. This paper first formaize the definition of domain certification. And then propose an approach, VALID, to achieve domain certification. The proposed approach is empirically evaluated on multiple datasets in various domains including TinyShakespeare, 20NG, MedicalQA. The experimetal results show that the proposed method is effective at achieving domain certification as defined in this work.", "questions": "See weaknesses section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This work introduces and formalize domain certification for characterizing the out-of-domain behavior of language models under adversarial attach, which is useful for language models tuned a specific domain.\n2. An approach called VALID is proposed to achieve domain certification. VALID bounds the probability of a LLM answering out-of-domain questions.", "weaknesses": "The main concern for this work is its limitations including the reliance on the domain generator which doesn't consider the model input. In addition, Theorem 1 assumes the certificate is useful given G is trained on in-domain data. However, as language models are usually pre-trained on large amound of text data, which ingests world knowledge into it. Therefore, model G can contains out-of-domain knowedge, which makes Theorem 1 extremely limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gvpuZIUGK2", "reviewer": "Reviewer_8eQm", "summary": "This work studies the problem of adversarial certificates for LLMs in domain specific deployment contexts. They define the problem of \"domain certification\" and derive a test time algorithm VALID based on a small guide/reference model, and a rejection sampling criterion, that provides such domain certificates. They quantify levels of tightness and permissiveness of their certificates using real data on out of domain and in domain samples respectively to argue for the potential utility their method in real deployments.", "questions": "Primary question (multi-part, single theme): \n\nInitially, I was going to comment that training guide LLM, G, on only \"good strings\" in T, will not necessarily give you a model that only assigns high likelihood to strings in T, and therefore there will likely exist some strings for which both L and G assign high likelihood but are in fact in T'... however, after reaching the Section 3.1 experimental setup detail that guide models G were to be parametrized by relatively tiny gpt2 style models, it became clear how this might work in practice. \n\n1. As a preliminary ask, \"Such a certificate with respect to G can be useful: As G is only trained on samples in DT ⊂ T, a dataset of domain T, it assigns exponentially small likelihood to samples that are in F.\"\nIs there an analytic or empirical argument that the authors can provide for this statement? It seems like both the benign and adversarial robustness of the certificate hinge on this assumption.\n\nNow, the section of future work describes a desire to explore using stronger models for G, which suggests that the authors don't necessarily see the _weakness of G_, eg. its inability to predict next tokens in any context beyond the small corpus on which it was fit, and maybe poorly even within that domain, as the actual crux of the method that _makes it work_. \n\n2. Was any ablation done about the model size for G and or the corpus size for training it? My hypothesis is that the stronger G becomes (more parameters and/or data), the wider set of strings it is able to model and therefor the closer L(y|x) and G(y) become, eventually depleting all the power of the algorithm. \n\n3. Overall, can the authors elaborate on any \"behind the scenes\" details that aren't in the draft or appendix on how G's role in the certificate was developed during the research and why the particular experimental choices (tiny gpt2's) for it were made?\n\nWhatever the answers are to this series of questions are, in my opinion, necessary additions to the draft to improve its clarity and soundness, as well as the ability for future work to build upon these results. Currently, my takeaway centers the identification of a weak G to compute an odds ratio against L as a way to perform rejection sampling for safety and alignment, and thus pitching the results a little more in this direction potentially improves the relevance and generalizability of the work (since the particular certificates/and algorithm might be limited in immediate applicability). \n\nSome relevant works that abstractly leverage the same odds ratio paradigm, though for different purposes, are Contrastive Decoding by Li et al. (2210.15097) which moderates policy L via a weaker policy L' at decoding time to improve output quality, and Direct Preference Optimization by Rafailov et al. (2305.18290) as a way to train a policy L to prefer samples in T rather than T'.\n\nMinor:\n\n4. L257, does \"OOD\" mean \"F\"? To match L 269 referring to other question domains as \"F\". Generally, maybe prefer one notation or the other to refer to the \"bad\" set everywhere.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Motivate the difficulty and real world implications of certifiable test time behavior for model LLM systems.\n2. Present a simple and clear algorithm to achieve their definition of domain certification.\n3. Identify a clever way to harness the limited abilities of very small, cheap to train, domain specific language models to implement test time rejection sampling favoring a constrained distribution of responses.", "weaknesses": "1. Work under very limiting constraints of a fixed set of bad strings F. This is a practical assumption, but pervasive limitation for this work and of any other attempts to characterize the output spaces of generative models with large vocabularies, and variable size outputs. See lines 132-137 for the authors' own description of this required narrowing of scope. This always leaves room for adversaries to circumvent certificates via finding inputs in the complement of the finite sample of F chosen, but still in T' such that they are falsely permitted by the system.\n\n2. It's not immediately clear how good the D_T to D_F ratios are in Table 1. Additionally, there seems to be a correlation between success/separation and the domain similarity, which is expected? Maybe the authors can discuss this further potentially highlighting a limitation of the approach based on how well separated good and bad behaviors are in a given domain.\n\n3. The generative results are weaker than the fixed sample set results, summarized lets say as weaker Youden's J for the same domain setting, and since this is the more realistic evaluation, it is important to caution that this approach isn't quite deployable at the moment. (On that note, lines 145-152 are more appropriate moved to the final paragraphs as a discussion section/future work/applications blurb)\n\nTogether, the problem formulation and empirical results leave the work in a somewhat middling position in terms of contribution to the field, but depending on the potential discussion surrounding questions below, this might be improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zNxNEYHKXz", "reviewer": "Reviewer_XBJU", "summary": "The authors propose a method to assess and mitigate the risk of Large Language Models (LLMs) deployed in particular tasks answering out-of-domain questions, a concept/framework which they formalize as domain certification. The proposed method, VALID, provides adversarial bounds as a certificate. Finally, this method is evaluated over 3 datasets.", "questions": "- How does this method compare to setting up restrictive (already existing) guardrail methods specific to a domain/task?\n- If the LLM is being deployed for a specific task (such as the tax report example used in throughout the paper), how easy is it to set up VALID? Do you use a finetuned version of L to form G? Do you use a smaller finetuned LLM? Is it trained from scratch? Is it a simple discriminator? This is something that may be clarified once source code is available, but in any case it would have been great if at least an anonymized repository was provided.  \n- What is contained in F′ ∩ T′? Seems unclear to me. Only semantically incoherent sequences? \n- VALID requires a secondary language model G, trained on in-domain data. In that case, since domain data is necessary anyway, wouldn't it be simpler to just quantify similarities between an output and the in-domain corpus within an embedding space, sparing the need for secondary model trained with in-domain data?\n- Around line 150, you mention \"The deployer might perform an a priori risk assessment and determine that they can tolerate the consequences of one out-of-domain response from a set D_F per year.\" But given your method depends on F for domain certification, and VALID depends on G, are there any guarantees that only one out-of-domain response will be produced per year? Seems like a very bold claim.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is interesting, well motivated and tackles a relevant problem. \n\nThe problem and concepts are well-defined. Domain certification is a interesting concept, and VALID's design appears well justified.   \n\nDespite the lack of discussion and comparison on existing guardrails this method benefits from providing some type of mathematical guarantees, which is a desirable property given recent legislation (particularly in the EU), as the authors mention.", "weaknesses": "The literature review should be improved. There is little to no discussion on related methods/tasks, and existing approaches to tackle this issue (if any). For example, a greater discussion on methods regarding LLM guardrails, and how they compare to your approach.\n\nDomain certification always requires the definition of F, which if I understood correctly, is used for the definition of domain certification and the experimental procedure, since VALID was motivated mainly by the \"certification through divergence\" approach. Adding some general recommendations on the definition of F in general terms, or for applied scenarios, would be interesting (e.g., I could find vast amounts of OOD data in benchmark datasets for a tax advisory LLM, but is there an efficient way to select a representative F?)  \n\nIt would be interesting to see how different definitions of the model G affect the quality of this method.\n\nThe experimental results lack a comparison to any benchmark or baseline method. Literature on LLM guardrails could be used as a way to compare the effectiveness of this approach. In my opinion, the main and most important weakness of this paper is the experimental setup, followed by the literature review, both of which should be substantially improved. \n\nI am accepting this paper under the assumption these weaknesses will be addressed, and am willing to improve the overall score at a later stage, depending on the quality of the revised version.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)