Noiser: Bounded Input Perturbations for Attributing Large Language Models
Authors: Mohammad Reza Ghasemi Madani
Aryo Pradipta Gema
Yu Zhao
Gabriele Sarti
Pasquale Minervini
Andrea Passerini
Venue: COLM 2025 (2025)
Keywords: Feature Attribution
Post-hoc Explanations
Large Language Model
LLMs
TL;DR: We introduce Noiser, a bounded perturbation-based method to explain LLMs predictions and tackle distribution shift. We also propose answerability, a new metric that evaluates the relevance of rationales without gold data or human evaluation.

Abstract: Feature attribution (FA) methods are common post-hoc approaches that explain how Large Language Models (LLMs) make predictions. Accordingly, generating faithful attributions that reflect the actual inner behavior of the model is crucial. In this paper, we introduce Noiser, a perturbation-based FA method that imposes bounded noise on each input embedding and measures the robustness of the model against partially noised input to obtain the input attributions. Additionally, we propose an answerability metric that employs an instructed judge model to assess the extent to which highly scored tokens suffice to recover the predicted output. Through a comprehensive evaluation across six LLMs and three tasks, we demonstrate that Noiser consistently outperforms existing gradient-based, attention-based, and perturbation-based FA methods in terms of both faithfulness and answerability, making it a robust and effective approach for explaining language model predictions.

Published as a conference paper at COLM 2025
Noiser: Bounded Input Perturbations for Attributing Large Language Models
Mohammad Reza Ghasemi Madani Aryo Pradipta Gema² Gabriele Sarti²
Yu Zhao? Pasquale Minervini Andrea Passerini*
University of Trento ?University of Edinburgh
CLCG, University of Groningen
Miniml.AI
Abstract
Feature attribution (EA) methods are common post-hoc approaches that ex- plain how Large Language Models (LLMs) make predictions. Accordingly, generating faithful attributions that reflect the actual inner behavior of the model is crucial. In this paper, introduce NOISER, perturbation-based FA method that imposes bounded noise on each input embedding and measures the robustness of the model against partially noised input to obtain the input attributions. Additionally, we propose an answerabilliy metric that employs an instructed judge model to assess the extent to which highly scored tokens suffice to recover the predicted output. Through a comprehensive evaluation across six LLMs and three tasks, we demonstrate that Noiser consistently outperforms existing gradient-basee based, attention- based, and perturbation-base FA methods in terms of both faithfulness and answerability, making it robust and effective approach for explaining language model predictions.'
1 Introduction
Transformer-based language models (Vaswani et al., 2023) are fundamental to the latest advancements natural language processing (Team al., 2024; Touvron al., 2023; Bai et al., 2023; DeepSeek-Al. 2025; OpenAl et al., 2024). However, they are often perceived as opaque (Rudin, 2019; Doshi- Velez Kim, 2017; Lipton, 2018), sparking significant interest in the development of algorithms that automatically explain the behavior of these models (Denil et al., 2015a; Sundararajan et al., 2017a; Camburu et al., 2018; Rajani et al., 2019; Luo et al., 2022).
Feature attribution (FA) techniques are popular post-hoc methods that generate token-level importance scores to highlight the contribution of each token to a prediction (Denil et al., 2015b; Jain et al., 2020; Kersten et al., 2021). The top-p% important tokens are typically con- sidered as the prediction rationale (Zaidan al., 2007; Sundararajan al., 2017b; De et al., 2020). The quality of a rationale is often evaluated using faillifllness metrics, which measure to what extent the rationales accurately reflect the downstream task on model predictions.
Perturbation-bassed FA methods aim explore neural networks by modifying the input of a model and observing the changes in the output to indicate which parts of the input are particularly important for inference. These methods are widely adopted in computer vision, leveraging the continuous nature of image inputs, where localized noise or masking preserves semantic coherence and avoids distribution shifts (Ivanovs al., 2021). In contrast, NLP models face inherent challenges due to the discrete structure of the text, where even minor perturbations whether token substitutions or embedding modifications can push inputs out-of-distribution (OOD), destabilizing predictions and confounding attribution analysis (Liu et al., 2019).
"Correspondence to: rezamadani.c ai@gmail. rigmail..cmm com. 'Qur code: https:/ thub. com/qasem /Noisser

---

Published as a conference paper at COLM 2025
This divergence underscores need for bounded perturbations in NLP, ensuring perturbed inputs remain in-distribution. Our work bridges this gap by exploring noise thresholds that alter token embeddings while preserving the original prediction to limit perturbation- induced OOD issues. Particularly, we introduce a perturbation-basen model's robustness against noisy papuds-examples created by introducing small alterations to the input embeddings without changing the model's original prediction- enabling reliable explanations grounded in the model's trained operational domain while quantifying feature importance through robustness to controlled perturbations. Our work makes the following contributions:
FA by exploring
We empirically show that NOISER consistently more faithful than nine popular FAs by conducting comprehensive experiments, covering three tasks and six LMs of varying sizes from three different model families;
We propose new plausibility metric, answerability, which measures the extent to which the top-p% attributed input tokens sufficiently support the target output. By leveraging language models, this metric assesses whether minimal subset of input tokens is adequate for generating the expected prediction, providing a simulatable alternative to human plausibility judgments.
2 Background
2.1 Generative Language Modeling
In generative language modeling, the input consists of sequence of tokens, denoted as X= [x XT 1]- The objective is to develop a model, Fe, that estimates the probability distribution P over the token sequence X. In this context, F0 represents a specific pre-trained generative language model characterized by parameters 0.
I-1
((A....,XTT-) = Fx() II
Fa(x; xo (x-1)
2.2 Input Importance for Generative LMs
Given a model Fo, our objective is determine the importance distribution of the input tokens for each predicted token xT, based on the preceding sequence X [xp. feature attribution method, denoted as ет, applied at position T, yields an importance distribution ST [So. ..., 1 corresponding to the target token XT, where higher value of s; indicates greater importance of the input token x; in predicting XT-
er (F X,)) + ST
2.3 Bounded Perturbations
Bounded perturbations refer small, structured uncertainties in mathematical systems where specified constraint limits the perturbation magnitude. These are critical for analyzing system robustness against disturbances while ensuring predictable behavior.
Perturbation is minor alteration to system, such as 8 added to nominal matrix A, resulting in A + 8. This captures uncertainties or disturbances. If the perturbation magnitude where > 0, it is called bounded perturbation. Consider the
is bounded |||| nominal system x = Ax. Under a perturbation ó, the system becomes:
(A (A+00 ō)x.
3
Our Method
Let n E Rmdddde denote a noise vector where each component n; ~ (0,11). We form noisy examples from original inputs by imposing small perturbations to the input embeddings,
2

---

Published as a conference paper at COLM 2025
such that the noisy input results in the model outputting incorrect answer. For this pur- pose, we first pass a prompt X [xor..., into Fa to collect the probability distribution, P, over the model's vocabulary with xy being the most likely output (i.e,, Fo(X) x7).
In the next step, we utilize a binary search algorithm to find the maximum scaling factor such that if we perturb the embedding of a targeted token with scaled k the model wouldn't change its initial prediction xy- Specifically, we set X; == x; + nscalld and let Fe to continue, giving us a set of corrupted probabilities PT, Because F& partially loses information about the corrupted token, the probability of xr from the first step would likely be lower in .
We repeat the process for each token until we obtain K = kko...,,ky where each k; is the maximum scaling factor such that we corrupt the embeddings x; using nscaledd = k; n, the model wouldn't change its original output. The mathematical representation of K is illustrated below:
K
(k: Vk > k;
Fo perturbeed))) # xT, Fo perturbed xT), i
1}
where X perturbedllk = [xo... (xi Dscaled) xx-1] is the input sequence in which x; is altered with sscoled Nbounded- The equation above indicates that each scale factor k; is such that for all values k greater than k; if we perturb using sclaed = k Ibbuunded to create a noisy input X pertuurbed]k Fo would return different output from the original one (x7).
In the final step, we find the kmin = min(K) to generate the final noise samples nscaled = kin n to add to each token embedding and obtain the token scores using the following:
S = (si = = p(X) - p(x perturbeeddkkaa } E fO,
Using kmin, we ensure to perturb the input enough to reach a flipping point in prediction to get the minimal set of features needed achieve this outcome. The intuition is that tokens with higher importance are more sensitive to noise injection, resulting in a larger reduction in the model's output likelihood.
To show the effectiveness of selecting kmin, we propose different boundings and measure their faithfulness. We analyse using maximum noise across tokens (kmax); ii) individual token maximum noise where k is different for each input token and is the maximum the model can tolerate (kmax per token); iii) norm-bounded setting where the noise vector n is divided by the expected value of the noise vector Lp norm, E |||n||]] and iv) random where is randomly selected from the uniform distribution. The details of each configuration are provided in Section 5.
Experiment
4.1 Model Data
In our study, we employ variants of Qwen (Bai et al., 2023), Gemma (Team et al., 2024), and Llama (Touvron et al., 2023) models. We choose our models to span from hundreds of millions few billion parameters as we want to explore how the model size affects the faithfulness of each FA. All models used are publicly available." 2
We use KNOWN d provided by Meng et al. (2023) and LONG-RANCE AGREEMENT (LONGRA; Vafa et al., 2021) to conduct our analysis. Besides, for long generation we utilize WIKIBIO (Lebret et al., 2016). The following is an instance from the KNOWN dataset.
We use checkpoints from the Huggingface library for each model. Dataset can be found at: ht https: /rome. baulab. info/data/dseed dstts///wst/ttonnn 1800.json
3

---

Published as a conference paper at COLM 2025
LeBron James professionally plays the sport of [basketball]
The LongRA dataset consists of word pairs that exhibit either a semantic or syntactic rela- tionship. Additionally, Vafa et al. (2021) incorporate a distractor sentence, which provides no relevant information about the word pair, to evaluate long- long-range agreement. An example from the LongRA dataset is shown below, with the distractor included in parentheses.
When my flight landed in Japan, I converted my currency and slowly fell asleep. (I had terrifying dream about my grandmother, but that's story for another time). I was staying in the capital, [Tokyo]
WIKIBIO a dataset consisting of Wikipedia biographies. We use the first two sentences as prompt, similar to Manakul et al. (2023). The model is then expected to continue generating the biography. This task inherently more open-ended compared to the previous two.
Super Mario Land is 1989 side-scrolling platform video game
The computation of each task's faithfulness is provided in Section 4.5.
4.2 Baselines
Following previous works, we compare our rationalization method to a variety of gradient- and attention-based baselines (Vafa et al., 2021). Inputx Gradient (Denil et al., 2015b) uses embedding gradients multiplied by the embeddings; Integrated Gradients (Sundararajan et al., 2017b) integrate overall gradients using a linear interpolation between a baseline input (all zero embeddings) and the original input. Gradient SHAP (Lundberg & Lee, 2017) compute the gradient w.r.t. randomly selected points between the inputs and a baseline distribution; DeepLIFT (Shrikumar et al., 2019) compares the activation of each neuron to its 'reference activation' and assigns contribution scores according to the difference. Sequential Integrated Gradients (Enguehard, 2023) extends Integrated Gradients by breaking down the input perturbation into sequential steps, computing gradients at each step, and aggregating them to provide more stable and interpretable attributions, Last Attention (Jain et al., 2020) uses the last-layer attention weights averaged across heads; Attention Rollout (Abnar & Zuidema, 2020) recursively computing the token attention in each layer, e.g., computing the attention from all positions in layer I; to all positions in layer I, where < <;; LIME (Ribeiro et al., 2016) trains a linear surrogate model using data points randomly sampled locally around the prediction. Occlusion (Zeiler & Fergus, 2014) involves systematically occluding different portions of the input and observing the impact on the output confidence.
4.3 Faithfulness Metrics
To assess whether a rationale extracted with a given FA is faithful, iee, actually reflects the true model reasoning (Jacovi & Goldberg, 2021), various faithfulness metrics have been proposed (Arras et al., 2017; Serrano & Smith, 2019; Jain et al., 2020; De et al., 2020). Sufficiency and comprehensivenes: (DeYoung et al., 2020) are two widely used metrics that effectively capture rationale faithfulness (Chrysostomou & Aletras, 2021; Chan et al., 2022b). Both metrics use hard erasure criterion for perturbing the input by entirely removing (i.e., comprehensiveness) or retaining (f.e., sufficiency) the rationale to observe changes in predictive likelihood. This hard criterion ignores the importance of each individual token, treating them all equally for computing sufficiency and comprehensiveness.
We evaluate rationales using soft sufficiency (Soft-NS) and comprehensiveness (Soft-NC) proposed by Zhao & Aletras (2023) to measure the faithfulness of the full importance distribution. Using these metrics, instead of entirely removing or retaining tokens from the input, we randomly mask parts of the token vector representations proportionately to their FA importance. The summation of Soft-NC and Soft-NS is considered as the final faithfulness score. For the detailed implementation of these metrics, please refer to Appendix A.
4

---

Published as a conference paper at COLM 2025
"LeBron James professionally plays the sport of "basketball"
LeBren
plays
Soor:
Of
Instruction
basketball
Score-O.96
0.0
0118
0,23
Judge
Mocovvs employed by 'CBS'
Leale
Moonves
employad
Instruction
X
Sooore000
G44
321
0.22
Judge
ScoreeO
Figure 1: Answerability metrics evaluation. To get the answerability metrics, the judge model is instructed to predict the completion token given limited set of tokens from the original prompt.
4.4 Answerability Metrics
The utilization of LLMs has emerged as prominent trend across numerous research domains (Peng al., 2023; Zhou et al., 2023a; Taori et al., 2023). With any given instructions, LLMs are expected to generate responses that align with these instructions (Chen et al., 2024; Li et al., 2024; Xu al., 2023; Longpre et al., 2023).
This capability, known as the "instruction following" ability, serves as key metric for assessing the effectiveness of LLMs (Chen et al., 2024; Zhao et al., 2024; Taori et al., 2023; Zheng et al., 2023). To facilitate more thorough assessment, several benchmarks have been introduced with a focus on instruction following (Zhou et al., 2023b; Qin al., 2024).
We exploit this progress in our answerability metric by framing attribution evaluation as an instruction-based completion task. To see whether the atfributions illustrate any meaningful association with predicted output, we extend our evaluation of FAs through prompt engineering. For this purpose, we aggregate the attribution scores of each word's sub-tokens to derive word-level scores. Then we select the top-p% most important words w.r.t their scores and feed these words as input to judge model along with a task prompt. The task prompt asks the judge model to predict the completion token using this limited set of words. Feldhus et al. (2023) offer a complementary perspective verbalizations, approach instead uses an LLM to directly quantify whether the selected tokens are sufficient for the prediction task.
they leverage
LLMs for generating theerpetabblity-enhaaeee
We evaluate attributions by computing the number of samples for which the judge model generates the correct output, which define the FAs answerability rate. Additionally, for these correctly predicted samples, we aggregate the word-level attribution scores to obtain the so-called answerability score. For both metrics, higher values indicate better performance. In our evaluation, a higher answerability rate means that a larger proportion of samples allow the LM judge to correctly predict the output using only the minimal set of tokens. Likewise, higher answerability score reteee-rfllctingg.. a greater aggregated attribution mass within that token set-indicates that the attribution method is more effective at isolating the minimal semantic requirements for prediction.
This evaluation pipeline specifically applies catassess such as KNOWN where the gold label is meaningful word that must be inferred from the input sequence. Figure shows an answerability evaluation example. The prompt used for this evaluation shown in Appendix F.
4.5 Implementation Details
None of our experiments involved training fine-tuning any language models. All FAs are built upon Inseq library (Sarti et al., 2023; 2024) except Last Attention and Attention Rollout, which we used the codebase from Zhao & Shan (2024). For NOISER, generate 10 different noise vectors during the corrupted run for more consistent results. Binary search is done in 10 steps, yielding the accuracy of z 0.001 for the scaling factor (k). For KNONW and

---

We chose the top-50% of Published as a conference paper at COLM 2025
KNOWN
Method
Qwen2-0.5B
Llama3.2-1B
Qwen2-1.5B
gemma-2-2B
gemma-2-99
Llama3-8B
Average
Last Attn
-0.0857
-0.0601
0.0607
0.1407
-0.2788
-0.0095
-0.0388
Rollout
-0.1161
-0.0471
0.1211
0.4624
-0.3607
-0.2625
-0.0338
SHAP
0.4946
0.3746
0.5390
0,3726
0,9203
0.1925
0,4823
IxG
012117
0.7059
0.4612
0.5233
1.0276
0,5891
0.5865
IG
0.2176
0.5428
0.5163
0.2015
1,0355
0.3759
0.4816
DIcPLIFT
0.3030
0.5473
0.5323
0.3557
0.8638
0.5174
0.5199
SIG
0.0361
0.3534
0,3003
-0.1879
0.7877
0.2755
0,2609
LIME
0.2439
0.5103
0.3826
0.3567
0.4832
0.6555
0.4392
Occlusion
0.1627
0.5373
0.2477
0.5341
NOISER
2.1854
1.3989
144000
1.4433
2.1767
2.2175
1.8103
LONGRA
Method
Qwen2-0.5B
Llama3.2-1B
Qwen2-1.5B
gemma-2-2B
gemma-2-98
Llama3-8B
Average
Last
1.9148
0.3255
-0.0110
-0.2382
-0.2382
1.0762
0.4715
Rollout
1.8517
0.2451
0.0802
-0.2643
-0.2643
1.2283
0.4794
SHAP
3.7970
1.2837
1.6276
1,9746
2.2769
0.7696
1.9549
IxC
3.8972
1.7299
1.5370
2.5803
2,5803
2.0796
2.4007
4.3388
1.3066
1.5495
1.3U23
1.3023
3.7190
2,2531
DIe
4.4991
1.7889
1.5512
2.7428
2.7428
2.1258
2.5751
SIG
3.8645
0.9272
1.1047
0.5412
0.5412
1.1618
1.3568
LIME
1.0765
0.2212
-0.4147
-0.1636
-0.1636
2.2995
0.4739
Occlusion
3.9424
1.9887
1.0145
3.4418
4.2240
NOISER
6.8055
4.8072
3.1779
4.2727
6.1681
5.1627
5.0657
WIKIBIO
Method
Qwen2-0.5B
Llama3.2-1B
Qwen2-1.5B
gemma-2-2B
gemma-2-99
Llama3-8B
Average
Last Attn
1.0605
0.6304
-0.7054
-0.2579
0.2815
0.5500
Rollout
-0.6404
0.5591
-0.7066
0.5085
0.3498
0.8785
0.1582
SHAP
1.4702
1.1672
1.1213
0.7966
3.1494
1.4063
1.5185
IxG
3.4273
1.8365
1.3942
1,5816
2.6047
1.3747
2,0365
2.4216
1.5797
0.6975
1.1909
4.1117
0.6876
1.7815
DeepLIFF
3.2207
1.6265
1.4590
1.4607
2.3006
1.2739
1.8903
SIG
3.7656
1.4300
2.0816
1.4256
5,2280
1.3620
2.5488
LIME
3.0009
0.5656
1.1714
0.7180
2.9527
0.8349
1.5406
Occlusion
5.1051
2.0019
3.8916
2.7232
4,9300
3,3885
3.6734
NOISER
8.7624
3.7385
4.9864
4.2527
Table Faithfulness scores across tasks.
LONGRA datasets, where the models must provide single output, we filter down samples to the ones that the model can correctly generate the gold output. See Appendix C for the details. For WIKIBIO, we generate 10 tokens for input completion. To obtain the faithfulness score in this task, we compute the faithfulness of each next token w.r.t. all the previous tokens and consider the averaged score as the final faithfulness. The judge model used to get the answerability metrics is 3.ama-3.3- 3-70-- Seeg-InnsructtTuur the most important words from the input prompt pt to get the answerability score and rate.
Results
Table 1 presents the faithfulness scores across different tasks. Following Zhao & Shan (2024), each score is computed as the logarithm of the ratio between the method's score and the random baseline. Consequently, scores below zero indicate less faithful methods than the random baseline, i.e., unfaithful. As shown in Table 1, faithfulness varies across different FA methods and generative models. Notably, NOISER consistently achieves higher faithfulness scores across all tasks and models, outperforming traditional FAs. This suggests that NOISER provides more reliable attributions, reinforcing its effectiveness in evaluating model faithfulness.
To demonstrate the effectiveness of selecting kmin, we compare its faithfulness performance against alternative bounding strategies across different models in Table 2. The results show

---

Published as a conference paper at COLM 2025
Scaling Factor (k)
Qwen2-0.5B
Llama3.2-1B
Qwen2-1.5B
gemma-2-2B
gemma-2-9B
Llama3-8B
Average
random
1.1519
1.0993
0.7165
1.2287
1.6007
1.4522
1.2082
None (k
1,0922
1,0219
0.6445
1.1844
1.4726
1.1070
1.0671
[||a/[2]
1.4849
1.3031
122617
1.7236
2,7905
1.7470
1.7185
[|||||=]]
0.8884
0.9515
1.1275
1,4300
1.4115
1,0993
Kmax per token
1.2230
0.9938
0.5850
1.2897
1,8359
1.0984
1.1710
Kmax Kmin
1.1962
1.0211
0.6515
1.1824
1.4753
1.1059
1.0886
2.1854
1.3989
1.4400
14433
2.1767
2.2175
1.8103
Table 2: Comparison of different boundings on the faithfulness score on KNOWN dataset.
Method
Qwen2-0.5B
Llama3.2-1b
Qwen2-1.5B
gemma-2-2b
gemma-2-9b
Llama3-8b
Average
Rate
Score
Rate
Score
Rate
Score
Rate
Score
Rate
Score
Rate
Score
Rate
Score
Last Attn
14%
0.0936
48%
0.2496
10%
0.0670
39%
0.2064
37%
0.1854
39%
0.208]
31%
0.1684
Rollout
8%
0.0527
13%
0.0649
8%
0.0557
9%
0.0457
22%
0.1033
27%
0.1364
16%
0.0812
SHAP
22%
0.1803
29%
0.1890
24%
0.1862
17%
0.1249
11%
0.0764
26%
0.1454
22%
0.1504
KGG
27%
0.2177
33%
0.2412
26%
0.1942
35%
0.2408
35%
0.2537
30%
0.2079
31%
0.2259
20%
0.1638
25%
0.1827
18%
0.1426
16%
0.1197
12%
0.0875
27%
0.1360
20%
0.1387
Deepl IFT
21%
0.1753
34%
0.2279
26%
0.1991
32%
0.2225
30%
0.2107
31%
0.2019
29%
0.2062
SIG
21%
0.1583
21%
0.1520
9%
0.0617
12%
0.0627
LIME
37%
01.2966
25%
0.1692
41%
0.3308
4.%
0.3003
50%
0.3291
36%
0.2423
39%
0.2784
Occlusion
53%
0,3689
49%
0.4224
48%
0.3152
50%
0.3726
%1%
0.3556
NOISER
55%
0.5063
37%
0.3665
43%
0.4099
43%
0.4102
49%
0.4497
41%
0.4858
45%
0.4381
Table 3: Answerability metrics on KNOWN dataset w.r.t. judge model top-1 predition.
that kmin consistently yields the highest faithfulness scores, confirming its superiority in preserving model behavior under noise perturbation.
Since kmi is model-dependent, we introduce norm-bounding as a flexible alternative, where the noise vector scaled based on the model's embedding size (see Appendix D). We further compare our approach with kmax, which applies the maximum k across all input tokens (max( K)), and a variant, kmax per token, which applies per-token maximum scaling factor. The latter performs slightly better, as it results in a less aggressive perturbation than the global kmax, reducing the likelihood of extreme changes in model behavior.
Additionally, we analyze the effects of unbounded scaling (k = 1) and random k, where k is sampled from a uniform distribution for each input sample. The consistently lower faithfulness scores in these settings highlight the necessity of proper bounding strategies to maintain faithfulness.
Overall, kmin is the only configuration that guarantees the model does not change its prediction under noise, making it the most reliable choice. The detailed computation of expected norm values is provided in Appendix D.
In addition to faithfulness, we monitored the runtime efficiency of NOISER and compared it with other FA methods. As expected, attention-based methods were the fastest, followed by muddient-based methods. On the other hand, perturbation-basel techniques are generally more computationally demanding. However, the runtime of NOISER can be further reduced by decreasing the number of binary search steps (e.g,, from 10 to 5), which yields a noticeable speed-up with only a slight trade-off in faithfulness.
The answerability rate and score are reported in Table 3. While NOISER achieves the highest answerability score in most cases and on average, Occlusion attains the highest answerability rate. This indicates that when NOISER attributions are deemed answerable (rate), the importance scores assigned to the top-p% tokens are significantly high, which is desirable. In contrast, Occlusion produces higher number of answerable attributions but with lower scores, implying that it does not assign as much weight to key tokens.
To provide more flexible analysis of answerability metrics, we examine cases where the gold prediction appears within the top-5 predictions of the judge model. Under this evaluation, the gap between NOISER's answerability score and those of other baselines widens, while its answerability rate also improves and approaches that of Occlusion, which achieves the best rate.
7

---

Published as a conference paper at COLM 2025
To further contextualize the answerability metrics, we also analyzed the answerability rate when all tokens (top-100%) are provided to the judge model. This represents an upper bound on the judge model's 's performance, offering insight into how often the judge model itself can recover the gold prediction using the full input. In our case, using all tokens, the judge model achieves approximately 84% accuracy when considering only the top-1 prediction, and around 92% accuracy when allowing for the gold output to appear a the top-5 predictions. This highlights the intrinsic limitations of the judge model and provides reference point when interpreting answerability rates obtained from top-p% subsets.
Another aspect regarding the FA methods' efficiency that we evaluate is their ability to identify the minimal set of tokens most relevant to the output. We visualize the importance scores assigned by each method to critical tokens in the LONGRA "country-capital" category. Additionally, we examine the distribution of importance scores across the distractor and main parts in Figure 2. As shown Figure 2, NOISER assigns the highest importance to critical tokens while effectively disregarding the distractor section, demonstrating stronger focus on the main part compared to the best-performing baselines.
0.6
Last Attn
Rollout
SHAP
8XG
16
DeepLIFT
SIG
LIME
Occlusion
Last Attn
SHAP
1XG
16
DeepLIFT
LIME
Occlusion
Figure 2: The aggregated score that each FAs put different parts of inputs from the "capital- world" subclass in LONGRA dataset. The red indicates the score assigned to the undesired part (e-g., distractor). The left image illustrates the aggregated score "country" +" "capital" token. The right image indicates the overall score on the main part against the distractor.
Finally, we analyzed the minimum proportion of top attributions required for each FA method to ensure that the judge model correctly predicts the original output. To determine this value, we first computed attributions for each sample using given FA method. Then, starting with the full set of tokens, we iteratively removed the least important tokens one by one until the judge model produced an incorrect prediction. We repeated this process across all samples, averaging the proportion of retained tokens to obtain the final minimum top-p% required for accurate prediction. A lower value indicates a more effective FA method in identifying the most relevant attributions. In this regard, Occlusion requires the least number of tokens overall, which aligns with the results Table 3, while Lime and NOISER take the second and third best place with minimal difference.
Related Works
Post-hoc explanation methods, such as FA techniques, are applied retrospectively by seeking to extract explanations after the model makes a prediction. Most FAs have been proposed in the context of classification tasks, where a sequence input = [xo...., x, is associated with true label and predicted label The underlying goal is to identify which parts of the input contribute more toward the prediction 1 (Atanasova et al., 2020; Wallace et al., 2020; Madsen et al., 2022; Chrysostomou & Aletras, 2022; Lei et al., 2016; Chan et al., 2022a; Ghasemi Madani & Minervini, 2023). Most FAs generally fall into gradient, attention, and perturbation-bant categories.
8

---

Published as a conference paper at COLM 2025
80
70
72.20
66,20
66.16
56.16 65.83
70:29
60
59,53
Min top-k
50
30
20
10
Last
Rollout
SHAP
IxG
1G
DeepLift lift
SIG
LIME
Occlusion
Noiser
Figure 3: Minimum top-p% attribution required for the judge model to retain the correct prediction across different feature attribution methods, Lower values indicate higher attri- bution accuracy, as fewer tokens are needed to maintain the original output.
Dataset
Input
Output
Known
LeBron
James
professionally plays the sport of
basketball
LongRA
When my flight landed in Japan converted my
Tokyo
had terrifying dream about my grandmother but that story for another time was staying in the capital Mario Land is 1989 side-scrolling platform video
currency and slowly fell asleep
WikiBio
Super
Nintendo
game
developed and
published by
Table 4: Example of NOISER attributions on different inputs.
Gradient-based methods derive the importance for each token by computing gradients w.r.t the input (Denil et al., 2015b). The resulting gradient captures intuitively the sensitivity of the model to each element in the input when predicting token w. While attribution scores are computed for every dimension of input token embeddings, they are generally aggregated at token level to obtain more intuitive overview of the influence of individual tokens.
Building upon this, Denil et al. (2015b) takes the input token vector and multiplies by the gradient (Inputx Gradient), while Sundararajan et al. (2017b) compares the input with null baseline input when computing the gradients w.r.t. the input (Integrated Gradients). FAs.
Nielsen al. (2022) offers a comprehensive overview of other propagation-
Attention-based methods are applied to models that include attention mechanism to weigh the input tokens. The assumption that the attention weights represent the importance of each token. These FAs include scaling the attention weights by their gradients, taking the attention scores from the last layer, and recursively computing the attention in each layer (Serrano & Smith, 2019; Jain al., 2020; Abnar & Zuidema, 2020).
Perturbation-based methods measure the difference in model prediction between using the original input and corrupted version of the input by gradually removing tokens (Lei et al., 2016; Nguyen, 2018; Bastings et al., 2019; Bashier et al., 2020). The underlying idea is that removing important tokens will lead the model to flip its prediction or

---

Published as a conference paper at COLM 2025
significant drop in the prediction confidence. For instance, the input token at position i can be removed, and the resulting probability difference Fx(X) F x;) can be used as an estimate for its importance. If the logit probability given to the original output does not change, we conclude that the i-th token has no influence. Differently, some perturbation- based techniques utilize modified model or separate explainer model to learn feature attributions (Ribeiro et al., 2016; Lundberg & Lee, 2017; Bashier et al., 2020; Hase et al., 2021). LIME (Ribeiro et al., 2016) and SHAP (Lundberg & Lee, 2017) fall into this category.
7 Conclusion
In this paper, we introduced NOISER, a perturbation-based input attribution method that employs bounded noise to address the distribution shift problem arising from the discrete nature text, aiming to explain language model predictions in generation tasks. Fur- thermore, we proposed answerability metrics, novel automatic plausibility evaluation metric that leverages an LLM to evaluate the relevance of attributed rationales to the target output in the absence of gold rationales or human evaluation. Through comprehensive experiments across three tasks and six LLMs, we demonstrated that NOISER consistently surpasses existing baselines in terms of both faithfulness and answerability rate. Notably, our approach requires no supervision, positioning it as promising direction for improving model interpretability and efficiency.
Acknowledgements
Reza was funded by the "borsa di studio post laurea per attività di ricerca" from the Depart- ment of Computer Science and Information Engineering of the University of Trento. Aryo Pradipta Gema was supported by the United Kingdom Research and Innovation (grant EP/ (P/S02431X/1) UKRI Centre for Doctoral Training in Biomedical AI at the University of Edinburgh, School of Informatics. Yu Zhao was supported by the UKRI Centre for Doc- toral Training in Natural Language Processing, funded by UK Research and Innovation (grant EP ESO042241 1). Pasquale Minervini was partially funded by ELIAI, an industry grant from Cisco, and a donation from Accenture LLP. This work was supported by the Edinburgh International Data Facility (EIDF) and the Data-Driven Innovation Programme at the University of Edinburgh. Gabriele is supported by the Dutch Research Council (NWO) as part of the InDeep project (NWA. 1292.19.99... Andrea Passerini was partially funded by the TANGO project, funded by the Horizon Europe Programme, Grant Agreement no. 101120763. Funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Health and Digital Executive Agency (HaDEA). Neither the European Union nor the granting authority can be held responsible for them.
References
Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers, 2020. URL https://arxiv.orgg 2ttp///055.000020
Leila Arras, Franziska Hor,, Grégoire Montavon, Klaus-Robert Müller, and Wojciech Samek. "what is relevant in text document?": An interpretable machine learning approach. PLOS ONE, 12())ee0181142, August 2017. ISSN 1932-6203. doi: 10.1371 /oo./raa..ooo...00 0181 142. URL http: http:://dddd doi. httt:::/ddk.di..ogg/11.... 1371 1or/jooralal/.....llaa.. pone. 0181142. .011.01111
Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, and Isabelle Augenstein. A diagnostic study of explainability techniques for text classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 3256- 3274, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/ v1 01/2020.emnlpmmain.22
URL https: pattps://aclante h/wp///occaatttooo..ooooo .arggyo.oo.r. org/2022 eo020.emnl--miin 263.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu,
10

---

Published as a conference paper at COLM 2025
Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Sheng- guang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen 2023.
technical report. Xiv preprint ar
Housam Khalifa Bashier, Mi-Young Kim, and Randy Goebel. RANCC: Rationalizing neural networks via concept clustering. In Proceedings of the 28th International Conference Compu- tational Linguistics, pp. 3214-3224, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics. doi: 10. 10.18653/vl 006653/v1/20222 w1/2/000.ooling-maiii main.286. URL col iiliing-nn wonggmain... 286.
Jasmijn Bastings, Wilker Aziz, and Ivan Titov. Interpretable neural predictions with differen- tiable binary variables. In Proceedings of the 57th Amnual Meeting the Association for Compu- fational Linguistics, pp. 9977,,977, Florence, Italy, July 2019. Association for Computational URL https::t//stt https:://allantholggy org 1284.
Linguistics. doi: 10.18653/ 10/553/v1//// v1/P19-1284.
Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. snli: Natural language inference with natural language explanations. In S. Ben- gio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Asso- ciates, Inc., 2018. URL https: httpss//pproceedingss werrunnnnsneeiiiine nwwr http://pppr.flepppppppeeeep htt.aaaaaabb22222200999922226660 Pappeeeeeeeeeefffaaa.aprrrre pdf.
Aaron Chan, Maziar Sanjabi, Lambert Mathias, Liang Tan, Shaoliang Nie, Xiaochang Peng, Xiang Ren, and Hamed Firooz. UNIREX: A unified learning framework for language model rationale extraction. In Proceedings of BigScience Episode #5 - Workshop on Challenges & Perspectives Creating Large Language Models, pp. 51-67, virtual+Dublin, May 2022a. Association for Computational Linguistics. doi: 10.18653/ v1 22022..ggscience-1..5 URL https://aclanthol///
org/2022.22222 wttp///22.bigcceeccee...
Chun Sik Chan, Huanqi Kong, and Liang Guanqing- A comparative study of faithfulness metrics for model interpretability methods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5029-5038, Dublin, Treland, May 2022b. Association for Computational Linguistics. doi: 10.18653/v1. / ortg/222..cll-aaoo. acl-long. 345.
acl-long.345 URL https: http:ss//allattholooo
Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, and Hongxia Jin. Alpagasus: Training better alpaca with fewer data, 2024. URL https: / http//aaiiiv.oraa/////2000/0/ 08701.
George Chrysostomou and Nikolaos Aletras. Enjoy the salience: Towards better transformer- based faithful explanations with word salience. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, PP. 8189-8200, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1 /225////v22222...o wat.002.emmlpmmmimm... URL https: htttp:://allantholloyy org/2021 emnlp-main. 645.
George Chrysostomou and Nikolaos Aletras. An empirical study on explanations in out- of-domain settings. In Proceedings of the 60th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 6920-6938, Dublin, Ireland, May 2022, Association for Computational Linguistics. doi: 10.18653/v1. /2022.ccl acl-long URL http::://elaaatoooog.//////2222.
DeepSeek-AL. Deepseek-r1: Incenniiiiinn reasoning capability in 1lms via reinforcement 12948.
learning, 2025. URL https://arxiivorr/aass
Misha Denil, Alban Demiraj, and Nando de Freitas. Extraction of salient sentences from labelled documents, 2015a
Misha Denil, Alban Demiraj, and Nando de Freitas. Extraction of salient sentences from labelled documents, 2015b. URL wtt.aaaiiii.oog////1//13/33 6815.
11

---

Published as a conference paper at COLM 2025
Jay DeYoung, Sarthak Jain, Nazneen Fatemn Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. ERASER: A benchmark to evaluate rationalized NLP models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4443 4458, Online, July 2020. Association for Computational Linguistics. URL https://aclaanthol..g org/2020 acl-main
doi: 10.18653/ /v1 h////v1/22002 408.
Finale Doshi-Velez and Been Kim. Towards rigorous science of interpretable machine learning, 2017.
Joseph Enguehard.
Sequential integrated gradients: a simple but effective method for explaining language models, 2023. URL https://aakti http:///aavvvvv.vaaassessee org/abs/2385. 15853.
Nils Feldhus, Leonhard Hennig, Maximilian Dustin Nasert, Christopher Ebert, Robert Schwarzenberg, and Sebastian Möller. Saliency map verbalization: Comparing feature importance representations from model-free and instruction-based methods. In Bhavana Dalvi Mishra, Greg Durrett, Peter Jansen, Danilo Neves Ribeiro, and Jason Wei (eds.), Proceedings of the Ist Workshop on Natural Language Reasoning and Structured Explanations (NLRSE), pp. 30-46, Toronto, Canada, June 2023. Association for Computational Linguis- URL https: hwwpaaaaaaannoooo..oooooooooooot org.
tics. doi:
wrg.2023.3322 2023. www..nnreee....ee.e.eee.ee wwrse-1....e. 1.4/.
Mohammad Reza Ghasemi Madani and Pasquale Minervini. REFER: An end-to-end rationale extraction framework for explanation regularization. In Jing Jiang, David Reitter, and Shumin Deng (eds.), Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), pp. 587-602, Singapore, December 2023. Associ- ation for Computational Linguistics. doi: 10.18653 111.6/////1/223.oollll1111 //aclanthology wwwp:gg..org/////0 2023. conll-1.48.
URL https:
Peter Hase, Harry Xie, and Mohit Bansal. The out-of-distribution problem in explainability and search methods for feature importance explanations, 2021. URL https://arxxv.oor
abs/2106. 005226666.00886
Maksims Ivanovs, Roberts Kadikis, and Kaspars Ozols. Perturbation-based methods for explaining deep neural networks: A survey. Pattern Recognition Letters, 150:228-234, 2021. ISSN 0167-8655. doi: http://tww.....eeee.......coo/ia
206.0000000 URL https:
//www.sciencediree.e.
Alon Jacovi and Yoav Goldberg. Aligning faithful intererrttttttrrtttoionnss with their social attribution. Transactions of the Association for Computational Linguistics, 9:294-310, 2021. doi: 10.1162/
tacl 00367. URL httph: h/tas///hllaaaoloooooooooo hwwpo/llggyyogg/2222222 org/2021. t..1-1.18..
Sarthak Jain, Sarah Wiegreffe, Yuval Pinter, and Byron C. Wallace. Learning to faithfully 001
rationalize by construction, 2020. URL
Tom Kersten, Hugh Mee Wong, Jaap Jumelet, and Dieuwke Hupkes. Attention vs non- attention for a shapley-based explanation method. In Proceedings of Deep Learning Inside Oud (DeeLIO); The 2nd Workshop on Knowledge Extraction and Indegration for Deep Learning Architectures, pp. 129-139, Online, June 2021. Association for Computational Linguistics. URL https: httt::::/aclannhooogy.ooor org/2021 deelio-1 1. 13.
doi: 10.18653 / v1 2202..eelior
Remi Lebret, David Grangier, and Michael Auli. Neural text generation from structured data with application to the biography domain, 2016. URL htt.hpss///axx..orrg////////////
07771.
Tao Lei, Regina Barzilay, and Tommi Jaakkola. Rationalizing neural predictions. In Pro- ceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 107-117, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653 /tt/////D////1111111 URL https: http://aaaantoooooorro//////////
Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason Weston, and Mike Lewis. Self-alignment with instruction backtranslation, 2024. URL https: // //arxiv.org www.vvv.rrg/bb//33s... 06262 06259.
12

---

Published as a conference paper at COLM 2025
Zachary C. Lipton. The mythos of model interpretability. Commun, ACM, 61((0))36643, sep 2018. ISSN 0001-0782. doi: (0.1145/3233233 URL https://doi.org/10 1145/3233231.
Shusen Liu, Zhimin Li, Tao Li, Vivek Srikumar, Valerio Pascucci, and Peer- Timo Bremer. Nlize: A perturbation-driven visual interrogation tool for analyzing and interpreting nat- ural language inference models. IEEE Transactions on Visualization and Computer Graphics, 25(1))651-660, January 2019. ISSN 1077-2626. doi: 10.1099/VVCG.228.22652222 URL https://doi... 1.. 1109/110.9.0/// 2018.0882000088 2865230.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoe V. Le, Barret Zoph, Jason Wei, and Adam Roberts. The flan collection: Designing org /
data and methods for effective instruction tuning, 2023. URL 136881
Scott Lundberg and Su-In Lee. A unified approach to interpreting model predictions, 2017. URL https:t/aaxii..oot httpi//..org////////////// 1775/1177777.77..
Siwen Luo, Hamish Ivison, Caren Han, and Josiah Poon. Local interpretations for explain- able natural language processing: A survey, 2022.
Andreas Madsen, Nicholas Meade, Vaibhav Adlakha, and Siva Reddy. Evaluating the faithfulness importance measures in NLP by recursively masking allegedly important tokens and retraining. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 1731-1751, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguis- tics. doi: 10.18653/ findings-emnlp. 125/.
125.. URL httpss//aclanttholpgg. wwwpgg...oooooooeeeee org/2022.
Potsawee Manakul, Adian Liusie, and Mark J. F. Gales. Selfcheckgpt: Zero-resource black- box hallucination detection for generative large language models, 2023. URL https: //arxiv.org/aass/303. 08896.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing 05262.
factual associations in gpt, 2023. URL http::://aaxiv.or//abba
Dong Nguyen. Comparing automatic and human evaluation of local explanations for text classification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), pp. 1069-1078, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/vvi (0665//v//II1 //////////////////// 1097. URL https: / /acaaaaaaaanntogo..oooooooooooae Perg/N88-1999
Ian Nielsen, Dimah Dera, Ghulam Rasool, Ravi P. Ramachandran, and Nidhal Carla Bouaynaya. Robust explainability: A tutorial on gradient-based attribution methods for deep neural networks. IEEE Signal Processing Magazine, 39(4):73-884 July 2022. ISSN 1558-0792. doi: 10.1109 0..109//msp.20223117777 URL http://dx wrg.ii..gg//... 11 1109/MSP. 2022. 3142719.
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapin Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brak- man, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han,
13

---

Published as a conference paper at COLM 2025
Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu J Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Paaraccandolo Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Fran- cis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thomp- son, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sher- win Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024. URL https:////axiii..ooo http://www..rrg//////33.../ 08774.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4, 2023. URL https://aavi http:aawwwv....ggbbba/aa/// wtg/ htt.a/////.eeeeee 032277.
Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. Infobench: Evaluating instruction following ability in large language models, 2024. URL 03601.
Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. Explain yourself! leveraging language models for commonsense reasoning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4932-4942, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653 (0.18653/v1/P111 /v1119--11 URL haclaa//aclanthhology
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?": Explaining the predictions of any classifier, 2016. URL https://arxxi.orrg/aasss 04938.
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead, 2019.
Gabriele Sarti, Nils Feldhus, Ludwig Sickert, Oskar van der Wal, Malvina Nissim, and Arianna Bisazza. Inseq; An interpretability toolkit for sequence generation models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics
14

---

Published as a conference paper at COLM 2025
(Volume 3: System Demonstrations), pp. 421-435, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/ vt///v11///// w122023.cl-ddmoo.. URL https:// aclanthology
40.
Gabriele Sarti, Nils Feldhus, Jirui Qi, Malvina Nissim, and Arianna Bisazza. Democratizing advanced attribution analyses of generative language models with the inseq toolkit. In xAI-2024 Late-breaking Work, Demos and Doctoral Consortium Joint Proceedings, pp. 289-296, Valletta, Malta, July 2024. CEUR.org. URL http://euuruws.oo//ol1133//77171 pdf.
Sofia Serrano and Noah A. Smith. Is attention interpretable? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2931-2951, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18633/w1 URÉ https://aclaanthology .raey..rg//11/ org/p19-12822
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences, 2019. URL https://arxxv..or wttp://wwwww.w.p//////////////// 02685.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks, 2017a.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks, 2017b. URL https://arxix: wwwpeewwwww..oreeeeeeeeeeeeeeeee org/ abs/170s. 01356.036336......
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunoii B. Hashimoto. Stanford alpaca: An instruction-folllwing Ilama model. h/tp
com/
alpaca, 2023.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Choo, Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George- Christian Muraru, Grigory Rozhdestvenskly, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikula, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross Mcllroy, Ruibo Liu, Ryan Mullins, Samuel I. Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojciech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy. Gemma: Open models based gemini research and technology, 2024. URL 2ttp//b//223..2222
https://arxiv wtgg/bs////aaeoooooooooo 2403 08295.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023. URL https::/arxxv..rrg// wwt./bo//////e// 2202. 13971.
Keyon Vafa, Yuntian Deng, David M. Blei, and Alexander M. Rush. Rationales for sequential htrg/abs/2100
predictions, 2021. URL htt.t:ttt://aeee
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. URL http://aaxivv.or/aab////// 03762.
15

---

Published as a conference paper at COLM 2025
Eric Wallace, Matt Gardner, and Sameer Singh. Interpreting predictions of NLP models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts, pp. 20-23, Online, November 2020. Association for Computational Linguistics. doi: 10. 1863//65v 22000.emmpptuutorriiiio 2020. 3
URL https: https://aalaaathlloooooo org/
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardIm: Empowering large language models to follow complex instructions, 2023. URL https: //appi://axivv http://xxvv..ogg//boaa/ee/ www.orsg/////. 2304. 12244.
Omar Zaidan, Jason Eisner, and Christine Piatko. Using "annotator rationales" to improve machine learning for text categorization. In Human Lanuuge Technologies 2007: The Confer- ence of the North American Chapter of the Association for Computational Linguisties; Proceedings of the Main Conference, pp. 260-267, Rochester, New York, April 2007. Association for Computational Linguistics. URL https: httt:s://cclaanhoooogg.o//////// N007
Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014, Pp. 818-833, Cham, 2014. Springer International Publishing.
Hao Zhao, Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Long is more for alignment: A simple but tough-to-beat baseline for instruction fine-tuning, 2024. URL https::/arxxvv.orggee..
048833 04833.
Zhixue Zhao and Nikolaos Aletras. Incorporating attribution importance for improving faithfulness metrics. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeling of the Associalion for Computational Linguistics (Volume 1: Long Papers), pp. 4732-4745 Toronto, Canada, July 2023. Association for Computational Linguistics. doi: acl-long.22 261.
URL https: /aclss///l//aatooooo.o.ooo org/2023.
Zhixue Zhao and Boxuan Shan. Reagent: A model-agmostic feature attribution method for generative language models, 2024. URL https://aaxxi wrtp://///g/bb//222.... 00794.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging lmm-as-a-judge with mt-bench and chatbot arena, 2023. URL https://arxiv.orr www.g//g//.ooooo...o h0t//bb/23300 05685.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. Lima: 11206.
Less is more for alignment, 2023a. URL http:://aaxvvv..gaabbb////ss
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-followingg evaluation for large language models, 2023b. URL
07911.
A Soft-NC and Soft-NS Metrics
The Soft-NC and Soft-NS metrics are defined as follows:
Soft-C(X,g.X') max X) p(0 X)) Soft-S(X.j.X') Soft-C(X,g,X''
(1)
(2)
where X' is soft-perturbed versions of X given the following instruction. For the embedding
vector X; E X and its FA score s; we modify the elements of x; using Equation (3).
x
X; O ej e; Bernoulligg)
(3)
16

---

Published as a conference paper at COLM 2025
where is a binary mask vector of size n (embedding size) and Bernoulli is parameterized with probability q:
S, q (1-s, s, if removing elements
if retaining elements
(4)
The normalized sufficiency and comprehensiveness are then computed using the following equations:
Soft-C(X,,,XX
Sof) = 1- S(X, 0.0)
(5)
(x) Soft-S(X, X') S(X,
(6)
1 -(,,0),)))
However, in generation tasks, the absence of predictive likelihood for the predicted label makes applying Soft-NS and Soft-NC challenging. Zhao & Aletras (2023) proposed using the Hellinger distance between prediction distributions over the vocabulary as measure of
changes in model predictions. They substitute p( X) -)) X') in Equation (1) with the
Hellinger distance. Given two discrete probability distributions, Px, = pi, Po. and Px = [p',,r....PP'u the Hellinger distance is formally defined as:
APx; H(Pxo V2
Σ
(VP
V P33s)
where Px, is the probability distribution over the entire vocabulary (of size z) when prompt- ing the model with the full-text X. Px' is for prompting the model with soft-perturbed text. For a given sequence input and a model of vocabulary size v, time step f, the model generates a distribution Px, for the next token The final Soft-NS and Soft-NC at step / for text generation are formulated as:
Soft-NS(X,x,, x; R) = max(0, AP, APxx)
APo,
APxiRA
Soft-NC(X,x; R)
(8)
APo,
where APo, is Hellinger's distance between zero input's probability distribution and full- text input's probability distribution. X' \\ R is the case of "if removing elements" " described in Equation (4).
B More Results
Table provides the detailed Soft-NS and Soft-NC scores of our experiments.
Table illustrates the answerability metrics on top-50% of feature attributions given the judge model top-5 predictions.
We also analyzed the correlation between the faithfulness score and the answerability metrics on KNOWN in Figure 4. The results show that IG, SHAP, DeepLIFT, IxG, LIME, and Occlusion exhibit similar faithfulness scores, with Occlusion achieving the highest answerability rate and score among them. Meanwhile, NOISER surpasses all baselines in both faithfulness and answerability score but falls short answerability rate, trailing Occlusion by 5%.
17

---

Published as a conference paper at COLM 2025
KNOWN
Method
Qwen2-0.55
1-mma32-1b
Qwen2-1.58
gemma-2-2b
gemma-2-4b
LJama3-5b
Sofi-NS
Soft-NC
Soft-NS
Soft-NC
Soft-NS
Soft-NC
Soft-NS
Soft-NC
Soft-NS
Soft-NC
Soft-NS
Soft-NC
Last Attn
0,0372
-0.1229
-0,030
-0,0301
0.0023
0,0581
0.1617
-0.0211
0.0321
-0.3109
-0.1188
0.1052
Rollout
-0.2567
0.1406
-04426
-0.0045
-0.0264
0.1475
0,3382
0.1042
0.0114
-0.3721
-0.5818
0,3194
SH1AP
07660
-0.1643
0.5388
-00809
-0.0050
0.3776
0,1933
0.7270
0.5310
IxG
-0.5373
0.7400
0.0079
0.6980
-00843
0,5435
-0.1152
0.6384
40.0265
1.0541
0.4730
0.5141
IG
-0.6136
08312
-0.1202
0.6629
-011299
0A372
-0.1328
0.3343
0.1540
0.8815
-0.3386
0.7144
DeepLIFT
-0.5430
0.4660
-0,800
0.6274
000826
0.6199
-0.12%6
0.4853
-0.0109
0.8747
0.0147
0.5027
SIG
-0.4989
0.5350
-000841
0.4375
-009664
0.3969
-0.1494
+00088
0.1730
0.6147
-0.1866
0.4600
LIME
-0.0221
012660
0.2210
0.0678
0,3149
0.2230
0.2100
0.2732
0.5439
0.109
Occlosion
0.489
0.2267
0.0635
0.3510
0,2119
NOISER
0,6785
5068
0248
1.1741
400264
144644
0.8805
1,2627
40.0233
2.2001
0.8045
14133
LONGRA
Method
Qwen2-0.58
1lamma32-1b
Qwen2-1.58
gemma-2-2h
gemma-2-9h
Llama3-8h
Soft-NS
Soft-NC
Soft-NS
Soft-NC
Soft-N
Soft-NC
Soft-NS
Soft-NC
Soft-N5
Soft-NC
Soft-NS
Soft-NC
Last Atmn
1.8522
0.0645
0.2522
0.0733
0.0828
-0.0938
-0.1553
-0,0799
40.15K3
-0.0799
0,0888
015874
Rollout
1,8511
-0.0025
0.0894
0.1557
0.0927
-0.0126
-0.1790
-0.0853
-0.1750
-0.0853
0.1052
1.123]
SHAP
1.5203
2.2767
-0.3457
-000233
1,699
0.1444
1.1972
25798
-0.1499
1,8718
2
2.2883
19406
IG
1.5378
2,8000
-0.3111
1.207
-0.1039
16557
0.3175
0.95-18
0.3475
0.9553
0.0639
3,551
DeepLIFT
1.6777
26811
-0.0516
18435
-0.1606
1.7118
0.2430
2. 2,4998
0.2430
2.4938
0.0657
2,0600
SIG
2.4897
1.3749
-0.3132
1,2404
-0.0297
1.1344
0.1009
0.4404
0.1009
0.4404
0.1895
0.9723
LIME
0.9895
0.0670
0.1343
0.0869
-0,1042
-0.0106
-0.0112
-0.1524
-0.1122
-0.1521
0.0923
2,2071
Occlusion
2.2561
1.682
0.3370
1.8517
-0,4008
1,4133
0.3351
2,8887
0.5531
2.9887
0.0124
4.2364
NOISER
2,0935
4.719
0.3222
4.4831
4.1367
0.5164
3.7563
1.1446
5:0235
0.4705
4.922
WIKIBIO
Method
Qwen2-0.55
L-1m
Qwen2-1.5B
gemma-2-2b
gemma--24b
Llama3-8b
Soft-NS
Soft-NC
Soft-NS
Seft-NC
Soft-NS
Soft-NC
Soft-NS
Soft-NC
Soft-NS
Soft-NC
Soft-NS
Soft-NC
Last Attn
0.8145
02459
0.1041
0.5263
0.0126
-0.7181
-0.1674
-0.1105
0.2246
0.0569
-0.0905
0.6405
Rollout
-0.6789
011885
0.0254
0.5336
0.0424
-0.7490
-0.0168
0.5253
0.2655
0.0843
-0.2114
1,0899
SHAP
0.6624
08078
0.0470
1.1202
-0.0233
1.1476
0,0071
0.7895
0.4254
2.7240
-0,0304
1,4688
IGG
2.2695
0.2109
1.1833
1.5496
0.4097
2.1950
0.2509
1.1238
0.5209
1,008
0.004
1.5749
-000332
0.7217
-0.0438
1.2318
0.5993
3.5124
-0.4733
1.1159
DeepLIFT
0.9232
2,2975
-0.0190
1.6755
0.0495
1.0066
0.0574
1.1033
0.3253
1.9753
0.1346
1.1393
SIG
1.4616
2,3040
-0.1815
16611
0.3493
1.7323
0.1572
1 1,2684
0.9031
4.3250
0.657]
0.7049
LIME
1.6490
1.3619
0.1118
0.4537
0.1616
1,0199
(.1260
0.5420
0.7891
2.1634
0.3183
0.5166
Occlusion
2,6100
2,1950
0.3736
1.6283
0.9353
2,9561
0.1395
2 2637
1.3045
3.6255
1.5921
1.761
NOISER
14619
0.6659
4,0016
1,5671
Table 5: Soft-NS and Soft-NC Scores Across Datasets.
Method
Qwen2-0.5B
Llama3.2-1b
Qwen2-1.5B
gemma-2-2b
gemma-2-9h
Llama3-8b
Average
Rate
Score
Rate
Score
Rate
Score
Rate
Score
Rate
Score
Rate
Score
Rate
Score
Last Attn
26%
0.1722
58%
0.3037
21%
0.1432
55%
0.2917
51%
0,2529
48%
0.2568
43%
0.2368
Rollout
21%
0,1445
37%
0.1836
23%
0,1604
23%
0.1086
36%
0.1621
40%
0.1980
30%
0.1595
SHAP
47%
0.3904
52%
0.3440
47%
0.3662
31%
0.2188
28%
0.2051
43%
0.2427
41%
0.2945
LGG
51%
0,4087
58%
0.4299
40%
0,3057
55%
0.3792
47%
0.3369
44%
0.3062
49%
0.3611
48%
0,4004
54%
0,3586
43%
0.3367
31%
0.2308
25%
0.1847
42%
0.2172
41%
0,2881
DeepLIFT
49%
0.4138
60%
0.4067
40%
0,3159
48%
0.3271
45%
0.3164
44%
0.2898
48%
0.3450
SIG
40%
0.3025
44%
0,2739
40%
0.3091
24%
0.1644
36%
0,2698
28%
0.1584
35%
0,2467
LIME
49%
0.3936
49%
0.3315
53%
0.4307
61%
0.4021
60%
0.3916
48%
0.3230
53%
0.3787
Occlusion
67%
0.4666
61%
0.4001
65%
0.5093
64%
0.4204
63%
0.4043
57%
0.4241
63%
0,4375
NOISER
65%
0.5996
58%
0.5435
57%
0.5474
66%
0.6245
60%
0.5400
54%
0.5962
60%
0.5752
Table 6: Answerability metrics on KNOWN dataset w.r.t judge model top-5 predictions.
C Datasets Statistics
Appendix C shows the number of true predictions by each model given KNOWN and LONGRA datasets.
Dataset
Qwen2-0.5B
Llama3.2-1b
Qwen2-1.5B
gemma-2-2b
gemma-2-9b
Llama3-8b
Known (1208)
661
828
774
830
822
875
LongRA (573)
140
160
170
209
148
165
Table 7: Number of true predictions captured by each model.
18

---

Published as a conference paper at COLM 2025
Falthfunnes
pelturt
UM
Palthfuiness
gime
getale
64
030
020
0.23
1,31
235
0.49
0.23
0.90
Answerability Score
Answerability Rate
(a)
(b)
Figure 4: Comparison of average faithfulness score with (a) average answerability score and (b) answerability rate.
D Bounding Computations
Since kmin is dependent the model, introduce norm-bounding the norm value of the noise vector n is different based on the model's embedding size (dmodel). To avoid different norm values for each sample in given data, use the expected norm value of the noise vector, E n plp], and use
E |[|n|||] as the final bounding for the noise vector.
In the following, we show the expected value of each norm given model with d embedding dimensions.
Let n E madel be random vector where each component n, (O, 1). Below, we derive the expected values of different norms and compare their properties.
The L2 norm (Euclidean Norm) is defined as follows:
Amadel L
| 1|
n2
where each n? follows chi-squared distribution with 1 degree of freedom, which results in the following:
n
dmodel
By Jensen's 's inequality and the Law of Large Numbers, for large dmodel
2 R3
/mddel
The Los norm (Maximum Norm) defined follows:
||| 1siddmode
max
|ng]
The cumulative distribution function (CDF) for |na] is F(x) erf The CDF for the
maximum of d samples is Fmax( (x) expected maximum for large dmodel approximates:
[F(x)] mmadel, Using extreme value theory, the
[o] Rs I2
E Answerability Evaluation Prompt
Below, we provide the prompt used for evaluating FAs' answerability.
19

---

Published as a conference paper at COLM 2025
Answerability Evaluation Prompt
# Task:
Given a set of words extracted from a prompt for a completion task, return single word as the most probable completion for the unseen prompt WITHOUT providing any explanation.
20


Reviews:
[{"review_id": "DBxZoX6nm9", "reviewer": "Reviewer_esjs", "summary": "The authors study the task of feature attribution for LMs. In particular, given a token history $x_1, ... x_{t-1}$, the goal is to assign a weight to each token in the history indicating the effect of that token on the model's prediction at step $t$.\nThey propose an approach called NOISER. If I understand correctly, NOISER determines the largest noise level which can be applied to the input embedding of each token in the history without changing the token predicted at step $t$. The authors use NOISER to compute token attributions on three datasets, and compare with a number of existing attribution approaches. They find that their approach outperforms all prior methods in terms of faithfulness. They also propose a new metric called \"answerability\" where they prompt an LM with the subset of tokens selected by a given feature attribution method, and check whether this prompted LM predicts the same next token as appeared in the original sequence.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors study the important problem of feature attribution for LMs. They propose a novel method and perform experiments on three datasets, comparing against a range of attribution baselines. They report that their method outperforms all previous attribution methods on all datasets.", "reasons_to_reject": "I unfortunately had trouble understanding parts of this paper, including:\n- The proposed method, particularly the notation.\n- The experimental setup, including the choice of datasets, choice and implementation of baselines, and definitions of key metrics.\n\nBecause of this, it was hard for me to interpret the results presented in the paper. See the next section for more detailed questions.", "questions_to_authors": "Questions are organized by section below.\n\nMethod (Section 3). If I understand correctly, given a token history $x_1, ... x_{t-1}$ and a model prediction for the token at step $t$, the goal is to assign a score indicating the importance of each token in the history on the model's prediction of the $t$th token. I think this is defined by $S$ on the equation in between lines 97 and 98, but I don't understand this equation. It says $s_i = p(X) - p(X_{perturbed|k_{min}})$. Where is the dependence on $i$ on the right hand side? I'm not sure how this assigns a unique score at each index $i$. Is each $x_i$ a discrete token or is it a continuous input embedding? The notation $p(X)$ makes me think each $x_i$ is a token, but at line 84 it seems to be an embedding. Similarly, the equation between lines 61 and 62 indicates that $F_{\\theta}$ assigns probabilities to token sequences, while its use at line 81 suggests that it _predicts_ a token. I think this section could benefit from another pass to clarify the notation and the writing.\n\nExperiment (Section 4).\nSection 4.1: In line 10 of the abstract, the authors mention that they use NOISER on 3 separate tasks. But from what I can tell there is only a single task -- next token prediction -- and three datasets used to evaluate it. Why use these three particular datasets, given that any text dataset could be used? How many instances are there in each dataset? How long is each prompt on average?\n\nSection 4.2: I don't understand how some of these baselines can be applied to transformer LMs. For instance, with the Occlusion method, what does it mean to occlude portions of an input text sequence? This is a computer vision concept if I understand correctly. Similarly, how is LIME applied in a language modeling setting? Some references or explanation would be very helpful.\n\nSection 4.3: I think the definitions of the key metrics used in the paper should be presented in the main paper body. What are their minimum and maximum values? What does it mean to \"randomly mask parts of the token vector representations\" at line 162? Are part of the input embeddings are set to 0 and then these are run through the full transformer? In lines 212-214, why report the log ratio against a random baseline instead of just reporting the values for the metrics?\n\nSection 4.4: It's not clear to me that answerability as defined by the authors is an appropriate measure of feature importance. For example, in the sentence \"LeBron James plays for the ____\", the most important words for predicting \"Lakers\" are arguably \"LeBron James plays\". But since \"Lakers\" is a poor completion for the prompt \"LeBron James plays\", wouldn't the proposed method assign this rationale a low answerability score?\n\nOne minor nit: at line 16, the year for Vaswani et al is incorrect.", "extra_scores": {}}, {"review_id": "XZp5GDykaY", "reviewer": "Reviewer_7SGD", "summary": "This paper proposes a new feature attribution method for faithfully measuring token contribution to a model’s prediction, and a plausibility metric that combines attribution scores and an LLM-judge to assess whether a subset of highly scored tokens (i.e. with high attribution) from the input are enough to recover the predicted output. The FA contrasts the predictive likelihood of the model by introducing bounded noise on each input embedding and partially noised input. The paper evaluates a series of LLMs of different families and sizes demonstrating the effectiveness of the proposed FA.", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The proposed FA is innovative and works really well across models and tasks.\n* The paper is really well written, and easy to follow.\n* The experimental setup is comprehensive and sound\n* Good job!", "reasons_to_reject": "* N/A", "questions_to_authors": "You should provide a discussion on the complexity/computational overhead in computing the attribution scores using your method.", "extra_scores": {}}, {"review_id": "UCLhzF6UHh", "reviewer": "Reviewer_CNV7", "summary": "This paper introduces NOISER, a perturbation-based feature attribution method for generative large language models. NOISER operates by injecting bounded Gaussian noise into the word embedding space of input tokens and measuring the model's robustness to such perturbations to quantify token importance scores. This method lies in using the minimal noise threshold that preserves model predictions, thereby identifying the most important tokens. The authors also propose a new metric \"answerability\" that evaluates whether a subset of top-k attributed tokens is sufficient to reproduce the original output using an external judge model. Experiments conducted on six LLMs and three tasks show consistent improvements in faithfulness and answerability over nine strong baselines.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written and very easy to follow. The writing is clear.\n2. The experiments are comprehensive and promising, which proves the efficiency of the proposed method.\n3. The method is easy to reproduce and the evaluation metric is easy to use. The newly proposed method and evaluation is well-motivated and easy to understand and be applied.", "reasons_to_reject": "1. The newly proposed evaluation metric \"answerability\" purely relies on another LLM, which might reduce the reliance of the evaluation metrics. Different evaluation models may have different prompt preferences. Therefore, it would be better to prove the robustness of these evaluation metrics, e.g., try different prompts or different models, and finally conclude which evaluation model is the best. \n2. It would be better to conduct a human evaluation and calculate the correlation between the LLM-based answerability score with the human-based score, to prove that such a technique is very correlated with humans.\n3. What is the runtime of this method? For each token, this method requires forward multiple times to find the $k_{min}$, which is time-consuming. It would be better to compare the running time between this method and previous baselines.\n\nI'd like to change my score if the authors can solve my concerns.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IMNSLIgGN0", "reviewer": "Reviewer_L9vK", "summary": "The paper introduces a new perturbation-based feature-attribution method called Noiser.\nThis method introduces noise into token embeddings, but restricts the magnitude of the noise vector such that the model's prediction doesn't change. To calculate the importance of each token, they measure the change in probability caused by adding the noise vector to each token's embedding.\nThe method is evaluated against a wide variety of other attribution methods, using two metrics, and for various pre-trained LLMs of different sizes.\nTo compare the plausibility of Noiser to the baseline attribution methods, the paper introduces an *answerability* metric. This metric measures how well a judge-LLM can give the correct answer to a question, based only on the top-k most important tokens (according to the attribution method).", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Improving perturbation-based attribution methods to avoid OOD perturbations is a worthwhile and relevant research direction.\n- The empirical results suggest that Noiser is an improvement over previous methods.\n- The *answerability* metric seems like a useful addition for evaluating attribution methods.", "reasons_to_reject": "- Given that the answerability metric is mentioned as a contribution, I would expect to see a defense of why this metric is suitable to measure plausibility. In general, a larger treatment of the assumptions underpinning this method seems warranted, these may include the following:\n   - the subset of tokens that makes for the most plausible (convincing to humans) explanation is the subset that is most predictive of the answer\n   - the judge-LLM gives an unbiased estimate of the predictiveness of the top-k tokens\n\n- The answerability results do not include a random baseline, I'd be interested to see what rates/scores could be obtained by selecting a random 50% of tokens.\n\n- The motivation given for Noiser is that it can prevent OOD perturbations, but this is not independently evaluated. I'm also missing, for example, the frequency with which various perturbation-based methods change the prediction, which should track their performance if the hypothesis is correct.\n\n- Only evaluated on autoregressive transformer-based models.", "questions_to_authors": "- I'm not sure the answerability score makes sense. Am I correct that it essentially just measures how diffuse/concentrated the importance scores are (I assume the scores are normalized to sum to 1, is that mentioned somewhere)? If correct, then any of the attribution methods could be transformed to be as concentrated as is desired, by raising the scores to a power $\\alpha$ and re-normalizing.\n\n- $k$ is used as a variable twice. Once for top-$k$, and once as the scaling factor for the noise. For clarity it probably makes sense to change one.\n\n- Did you try values for $k$ other than 50%? I'd be interested to see if occlusion's performance drops off quicker than Noiser as $k$ approaches 0%.\n\n- Did you calculate the rate at which the judge-llm can answer the question when you set k=1. So including all the (shuffled) tokens?\n\n- Typos:\n  - caption Table 3: predition\n  - line 203: KNONW", "extra_scores": {}}]

Author rebuttal:

Meta-review: In this paper the authors propose an approach for attributing token-level predictions in LLMs to previous tokens in the context. The approach works by introducing bounded Gaussian noise into context token embeddings, and measuring the corresponding change in output prediction probability. The approach is evaluated on against a variety of existing attribution methods with respect to faithfulness and “answerability,” a new metric based on the extent to which the top-k attributed context tokens are sufficient for a judge LLM to reproduce the original output. Experimental results demonstrate that the approach out-performs previous work across a variety of autoregressive LLM families and sizes.

Reviewers agreed that the proposed method and evaluation metric were useful, easy to understand and to apply, and that the approach seemed to work well. Most reviewers felt that the paper was well-written and clear. Reviewers expressed concerns about the proposed metric’s reliance on an LLM, and the corresponding flaws that come with such model-based evaluations, such as sensitivity to prompts, and that the paper lacked a human evaluation to validate the metric. One reviewer also found the paper hard to follow. Overall, the reviewers were satisfied with the authors' responses to their concerns, and all agreed that the paper was worthy of acceptance despite its flaws.

[Automatically added comment] At least one review was discounted during the decision process due to quality]
Decision: Accept