B-cos LM: Efficiently Transforming Pre-trained Language Models for Improved Explainability
Authors: Yifan Wang
Sukrut Rao
Ji-Ung Lee
Mayank Jobanputra
Vera Demberg
Venue: Accepted by TMLR ()

Abstract: Post-hoc explanation methods for black-box models often struggle with faithfulness and human interpretability due to the lack of explainability in current neural architectures. Meanwhile, B-cos networks have been introduced to improve model explainability by proposing an architecture that removes bias terms and promotes input-weight alignment. Although B-cos networks have shown success in building explainable systems, their application has so far been limited to computer vision models and their associated training pipelines. In this work, we introduce B-cos LMs, i.e., B-cos Language Models (LMs) empowered for natural language processing (NLP) tasks. Our approach directly transforms pre-trained language models into B-cos LMs by combining B-cos conversion and task fine-tuning, improving efficiency compared to previous methods. Automatic and human evaluation results demonstrate that B-cos LMs produce more faithful and human interpretable explanations than post-hoc methods, while maintaining task performance comparable to conventional fine-tuning. Our in-depth analysis explores how B-cos LMs differ from conventionally fine-tuned models in their learning processes and explanation patterns. Finally, we present a first exploration of transforming decoder-only models to B-cos LMs for generation tasks. Our code is available at https://github.com/Ewanwong/bcos_lm.

Published in Transactions on Machine Learning Research (12/2025)
B-cos LM: Efficiently Transforming Pre-trained Language Models for Improved Explainability
Yifan Wang
Sytuann@isst.uuiisaarrraa de
Saarland University, Saarbrücken, Germany
Sukrut Rao
sukrut. sukrut. mukrul.ruoommppiimmf.mmmp
Mar. Planck Institute for Informatics, Saarland Informatics Campus, Suarbrücken, Germany
Ji-Ung Lee
jt-ngg banng.lleeumi--aaaaadd.d.o
Saarland University, Saarbrücken, Germany
Mayank Jobanputra
mayankk@lst.uni-suardle. de
Saarland University, Saarbrücken, Germany
Vera Demberg
vera@ilst. urrn@lst.umiisaarland de
Snarland University, Saarbrücken, Germany
Max Planck Institute for Informatics, Saarland Informatics Campus, Suarbrücken, Germany
Reviewed on OpenReview: https: // openreview. net/ pormm id= c180UH80g8
Abstract
Post-hoc explanation methods for black-box models often struggle with faithfulness and human interpretability due to the lack of explainability in current neural architectures. Mean- while, B-cos networks have been introducod to improve model explainability by proposing an architecture that removes bias terms and promotes input-weight alignment. Although B-cos networks have shown success in building explainable systems, their application has so far been limited to computer vision models and their associated training pipelines. In this work, we introduce B-cos LMs, i.e., B-cos Language Models (LMs) empowered for natural language processing (NLP) tasks. Our approach directly transforms pre-trained language models into B-cos LMs by combining B-cos conversion and task fine-tuning, improving efficiency compared to previous methods. Aulomatic and human evaluation results demonstrate that B-cos LMs produce more faithful and human interpretable explanations than post-hoc methods, while maintaining task performance comparable to conventional fine-tuning. Our in-depth analysis explores how B-cos LMs differ from conventionally fine-tuned models in their learning processes and explanation patterns. Finally, we present first. exploration of transforming decoder-only models to B-cos LMs for generation tasks. Our code is available at https:/ httg/ggtttbhhoooo htatthuub..om//wwwwnnnng. brrrawwoog/bbos... bcos Im
1
Introduction
Pre-trained language models (PLMs) such BERT (Devlin al. 2019 and GPT [Radford et al. 2019 Brown
et al. 2020
OpenAl 2023 have significantly advanced performance across plethora of NLP tasks Wang
ct al.
2018 Gao et al. 2023). However, their complex architectures and black-box nature make understanding
their behavior a persistent challenge (Bomnasani et. al. 2021). To address this, research has increasingly focused on understanding model predictions various natural language understanding and generation tasks using different forms explanations, such as input-basod explanations (Feng et al. 2024 Wei Jie al. 2024
Jiang al. 22024 Madsen al. 2024 Yin & Neubig 2022 Deiseroth et al. [023 Abbasi al. (2025), natural Wang et al. 2025a), and concept-based explanations (Yu et al.
language explanations (Ramnath et. al. 2024

---

Published in Transactions on Machine Learning Research (12/2025)
Example (a) - label: not toxic, prediction: not toxic
Incorrect attribution
Conventional LM
[CLS]
user
funny
guy
[SEP]
B-cos LM (ours)
[CLS]
user
funny
guy [SEP]
Irrelevant
Example (b) label: not toxic, prediction: not toxic
attribution
Conventional LM
[CLS]
why
does
smell
like
black sesame
mochi
[SEP]
B-cos LM (ours)
[CLS]
why
does
smell
like
black sesame mochi
[SEP]
Figure 1: Visualization of Wxx)x in a conventionally finc-tuned model (Conventional LM) and B-cos LM. Green (red) indicates the positive (negalive) impact of tokens on the prediction. In both examples, both models correctly predict not toric. In the Conventional LM, "funny" is incorrectly assigned negative attribution in example (a), and in example (b), irrelevant words like "why" and "smell" are highlighted, making the explanations unfaithful and less interpretable. Examples and explanations are drawn from HateXplain. See $3 for details on how W(x)x is computed.
2024
Raman et al. 2024). Among others, input-based explanations, often referred to as rationales, aim to
reveal how specific inputs influence a model's prediction (Arras et al.) 2019 Atanasova al. 2020 Lyu et al., 2024 In this work, we focus on input-based explanations, they offer the most direct insight into model behavior and are often mandated by laws, such as the EU Artificial Intelligence Act.
Most input-based explanation methods for neural models are post-hoc, meaning that they attempt to explain a model's behavior only after it has been trained and deployed (Lyu et al. 2024). While these methods are widely used and casy to apply, they have been shown to produce unfaithful explanations that do not accurately reflect. the model's s actual reasoning process (Kindermans et al., 2019 Şlack et al.] 2020 Pruthi let al. 2020 Ye et al., 2025p. They also struggle with human interpretability, making it difficult for users understand the model's iss reasoning (Smilkov et al.) 2017 Ismail et al. (021). Prior rescarch suggests that these limitations stem from lack of inherent explainability in current models, that is, the model's ability to generate faithful and interpretable explanations by design (Kindermans et al., 2018 Alvarez Melis & Jaakkola, 2018 Rudin 2019). As result, improving model explainability is crucial for producing explanations that are both reliable and useful to users Figure 1 provides examples illustrating this issue.
To overcome these limitations, we introduce B-cos LM, a dynamic linear model that learns the most task-relevant patterns through increased input-weight alignment pressure. Building upon B-cos networks that were first introduced by [öhlle et al. (2022) for computer vision, ensure the explainability of B-cos LMs through mathematically grounded architectural and computational adaptations, with specialized architectural modifications and training pipclines tailored for NLP tasks.
We conduct comprehensive empirical experiments using encoder-only models on classification tasks. Our focus on classification motivated by its prevalence in high-stakes applications, such as loan approvals, hiring decisions, and hate speech detection, where explainability is crucial and often legally mandated. Encoder-only models have also seen renewed interest in the research community (Warner et. al. 2024 Breton el. al. 2025 Chaffin 2025 and they remain the standard architecture for text classification tasks and continue to perform competitively compared to large language models (LLMs) (Zhao et al.) 2024). Beyond that, we also explore applying B-cos LMs to decoder-only models for generation tasks and show that B-cos LMs can be extended to variety of tasks and the latest model architectures. Our contributions are as follows:
We propose B-cos LM, a novel model with enhanced explainability. Automatic and human evaluations demonstrate that B-cos LMs generate more faithful and human interpretable explanations than post-hoc explanations while maintaining strong task performance.
2. We investigate different strategies for transforming PLMs into task-specific B-cos LMs. Our findings show that combining lask fine-tuning and B-cos conversion is the most elficient approach, leading to faster convergence than previous B-cos methods and conventional fine-tuning.
Connidering the evolving definition these terms past. literature, we provide detailed definition in Appendix
2

---

Published in Transactions on Machine Learning Research (12/2025)
3. We thoroughly investigate how B-cos LMs differ from conventionally fine-tuned models and examine how alignment pressure influences their behavior.
4. We are also the first to explore the transformation of decoder-only models to B-cos LMs for generation tasks, providing a step towards a broader application of B-cos LMs in the era of LLMs.
2 Related Work
Various methods have been proposed lo provide post-hoc explanation ions for neural model predictions (Atanasova et al. 2020). These methods can be broadly categorized based on how
Post-hoc Explanation Methods
they generate explanations: gradient-based (Simonyan al.,
2014 Kindermans al.
2016
Sundararajan
et al.
2017
Enguehard| 2023), propagation
bastetoon--adddddd based (Bach al.
2015 Shrikumar et al. 2017 Springenberg
et al.
2015
Ferrando et al.
2023 Modarressi et al. 2022 2023), and perturbation- based methods (Li
et al.
2016
Ribeiro et al. 2016 6 Lundberg & Lee. 2017 Deiseroth et al., 2023). Besides, the attention
mechanism
Bahdanau et al.
2015) is often viewed as an explanation, particularly in transformer-
models Vaswani et al.]
2017 While most. existing work focuses on understanding model predictions in
classification settings, rocent efforts have also aimed to explain model behavior in generation tasks, including
sentence completion (Yin & Neubig, 2022 Ferrando al., 2023), question answering (Enouen et al. 2024).
and summarization (Cohen- Wang al. 2024).
Although post-hoc methods have boen widely used, numerous studies have shown that they lack faithfulness, often failing to capture the true decision-making process of the model (Kindermans et al. 2019 Jain &
Wallace 2019 Slack et al. 2020 Pruthi et al., 2020 Ye et al. 2025). Furthermore, they are noisy and may select irrelevant information, leading to explanations that cannot be interpreted by humans (Smilkov et al.,
2017 Ismail et al. 2021
From Post-hoe Explanations to Explainable Models Prior rescarch suggests that the lack of faithful- ness and human interpretability in post-hoc explanations arises from the fundamental lack of explainability in modern neural models, which are typically optimized solely for task performance (Kindermans
al., Atanasova al. 2022). In response, various efforts have been made to enhance model explainabilin bility. Some works have introduced constraints that improve specific explanation properties, such as laithfulness Tutek & Suajder 2022 Moradi et al. 2020 2021 Barkan et al. 2024), consistency (Alanasova) et al. 2022). locality (Alvarez Melis Jaakkola 2018 and plausibility (Ismail et al. 2021). However, as these constraints are typically imposed as regularizers, their effectiveness in improving explanation quality is not guaranteed (Pruthi al., 2020). Others have proposed self-explanatory model architectures such as rationale-based models that utilize an "explain-hhen-prediitt pipeline, where one module selects rationales for another to make predictions based on them (Lei et al., 2016). Although seemingly transparent, both components rely on neural networks, making the rationale extraction and utilization processes opaque (Zheng et al., 2022 Jacovi & Goldberg 2021). Besides, such models may face optimization challenges that limit their practicality in real-world tasks (Lyu ct al. 2024).
2018 Rudin 2019
To lackle these shortcomings, Böhle el al. (2022) proposed B-cos networks. Unlike methods that impose external constraints, B-cos networks improve explainability through mathematically grounded architectural and computational adaptations. Morcover, these adaptations are designed drop-in replacements for conventional model components, making B-cos networks easy to train with minimal performance loss. Most recently, Arya et al. (2024) explored B-cosification techniques to convert existing models into B-cos models, which reduces the training costs of adopting B-cos architectures.
Despite their successful application in vision tasks, B-cos networks have yet to be explored in NLP, where input modalities and training paradigms differ significantly. In this work, we adapt B-cos models for the language domain, integrating them efficiently into NLP pipelines.
3 Methodology
In this section, we outline the architecture and training process of B-cos LMs and how their design ensures faithful and human interpretable explanations. We first. introduce B-cos networks (S (.1 and then describe
3

---

Published in Transactions on Machine Learning Research (12/2025)
how transform PLMs to task-specific B-cos LMs ($ 3.2 Finally, we demonstrate how to generate explanations from B-cos LMs ($ 3.3 Notations used in the work are detailed in Appendix B
3.1 B-cos Networks
Complex neural networks can be interpreted as generalized linear models (Nair & Hinton 2010 Alvarez Melis & Jaakkola, 2018; Srinivas & Fleuret 2019). For each input x, the network effectively applies a linear transformation: f(x) W(x)x b(x), where both the weight W(x) and bias b(x) depend on x. Given that many activation functions are (approximately) piecewise linear, the overall network can be viewed as (approximately) piecewise affine (Alvarez Melis & Jaakkola 2018). Earlier work refers to such models as dynamic linear models (Böhle et al. 2021 2022), highlighting the fact that the weight and bias terms dynamically change according to x.
Under this dynamic lincar perspective, the linear mapping ((x) can be seen as attributing model prodictions to individual input features, and Wxx)xx; can be seen the contribution of feature x; to the model prediction. However, two challenges hinder the direct use of this interpretation. First, W(x) alone provides an incomplete and unfaithful model summary, since f(x) & W(x)x due to the presence of the bias term bxx), and incorporating b(x) into explanations is highly non-trivial (Wang et al.) 2019). Second, W(x))x is often difficult for humans to interpret, as (xx does not necessarily align only with task-relevant input patterns (Smilkov et al. 2017) and therefore yields noisy and irrelevant explanations. Figure [ illustrates these challenges. To address these issues, Böhle et al. (002)) introduced B-cos networks by replacing the conventional linear transformation:
f(x; w. b) = w + |||||x||oo(xx w) +
(1)
with a B-cos transformation:
B-cos(x; w) = Tx x x |cos(x,, w) )
(2)
|| [|]|||||//cos[f )1B sgn(cos(xx w))
where w is a scaled version w with unit norm and sgn denotes the sign function.
B-cos(x: w) can be seen as a linear transformation of x with the dynamic linear weight w(x) = |cos(x, B) x w. The absence of b(x) ensures the completeness of summary w(x). We show that all components of transformer models can be viewed as or easily converted to bias-free, dynamic linear modules in Appendix C and demonstrate that this completeness extends to an entire network composed of such modules in $ 3.3. Moreover, since the B-cos module output is bounded by ||x||, the weight w must align closely with task-relevant patterns to achieve a high cosine similarity and strong activation, especially under additional alignment pressure (B>1). This drives the model to assign greater weight. Lo the most relevant. features when optimizing target output probabilities, promoting the learning of representative patterns during training. Consequently, during explanation generation, task-relevant features X₁ receive higher attribution W(x;)x; due to stronger alignment, while irrelevant features receive lower attribution, suppressed by weaker alignment and the exponential scaling. For a more detailed discussion of how the B-cos transformation enhances faithfulness and human interpretability, please see Böhle et al. 2022 2024].
While early B-cos models were trained from scratch, Arya et al. 2024 recently introduced B-cosification, an efficient method to obtain B-cos models. This approach first modifies conventional models with task capacities adopt the B-cos architecture, followed by fine-tuming on downstream datasets for B-cos conversion. B-cosified models generate explanations faithful and interpretable as B-cos models trained from scratch but at much lower training cost. However, directly applying B-cosification to LMs is non-trivial and inefficient due to the significant differences in model architectures and training pipclines.
3.2 B-cosification for LMs
In this section, we present our B-cosification approach for LMs. We summarize the differences between B-cosification for L.Ms, its counterpart for vision models, and conventional fine-tuning in Table
4

---

Published in Transactions on Machine Learning Research (12/2025)
Property
Couventional Fine-tuning
B-cosification for vision
Arya et al. 2024
B-cos LM (ours)
Bias Verous
yes
(alignment pressure)
1:23 1.5 ideatity
Pred. Head Activations
tanh
Prior
to
yes B-cos conversson
Bo
Training objectives
Task fine- suning
Task fine- tuming & B-cos conversion
Table 1: Comparison between conventional fine-tuning, B-cosification for vision models and B-cosification for language models (B-cos LM). Conventional fine-tuning and B-cosification for vision follow the configuration of BERT for sequence classification and CLIP (Radford et al. 2021], respectively (cf. 3 for details).
3.2.1 B-cos Adaptations
Given a conventional model, we first modify its architecture and computation to integrate the B-cos framework.
Architectural Adaptations For completeness and faithfulness of explanations, we follow Arya et al. 2024 and remove bias terms in models, including those in the affine transformations of layer normalization and attention blocks. Additionally, a prediction head is typically added on top of transformers before fnn-tuning for downstream tasks in the NLP pipeline. This head often includes activation functions that are not (approximately) piecewise linear, such sigmoid and tanh. To accommodate the unique architecture of LMs, we remove all activation functions in the prediction heads, as they generate explanations that are not locally deference-boundee (Alvarez Melis & Jaakkola 2018) and introduce numerical instability during explanation gencration. Our experiments show that the added non-lincarity from B>1 could compensate for this removal.
Introducing B-cos Computation To promote input-weight alignment and improve human interpretability of explanations, we replace linear transformations with B-cos transformations in $ 3. For a more efficient. B-cosification, B-cos layers are initialized with the corresponding weights of the original model.
3.2.2 Fine-tuning
The B-cos adaptations above modily the architecture and computation of models, requiring fine-tuning to restore their capabilities and adapt to alignment pressure. Following the "pre-train then fine-tune" paradigm, which is frequently utilized in NLP tasks, we directly transform PLMs to B-cos LMs, rather than adapting task-specific models as done in previous work (Arya et al. 2024.. This fundamental difference in the training pipeline adds complexity to B-cosification for LMs, as the objective involves both B-cos conversion and task fine-tuning. While there are multiple ways to conjoin these two steps (cf. 5 we find that the most efficient way is to combine them by first applying B-cos adaptations to PLM and then fino-tuning it downstream task. Following Böhle et al. 2022). we the binary cross-entrops (BCE) loss instead of the conventional cross-entropy loss, as it explicitly maximizes the absolute target logits and strengthens the alignment pressure. We provide an extensive comparison of different B-cosification setups in $ 5
3.3 Computing B-cos Explanations
Once trained, the B-cos LM can generate explanations that faithfully summarize its decision-making process during inference. As all components dynamic linear with no bias terms (cf. Appendix C), the entire model computation can be expressed as sequence of matrix multiplications, which can be completely summarized as single dynamic linear function:
X)X 11 (A,))
3)
Note that a residual connection of Wxx) x with x E gn and W(x) Rx is mathematically equivalent to single dynamic lincar transformation of (W(x) + )x). Considering the textual inputs specific to LMs, we attribute the model's predictions to the embedding representations. Specifically, to quantily the contribution of a token to a model prediction, we compute the dot product W(x))x, between its embedding x; and the
Arya al. 2024 used single linear layer top of CLIP so the prediction head activation tion not applicable their setup.
5

---

Published in Transactions on Machine Learning Research (12/2025)
corresponding dynamic linear weight (xx)) for the target class logit. For the remainder of the paper, we will refer to such explanations as B-cos explanations.
4 Experiments
We evaluate the lask performance of B-cos LMs and faithfulness of B-cos explanations with automatic evaluation across various tasks and PLMs. In addition, conduct human evaluation study to compare the human interpretability of B-cos explanations. as well as human evaluation study, respectively. $4.4 provides a qualitative analysis. Finally, we conduct an ablation study in $4.5 More details on the experimental setup and baseline methods are provided in Appendix D and a comparison of computational efficiency is provided Appendix E
$4.1
4.3
describe our automatic evaluation setup, results,
4.1 Experimental Setup
Datasets and Models Our experiments use three datasets: AG News (topic classification, Zhang et al.
2015), IMDB (sentiment analysis. Maas al.) 2011), and HateXplain (hate speech detection, Mathew et al,
2021 BERT (Devlin et al. 2019, DistilBERT (Sanh et al.) 2019), and RoBERTa (Liu et al. 2019) used
as the basis for conventional fine-tuning and for obtaining B-cos LMs. We set B-1.25 for IMDB and B=1.5
for AG News and HateXplain datasets.
Baselines We compare B-cos explanations against a diverse set of post-hoe explanation methods: Attention Bahdanau al.) 2015). InraieGrr (IxG, Kindermans al. 2016), Sequential Integrated Gradients
SIG, Enguehard 2023 DecompX (Modarressi al. 2023), Shapley Value Sampling (ShapSampl, [Stumbel]
& Kononenko 2010), and LIME (Ribeiro et al. 2016 We also apply these methods to model trained with Saloss (Chrysostomou & Aletras 2021b, which introduces faithfulness regularization. This setup enables a direct comparison belween B-cos LMs and models specifically optimized for explainability. For embedding- explanation methods, we aggregate attributions by summing across all embedding dimensions.
Faithfulness Metrics For more comprchensive evaluation, we employ two different methods to assess faithfulness. First, report two perturbation-baser metrics YDeng et al. 2020):
Comprehensiveness (Comp) measures the average drop in predicted class probability after masking out the top A% most important tokens in the explanation. higher score indicates better faithfulncss. Sufficiency (Suff) measures the average drop in predicted class probability after keeping only the top 4% tokens. A lower score indicates better faithfulness.
To avoid arbitrary choices of k, compute Comp and Suff for multiple values (k 10, 20, 90) and summarize them using the Area Over the Perturbation Curve (AOPC, DeYoung et al. 2020).
In addition, we introduce a new faithfulness metric called Sequence Pointing Game (SeqPG), inspired by the grid pointing game in vision tasks (Böhle et al. 2021
Sequence Pointing Game (SeqPG.. We evaluate models on synthetic sequences composed of segments associated with different classes. To assess faithfulness, we mcasure the proportion of positive altribution assigned to the corresponding segment of each class and compute their average. A higher score indicates better faithfulness.
Compared to perturbation-based metrics, SoqPG does not rely on perturbations and thus avoids the potential distortions rntrodee by loken masking. When constructing SeqPG examples, we truncale each segment. to a fixed length and randomize segment order to control for length and position effects. We generate synthetic examples using correctly and most confidently classified test instances. SoqPG can be seen as standardizod version of hybrid document evaluation (Poerner al. 2018). We provide an example of SeqPG in Figure and more details in Appendix F

---

Published in Transactions on Machine Learning Research (12/2025)
4.2 Automatic Evaluation Results
Task Performance Figure 2 shows the accuracy of conventionally fine-tuned, Saloss and B-cos BERT across three datasets (we provide results for DistilBERT and RoBERTaa in Appendix Gj. On AG News and HateXplain, B-cos LMs performs on par with conventional modcls, with only a minor drop (~1%) in accuracy. They also outperform Saloss models on these datasets. Only for IMDB, we find a slightly larger drop of 3.06% compared conventional BERT, though the performance remains strong overall.
Faithfulness Results Table 2 shows the faithful- scores for post-hoc explanation methods con- ventionally fine-tuned and Saloss BERT models, as well as B-cos explanations from B-cos BERT. The re-
94.76
92.72
93.81
93.56
91.00
Saloss DCRT
Accuracy
80
78.4
sults show that B-cos explanations are consistently and substantially more faithful than post-hoc methods across all models and datasets. On average, B-cos explanations oulperform the strongest post-hoc meth- ods conventional models by 14.63 points in Comp and achieve negative Suff scores, indicating that the identified important tokens alone enable even more confident predictions. B-cos also shows significant. gains in SeqPG. While Saloss improves faithfulness for some post-hoc methods over ionaentional models, it still lags markedly behind B-cos LMs. Similar trends are observed for other PLMs (Appendix H as well. Although we do not include rationale-baset models in the main experiments because they typically require additional supervision, a supplementary comparison in Appendix D shows that B-cos BERT still outperforms rationale-based model on HateXplain.
75
70
AG news
IMDB
HateXplain
Figure 2: Mean accuracy of conventionally fine-tuned, Saloss and B-cos BERT averaged over three runs. B-cos models perform comparably conventional models on most tasks.
AG News
IMDB
HateXplain
Model
Method
Comp (t)
Suff (1)
SeqPG (1)
Comp (t)
Suff (1)
SeqPG (1)
Comp (1)
Suff (U)
SegPG (t)
Attention
24.40
8.09
50
26.84
14.56
50
27.6-
13.83
50
IxG
15.28
10.19
45.41
18.29
16.96
49.42
19.16
18.90
47.24
SIG
27
3.40
64.77
29.34
14.05
59.09
37.31
5.10
66.38
Conv. BERT
DecompX
52.16
0.92
84.48
57.94
2.41
63.27
44.86
2.72
66.76
ShapSampl
43.96
0.46
82.87
58.29
2.44
71.29
44.86
2.43
67.17
LIME
44.95
0.06
80.28
51.45
6.07
60.15
22.64
14.30
57.61
Attention
34.73
3.65
50
27.59
13.64
50
34.95
26.26
50
IxG
14.98
12.66
51.01
24.19
16.30
49.02
26.61
30.94
50.74
SIG
16.70
8,22
63.74
45.44
8.48
51.96
44.53
21.50
51.70
Saloss BERT
DecompX
59.37
0.30
75.34
59.42
5.38
62.02
58.71
13.23
65.17
ShapSempl
37.73
0.77
73.96
65.38
3.17
70.23
57.05
15.10
72.36
LIME
53.18
2.37
76.16
53.31
6.32
58.65
21.73
21.96
55.71
B-cos BERT
B-cns
64.22
-1.26
87.92
74.18
-2.87
70.43
59.66
-4.89
77.57
Table 2: Faithfulness evaluation for conventionally finc-tuned, Saloss, and B-cos BERT across three datascts.
The best results are in bold. B-cos explanations are consistently more faithful than post-hoe explanations from both baseline models.
4.3 Human Evaluation
Contrary to previous B-cos studies that rely solely on automatic evaluations to assess explanations, we conduct the first human study to better evaluate the human interpretability and agreement of B-cos explanations. We compare them against three strong post-hoe explanation methods conventional BERT models.
Following the practice in Enguehard 2023) and Yue et al. (2022) randomly select 50 instances, respectively, from AG News and HateXplain where the B-cos and conventional models make the same prediction. Five annotators then rate the explanations in terms of human interpretability (how well they understand them) and human agreement (how much they agroe with them) on a scale of 1-5. Further details on the evaluation criteria, rating scales, annotator instructious, and example annotations can be found Appendix J

---

Published in Transactions on Machine Learning Research (12/2025)
Figure B shows that B-cos explanations have a better human interpretability and exhibit greater alignment with human reasoning than post-hoc methods, even though they are not. directly optimized for human agreement. Paired t-tests with Bonnerroniicorreeteed significance level 0.05 0.0083 (Bonferroni] 1936) shows that the improvements of B-cos explanations statistically significant (p < a:) for both metrics.
B-cos
ShapSampl
Decompx
3.48
3.16
Rating
4.4 Qualitative Analysis
Human Interpretability
Human Agreement
provides providds example of B-cos and other (post-hoc) explanations. It can be seen that the B-cos explanation highlights important tokens correctly with little focus on irrelevant ones. In contrast, ShapSampl attributes the highest importance to the [SEP] token and provides only little useful information. Meanwhile, DecompX extracts a significant amount of irrelevant information. Overall, the B-cos explanation is more interpretable humans by providing clearer and more relevant attributions compared the post-hoc explanations. We
Figure Human evaluation reveals that B-cos explanations have better human interpretability and human agreement than baseline methods.
provide more examples and analyze undesired explanations from B-cos LMs in Appendix K
Label: Kci/Tech, Model prediction: Sci/Tech
B-cos
(CLS) microsoft to help users prep for patching microsoft said today that plans to give customers three days
monthly security updates
prepare
related software patches [SEP]
ShapSampl [CLS] microsoft help users prep for patching microsoft said today that it plans to give customers three days
advance notice about its monthly security updetes help them prepare to install related software patches
DecompX
CCS
microsoft help prap for patching microsoft said today that plans give
three
advance notice about its monthly security updates to help them prepare to install related software patches
[SEP)
Figure 4: Examples of B-cos explanations (B-cos BERT) as well as ShapSampl and DecompX explanations (conv. BERT) from AG News. Green (red) indicates the positive (negative) impact of tokens on the prediction. The B-cos explanation highlights only relevant tokens and is more interpretable to humans.
4.5 Ablation Study
To better understand B-cos LM,, we conduct an ablation study evaluating the impact of key design choices task performance and explanation faithfulness.
Acc (t)
Comp (t)
Suff (J)
SeqPG (t)
Full system
78.64
59.66
-4.89
77.57
w/o aligument pressure (B-1)
78.07 (0.57)
57.19 (2.44)
-2.57 (2.32)
70.18 (7.39)
w/o BCE training
79.00 (0.36)
49.22
(10.44)
-7.91 (3.02)
79.21 (1.64)
w/o architectural adaptations
77.65 (0.99)
52.23 (7.43)
-3.80 (1.09)
74.30 (3.27)
w/o dynamic linear weights (IxG)
78. (0.00)
44.93 (14.73)
-0.60 (4.29)
53.57 (24.00)
W(x)x from conv. model
80.77 (2.13)
44.92 (14.74)
2,80 (7.69)
70.20 (7.37)
Table 3: Ablation study of key designs in the B-cos BERT model on HateXplain. Green (red) values in parentheses indicate the results are better (worse) than the full system.
In Table 8 we find that removing alignment pressure (using B=1) degrades both task performance and explanation faithfulness. Replacing cross-entropy with BCE loss has little effect classification accuracy,
8

---

Published in Transactions on Machine Learning Research (12/2025)
but improves faithfulness in perturbation- based evaluations. Architectural adaptations, including removing bias terms and eliminating activation functions in prediction heads, are also critical for enhancing task performance and explainability. Besides, we observe numerical instability when generating explanations without these architectural adaptations, as the dynamic lincar weights for sigmoid and tanh (sigmoid(x) x xx and lanh(x) x x- become unstable when x is close to zero.
In addition to ablations of model design and training components, we also evaluate alternative explanation methods. Replacing the dynamic linear weights W(x) with gradients (equivalent to IxG) yields less faithful explanations on B-cos LMs. Besides, directly extracting B-cos-like explanations, W Wxx)x, from a conventional model results in worse faithfulness compared to those from B-cos LMs.
5 Comparison of B-cosification Setups
Transforming PLMs into task-specific B-cos LMs involves two key objectives: task fine-tuning and B-cos conversion. While our main experiments combine these two phases, they can also be performed separately. To assess their effects, we compare two alternative training sctups:
ream task. B-cos adaptations then applied, followed by further fine-tuning on the same task for B-cos conversion. This setup is equivalent to
Task then B-cos: PLMs are first fine-tuned
Arya el al. (2024) who apply B-cosification to models with downstream task capabilities.
B-cos then task: B-cos adaptations are applied to PLMs first, followed by unsupervised pre-training to enhance B-cosification. The pre-trained B-cos models are then fine-tuned on the downstream task.
We evaluate these selups against the B-cosification ap- proach used in our main experiments (B-cos LM) and compare task performance, faithfulness, and training of-
Setup
Epochs Ace (1) SeqPG (1) Steps (K)
Conv. LM
5
94.06
6.67
ficiency (cf. Appendix D for B-cos pre-training details). We also report results for conventional fine-tuning (Conv. LM) and randomly initialized B-cos models (B-cos from scratch). All experiments are run on IMDB with B=1.25 for B-cos models, with results averaged over three runs.
B-cos LM
5
91.00
70.66
4.33
B-cos from scratch
88.25
60.92
4.33
1+4
91.17
70.01
1+5
2+3
91.30
70.48
34-3.33
Task then B-cos
3+2
91.38
70.83
443
4+-1
89.56
70.66
5+11
5++5*
91.27
70.78
6.67+3333
Table 4 shows that B-cos LM requires fewer training steps to reach optimal validation performance than conventional fine-tuning. Training B-cos LM from scratch results in worse accuracy and faithfulness, showing the importance of good parameter initialization. Among the two setups that separate task sue-tuning and B-cos conversion, Task then B-cos achieves results similar to B-cos LM but ro- quires more total training steps. B-cos then task initially performs worse under the same training budget. How- ever, with additional pre-training epochs, it surpasses other B-cosification setups in both task performance and faithfulness. Overall, we find that combining task fine- tuning and B-cos conversion is the most efficient approach.
1+-4
90.64
67.07
2+:3
91.01
65.97
3+-4
3+-2
90.50
68.48
4++3
B-cos then task
4+-1
89.18
69.92
6+1
5+-5*
91.45
71.86
745.33
10+-5*
92.19
73.44
15-+6.33
20++5*
92.88
75.01
31+6
Table 4: Training epochs, accuracy, explanation faithfulness, and convergence steps for different B-cosification setups. For two-phase methods, we report epoch distribution and convergence steps per phase. * marks additional training epochs.
However, with sufficient pre-training, B-cos then task can produce more performant and explainable models.
6 Impact of B-cosification and B Values
For a deeper understanding of how B-cosification and alignment pressure parameter B affect model performance and bchavior, compare conventional and B-cos BERT trained on HateXplain across different B values. We also provide an empirical analysis of the impact of B on input-weight alignment in Appendix L
Model Performance Figure shows the effects of varying B on the task performance and explanation fnithfulness. Classification accuracy initially improves slightly B increases from 1 to 1.25, benefiting from
9

---

Published in Transactions on Machine Learning Research (12/2025)
the extra non-linearity introduced by B>1. However, beyond this point, accuracy declines higher alignment pressure reduces model flexibility. A similar trend is observed for Comp, which peaks around B=1.5 before decreasing. This differs from previous findings in vision models (Böhle et al. 2022), which we attribute to the high sparsity of explanations at larger B values. As alignment pressure increases, fewer tokens receive altribution scores that are not close lo zero, leading to poor token importance calibration and worse Comp scores. The effects of on other metrics are similar and can be found in Appendix
Explanation Entropy Figure also reveals a negative correlation between explanation entropy and B, indicating that higher alignment pressure leads to sparser explana-
4.0
90
Accuracy
Entropy
Comp
3.5
Accuracy 0%6 Come
tions. This aligns with our expectations: larger B ampli- fies the differences between dimensions in |cos(x, W) [B-1 of BB-co layers (Equation 2 and the dynamic linear weight assigns more distinct attributions to input features. As a result, explanations become more concentrated, where only a few tokens receive high attributions, while most remain close to zero (ef. Appendix N for an example).
70-
60-
2.5
40-
2.0
1.00
1.25
1.50
1.75
2.00
225
2.50
Model Bias PLMs often acquire biased patterns present in their training data (Wang & Demberg 2024a bj. Since B-cos LMs with larger B values rely on fewer to- kens for prediction, we investigate whether this may cause them to overfit and learn biases in the data. For this, we examine label bias and word-level spurious correlations using HateXplain, where approximately 60% of training and test examples have positive labels and societal biases are present. Figure shows that a larger B value (B=2.5) reduces the model capacity, leading to substantially higher positive rate in predictions and therefore lower class-balanced accuracy. Moreover, the B=2.5 model assigns higher attributions to non-semantic (CLS) and [SEP] tokens, indicating reduced reliance mcaningful content. Notably, this label bias is not observed in the balanced AG News and IMDB dalasets.
Figure 5: Varying B for B-cos BERT (HateX- plain). Accuracy and Comp both peak around B=155, while explanation entropy negatively cor- relates with B.
We also find that B-cosification, particularly with large B, amplifics reliance on spurions correlations. For example, the prediction positive rate for examples with the word "black" rises from 49. 19.02% in the set and 52.94% the conventional model to 59.80%, 56.86%, and 73.53% in B- cos LMs with B=1, 1.5, and, 2.5, respectively (we provide example in Appendix O However, the faithfulness and interpretability of B-cos explanations facilitate the detection of spurious correlations and can effectively guide models toward reducing them (Rao al. 2023 Balkir ct al. 2022. Wang al. 2025b). We leave the exploration of B-cos LMs for bias detection and mitigation to future work.
80
Add birioon Ratlo
55
50
Cons.
6-1
8-1.5
B-2,5
True Label Positive Rate
Balanced Accuracy
Predictin Positive Rate
[CLS]4 [SEP) Attributian
Figure 6: Comparison of conv. BERT and B-cos BERT with different B values. The attributions to [CLS] and [SEP] tokens indicate that B- cos LMs with large overfit to the non-semantic label distribution.
7 B-cosifying Decoder-Only Models for Generation Tasks
LLMs are increasingly used general-purpose assistants, with most based decoder-only architectures ZZao ct al. 2023: Minace et al. 2024). While our primary focus is on classification tasks using encoder-only models, we also exlend B-cosification to decoder-only models for generation tasks to demonstrate the broader applicability of B-cos LMs. Specifically, we apply B-cosification to two decoder-only models, GPT-2 small (Radford al. 2019 referred as GPT-2 afterwards) and Llama-3.2-1B (Dubey et al. 2024] referred Llama-3.2 afterwards), and evaluate their language modeling performance and explanation quality on two ioneration tasks. For more details on the datasets, experimental setup and baseline models, Appendix [)
10

---

Published in Transactions on Machine Learning Research (12/2025)
B-cosification Setup Given the complexity of mod- eling natural language, we use a small B value of 1.1. We do not B-cosify the language head, as its parameters are tied with the embedding layer. We use the standard cross-entropy loss instead of BCE, since the unnormalized language head weights could otherwise grow arbitrarily large to minimize the loss. To convert GPT-2 and Llama- 3.2 to B-cos LMs, we apply B-cos adaptations and further train them on 500,000 and 4,000,000 sentences from Open- WebTex respectively.
Probability Gap (1)
Model
PPL (U)
BLiMP
IOI
GPT-2
0.0055
0.3351
3.10
B-cos GPT-2
0.0059
0.3265
3.04
Llama-3.2
0.0058
0.4652
2.51
B-cos Llama-3.2
0.0065
0.5021
2.64
Table 5: Language ability results for vanilla and B-cos decoder-only models. Scores where B-cos LM outperforms their vanilla counterparts are in bold. B-cos LMs show language modeling ability comparable to vanilla models. Results for each subset can be found in Table 13 in Appendix P
Datascts For explanation evaluation, we use the BLiMP dataset (Warstadt et al.) (020) to assess ex- planations for linguistic phenomena, and the Indirect Object Identification (TOT) dataset (Brian Muhia, (022)
to test models' reasoning about object identificatinn. Following Ferrando et al. 2023 we use nine subsets of BLiMP. Each example in both datasets consists of sentence prefix followed by tar- get and a foil next word prediction, differing in whether they align with the phenomenon or abil- ity of interest. Ground truth evidence is provided to support either grammatical correctness or cor- rect object identification. Examples of these datasets can be found in Table 7 in Appendix D
Metrics and Baselines We evaluate explanation qual- ity using Mean Reciprocal Rank (MRR), where higher scores indicate stronger alignment with the ground truth evidence. To assess language modcling abilitics of modcls, we report two metrics: (1) the probability gap between target and foil predictions, and (2) perplexity (PPL) on held-out corpus. Following Yin & Neubig (2022).
GPT-2 MRR (t)
Llama-3.2 MRR (t)
Method
BLMM
T0T
BLiMP
TOT
Random
0.5130
0.2360
0.5132
0.2328
Grad Norm
0.5465
0.8599
0.5504
0.3637
IxG
0.4750
0,1112
0.5303
0,1034
Occlusion
0.6365
0.8517
0.6201
0.4767
Logit
0.7307
1.0
generate contrastive explanations that explain why the model predicts target tokens instead of foil tokens, and compare B-cos explanations against several baseline meth- ods: L1 gradient norm (Grad Norm), IxG, Occlusion, and two propagation- based methods Logit and ALTI Logit from Ferrando al. 2023
ALTI Logit
0.7391
1.0
B-cos
0.7561
1.0
0.6969
0.9913
Table 6: Alignment results (MRR) on BLiMP and TOI. Logit and ALTT Logit results are replicated from the original paper (Ferrando et al. 2023 Best scores are marked in bold. B-cos expla- nations achieve the best alignment with ground truth evidence. Results for each subset can be found in Table (4and Table 15 in Appendix
Results Table 5 shows that B-cos GPT-2 and B-cos Llama-3.2 models achieve strong language modeling per- formance comparable to their vanilla counterparts. Be- sides, Table 6 demonstrates that B-cos explanations exhibit better alignment with ground truth across tasks and models, indicating improved explainability of B-cos docoder-only LMs. Although the current B-cosification pipeline requires additional training, future work could explore more efficient approaches that reduce training overhead or integrate B-cosification into the pre-training phase. Overall, we believe B-cos decoder-only models are well-suited for tasks where explainability is critical and represent promising direction for building more transparent and reliable LLMs.
8 Practical Guidance for Training B-cos LMs
Based on our experiments and analyses, we provide the following guidance on configuring B-cos LMs:
B-cosification Setup For efficient B-cosification of encoder-only models, we recommend combining B-cos conversion and task fine-tuning. However, if resources permit, an additional B-cos pre-training stage can further enhance both task performance and explanation faithfulness. For decoder-only models, B-cosification
11

---

Published in Transactions on Machine Learning Research (12/2025)
can be applied to pre-trained models with unsupervised training data, enabling their conversion into B-cos LMs with improved explainability.
Choice of B Value Several factors influence the selection of an appropriate B value:
(1) Model capacity and explanation sparsity: Excessively large values can reduce model capacity and produce overly sparse explanations. We therefore recommend setting B within the range of 1-2.
(2) Task complexity and language diversity: For complex tasks with varied language use (e.g., online forum data with diverse language styles), smaller B values are preferable, they preserve model flexibility and capture more useful features. For other tasks, larger B values can improve explainability.
(3) Bias considerations: Be mindful that B-cosification may amplify biases that the model learns from biased training data. When this risk is present, consider choosing smaller B values and applying bias mitigation methods if necessary.
9 Conclusion
In this work, we introduce B-cos LM, a bias-free dynamic linear model that learns task-relevant patterns through increased input-weight alignment pressure. B-cos LMs generate more faithful and human interpretable explanations while maintaining strong task performance and fast convergence. We further explore adapting decoder-only models into B-cos LMs for generation tasks and show that, with additional training, they match the language modeling performance of conventional models while providing better explanations. Finally, based on our systematic analysis, we provide practical guidelines for effectively transforming PLMs into B-cos LMs.
10 Limitations
This study has certain limitations that should be acknowledged. First, the automatic evaluation metrics we use may not fully capture the faithfulness of different explanation methods (Feng al. 2018 Lapuschkin et al. 2019). However, since there is no universal consensus on the most reliable evaluation metrics, this remains an open challenge in explainability research.
Second, we find that B-cos explanations do not consistently capture token interactions within multi-token phrases. For example, a negation phrase like not good tends to receive an overall attribution score that aligns with its meaning (e.g.; a negative score for positive sentiment), but the individual token scores within the phrase vary across contexts. In some cases, the word good may receive either positive negative scores across different examples, even when the overall sentiment remains the same. Similar issues arise in other methods, suggesting a broader limitation of token-level rationales in capturing compositional semantics.
Finally, B-cos explanations, like all input-based approaches, are not suitable for all tasks. They are particularly suitable for NLP tasks where predictions can be directly attributed to specific words or phrases. Examples include various text classification tasks, such as sentiment analysis and toxicity detection, where the presence absence of certain tokens often provides clear evidence for the model's decision. Conversely, for tasks requiring multi-step reasoning external knowledge (e-g., commonsense reasoning, fact verification), token- level rationales become less informative. Higher-level explanations, such concept-based natural language explanations, may instead offer more meaningful insight into the model's 's reasoning.
Acknowledgements
This work was funded in part by the Deutsche Forschungsgenniisslaatt (DFG, German Research Foundation) GRK 2853/1 'Neuroexplicit Models of Language, Vision, and Action' project number 471607914. We also thank all those who provided valuable feedback and suggestions during the development of this work. Their input has been instrumental in improving its quality.
12

---

Published in Transactions on Machine Learning Research (12/2025)
References
Sina Abbasi, Mohammad Reza Modarres, and Mohammad Taher Pilchvar. NormXLogit: The head-on-top never lies. In Christos Christodooulpouulss, Tannoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp.
Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176-3322 doi: 20.18653/v1/2000 http://112222.mllppppppp www.p-mmin.nnaeaee.mmm...m.m. 176n. URL https://aclaathholog/) org/2025. emmle-mannlppmmmmm 1769/
David Alvarez Melis and Tommi Jaakkola. Towards robust interpretability with self-cxplaining neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL
www.rng....uuri..... pwweurips.ccpppprrrr pdf
Leila Arras, Ahmed Osman, Klaus-Robert Müller, and Wojciech Samek. Evaluating recurrent neural network explanations. In Tal Linzen, Grzegorz Chrupal a, Yonatan Belinkov, and Dieuwke Hupkes (eds.), ), Proceedings of the 2019 ACL Workshop BlackboxNLPP Analyzing and Interpreting Newral Networks for NLP, pp. 113 126, Florence, Italy, August 2019. Association for Computational Linguisties. doi: 10.18633/v1/W19-46 URL httpp://aclanthollog) http://chhhbbblllo....gggg//-aa
Shreyash Arya, Sukrut Rao, Moritz Böhle, and Bernt Schiele. B-cosification: Transforming deep neural networks to be inherently linterpretable. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquct, J. Tomezak, and C. Zhang (eds.), Advances in Newral Information Processing Systems, volume 37, pp. 12756-62784 Curran Associates, Inc., 2024. URL http:://pprooceaiags..eu.. httpppscccpppppppppppeeeee files/ fppp:0000111777760r.222426666661
Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, and Isabelle Augenstein. A diagnostic study of explainability techniques for text classification. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 3256-3274, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/ v1 (1222000emmllpp wwwnl-mmiinnnn6665 main.263. URL
emnlp-main 263/
Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, and Isabelle Augenstein. Disgnostics-guidee explanation generation. In Proceedings of the AAAT Conference Artificial Intelligence, volume 36, pp. 10445 10453, 2022.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(():e0130140, 2015.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Yoshua Bengio and Yann LeCun (cds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL attp://arxiv. www///g//////eeeeeee..eeeeeeeeet abs//409 0473.
Esma Balkir, Svetlana Kiritchenko, Isar Nejadgholi, and Kathleen Fraser. Challenges in applying ex- plainability methods to improve the fairness of NLP models. In Apurv Verma, Yada Pruksachatkun, Kai- Wei Chang, Aram Galstyan, Jwala Dhamala, and Yang Trista Cao (eds.), Proceedings of the 2nd Workshop on Trustworthy Natural Language Processing (TrustNLLP 2022), pp. 80 30-92, Seattle, U.S.A., URL
July 2022. Association for Computational Linguistics. doi: 10.18653/ v1 /22.222.tttstnllp....222
trustalp-1.8/
Oren Barkan, Yonatan Toib, Yehonatan Elisha, Jonathan Weill, and Noan Koenigstein. LLM explainability via attributive masking learning. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.). Findings of the Association for Computational Linguisties: EMNLP 2024, pp. 5522-9537, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v v2/2024 2.02....iinng-oooo. crmiingsemmlp55566. URL httppe://aclattaooggg.rr//////// findings--niiiinsi-e emmlp.56//
13

---

Published in Transactions on Machine Learning Research (12/2025)
Moritz Böhle, Mario Fritz, and Bernt Schicle. Convolutional dynamic alignment networks for interpretable classifications. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognilion, pp. 10029 10038, 2021.
Moritz Böhle, Mario Fritz, and Bernt Schicle. B-cos networks: Alignment is all we need for interpretability. In Proceedings of the IEEE/CH /CVF Conference on Compuler Vision and Pollern Recognation (CVPR), pp- 10329 10338, June 2022.
Moritz Böhle, Navdooppal Singh, Mario Fritz, and Bernt Schicle. B-cos alignment for inherently interpretable cnns and vision transformers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ B. Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolisson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annic Chen, Kathleen Croel, Jared Quincy Davis, Dorottya Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren E. Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, and et al. On the opportunities and risks of foundation models. CoRR, abs/2108.07255 2021. URL https: h http///xxvv..rrr/ abs/ awwpaa//...//////////////aaaaaea 22////100000222
Carlo Bonferroni. Teoria statistica delle classi calcolo delle probabilita. Pubblicazioni del R Istâtuto Superiore di Scienze Economiche e Commericialii di Firenze, 8:3-62, 1936. doi:
9781412961258.n455
Lola Le Breton, Quentin Fournier, Mariam El Mezouar, and Sarath Chandar. Neobert: A next-generation BERT. CoRR, /bs/2502.19577 2025. doi: (0448500/ARXIV222222.15 URL https://doi.org//10 48550/arXiv 2502. 19587
Brian Muhia. ioi (revision 223da8b), 2022. URL httpp://hhgcgggfo..o.o//ttttttte
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Hemighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Tlya Sutskever, and Dario Amodei. Language models are few-shot learners. H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Newral Information Processing Systems, volume 33, pp. 1877 1901. Curran Associates, Inc.,, 2020. URL /tttps::/
Antoine Chaffin. Reason-moderncolbeet, 2025.
URL
Modeon-NodernooLBEERT
George Chrysostomou and Nikolaos Aletras. Enjoy the salience: Towards better transformer-based faithful explanations with word salience. In Maric-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 8189 8200,8200, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2001 http:///11/22.memllppmem eennl--mainnn main.645.. URL httpp::/acclanthologge org/ 2021. wwwnmmmmannnnnmmmmmeeaamaammmmat 645/
Benjamin Cohen- Wang, Harshay Shah, Kristian Georgiev, and Aleksander Mądry. Contextcite: Attributing model generation to context. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, Tomezak, and C. Zhang (ods.), Advances in Neural Information Processing Systems, volume 37, pp. 95764-95807. Cur- ran Associates, Inc., 2024. URL /tttot//////////oeeeeee www.:eeedddnnnn.uis.s.oomm aateatasee.saaaaaaaaeeeeeeeereee http://...........ooppp.o.pp/ppp
14

---

Published in Transactions on Machine Learning Research (12/2025)
Björn Deiscroth, Mayukh Deb, Samuel Weinbach, Manuel Brack, Patrick Schramowski, and Kris- tian Kersting. Atman: Understanding transformer predictions through memory efficient atten- tion manipulation. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 63437-63460. Cur- ran Associales, Inc., 2023. URL httt:p:///poocedng..eeeeirre ctp.pp...ccppprrrreeee fillss/paper/223//ii1a 08861220000cee66ee11066666666669
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (ods.), Proceedings of the 2019 Conference of the North American Chapler of the Association for Computational Linguistios: Human Language Technologies, NAACI-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pp. 4171 4186, Association for Computational Linguistics, 2019. doi: 11886//V1/1199/ 1423. URL https: http:///oooorro//////e 10. 18653//1//19--///
Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. ERASER: A benchmark to evaluate rationalized NLP models. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistios, pp. 1443-4458 Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18653/v1 20/533vv12022/ /2020000c0--mmin.44.. URL htttps://aclaatthollggg wrg. aac2..acll1...................l. acl. 1aammmi..000 408/
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archic Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia- Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Fhab AlBadawy, Elina Lobanova, Emiil Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Lliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jacwon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadcokar, Joct Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. The Ilama 3 herd of models. CoRR, abs/ abs/2407.222 21783, 2024. doi: 005550//ARXIV22407.0202
URL https://doi.org::: 41////0..8555//ra1.. 2407 21783
Joseph Enguehard. Sequential integrated gradients: a simple but effective method for explaining language and Naoaki Okazaki (eds.). Findings of the Association
models. In Anna Rogers, Jordan for Computational Linguistics: ACL 2028, pp. 7555-7565, Toronto, Canada, July 2023. Association for Computational Linguisties. doi: 10.18653/v1 2t1//2222...nnnngge. 2023. Findings-acl.4777
URL httpp://aclaanthologyy org/
James Enouen, Hootan Nakhost, Sayna Ebrahimi, Sercan Arik, Yan Liu, and Tomas Pfister. TextGeSHHP: Scalable post-hoc explanations in text generation with long documents. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp- 13984 14011, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/ v1 fin2024.fnninns- wac.iinngssaal8222222 acl.83. URL https://aclantho/// wttp://ttooooooooo////2././. wrg/2024. f findings-acll.... ac1.832/
Kawin Ethayarajh. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNNIL) pp. 55-65, Hong Kong, China, November 2019. Association for Computational Linguisties. doi: 10.18653/v1/D19- 1006. URL
15

---

Published in Transactions on Machine Learning Research (12/2025)
Shi Feng, Eric Wallace, Alvin Grissom II, Mohit Iyyer, Pedro Rodriguez, and Jordan Boyd-Graber Pathologies of neural models make interpretations difficult. In Ellen Riloff, David Chiang, Julia. Hockenmaier, and Jun'ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3719-3728, Brussels, Belgium, October-Novemberr 2018. Association for Computational Linguistics. doi: 10. 10.18653/v1 /153//11/188-407 URL httas://aclaathhooogg.o////////
Zijian Feng, Hanzhang Zhou, Zixiao Zhu, Junlang Qian, and Kezhi Mao. Unveiling and manipulating prompt influence in large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL httpp:://peerrevier net/ forum? Dorum?id=ap1Byuutii
Javier Ferrando, Gerard I. Gállego, Toannis Tsiamas, and Marta R. Costa-jussà. Explaining how transformers use context build predictions. In Anna Rogers, Jordan
and Naoaki Okazaki (ods.), Proceedings of the 61st Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pp. 5486 5513, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.18653/ vv1 a00..000..-0.1...300111. URT. sttta://aclaataaoogg.o//////// 2ww.2220...c1ooogg.. 1010
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noachh, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023. URL httts:://enod:: httpp///o..orgeeeerr//////6--a50
Dan Hendrycks and Kevin Gimpel. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR, abs/1606.000 1606.08415, 2016. URL
08415
Aya Abdelsalam Ismail, Hector Corrada Bravo, and Soheil Feizi. Improving deep learning interpretability by saliency guided training. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (cds.), Advances in Neural Information Processing Systems, volume 34, pp. 26726-26739. Cur- ran Associates, Inc., 2021. URL httt::://ppoccedngg...eeeee newripa. htppppp....ppp....ppperr.rrr fillss/aaper/022//E1 SCcC33E6668008B8000ccc4444444443 pdf.
Alon Jacovi and Yoav Goldberg. Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? In Dan Jurafsky, Joyce Chai, Natalic Schluter, and Joel Tetreault (ods.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4198 4205, Online, July 2020. Association for Computational Linguistics. doi: (0.18633/v1/2020. mcln URL https://aclantthology. org. 2020 acl-main.....m...aaa..a 386/
Alon Jacovi and Yoav Goldberg. Aligning faithful interpretations with their social attribution. Transactions of the Association for Computational Linguistics, 9:294-310, 2021. doi: 1 a 00366. URL
https://aclantholloo oerlloooooggoooo.o.o ottg///2........ooo. tac1-1..c.1.......
Sarthak Jain and Byron C. Wallace. Attention is not Explanation. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 3543 3556, Minneapolis, Minnesota, June 2019. Association ion for Computational Linguisties. 1357. URL
doi: 10.18653/v
Han Jiang, Junwen Duan, Zhe Qu, and Jianxin Wang. MARE: Multi-aspect rationale extractor on unsupervised rationale extraction. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proccedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 11734 11745, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1 210633//11/2244.emmpmmmmaa URL
Picter-Jan Kindermans, Kristof Schütt, Klbus-Robert Müller, and Sven Dähne. Investigating the influence of noise and distractors on the interpretation of neural networks. CoRR, abs/1611.07270, 2016. URL
http://arxiv. http://aaxv..orr///////1...11 wttp.///bbabb/b...0777777/
16

---

Published in Transactions on Machine Learning Research (12/2025)
Pieter-Jan Kindermans, Kristof T. Schütt, Maximilian Alber, Klaus-Robert Müller, Dumitru Erhan, Been Kim, and Sven Dähne. Learning how to explain neural networks: Patternnet and patternattribution. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30
May 3, 2018, Conference Track Proceedings. OpenReview.not, 2018. URL httpss://pperreiieeeeeee neet
Torum?2d=Hkm7CCBlT
Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. The (un) reliability of saliency methods. Explainable Al: Interpreting, explaining and visualizing deep learning, pp. 267-280, 2019.
Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Mclníkov, Natalia Kliushkina, Carlos Araya, Siqi Yan, and Orion Reblitz-Richarssnn. Captum: A unified and generic model interpretability library for pytorch. CoRR, abs/2009.07596, 2020. URL
https://arxiv. wwwg/ag/////oooee..oo.eee 200900009900066
Isaac Lage, Emily Chen, Jeffrey He, Menaka Narayanan, Been Kim, Sam Gershman, and Finale Doshi- Velez. An evaluation of the iuman-interprettablity of explanation. CoRR, abs/1902.00006, 2019. URL
http://arxiv.org//
Sebastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grégoire Montavon, Wojeiech Samek, and Klaus-Robert Müller. Unmasking clever hans predictors and assessing what machines really learn. Nature communications, 10(1):1096, 2019.
Tao Lei, Regina Barzilay, and Tommi Jaakkola. Rationalizing neural predictions. In Jian Su, Kevin Duh, and Xavier Carreras (eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 107 107-117, Austin, Texas, November 2016. Association for Computational Linguistics. doi: (0.18653/v1/D16 1011. URL
Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. On the sentence embeddings from pre-trained language models. In Bonnic Webber, Trevor Cohn, Yulan Hc, and Yang Liu (eds.). Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9119 9130, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1 http://///2200.mmmmmmmmm..... URL https://acclantaaooggg..or htt.:///g...rr/220220mmm11ma1a1a emnlp-main. 733/
Jiwei Li, Will Monroe, and Dan Jurafsky. Understanding neural networks through representation erasure. CoRR, abs/ abs/1612.0022D) 2016. URL. http://aaxxiv.org/bbsat 00220000000012022002002
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zeutlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR,
2019. URL http: http://aarri..rg//aa///000. 11692 11692
Scott M Lundberg and Su-In Lee. A unified approach interpreting model predictions. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett (eds.), Aduances in Neural Information Processing Syslems, volume 30, pp. 1-10. Curran Associates, Inc., 2017. URL https::/ppooceeddngg mewrridiinnnnnnuiis...... cc/paper files//paper///1////ee 62206620996622216444466667776-13
Qing Lyu, Marianna Apidianaki, and Chris Callison-Burch. Towards faithful model explanation in NLP: A survey. Computational Linguistics, 50(2):657-723, June 2024. doi: 10.1162/coli a 00511. URL
http:://accannhooog..rrroor
c1-2.6/
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Polls. Learning word vectors for sentiment analysis. In Dekang Lin, Yuji Matsumoto, and Rada Mihalcea (eds.), Proceedings of the 49th Annual Meeting of the Association for Computational Linguisties: Human Language Technologies, pp. 142- 140- Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL httpp:://aclaattaooooooooo/// wtt.:poooo..oo.1..11..151111
17

---

Published in Transactions on Machine Learning Research (12/2025)
Andreas Madsen, Sarath Chandar, and Siva Reddy. Are self-explanations from large language models faithful? In Lum-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 295 337, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1, 1002...innnns--...... URL https://aclanthaae .ogg//gggooooo/o////oooooo 2024. f wwwdingnnn-sll....
Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. Hatexplain: A benchmark dataset for explainable hate speech detection. In Proceedings of the AAAT conference on artificial intelligence, volume 35, pp. 14867 14867-14875, 2021.
Shervin Minace, Tomás Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. Large language models: A survey. CoRR,, abs/2402. 2222220001900 2024. doi: 10. 00.48550/ARXII0 2402.06196. URL https: org/10 48550/ arXiv. 2402. 06196
Ali Modarressi, Mohsen Fayyaz, Yadollah Yaghoobzadch, and Mohammad Taher Pilchvar. GlobEnc: Quanti- fying global token attribution by incorporating the whole encoder layer in transformers. In Marine Carpuat, de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, pp. 258-271, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v1 1 m0022..acclmmmin. URL hattps://aclanteee
wwt.gyy..gg//2222// www.l-main.nnn.mmmmmmmmm 19/
Ali Modarressi, Mohsen Fayyaz, Ehsan Aghazadeh, Yadollah Yaghoobzadeh, and Mohammad Taher Pile- hvar. DecompX: Explaining transformers decisions by propagating token decomposition. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Popers), pp. 1649-2664 Toronto, Canada, July 2023. Association for Computational Linguisties. doi: 10. wt..1865/v1122222 acl-lon... Mooo....9 URL https://aclanthology/ org.
www.:/g..........ee 2023 acl-2aaaccllaee acl-long. 149/
Pooya Moradi, Nishant Kambhatla, and Anoop Sarkar. Training with adversaries to improve faithfulness of attention in neural machine translation. In Boaz Shmueli and Yin Jou Huang (eds.), Proceedings of the 1st Conference of the Asia-Pacific Chapler of the Association for Compulational Linguisties and the 10th International Joint Conference on Natural Language Processing: Student Research Workshop, pp. 93-100, Suzhou, China, December 2020. Association for Computational Linguistics. doi: 10.11833/1122022 aacl-srw.14 URL https://aclanthoology/ wwwp:/20..ooooo.. http:22020.aaallsaaeeaa aacl- srwl.sssw... 14/
Pooya Moradi, Nishant Kambhatla, and Anoop Sarkar. Mcasuring and improving faithfulness of attention in neural machine translation. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 2791-2802, Online, April 2021. Association for Computational Linguisties. doi: 10.18653/v1/ URL 2021.eacl- mandd 2433.
www.aaaooooooooo//....o.oo erc/2221.eacllmminn 2243
Vinod Nair and Geoffrey E. Hinton. Rectified lincar units improve restricted boltzmann machines. In Johannes Fürnkranz and Thorsten Joachims (eds.), Proceedings of the 27th International Conference on Machine Learning (ICML-10), June 21-24. 2010, Haifa, Israel, pp. 807 8144 Omnipress, 2010. URL
OpenAl. GPT-4 technical report. CoRR, abs/ aabs/2303.087744, 2023. doi: 10.48550/ARXII 000550//AXXIV2280800777.. 2303.08774. URL https://aoi.orr//1 48550/arxxxx. 2377
Nina Poorner, Hinrich Schütze, and Benjamin Roth. Evaluating neural network explanation methods using hybrid documents and morphosyntactie agreement. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 340 350, Melbourne, Australia, July 2018 Association for Computational Linguistics. doi: 10.18653/v1, ///118 1032. URL https://aclanthoology www.://w.....-/--///// 1032/
Danish Pruthi, Mansi Gupta, Bhuwan Dhingra, Graham Neubig, and Zachary C. Lipton. Learning to deccive with batee explanations. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4782-4488 4793,
18

---

Published in Transactions on Machine Learning Research (12/2025)
Online, July 2020. Association for Computational Linguistics. doi: 10.18653/11/2000.accmmm URL https://aclanthoolgyy .ewgg/ogg//// wwg.g020..0......o acl minelmmann.444
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748 8763. PMLR, 2021.
Naveen Janaki Raman, Mateo Espinosa Zarlenga, and Mateja Jamnik. Understanding inter-concept relation- ships in concept-based models. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 42009-42025. PMLR, 21 27 Jul 2024. URL https: http:////cccceess.cooo wet.pmmmmppppeeeee www.ppreessSS he////235///eeee 2//35/rama24a.... htmm
Sahana Ramnath, Brihi Joshi, Skyler Hallinan, Ximing Lu, Liunian Harold Li, Aaron Chan, Jack Hessel, Yejin Choi, and Xiang Ren. Tailoring rationaatisl with multi-reward distillation. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024.
OpenReview.net, 2024. URL httpp:://ppenrevie: httpps//ppppooooeeee.eoooooooooa pot//orrumid
Sukrut. Rao, Moritz Böhle, Amin Parchami--raghi, and Bernt Schiele. Studying how to efficiently and effectively guide models with explanations. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 1922 1933, October 2023.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 'why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144, 2016.
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence, 155)2206 215, 2019.
Punyajoy Saha, Divyanshu Sheth, Kushal Kedia, Binny Mathew, and Animesh Mukherjee. Rationale-guided few-shot classification to detect abusive language. In Kobi Gal, Ann Nowé, Grzegorz J. Nalepa, Roy Fairstcin, and Roxana Radulescu (ods.), ECAI 2023 26th European Conference Artificial Intelligence, Seplember 30- Oclober 4, 2023, Kraków, Poland Including 12th Conference on Presligious Applicalions of Intelligent Systems (PAIS 2023), volume 372 of Frontiers in Artificial Intelligence and Applications, pp. 2041 2048. IOS Press, 2023. doi: 10.3233//AAAA234497 URL https: /tttp::/ddo.oorg/1..o 00g/10..3333//1122345993
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01100, 2019. URL http:://axiiii wwwpexiii..oo//bbae wrgg/bs///9/
01108
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International conference on machine learning, pp. 3145 3153. PMIR, 2017.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference on Learning Representaatins, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Workshop Track Proceedings, 2014. URL http:::/aaxivv.rrgg.e wttp://..rr///ab/1....00000
Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju. Fooling lime and shap: Adversarial attacks on post hoc explanation methods. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pp. 180-186, 2020.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda B. Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. CoRR, abs/1700.003// 2017. URL htttp::/aardv..orgbbbb/////
19

---

Qianli Wang, Tatiana Anikina, Nils Feldhus, Simon Ostermann, Sebastian Möller, and Vera Schmitt. Cross- refine: Improving natural language explanation generation by learning in tandem. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert (eds.), Proceedings of the 31st International Conference on Computational Linguistics, pp. 1150-1167, Abu Dhabi, org/2025 Published in Transactions on Machine Learning Research (12/2025)
Jost Tobias Springenberg. Alexey Dosovitskiy, Thomas Brox, and Martin A. Riedmiller. Striving for simplicity: The all convolutional net. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Hepresentations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Workshop Track Proceedings, 2015. URL htttp://arxiv.ort ab////a2../////////////////////a 1412.681000
Suraj Srinivas and François Fleuret. Full- fill-gradient representation for neural network visualization. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alchh-BBu,, F. Fox, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https: pttpps://proceedingg Paper .dd
Erik Strumbelj and Igor Kononenko. An efficient explanation of individual classifications using game thoory.
The Journal of Machine Leurning Research, 11:1-18, 2010.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 3319-3328. PMLR, 06-11 Aug 2017. URL https://procceddngs. mlr. /ttps///////uaaaaraaaaaaaaaa html
Martin Tutek and Jan Snajder. Toward practical usage of the attention mechanism as a tool for interpretability. IEFE access, 10:47011-47034 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998 6008, 2017. URL https://proceeddngg. ien neurips www.aaat-bssatt.....a html
cc/paper/200 2aapper/217/aaah//eeeee
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Tal Linzen, Grzegorz Chrupat a, and Afra Alishahi (eds.), Proceedings of the 2018 EMNLP Workshop BlackborNLP: Analyzing and Interpreting Newral Networks for NLP, pp. 353 355, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10.18653/v1, 1111//11-54540 URL httpp:://clantth///
UAE, January 2025a. Association for Computational Linguistics. URL 77/
Shengjie Wang, Tianyi Zhou, and Jeff Bilmes. Bias also matters: Bias attribution for deep neural network explanation. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 6659 6667. PMLR, 09-15 Jun 2019. URL httpp::/ www.mmrrrrsrss... ppress/v97/wang11ppp htmll
Yifan Wang and Vera Demberg. A parameter-officient multi-objectivve approach to mitigate stereotypical bias in language models. In Agnieszka Faleńska, Christine Basta, Marta Costa-jussà, Seraphina Coldfarb-Tarrant and Debora Nozza (eds.), Proceedings of the 5th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pp. 1-19, Bangkok, Thailand, August 2024a. Association for Computational Linguistics. doi: URL
Yifan Wang and Vera Demberg. RSA-control: A pragmalics-groundee lightweight controllable text generation framework. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), ), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 5561-5582, Miami, Florida, USA, November 2024b. Association for Computational Linguistics. doi: 10.18653/ 2111663//1/2224...0 2024..mmnlp-main.3 URL http:://acllathhoooy..or///
maanp--main... 318/
20

---

Published in Transactions on Machine Learning Research (12/2025)
Yifan Wang, Mayank Jobanputra, Ji-Ung Lee, Soyoung Oh, Isabel Valera, and Vera Demberg. Bridging fairness and explainability: Can input-based explanations promote fairness in hate speech detection? CoRR, abs/2509. 22291, 2025b. doi: 10.48550/ 10.44550/AARIV.... (TKXIV..509.2220 arXiv. 2509. 22291
URL https://doi.
Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and Tacopo Poli. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuming and inference. CoRR, (2s1/244..11 13663, 2024. doi: 10.48550/ /A////XXXIV...... URL https://doi. /ttt:s::/doi.orr//.... 48550/arXiv. 2412.13663
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R Bowman. Blimp: The benchmark of linguistic minimal pairs for english. Transactions of the Association for Computational Linguistics, 8:377 392, 2020.
Yeo Wei Jie, Ranjan Satapathy, and Erik Cambria. Plausible extractive rationalization through semi- supervised entailment signal. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 5182-50 5192, Bangkok, Thailand, August 2024. Association for Computational Linguisties. doi: 10.18653/ /tt///222...nnss-a/m. //aclanthology.orge http/2224.ffiiee findings-acl.3077 307/
URL https:
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Picrric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pp- 38-45, Online, October 2020. Association for Computational Linguistics. URL https://wwe acwweb..leb.... wrgg/anthologgg/200000 emm1p-mmmllppmm.mm eww.p-demos.pppe pemos.o.. 6.
Mengyu Ye, Tatsuki Kuribayashi, Goro Kobayashi, and Jun Suzuki. Can input attributions explain inductive
reasoning in in-context learning? In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Moham- mad Taher Pilehvar (eds.), Findings of the Association for Computational Linguistics: ACL 2025, pp. 21199- 21225, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8 891766256-5 doi: 10. 10.18653/v1 222////vv222.....
URL https://aclatthoooogy htt.://wwwwoooooooor/o.//....... wtt.222.....nndggn findingnnn----. acl. 1092/
Kayo Yin and Graham Neubig, Interpreting language models with contrastive explanations. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp- 184-198, Abu Dhabi, United Arab Fmirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1122222.mmm//. wmn.p- 14. URL https: \/aclanthology.o org/2022.
14/
Xuemin Yu, Fahim Dalvi, Nadir Durrani, Marzia Nouri, and Hassan Sajjad. Latent concept-based explanation of NLP models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 12435 12459, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1 /2222/22m..mmmmmate 6622. URL https://aclanthoot .rg org.oogg///02 emnlp-main. 692.mmninn..nnn/..m.
Linan Yue, Qi Liu, Yichao Du, Yanqing An, Li Wang, and Enhong Chen. Dare: Disentanglement- augmented rationale extraction. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and
A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 26603 26617. Cur- ran Associates, Inc., 2022. URL
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character- convolutional networks for text classification. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (ods.), Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL /ttpap//////////oedddigggg
wwa.ggs.euuii...o cc/
21

---

Published in Transactions on Machine Learning Research (12/2025)
Hang Zhao, Qile P. Chen, Yijing Barry Zhang, and Gang Yang. Advancing single- and multi-task text classification through large language model fine-tuning. CoRR, abs/2412.085877, 2024. doi: 10.48550/ ARXIV.24122 2412182212.08888.. URL https://doi / 48550/arxiv. 2412.08587
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nic, and Ji-Rong Wen. A survey of large language models. CoRR, abs/2303. 22203.18222 2023. doi: 0.48550/ARXIV/ 200.3222223.32 18223. URL https: 7/doi org/ 11..1...8555 00..4485////// arriv. 2303. 18223
Yiming Zheng, Screna Booth, Julic Shah, and Yilun Zhou. The irrationality of neural rationale models. In Apurv Verma, Yada Pruksachat Kai- Wei Chang, Aram Galstyan, Jwala Dhamala, and Yang Trista Cao (eds.), Proceedings of the 2nd Workshop on Trustworthy Natural Language Processing (TrustNLP 2022), pp. 64-73, Seattle, U.S.A., July 2022. Association for Computational Linguistics. doi: 10. 10.18653/v1/ 2022.trustnlp- URL
wwwp:atthooooooooooooooooo 2022. trustnlp-1.6/
A Terminology
To ensure clarity, we define key terms used in this work as follows:
Faithfulness The extent to which an explanation accurately reflects the model's actual reasoning process (Jacovi & Goldberg 2020). A faithful explanation should directly correspond to the internal mechanisms that led to the model's prediction.
Human Interpretability The ease with which a person can understand the model's reasoning from the explanation (Lage et al. 2019). A highly interpretable explanation should be clear, concise, and focused on relevant information while avoiding unnccessary or distracting information. However, an explanation that is easy for humans to interpret may not necessarily reflect the model's actual reasoning process or align with human reasoning patterns.
Human Agreement The dogrec to which a model's explanation aligns with the reasoning a human would use for the same prediction. A high-agreement explanation should follow intuitive, logical reasoning patterns similar human decision-making.
Explainability The extent to which a model's computations can be faithfully explained and its learned patterns understandable to humans. A highly explainable model should yield explanations that are both faithful to its actual reasoning process and interpretable to humans.
B
Notation
In this paper, use lowercase letters for scalars (e.g,, b), bold lowercase letters for vectors (e.g., w, x), and bold uppercase letters (W) for matrices.A A special case is the alignment pressure parameter, denoted by the non-bold uppercase letter B, to distinguish it from the bias term in lincar layers. We use bold uppercase letters X and A to denote a sequence of model inputs or hidden state activations. In $ 3 we use x to denote the input when a function is applied to each element of the input sequence separately. contrast, we use X or A when the function involves interactions between elements, such as in the attention mechanism.
C Dynamic Linear Representation of Model Components
Here describe how each model component in transformers can function be converted to bias-free, dynamic linear module in B-cos LMs.
B-cos Layers B-cos layers are designed as bias-frce, dynamic linear modules with a dynamic linear weight matrix W(x) = cxos(x, W @ W. Here, & scales the rows of the matrix W its right by the scalar entries of the vector to its left.
22

---

Published in Transactions on Machine Learning Research (12/2025)
Non-linear Activation Functions In transformer models, non-linearity is typically introduced using (approximately) piecewise linear activation functions, such as ReLU (Nair & Hinton 2010) and GELU [Hendrycks & Gimpel 2016). These functions can be easily interpreted as linear transformations with input-dependent weights. For example, GELU(x) = x (0.5 + 0.5 x erf(x/v2) can be interpreted as lincar transformation where the second lerm acts as the dynamic linear weight.
Attention Blocks Böhle et al. (2024) showed that attention computations can be seamlessly integrated into B-cos networks as dynamic linear module:
Att Q.K. V) = softmax X T Q2T - w(x)X
(4)
For multi-head self-attention (MSA), the output can be viewed as the concatenation of the outputs from H attention heads, followed by linear projection with matrix U:
MSA(X)= = U (X)X, (X)X)
(5)
Since this operation maintains a dynamic linear structure, the multi-head attention block remains a dynamic linear module.
Normalization Layers Following Böhle al. 2024 we apply bias-free normalization layers ensure completeness of explanations:
Axrmm(x,23) x37))
x
x
(6)
/vvar (X)
where X' represents a batch or sequence of inputs and * is the dimension along which the mean ()) and variance var(.) are computed (e.g., across the batch or layer). Unlike standard normalization, this variant omits the bias term in the affine transformation to preserve explanation completeness. If a running mcan estimate is used during inference, the centering term (X), is also removed. This yields bias-free, dynamic
linear transformation with weight vvrr," (X)
D Implementation Details
Fine-tuning Setups For all PLMs used in the experiments, we use the uncased base version from huggingface (Wolf et al., (020). For both conventional models and B-cos LMs, we train them for epochs with 10% linear warm-up steps on the downstream task datasets. The learning rates are set to 2e-5 for IMDB and HateXplain, and 3e-5 for AG News. All models use batch size of 16 and maximum sequence length of 512. For validation, randomly sample half of the test set from IMDB and AG News.
Baselines For IxG and ShapSampl, we use the Captum (Kokhlikyan et al. 2020) implementations We implement the Attention method ourselves, and LIME is sourced from the lit library For Decomp and I we use their official implementations with default configurations. The number of samples is set to for ShapSampl and 3,000 for LIME, with [MASK] as the baseline token. For all explanation methods at the attributed to the combined sum of word, position, and token type
embedding level, model predictions embeddings (if applicable). In the main experiments, we compute token attribution scores by summing over all embedding dinensions, as this approach demonstrates betler faithfulness results than using the L2 norm.
For Saloss models, we use the official codebase with default hyperparameters to train BERT and RoBERTa on AG News, IMDB, and HatcXplain. DistilBERT is not included, as is not supported by the codebase.
In Section 7 we follow Ferrando et al. 2023 to generate contrastive explanations that highlight why the models predicts the target token instead of the foil token. For Occlusion explanations, we use the [PAD]
hatpps:/captuumaa//aiii
http:://gghhh..cc///oaaaaasyyypp
http:///ghhuu.ca///ehhaagaaaaaas interpret
23

---

Published in Transactions on Machine Learning Research (12/2025)
token to perform occlusion, instead of a zero vector done in Yin & Neubig (2022) and Ferrando et al. (2023). Using zero vectors distorts the input distribution and, in generative settings, can influence predictions differently depending on position. To avoid such positional effects, we instead occlude using the in-distribution embedding of the [PAD] token.
SCqPG Examples When constructing examples for SoqPG, we set the sequence length to 50 for AG News, 256 for IMDB, and 25 for HateXplain, aligning with their median lengths. Ouly examples longer than these thresholds are selected, and they are truncated to construct synthetic examples. Additionally, we only use examples that are correctly predicted with minimum confidence of 75% after truncation. For fair comparison, we evaluate Saloss models and B-cos LMs on the same sets of examples constructed based on the predictions of the corresponding conventional models.
Automatic Evaluation Setups For task performance evaluation, we use the complete test set for cach task. For faithfulness evaluation, we conduct perturbation-based evaluations on 2000 test examples and SeqPG on 500 test examples for AG News and IMDB. For HateXplain, we use the full test set for perturbation- based evaluation (1,924 examples) and construct 269, 310, and 308 SeqPG examples from it using BERT, DistilBERT, and RoBERTa, respectively. In the perturbation- based evaluation, the [CLS] token is never perturbed because it is used directly to make predictions.
B-cos Pre-training For B-cos pre-training in $ we set B=1.25 and further pre-train the model on 25,000 sentences from the Wikipedia datase using masked language modeling loss with learning rate of 1e-4 and a 15% masking ratio. We do not B-cosify the language head, as its parameters are tied with the embedding layer. Pre-training uses the standard cross-entropy loss rather than binary cross-entropy loss, since the unnormalized language head weights could otherwise grow arbitrarily large to minimize the loss.
Decoder-only Models B-cosification We use the GPT-2 small and Llama-3.2- models from huggingface. As with the encoder- basod models, we do not B-cosify the language head and use cross-cntropy loss for GPT-2 and Llama-3.2 training. B-cos adaptations are first applied and both models are then trained on 500,000 and 4,000,000 sentences, respectively, from the OpenWebText dataset for one epoch, using a learning rate of 5e-4. For GPT-2, we use batch size of 16 and a maximum sequence length of 512; for Tlama-3.2, we use a batch size of 128 and a sequence length of 1024. Perplexity is evaluated on a held-out OpenWeb Text subset. of 10,000 sentences using a maximum sequence length of 512.
BLiMP Subsets We follow Ferrando et al. 2023 to use the following nine BLiMP subsets with cor- responding IDs. aga: anaphor gender agreement; ana: anaphor number agreement; asp: animate subject. passive; dna: determiner noun agreement 1; dnai: determiner noun agreement irregular 1; dnaa: determiner noun agreement with adj 1; dnaai: determiner noun agreement with adj irregular 1; npi: npi present 1; darn: distractor agreement relational noun. Examples of these datasets and the IOI dataset can be found in Table 7
Dataset
ID
Example
Anaphor gender agreement Anaphor mumber agroement Animate subject passive Determiner noun agreement
aga
Katherine can't help herself himself.
Susan revealed herself / themsslves.
asp
Amanda respected by some waitresses / pictures.
dna
Raymond sclling this skoteh / sketches.
Determiner noun agrocment irregular Determiner noun agreement with adjectives
dnai
Adam hadn't discussod these analyses / analysis.
duaa
Rebeeca was criticizing those good documentaries / documentary.
Determiner noun agreement with adjectives irregular NPI present
dnaai
Some waiters broke this lost foot / feet.
upi
Even Suzanne has really / ever joked around.
Distractor agreement relational noun
dam
A niece most senators hasn't / haven't descended most slopes.
Friends Juana and Kristi found a mango at the bar.
Indirect Objoct Identification
101 Kristi gave it to Juana/ / Kristi,
Table 7: Examples from the BLiMP and 1OI datasets. Green (red) indicates target (foil) predictions. Ground truth evidence for the correct continuations is underlined.
24

---

Published in Transactions on Machine Learning Research (12/2025)
Label: Scú/tech
could
vince very
system shaped by
say space
space
[SEP]
Target Sc//Tcch (CLS)
prove plus nots roported
Vince canter very surprises me given [SEP)
solar system shaped by brush astronomers
say space
com
space
(SEP)
Figure An example SeqPG from AG News (using B-cos BRRT), Green (red) indicates the positive (negative) impact of tokens on the prodiction. The example consists of two sequences with different labels (Sports and Sci/tech), separated by the [SEP] token after the first sequence. Explanations are generated for each label, and the proportion of correctly attributed positive tokens is averaged across both labels to compute the SeqPG score for this example.
Compute Infrastructure Unless stated otherwise, all experiments are conducted on single NVIDIA H100 GPU. Training one epoch of B-cos BERT takes approximately 40 minutes on AG News, 10 minutes on IMDB, and minutes on HateXplain.
E
Explanation Efficiency
Beyond improved faithfulness and human interpretability, B- cos explanations are also efficient lo extract. Comparing their computational costs with strong post-hoc methods shows that B-cos explanations
Method
Time (s) Memory (GB)
ShapSampl
37.22
21.95
LIME
6.82
21.96
the most efficient in both time and memory usage (Table D). Post-hoc and B-cos explanations generated from the conventionally fine-tuned and B-cos BERT models on LMDB, respectively.
SIG
67.46
29.09
DecompX
0.76
48.38
B-cos
0.08
2.82
Table 8: Computational costs per example of generating explanations for 100 instances using an NVIDIA H100 GPU (batch size 1). B-cos explanations (bold) are at least 9x faster and require at most of VRAM.
F SeqPG Example
Figure presents a SeqPG example from AG News using B-cos BERT. For better visualization, each segment is truncated to
Lokens here instead of 50 used in the experiments. Unlike
the hybrid document evaluation proposed by Poerner et al. (2018), our approach explicitly controls segment length and position ensure fair comparison. Additionally, we measure the proportion of correctly assigned positive attributions rather than relying solely the highest attribution value.
G Task Performance of Other B-cos LMs
100
95
94.59
93.99
93.38
91.43
Conv. DistilBERT B-cos DistilBERT
90
Accuracy (%)
85
80
78.19
78.14
75
70
65
60
AG news
IMDB
HateXplain
Figure& 8: Mean accuracy of conventionally fine-tuned and B-cos DistilBERT models averaged over three runs. B-cos models perform comparably to conventional models on most tasks.
25

---

Published in Transactions on Machine Learning Research (12/2025)
94.78
93.88
95.51
94.02
Conv. RoBERTa
95
91.35
88.95
Saloss RoBERTa
Accuracy (%)
90
B-cos RoBERTa
85
79.63 78.05 78.41
80
75
70
65
60
AG news
IMDB
HateXplain
Figure 9: Mean accuracy of conventionally fncetuned and B-cos RoBERTa models averaged over three runs. B-cos models perform comparably to conventional models on most tasks.
Figures g and g illustrate the task performance of conventional and B-cos DistilBERT and RoBERTa across
datasets. Consistent with findings from BERT models (ef. Figure 2. B-cos LMs exhibit strong performance comparable conventionally fine-tuned models.
H Faithfulness Evaluation of Other B-cos LMs
Tables and present the faithfulness ovaluation results for DistilBERT and RoBERTa. The findings
are consistent with our main experiments (cf. Table 2), confirming that B-cos LMs produce more faithful
explanations compared to post-hoc explanation methods.
Model
Method
AG News
IMDB
HateXplain
Comp (f)
Suff (1)
SegPG (1)
Comp (?)
Suff (4)
SegPG (t)
Comp (t)
Suif (1)
SeqPG (*)
Atlention
26.36
5.37
50
31.62
10.46
50
30.56
14.67
50
IxG
19.29
6.21
53.71
23.78
12.38
49.23
25.13
18.08
46.60
Conv. DistiBBERT
30.78
1.63
67.87
47.16
5.45
60.66
41.11
4.23
58.55
DecompXX
ShapSampl
52.56
-0.56
82.61
63.29
2.91
70.27
48.73
0.87
61.41
LIME
52.59
-0.56
77.64
58.6
5.12
61.11
31.61
12.94
56.49
D-cos DistilBERT
B-cos
61.93
-1.01
86.78
76.26
-1.28
72.68
57.2
-4.49
74.89
Table Faithfulness evaluation for conventionally fine-tuned DistilBERT and B-cos DistilBERT across three datasets. The best results in bold. We find that B-cos explanations consistently more faithful than post-hoc explanations from both models.
Comparison to Rationale-Based Models
We compare B-cos LMs to one rationale-based, explain-then-predict BERT model, RGFS-SA (aha et al. on HateXplain. This model leverages human rationales as additional supervision during training. As shown in Table [ although the RGFS-SA model brings improvement over the conventional BERT model, it generates considerably less faithful rationales compared to B-cos explanations.
2023 10
J Human Evaluation Details
In the human study, we select only examples shorter than 25 tokens for HateXplain and 40 tokens for AG News to improve visualization. Additionally, we replace [CLS] and [SEP] with ## to make the examples more understandable for lay users. Below, we provide the instructions along with detailed description of the criteria and scoring used in our human evaluation. In our human study, 92% of AG News examples and 80%
htttps:://mugiinggacc co/Hate- httpa//eweppppaeeeeeaaaeaaaaaaee otap://aatttnllleeerrattttooto
26

---

Published in Transactions on Machine Learning Research (12/2025)
AG News
IMDB
HateXplan
Model
Method
Comp (1)
Suff (4)
SeqPG (f)
Comp (1)
Suff (4)
SeqPG (t)
Comp (1)
Suif (1)
SeqPG (f)
Attention
22.17
3.80
50
25.26
5.84
50
32.94
7.52
50
IGG
11.33
7.51
11.15
16.15
11.53
47.20
21.10
15.16
50.59
SIG
19.64
1.63
66.43
38.14
2.13
59.04
44.21
-1.42
66.
Couv. RoBERTa
DecompX
50,00
-0.84
90.38
49.24
0.65
72.80
46.94
-1.42
70.16
ShapSompl
35.63
-0.68
78.31
43.32
1.83
65.85
44.83
-1.30
67.
LIME
19.28
2.85
66.73
21.07
8.32
50.81
27.97
11.38
58.53
Attention
40.69
2:77
50
24.51
4.33
50
47.04
7,83
50
IxG
6.19
27.30
52..46
10.98
12.30
47.92
22.78
25.78
49.49
SIG
27.22
56.84
11.53
13.76
62.10
43.77
5.02
58.67
Saloes RoBERTa
DecompX
61.46
0.16
74.20
0.10
74.41
2.47
65.63
ShapSampl
34.48
64.67
48
63.04
64.53
LIME
15.93
5.03
55.17
18.01
6.47
50.94
29.62
15.78
56.00
B-cus RoBERTa
B-cos
86.63
74.05
74.01
Table 10: Faithfulness evaluation for conventionally fine-tuned RoBERTa, Saloss RoBERTa and B-cos RoBERTa across three datascts. The best results are in bold. We find that B-cos explanations are consistently more faithful than post-hoe explanations from both models.
Model
Method
Accuracy (t)
Comp (t)
Suff ())
SeqPG (1)
Conv. BERT
Attention 80.77
22.64
13.83
50
RGFS-SA BERT
Rationale
80.09
36.11
16.54
50
B-cos BERT
B-cos
78.64
59.66
-4.89
77.57
Table 11: Performance of conventional, RGFS-SA and B-cos (B-1.5) BERT models HateXplain. SeqPG is consistently 50 for rationale-based models, as their explanations are class-agnostic, similar to attention. The rationale-based RGFS-SA model generates less faithful explanations than B-cos BERT.
of HateXplain examples contain correct model predictions; in the remaining cases, explanations are supposed to support the wrong predictions.
WARNING: SOME CONTENT IN THIS QUESTIONNAIRE IS HIGHLY OFFENSIVE.
Prerequisites:
Proficiency in English is required for this evaluation task. If you do not meet this criterion, please do not proceed.
We invite you to review 100 examples where LMs perform classification tasks and provide explanations for their predictions.
The first 50 examples come from a hate speech detection task, where the model predicts whether a text toxic or not. toxic.
The last 50 examples come from topic classification task, where the model categorizes text into one of four topics: sports, world, business, or sci/tech.
For each example:
The model's prediction is shown along with four explanations justifying the prediction.
The order of the explanations is randomized prevent bias.
Words highlighted in green indicate words that had positive influence on the prediction, while words in red indicate words that had negative influence. The intensity of the color reflects the strength of the impact.
Important: The model's prediction may be incorrect. Your task to evaluate the explanations based on how well they support the modcl's prediction, not the true labels.
Evaluation Task:
After reviewing cach example, please rate the the human interpretability and human agreement of the four explanations on scale of 5. Refer the definitions and rating scales provided below when making your assessments.
27

---

Published in Transactions on Machine Learning Research (12/2025)
Reasoning Rating Explanation
Model prediction: Not Toxic
incorrect positive evidence for Not Toxic, -human agreement
Explanation 2: ## just had
kabitch to the jeep ##
Explanation ## juct bad
labitch the icen #ift
focus on relevant tokens, +human interpretability
Explanation Just nad
Kabitcn to the Jeep
Figure 10: An example shown to participants that demonstrates how to rate explanations.
Human Interpretability: How easily person understand the model's reasoning based on the explanation. A highly interpretable explanation should be clear and easy to follow, focus on relevant words and avoid unnecessary or distracting details.
1. Not Interpretable: The explanation is unclear, noisy, or provides no meaningful insight.
2. Slightly Interpretable: Some clues are present, but the explanation is too sparse, irrelevnt, or confusing.
3. Moderately Interpretable: The explanation contains useful information but is cluttered with noise irrelevant details.
4. Highly Interpretable: The explanation is mostly clear, with minimal irrelevant highlights.
5. Completely Interpretable:
The explanation fully transparent, highlighting only the most relevant words, making the model's reasoning fully clear.
Human Agreement:
How closely the model's explanation aligns with the reasoning human would use for the same prodiction. A high-agreement explanation should follow logical, intuitive reasoning and align with typical human decision-making patterns.
1. No Agreement: The explanation contradicts human reasoning or lacks logic.
2. Low Agreement: The explanation bears some resemblance to human reasoning but includes major inconsistencies.
3. Moderate Agreement: The explanation partially aligns with human reasoning, yet
contains notable differences.
4. High Agreement: The explanation largely aligns with human reasoning, showing only minor discrepancies.
5. Complete Agreement: The explanation fully matches human reasoning, following logical and intuitive path that human would naturally use.
We also provide participants with examples to illustrate the reasoning behind rating explanations. One such example is shown in Figure I0 Additionally, Figure I1 presents an example of model prediction and its explanations as displayed to participants during the study.
K More Examples of B-cos Explanations
We provide two more examples of B-cos and other (post-hoc) explanations from AG News in Figure 12 Consistent with our findings in $ 4.4 B-cos LMs provide more human interpretable explanations.
Figure 1 presents two examples where B-cos explanations do not align with human expectations. As B-cos explanations faithfully summarize model computations, they may also reveal spurious correlations from the
28

---

Published in Transactions on Machine Learning Research (12/2025)
Words highlighted in green indicate words that had positive influence the prediction, while words red indicate words that had negative influence. The intensity of the color reflects the strength of the impact.
## symbols mark the beginning and end of the text.
Possible classes:
Toxic: The text contains language that is offensive, derogatory, or harmful toward individuals or groups, including insults, slurs, threats, or dehumenizing statements.
Not Toxic: The text does not contain harmful intent or offensive language, expressing opinions, criticism, or discussions in respectful and non-threatening manner.
Model prediction: Not Toxic
Explanation ##
number
number
people are removing shoes ##
Explanation 2: ##
number
number people are removing shoes #
Explanation 3: ##
number ]
number
people are removing shoes ##
Explanation ## number
number
people are removing shoes ##
Figure 11: An examples of a model prediction and its explanations presented to participants.
training data and occasionally produce unrcasonable rationales. For example, in the first case, the token [Juser]' receives high attribution, likely due to recurring patterns in the data. Another observed error is polarity misinterpretation: in the second case, the token "niggas" incorrectly highlighted as positive evidence for a non-toxic prediction. Similar types of undesired behavior have also been reported for other explanation methods.
L Impact of B on Input- Alignment
To analyze how B-cosification and alignment pressure inftuence the behavior of B-cos LMs, compute the alignment (cosine similarity) between each input and its corresponding weight in B-cos modules across all layers. This analysis is performed on 100 examples from the HateXplain dataset. In Figure 14 we plot different percentiles of inpgh- aligument for conventional and B-cos BERT models with varying B values. For better visualization, we display only the 10th 90th percentiles.
Overall, larger values generally lead to stronger input-weight alignment compared smaller B and conventional models, as evidenced by the curves for B=1.5 and B=2.5 lying above those for the conventional model and B=1. However, the alignment pattern becomes Specifically, at B-2.5, the most aligned input-weight pairs exhibit higher alignment than in other models, but some pairs show very low aligument. This result may arise because certain weights are highly optimized for specific input patterns, leading to poor alignment with others, particularly later layers where input features become more anisotropic (Ethayarajh) 2019 Li al.) 2020 As result, some outputs from the B-cos layers are highly negative. When these outputs are fed into GELU activation functions, their dynamic weights approach zero, making the explanations more sparse.
complex when comparing B=1.5 and B=2.5.
29

---

Published in Transactions on Machine Learning Research (12/2025)
Label: World, Model prediction: World
B-cos
[CLS]
fran deploys new missile tehran iran added one more missile to its military arsenal and the defense minister
saturday country was ready confront any extermal threat [SEP]
ShepSempl [CLS] deploys missile tehran added
missile military arsenal and the defense minister
said saturday his country was ready to confront any external threat SSEP
DecompX
deploys new tehran added more
military arsenal and defense minister
said saturday his country was ready confront external threat [SEP]
Label: Sci/Tech, Model prediction: Sci/Tech
B-cos
cisco and microsoft
2004 cisco systems yesterday announced
customer relationship management crm communications connector for microsofts crm offering [SEP]
ShapSampl [CLS] cisco microsoft partner cmm /
systems yesterday announced new
customer relationship management
communications connector for microsofts offering
[SEP]
DecompX
[CLS] eisco and microsoft partner for cm 24 2004
cisco systems yesterday announced
new
customer relationship management communications
offering [SEP]
Figure 12: More examples of B-cos explanations (B-cos BERT) as well ShapSampl and Decomp. explanations (BERT) from the AG News dataset. Green (red) indicates the positive (negative) impact of tokens on the prediction. As can be seen, the B-cos explanation highlights only relevant tokens and is more interpretable to humans.
Label: Model prediciion Toxle
you are missing essential premise whether any immigrants committed any
[SEP)
Label: Not Taxic, Model prediction: Not Toxic
[CLS] it ain even noon yet and niggas lying [SEP]
Figure 13: Two undesired B-cos explanations from B-cos BERT on HateXplain. Green (red) indicates the positive (negative) impact of tokens the prediction. B-cos explanations can be influenced by spurious correlations learned by the model and may occasionally misattribute the contribution of certain tokens.
M Effects of B on Other Metrics
Table 12 presents the complete results on how B values affect task performance, explanation faithfulness and explanation entropy, as shown in Figure 5 Similar to Comp, SeqPG scores also decline with higher alignment pressure. This could also be attributed to the high sparsity of explanations. As increases, fewer tokens receive attribution scores that are not close to zero, and some SeqPG examples, B-cos LMs may attribute predictions to a single segment. This can lead to numerical instability when computing the positive attribution ratio.
B
1.00
1.25
1.50
1.75
2.00
2.25
2.50
Acc (t)
78.57
79.23
78.10
77.41
77.48
70.44
73.55
Comp (t)
55.09
58.99
59.64
59.23
54.44
35.80
27.11
Suff (U)
-4.25
-5.71
-5.47
-5.84
-6.69
-7.23
-5.47
SeqPG (t)
69.75
77.26
77.79
77.67
76.79
76.68
77.25
Eutropy
3.09
2.79
2.58
2.35
2.28
1.98
1.89
Table 12: Task performance, explanation faithfulness, and explanation entropy of B-cos BERT models HateXplain with different B values. Results averaged over three runs. Similar to Figure 5 task performance and explanation faithfulness peak around B=1.5, while explanation entropy correlates negatively with B.
30

---

potantial liability for Published in Transactions on Machine Learning Research (12/2025)
Layer
Layer 6
0.04
0.04
Conv.
Conv.
0.02
B=1
0.02
B=1
Alignment
B=1.5
Alignment
B=1.5
0.00
2=2.5
0.00
B=2.5
-0.02
00.02
-0.04
-004
-0.66
20
40
60
80
20
40
60
80
Percentile
Percentile
Layer 9
Layer 12
0.04
Conv.
0.04
Conv.
B=1
B=1
Alignment
0.02
B=1.5
Alignment
0.02
B=1.5
0.00
B.2.5
0.00
B.2.5
-0.02
00.0
0.04
-0.04
60
80
20
40
60
Percentile
Percentile
Figure 14: Percentiles of weight alignment in B-cos modules across selected layers of conventional and B-cos BERT models with different values (HateXplain).
Label: Sci/Tech, Model prediction: Sci/Tech
B-1
[CLS] viruses
blame microsoft last year we explored the question microsoft
39
potential
liability for
software flaws exploited by viruses and other forms malware
[SEP]
[CLS)
microsoft year we explored question microsoft potential liability
exploited viruses and
[SEP]
B-2.5 viruses
blame microsoft
last explored the question of microsoft
software flaws exploited viruses and other forms malware
[SEP]
Figure 15: B-cos explanations (B-cos BERT) on AG News with different B values. Green (red) indicates the positive (negative) impact of tokens on the prediction. As B increases, B-oos LMs produce sparser explanations, with fewer tokens receiving significant attribution scores.
N B-cos Explanations with Different Values
Figure 15 illustrates that with increased alignment pressure, B-cos LMs learn fewer but task-relevant features. Consequently, they produce sparser explanations, with fewer tokens receiving significant attribution. This finding aligns with the statistics presented in 6
O Example of Model Bias
In the example shown Figure 16 models become increasingly confident the incorrect prediction as B increases, with attributions primarily assigned the word "blacks". Moreover, simply replacing "blacks" with "whites" results in sharp drop confidence, which demonstrates a growing reliance on spurious correlations with increased alignment pressure. The observation further confirms findings in $6
31

---

Published in Transactions on Machine Learning Research (12/2025)
Label: Not Toxic, Model prediction: Toxic
B=1: Prediction confidence=69.98 Confidence after perturbation=53.91
[CLS] america so racist that they think asians are smarter than blacks and hispanics
B=1.5: Prediction confidence=92.54 Confidence after perturbation=8.93
[CLS] america so racist that they think asians are smarter than blacks and hispanics
2=2.5: Prediction confidence=99.70 Confidence after 6erturbation=6.44
[CLS] america is so racist that they think asians are smarter than blacks and hispanics
Figure 16: Example of how larger values lead B-cos LMs to learn word-level spurious correlations. Green (red) indicates the positive (negative) impact of tokens the prediction. Higher alignment pressure increases the reliance of B-cos LMs on spurious correlations in the data. In this example, perturbation involves changing "blacks" to "whites".
P Decoder-Only Model Results
Table 13 presents the average probability gaps between target and foil predictions every datasct from different vanilla and B-cos models. Table 14 and Table 15 contain MRR results every dataset in BLiMP and TOT for GPT-2 and Llama-3.2 models, respectively.
Probability Gap (t)
Dataset
Vanilla GPT-2
B-cos GPT-2
Vanilla Llama-3.2
B-cos Llama-3.2
aga
0.0120
0.0170
0.0171
0.0196
ana
0.0152
0.0189
0.0140
0.0170
asp
0.0007
0.0008
0.0016
0,0009
dna
0.0011
0.0011
0.0011
0.0017
dnai
0.0021
0.0006
0.0011
0.0013
dnaa
0.0014
0.0012
0.0012
0.0017
dnaai
0.0091
0.0058
0.0077
0.0072
npi
0.0015
0.0002
0.0002
0.0001
darn
0.0067
0.0078
0.0080
0.0085
IOI 0.3351
0.3265
0.4652
0.5021
Table 13: Probability gaps between target and foil next token predictions from vanilla models and B-cos LMs on every datasct.
32

---

Published in Transactions on Machine Learning Research (12/2025)
Dataset Random
Grad Norm
IxG
Occlusion
Logit
ALTI Logit
B-cos GPT-2
aga
0.6875
0.7927
0.7910
0.7513
0.827
0.964
0.8764
ana
0.7056
0.6753
0.7387
0.5957
0.817
0.976
0.7532
asp
0.3818
0.7512
0.4086
0.4374
0.386
0.499
0.4939
dna
0.4608
0.3629
0.3869
0.9030
0.737
0.646
0.9308
dnai
0.4626
0.4077
0.4317
0.8395
0.711
0.637
0.8596
dnaa
0.4103
0.2632
0.3214
0.6557
0.951
0.807
0.7798
dnaai
0.4074
0,2632
0.3392
0.6167
0.9
0.757
0.7601
npi
0.6121
0.7854
0.4948
0.4775
0.445
0.417
0.4573
darn 0.4888
0.6170
0.3627
0.4247
0.802
0.949
0.8936
1OI
0.2360
0.8599
0.1112
0.8517
1.0
1.0
1.0
Table 14: MRR Alignment of different explanation methods GPT-2 small predictions every dataset.
B-cos explanations are extracted from the B-cos GPT-2 model. Logit and ALTI Logit results are duplicated from Ferrando et al. 2023)).
Dataset Random
Grad Norm
IxG
Occlusion
B-cos Llama-3.2
aga
0.6868
0.6030
0.5928
0.7811
0.8485
ana
0.7037
0.5955
0.6432
0.6072
0.8535
asp
0.3842
0.7694
0.4537
0.3670
0.6108
dna
0.4615
0.4598
0.4898
0.8352
0.6743
dnai
0.4630
0.4542
0.5043
0.7671
0.6652
dnaa
0.4112
0.4299
0.4750
0.6115
0.6308
dnaai
0.4075
0.4221
0.4498
0.5758
0.5563
npi
0.6123
0.6367
0.7062
0.5154
0.6264
darn 0.4884
0.5828
0.45787
0.5210
0.8065
TOI 0.2328
0.3637
0.1034
0.4767
0.9913
Table 15: MRR Alignment of different explanation methods on Llama-3.2 predictions every dataset. B-cos explanations are extracted from the B-cos Llama-3.2 model. As Llama models are not supported in Ferrando
et al. 2023), we do not include their results.
33


Reviews:
[{"review_id": "DwWNVdBywY", "reviewer": "Reviewer_PXTs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "## Strengths\n1. This paper first introduces B-cos netowrks to the language model setting, specifically the classification task and generation task, which help improve the language model's explanability with minimal accuracy scrafice.\n2. This paper prevides comprehensive evaluation, including both automatic metrics and human evaluation studies demonstrating improved explanation faithfulness and interpretability compared to post-hoc methods. Also various ablation study setting for B values, and training setups. This provides the community enough insights and refernece in the future developement.\n\n## Weaknesses\n1. The selection of B values for different tasks, as shown in the paper, may vary and it usually takes additional experiements to determine the best value that balance the accuracy and explanability. It's will be good to see whether B-value can be also included as a trainable paramtere to dynamicly select during the adaption process.\n2. For", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yeah, the claims are supported by comprehensive experiments mentioned in section 4, 5, 6, 7, including auto-metrics like comprehensiveness, sufficiency, and human evaluation, case visualzation, and ablations studies.\nFor exmaple:\n1. The authors cliam B-cos LMs produce more faithful explanations than post-hoc methods. The author prove this by results in Table 2, 8, 9\n2. The authors claim that after fine-tuning LM to a B-cos LM, the accuracy is still comparable to conventional methods, this is supported by Figure 2, though there is a little performance drop on IMDB (3 points) compared to the conventional LM method.\n3. The authors also claim that it's more efficient in terms of both time and memory occupation, as specified in Table 11.", "audience_yn": "Yes", "audience_explanation": "This paper presents a way to make langauge model's more explanable and really did a comprehensive experiments to validate the sentitiveness of various hyperparameters like B values. This is proven to be a quite efficient way as specified in Table 11, though requring additional fine-tuning to transforming a LM into a B-cos LM. Therefore, this paper first introduce B-cos way into the language model and can be interesting to those people who focus on the langauge model explanability, like explaination to customers, etc. \n\nHowever, this paper may also be limited in the generation task as all the tasks in the paper are classfication tasks (even for generation models). How does B-cos LM explain those long-form generation tasks is not discussed. In fact, generative LMs may actually self-explain it's response through the long response generated. How to position B-cos LM in generation tasks may be a undetermined topic.", "requested_changes": "1. Although Figure 5 provides detailed ablation study about the B vallue  selection, I think the authors should write a guidance based on the experiments insights about the selection of B-cos LMs under different task setting, so people don't need to do a bunch a ablation studies to determine a good value for them.\n2. Are there any failure cases where B-cos LM make explicitly wrong explanation and this can help people understand better about when B-cos LM not work and better understand its behavior.", "broader_impact_concerns": "As a explanation method, B-cos LM may have a possibility to give wrong explanation and thus mislead the user in understanding the LM's behavior.", "additional_comments": "1. How does the approach scale to larger language models (e.g., models with billions of parameters)?\n2. How does B-cos LM generalize and work for those long-form generation task or they may only work in the classfication task?", "extra_scores": {}, "extra_text": {}}, {"review_id": "nCUaSX8lHP", "reviewer": "Reviewer_hr5c", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper extends the B-cos method to language models by removing bias terms to enhance model interpretability. The proposed approach first applies B-cos architectural modifications to a PLM and then fine-tunes it on downstream tasks. The resulting B-cos LM can generate explanations during inference, showing the contribution of inputs to decisions.\nStrengths:1.Experiments were conducted on both encoder-only models for classification tasks and decoder-only models for generation tasks, demonstrating the method ’s versatility.2. Comprehensive experimental analysis was performed on applying the B-cos method to language models, highlighting the roles of different components.\nWeaknesses:1. Compared to conventionally fine-tuned methods, B-cos shows lower accuracy . Isn ’t accuracy more important than interpretability, especially when the evaluation of interpretability is not yet fully reliable? 2.The classification tasks lack relatively complex test sets, such as arithmetic or reasoning tasks, and the same is true for generation tasks. It would be interesting to see results on more challenging datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Sections 4.2 and 4.3 experimentally verify the advantages of B-cos LM in terms of interpretability. Section 5 compares different conversion strategies, demonstrating the effectiveness of combining task fine-tuning with B-cos conversion. Section 7 validates the method ’s generality on generation tasks with decoder-only models.", "audience_yn": "Yes", "audience_explanation": "The method of applying B-cos to language models in this paper provides a more intuitive and clearer perspective for explaining the model ’s decisions, highlighting the contribution of inputs to the final prediction, and offering meaningful value in terms of model interpretability.", "requested_changes": "1. It is important to know whether the method can achieve similarly good results on more challenging test sets. This approach can intuitively reveal how the model focuses on key information in complex tasks.\n2. The consistent drop in accuracy compared to conventionally fine-tuned methods across different test sets needs a reasonable explanation . Although the faithfulness results are good, isn ’t accuracy more important?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cDmF2Dt3lr", "reviewer": "Reviewer_KuxQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Pre-trained language models achieve strong results in NLP tasks, but they are black-box systems. Post-hoc explanation methods (such as gradients, LIME, or SHAP) often generate unfaithful and hard-to-interpret explanations, making it difficult to trust their reasoning in sensitive domains. B-cos networks remove bias terms and enforce input-weight alignment helping improve explainability. The paper introduces B-cos LLMs, language models modified with the B-cos architecture. This makes explanations inherently faithful rather than post-hoc approximations. The paper presents and examines various strategies for transforming PLMs into task-specific B-cos LMs and shows that combining task fine-tuning and B-cos conversion is the most efficient approach.", "claims_and_evidence_yn": "Yes", "claims_explanation": "+ B-cos LM architecture for NLP that removes the bias term and used dynamic linear modules that yield faithful token-level attributions. The transformer components (attention, normalization, prediction heads) are adapted to bias-free B-cos modules. The linear laters are replaced with B-cos transformations that enforce cosine similarity alignment with input features. Fine-tuning with binary cross-entropy loss strengthens alignment. Explanations are derived directly from the learned input-weight alignment.\n\n+ The paper shows that combining B-cos conversion and task fine-tuning yields faster convergence and better efficiency than prior methods.\n\n+ Demonstrates superior explanation faithfulness and human interpretability across multiple datasets (AG News, IMDB (sentiment), HateXplain (hate speech)).  B-cos explanations outperform all post-hoc methods, with significant gains in faithfulness and human agreement. Ablation studies show that alignment pressure (B > 1) and architectural changes are crucial. Human studies confirm explanations are clearer and more aligned with human reasoning.\n\n+ The extension to decoder-only models (e.g., GPT-2, Llama-3.2) for generation tasks, showing applicability beyond classification. Decoder-only B-cos models match language modeling ability of GPT-2 and Llama-3.2 while producing more faithful explanations.", "audience_yn": "Yes", "audience_explanation": "The audience will be interested in this paper because it tackles one of the most pressing challenges: making powerful pre-trained language models explainable without sacrificing performance. \n\nCurrent explanation methods are mostly post-hoc, often unfaithful to the model’s true reasoning, and difficult for humans to interpret. By introducing B-cos LMs, the authors provide a way to intrinsically embed explainability into the model architecture through bias-free transformations and input-weight alignment, while retaining nearly the same task performance. This approach is practical since it efficiently converts existing PLMs rather than requiring training from scratch, and timely, given increasing regulatory and societal demands for transparent AI. Moreover, the extension of B-cos LMs to decoder-only models shows promise for explainability in generative tasks, an area of growing importance as LLMs become central to real-world applications.", "requested_changes": "The paper's presentation can be improved by reducing discussion covered by prior work on B-Cos networks and focusing on the paper's central contributions. For example, adding explanation about why B was set to different values for different datasets would be useful.", "broader_impact_concerns": "There are no broader impact concerns. \n\nBeyond improving faithfulness and interpretability, B-cos explanations can also help uncover spurious correlations and biases, making them valuable not only for model trustworthiness but also for fairness and safety research, which are core concerns across the ML community. So, this work has positive broader impact.", "additional_comments": "No other comments.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)