Correcting misinterpretations of additive models
Authors: Benedict Clark
Rick Wilming
Hjalmar Schulz
Rustam Zhumagambetov
Danny Panknin
Stefan Haufe
Venue: NeurIPS 2025 poster ()
Keywords: explainability
correctness
benchmarking
additive models
fairness
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We provide a generalisation of activation pattern-based approaches to correct the interpretations of additive models, particularly in the presence of suppressors and correlated noise.

Abstract: Correct model interpretation in high-stakes settings is critical, yet both post-hoc feature attribution methods and so-called intrinsically interpretable models can systematically attribute false-positive importance to non-informative features such as suppressor variables.
Specifically, both linear models and their powerful non-linear generalisation such as General Additive Models (GAMs) are susceptible to spurious attributions to suppressors. 
We present a principled generalisation of activation patterns - originally developed to make linear models interpretable - to additive models, correctly rejecting suppressor effects for non-linear features.
This yields PatternGAM, an importance attribution method based on univariate generative surrogate models for the broad family of additive models, and PatternQLR for polynomial models. 
Empirical evaluations on the XAI-TRIS benchmark with a novel false-negative invariant formulation of the earth mover's distance accuracy metric demonstrates significant improvements over popular feature attribution methods and the traditional interpretation of additive models. 
Finally, real-world case studies on the COMPAS and MIMIC-IV datasets provide new insights into the role of specific features by disentangling genuine target-related information from suppression effects that would mislead conventional GAM interpretations.

Correcting misinterpretations of additive models
Benedict Clark'
Rick Wilming'
Hjalmar Schulz?
Rustam Zhumagambetov' Danny Panknin' Stefan Haufe 1,2,3
Physikkaissh Technische Bundesanstalt, Berlin, Germany
Technische Universität Berlin, Germany
Charité Universitätsmediinn, Berlin, Germany
Abstract
Correct model interpretation high-stakes settings is critical, yet both post-hoc feature attribution methods and so-called intrinsically interpretable models can systematically attribute false-positive importance to non-informative features such suppressor variables. Specifically, both linear models and their powerful non- linear generalisation such as General Additive Models (GAMs) are susceptible spurious attributions to suppressors. We present principled generalisation of activation patterns originally developed to make linear models interpretable to additive models, correctly rejecting suppressor effects for non-linear features. This yields PatternGAM, an importance attribution method based on univariate gener- ative surrogate models for the broad family of additive models, and PatternQLR for polynomial models. Empirical evaluations on the XAI-TRIS benchmark with novel False-negative invariant formulation of the earth mover's distance accu- racy metric demonstrates significant improvements over popular feature attribution methods and the traditional interpretation of additive models. Finally, real-world case studies on the COMPAS and MIMIC-IV datasets provide new insights into the role of specific features by disentangling genuine target-related information from suppression effects that would mislead conventional GAM interpretations.
Introduction
As machine learning (ML) models are increasingly used in high-stakes domains, the need for reliable model explanations has grown tandem, with much of the work in explainable AI (XAI) has focused on interpreting ML models through post-hoc attribution techniques. Parallel to this, simple model architectures such as Generalised Additive Models (Hastie Tibshirani 1986 Rudin 2019). which are thought of as 'inherently interpretable have become popular. GAMs non con-linearly model the output as sum of smooth functions applied to each input feature, followed by link function -typically the logit or softmax in classification settings. This architecture is often considered interpretable in the sense that each component function also known as shape function can be visualised to illustrate how the feature contributes to the model's prediction. However, how models use specific features, and therefore, how model coefficients or shape functions must be interpreted, depends non-trivially on the data generating process, even when models are decomposable into simple terms. It is often crroneously assumed that non-zero coefficients corresponding to a feature or feature pair imply that these features carry information about the prediction target. However, depending the joint causal structure of the model's in- and outputs, optimal models may assign significant non-zero weight to non-informative features and, conversely, zero weight to informative features (Haufe al. 2014). It also possible that a feature enters the model with a polarity that opposes the correlation between the feature and the target. Such effects be due to suppressor variables, which variables that statistically unrelated or weakly related to the prediction target but help the model to remove variance from other variables, therefore improving its predictions (see, e.g., Conger 1974 Weichwald et al. 2015 Haufe et al. 2024).
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

In the context of linear models, Haufe et al. (2014) highlighted that model weights cannot be interpreted as typically desired unless adjusted by the data covariance, which yields the linear activation pattern, or simply Puttern. Without such adjustment, suppressor variables can receive arbitrarily large weights of any polarity, misleadingly signalling importance. Such misleading attribution is also systematically observed in a large variety of popular feature attribution methods applied to Bayes-optimal model in the presence of suppressors Wilming et al. (2023). Empirically, Wilming et al. (2022) showed that the Pattern approach outperforms other XAI methods in linear problem settings in the presence of suppressors, while Clark et al. (2024b) extended this analysis to lonelinear problems. Addressing the shortcomings of existing work on interpretable non-line non-linear models, our contributions are as follows:
1. We introduce a framework for computing univariate and bivariate feature importances turn the broad class of non-linear GAMs into models that can be interpreted in terms of practically relevant associations between features and prediction targets.
2. We instantiate this framework for multiple arts of models, resulting in two new explanation approaches: PatternQLR (for quadratic models) and PatternGAM (for general additive models).
3. We empirically validate our methods on the XAI-TRIS benchmark suite, reporting superior performance of PattemGAM and PatternQLR over GAMs and existing attribution methods.
4. We present novel metric for assessing the performance of feature attributions, negative invariant earth mover's distance accuracy (FNI-EMDA), to overcome the tendency of vanilla HMD metrics to penalise false-negative attributions and operate over narrow resolution.
5. We apply our approaches to two real-worl world datasets for the prediction of recidivism risk and in-hospital mortality. Using PatternGA.M, we unearth new insights into the role of sensitive attributes in the prediction process, thereby revealing critical misinterpretations of conventional GAM-based interpretations that can be attributed to suppression.
2 Background: additive models and suppressor variables
Generalised linear models (GLM) g [|yl[]) = w T x x where x E RP are input features, are often seen as inherently interpretable due to their simple structure, assigning one weight w; to each feature Iz. Consider the two-dimensional linear structural model introduced by /Wilming et al. (2023):
x ay + 17
where is the target label, = (1,0) is the signal activation pattern defining how the signal is represented each feature, and 77 ~ (0, Σ,). The data covariance defined as Ex =
+1 cri2 for non-negative standard deviations T1 and 72. Then, with P1 (1,0) and
cTiT2 r
#2 = (- the weights of the Bayes-optimal GLM are calculated as w E, (#1 p2).
which derives to w (a, -acr-acr//all / for suppressor feature contains no diserimiiiii information about y, it must still receive non-zero weight when the correlation between features is non-zero. This misleading attribution to suppressors is corrected by the activation pattern or Pattern (Haufe et al. 2014), defined as E.w Ex,) -1 which is the ordinary least-squares (OLS) estimate of the coefficients a, of the univariate models a,ÿ +. = Ij and simplifies to the univariate solutions a; Cov[z,, 0) Varlū] -1. In the present case, á (1, T) correctly reflecting the lack of statistical association between 222 and y and forming the desired data-aware global explanation.
(1 (cri (or1/ra)²)-1/2 showing that, while the
Generalised Additive Models Generalised additive models (GAMs,Hastic & Tibssirani 1986
Lou et al. 2013 Nori et al. 2019) model the prediction target y E IR as
D + JCc
E [Ely)))
GAAM (x.)
(2,,
(2)
where JGAM and fik (GAM are univariate and bivariate shape functions applied to single features and feature pairs, respectively, and where g is link function, often chosen as the logit function in binary

---

classification tasks and the identity function in regression tasks. In the following, we focus on binary classification, although all presented results should hold for other suitable choices of link functions.
In the original GAM and GA approaches (Hsstie & Tibshirani 1986 Lou et al. 2013), each shape function is represented as linear combination of basis functions such splines. Explainable Boosting Machines (EBMs) replace the spline functions with tree-based bagging and boosting techniques for particularly expressive and 'wiggly' functions, whereas Neural Additive Models (NAMs) AAgaraal et al. D replace the spline- or tree-based shape functions with small neural networks. Model parameters are fitted using training data D = {[x((1('') enabling coordination among the individual functions during optimisation. In the following, we assume that Ø|x]= = 0 and Vi Var[z:) > 0. Expectations and variances are taken across training samples. We refer to single entries v and pairs (R) of x as univariate and bivariate input features, respectively, and to fj and fy.k as univariate and bivariate functions. Note that we largely refrain from referring to these functions as modelling interactions per se, as the actual computation carried out depends on statistical properties of the input features Ij and IĄ.-
(x(0), , ))
We here choose NAMs our main additive model architecture, where we extend the original approach to also include bivariate terms fGAM Practically this amounts to adding an additional two-input sub-network with one output per modelled feature combination. In practice, rather than choosing the entire quadratic set of all possible bivariate feature combinations, Lou al. (2013) propose and Nori et al. (2019) make use of the FAST algorithm, which selects bivariate terms based on the residual of the best current additive model in feature space. Feature pairs are iteratively added to the model until there is no gain in accuracy. This approach is adopted here as well.
Fitted shape functions fcAM are often visualised to show how the model output varies as the value of a single feature or feature pair changes. Non-zero shape functions are interpreted as indicating a statistical association between feature and target, and their scales and signs are interpreted as strengths and directions of the associations (Lou et al.] 2012 Caruana et al. 2015 e.g.... Following HHaufe et al.) (2014), these interpretations are theoretically ly unjustified and potentially misleading. This is because GAM shape functions reflect how individual features need to be transformed to yield the desired target- but not based on statistical association with the target. In the two-dimensional suppressor setup introduced above, fitting a GAM with the identity link function g leads to a non-zero linear function in f2 FG (zz). Even though 22 is not discriminative in isolation, its statistical association with 21 (the true signal carrier) through the shared noise component causes the GAM to model a non-zero functional effect for 22, resulting in spurious attribution of importance.
Existing Non-linear Pattern Approaches Thus far, no analogy to the linear activation pattern has been proposed for GAMs, while proposals to make other families of non-linear models interpretable present disadvantages. PatternNet and PatternAttribtion are two well known approaches which apply the principles of Pattern to non-linear models, aiming to trace signal flow in deep neural networks (Kindermans et al. 2018). While reducing to Pattern in linear cases, both have been shown to have degraded performance on controlled non- -inerr benchmarks, particularly when correlated background noise acts as a suppressor (Clark et al.) 2024b). The Kernel Pattern method of Zhang et al. (2024) computes the Pattern in kernel space, but due to the pre-image problem (Bakir al.) 2003 Honeine & Richard 2009). the corresponding input-space Pattern is not uniquely defined and must be estimated. This reverse mapping is often ill-posed, unstable, or out-o--distribution, and the method captures only implicit kernel-induced relationships (e.g., similarity) rather than explicit or flexible feature combinations like those learned by additive models with explicit bivariate terms. Finally, recent work by Gjølbye et al. (2025) uses the Pattern concept in the context of locally linear explanations.
3 Methodology
In the following, extend the Pattern approach make two families of additive non-linear models, quadratic logistic regression (QLR) and GAMs, interpretable with respect to the Statistical Association
Property (Wilming et al. 2022 2024).
3.1 Pattern Quadratic Logistic Regression (PatternQLR)
We first extend the idea of activation patterns to a polynomial expansion of the input x up to degree 2, consisting of all first-order terms and all unique pairwise products
3

---

2 QLR = xxx)= =
(3)
Note that we include only one of the terms dy and 2g for each feature pair (i.e., we assume fixed ordering such that j < k), avoiding redundant terms due to symmetry. This results in DQLR 2D- + (D² - / (D linear, D quadratic and (D² -2)/2 pairwise) overall features that ,QLR
capture all uni and bivariate effects. For simplicity, we define f (x;) = IJJJ J (z,, = 2y
and fouR (x) = 2; QLR This Quadratic Logistic Regression (QLR) mapping approach (if we were to use the full non-redundant set of pairwise products) is equivalent to degree-2 polynomial kernel kxx,,x,) = (xx; x, + e)² with 7 = 1,c c = 0. Both the QLR and this particular polynomial kernel can be viewed as additive models with multiplicative feature combinations.
To solve the classification problem, we train a linear classifier (e.g., linear logistic regression, LLR) on z QLR From the obtained weight vector WQLR in z-space, we can compute the QLR activation pattern (PattemQLR) a PQL.R Cov Cov[Z01R g( )] Varigg (ij
where o(jQIR) wQI. ZQ1R + UQUR
This is again equivalent to fitting univariate models d;
PQLR PQLR (()) us
QLR post-hoc with OLS.
In cases where all features of the extended feature vector 2 QLR possess similar scale, the scale of the a; PQUR also comparable across features and can importance. This is the case in the neuroimaging context, in which Pattern has been introduced, where features typically correspond sensors of the same type and are measured in the same units (Haufe et al. 2014). If features are on different scales (e.g. correspond to different physical quantities), the coefficients df
as quantitative measure of feature
PQLR are not quantitatively comparable across features as well as across linear to
quadratic terms. Here we address this issue in two ways. First, standardise each 2; 201R mean and unit variance based on the training data prior to Pattern computation. Second, we derive additional feature importance metrics that scale-invariant. To this end, fit DQUR univariate LLR PQI QLR
+ dDOUR where the univariate PatternQI.R shape functions f ,PQLR (x) evaluate to f. P (x) = 1 PQLR x, + &OLR for univariate features and to (PQLR(() (x) = PQLK zzzx + df PQLR for bivariate features. Rather than interpreting scale-dependent coefficients POLR directly, we here interpret shape functions S.QUR which can be visualised as functions of the underlying input- features žj and Fy. Note that PattemQLR shape functions are fit to the true class labels y instead of their QLR estimates gOLR as done in the Pattern approach.
models of the form o( F [l]lx)) = J POI (x) bi 2
3.2 PatternGAM
For the fitted additive model Eq. ( we obtain DGAM D + (D² -D/22 (D univariate and (D2 / bivariate) non-lineat features produced by shape functions f f( (x) that are jointly leamed. We collect these in
(4) where, in practice, z 2GM is estimated the mean over K 100 NAM fits with random initial- isations, and a subset of bivariate terms are selected by FAST. Summing up the entries of z GAM according to Eq.
z ZGAM_ (x) =
21, 12), (bD
2 formally amounts applying a linear logistic regression model with fixed I. We observe, however, that this results in slightly suboptimal models, pre- sumably due to the model averaging process. Therefore, we re-estimate GAM using LLR leading to adjusted features 2, 2.A2 +- u w.A. 2GAM, which are used in all subsequent steps. After a fur- ther standardisation of the 2 2GGMM we again calculate the PatternGAM activation pattern vector as a PGAM = Cov(z GAM _g g(GGAM ([GA)]]arr gf GG Similar to PatternQLR, we also fit separate univari- ate LLR models for each of the DGAM features, where F.GAM (I,) 8PGAM ff fGAM (22,) d. dPGAM for
weights GAM
(22,)
univariate features and f.GAM (Iz; mx) = 1/PGAM PGAM (I.,, xx) + JPGAM for bivariate features.
3.3 Feature importance metrics
Our main object of feature interpretation PatternQLR and PatternGAM shape functions I PQLR//GAM evaluated as functions of uni- bivariate features, which we compare their QLR/GAM counterparts / QIRIGAM In addition, we use the following scalar metrics of feature importance. Activations patterns: We use the absolute value of the PattemmLLR/PatemmCC coefficients

---

POLR/PGAM as estimated from standardised features 2,
QLRIGAM, PAT, (J+ (F = 10 POLRIGGAM
Scale: We evaluate the standard deviation of shape function values as SD; = Var[fi(x)]*
across training samples for both the original QLR and GAM shape functions ft f! and their PatternQRR and PatternGAM counterparts f P
Discriminability: We use the area under the receiver operating curve (AUROC) to assess the univariate discriminability of each univariate or bivariate shape function. We define DISCR,(f:: 2(AUROC(fi. v) 0.5), where positive associations with the target are reflected by scores between and 1. Being a correlation measure, DISCR invariant to rescaling of fi, and presents a general way to avoid suppressor misattribution in additive models.
Intersection: Our main interpretation goal is identify features or feature pairs that are both informative w.n.t the target well as 'used' by the model. The former can be ensured by measuring importance either with the PAT or DISCR metrics, or by evaluating SD for fPQRR/GGM As QLR and GAM models have no incentive to estimate non-zero shape functions for unnecessary, e.g. redundant, features, the latter is also ensured in principle. In practice, however, such features may receive small non-zero weights, which could preserve any potential statistical association with the target. Such fea- tures will receive strong importance according to all the above metrics, despite not being used by the model. To counteract this behavior, we also introduce PROD; = S); (... SD;
PQL P
Finally, while all importance metries evaluated on both univariate and bivariate features, restrict our statistical evaluation to input-space. To this end, we aggregate univariate and bivariate feature importances. We define IMP, IMP: for 2, = f(zz) and IMP,, IMP; (ft) for 2; = fi(z,, ;4),, where IMP- = {PAT, SD, DISCR} } (analogously for INTER). The aggregated input- space importance IMP AGG E IR is denoted by IMP_ LMP_AGG- = maxk45 (IMP ,AIR where
IMP, IMP PAI_PAIR,,A J.b max (IMP J.K, IMP,)
IMP, < IMP;
(5)
else,
with f= fk. for the case > k. and where IMP (PAT, SD, DISCR, INTER). We compare these metrics with existing feature attribution methods acting as baselines (c.f. Section 4.1). For local methods, we define the global attribution as the average absolute local attribution over all samples. Finally, denoting by s E RD a global attribution, we rectify all entries by taking their absolute values, followed by division by the sum of all entries, thereby ensuring that Vi 2 0 and >, 8; =
3.4 Theoretical Properties PatternQLR and PatternGAM
Let (D) be method capable of generating global input-space attributions E R.
Definition 1 (Statistical Association Property, SAP, Wilming et al. (2022, 2024)) A feature attri- bution method s possesses the SAP if any significant non-zero importance attribution to univariate feature žj indicates statistical association with the target: 8j indicates importance = Ij J y.
The SAP rules out that non-informative variables, including suppressors, are assigned significant importance, which is prerequisite for correct interpretations and use of attributions in downstream tasks such as model debugging, scientific discovery, and counterfactual analysis (Haufe et al. 2024).
Theorem The following quantities possess the SAP.
1. PutIemQLR/GAM shape function coefficients b, POLR/PGAM for (possibly) non-linear univariate features 2;
QURIGAM
= -f (xzz)-
2. Under conditions, PatterQLLR/GAM coefficients a. POURPGGM for 2 .URIGAM S OLRIGAM (I.z)- PattemQLRR/GAM shape functions f
(x,)) and their standard deviations SD (POLRGGAM Discriminabilityy metrics applied to univariate functions f (xzz), e.g., fGAM
Theorem The original GAM shape functions J.GM for 2, Fz and their standard deviations (D(GG do not possess the SAP.
The proofs are presented in Appendix A
3.5 Explanation Performance Metries
We conduct ground-truth benchmarking of univariate and bivariate feature attributions using the XAI-TRIS benchmarking suite (Clark al. 2024b see also Supplementary Section In the LIN
5

---

and MULT XAI-TRIS scenarios, the features 2, for which the SAP holds are known by construction and are collected in the set A where j A @ 2; 1 y. Analogously, we define the ground set of pairwise interactions I+ for the XOR scenario. In this work, we aggregate univariate and bivariate importance attributions in input space for benchmarking purposes. In this setting, the ground-truth is given by the set T+ = {j j E A+ V 3k (j,k) E I+} containing all univariate input features that are either in A or part of feature pair that is in I+, We further define the binary vector t+ E RD, where 1 = 1/IT+1 I + 2 E T and E 0 otherwise. Based on the known ground truth, the explanation performance
a given explanation S is assessed using the following metrics. Importance Mass Accuracy (IMA): Defined (MA T+) (EieT+ sx) / (E/ sa). this is the proportion of explanation 'mass' assigned to known ground-truth features compared to the total importance attributed to all features (Arras et al. 2022 Clark et af., 2024b).
False-Negative- Mee-Negaatvv-Invaiinn Earth Mover's Distance Accuracy (FNI-EMDA): For image data, the Euclidean distance between input pixels can be taken into account when evaluating explanation performance (Clark al., 2024b). Given such a distance C E DDxD, the earth-mover's distance (EMD) between the continuous-valued attribution s and the ground-truth importance t is defined as
EMD(s, tt, CT, FERDXD
min ΣΣT,
Tu,oCuu s.t. T1=s, -I=sTri==; k=1==
where r E BDD is transport plan. Unlike IMA, the EMD can readily distinguish between minor local displacements of ground-truth importance and globally altered importance patterns. EMD, however, requires that the explanation is transformed into the complete ground truth; that is, to avoid penalisation, XAI methods need to assign equally high importance to every ground-truth input feature. In practice, subsets of important features may form equally valid explanations -just as doctors, models may not need to use every symptom or available measurement to diagnose a patient. False- Negative-Invarl Invariant EMD Accuracy (FNI- (FNI-EMDA) addresses this shortcoming by setting the transport costs between all ground-truth -trth features to zero, which is achieved via the modified Euclidean distance Ch C.. (1 I(u E T Au E T)) FNI-EMDA is then defined as the negative normalised EMD with modified ground distance C': FNI-EMDA t+ C') Ómax max,[(" [C't+) is the worst-case EMD for given ground-truth 6+. This novel metric does not unfairly penalise subset-style attributions, illustrated in Appendix Section[[] and improves on the original formulation of EMD accuracy by providing a higher resolution of possible resulting scores. We analogously define EMD Accuracy (EMDA) by replacing C' again with the unaltered Euclidean distance C.
EMD(s,t C')//mmx where
4 Results
Code for all experimental results is provided") with details on usage, data availability, and compute required available Appendix Section B. We acknowledge the use of LLMS (ChatGPT 4o and Gemini 2.5 pro) for early prototyping. plot structuring and formatting, however all uses have been verified and tested (and compared with known results where possible).
4.1 Experiments on XAI-TRIS synthetic ground truth imaging data
Agarwal et al. (2021) motivate the use of NAMs for computer vision tasks due their potential intelligibility. The XAI-TRIS (Clark et al. 2024b) benchmark datasets provide visually compelling binary classification scenarios with known signal patterns, shown as 'tetrominos' (Golomb) 1996). Here, tetromino signals a are combined additively or multiplicatively with two different types of noisy backgrounds 77: white noise, correlated noise smoothed with a Gaussian filter. Different binary classification scenarios are presented linear additive noise task (LIN), a multiplicative noise task (MULT), and an XOR task with additive noise. In all cases, signal patterns a are combined with a background noise component 77, where the background is either white (Gaussian) noise, or correlated noise (white noise smoothed by a 2D Gaussian filter). The correlated background noise induces suppressors due to the spatial overlap of the background noise and the tetromino components. Here, we extend this with an explicit distractor pattern d that spatially overlaps the signal tetrominos,

---

again acting as suppressor. The full details and visualisations of data generation, model parameters, and experimental setup can be found in Appendix
-34
(a) LIN DIST CORR
(b) MULT DIST CORR
Figure 1: GAM vs PatternGAM shape functions for a NAM trained on the LIN (left) and MULT (right) XAI-TRIS scenarios with dedicated distractor patterns and the correlated (CORR) noise background. Grey shaded features represent class-informative ground truth tetrominos, which completely overlap with the non-informative distractor (suppressor) patterns (in orange). While the raw shape functions, used as the traditional global explanation of additive models, highlight noise features and distractors, PattemnGM correctly removes their influence on the explanation.
In the LIN and MULT scenarios, no interactions are present by construction, for which reason do not include explicit bivariate feature terms in the underlying model. [ the raw shape functions for NAM trained over these scenarios, where a dedicated distractor (suppressor) pattern, shaded in orange, overlaps with the tetromino signals in grey. One can that, when PatternGAM is applied, the shape functions of irrelevant features are nullified, overcoming misleading attribution of importance to distractor features seen in the original GAM shape functions.
(a) SGAM
(b) PGAM)
(c) DISCR(FGAN
Figure 2: Spatial maps of bivariate feature importances for 256 feature pairs in the XOR-DIST-CORR XAI-TRIS scenario. Heat indicates higher importance and red contours mark ground truth features. Importance of the original NAM shape functions in (a), measured by the SD metric, shows diffuse and noisy characteristics, whereas PatternGAM importance (b) concentrates within the ground-truth regions, showing the cross-tetromino interaction pattern of the underlying XOR problem. Similarly, the discriminability map (c) highlights ground-trut truth feature pairs as the most informative.
Figure 2 shows spatial importance maps for bivariate feature combinations in terms of the standard deviation (D() and LD(PPGAM) of the NAM and PatternGAM shape functions, as well as the univariate discriminability DISCR(FGAM) This is shown for the XOR-DIST-CORE XAI-TRIS

---

setting trained with 128 bivariate terms as selected using FAST. The 'picture-in in picture' style image shows how each feature of the 8x8 input image is estimated to interact with all other features. For SD(GGAM), ). limited structure is visible; however for SD(SRGAM) and DISCR( (GGM) we see the ground-truth XOR interaction between the T-shaped tetromino in the top left and the I--shaped tetromino in the bottom right reflected in the importances attributed to the corresponding pixel pairs.
We further qualitatively and quantitatively evaluate the global explanations of PatterGGAM and PatternLLR with those of the original NAM, an EBM. Kernel Pattern, SHAP (Lundberg & Lee 2017). Integrated Gradients (Sundararajan et al. 2017). PatternNet and PattermAttribution. The latter four are applied to four-layer Multi-layer Perceptron (MLP). Appendix Figure 8 shows that PattemGAM and PattemQLR tend to highlight truly important features more consistently than all other approaches. Interestingly, PatternNet and Kernel Pattern highlight smoothed correlated noise in several CORR scenarios, and all methods that do not possess the SAP can highlight significant attribution to the distractor features.
Tables pressnts results of quantitative assessment of the explanation performance of all attributions on the XAI-TRIS data using the FNI-EMDA metric. Analogous results for the IMA and regular EMDA metries are shown in Appendix Tables 3 and Generally, the standard interpretations from the NAM and EBM as well as the traditional post-hoc feature attribution methods SHAP and Integrated Gradients have worse performance than Pattern-based methods, showing false- colsi-posstive attribution to suppressor and otherwise uninformative features.
Medial
WHITE
COKS
LIST WHITE
DIST
CORK
WHITE
CORX
DIST WHITE
DIST CORK
WHITE
COKK
DIST WHITE
DIST CORK
E10 21101
01:0:2 1035
1150 = E111
11:02
8133
ETA2 2101
1330:21005
11.04 6011
1151 1 8131
0131 [11
0.20 10:14
035022 1011
2.100
0183
EU31
Exll
1u09 2:0.0
0.75. 11331
018522
0.72 0 (13)
1:11 1.000
0.80-2 0.11
0153
0148:
022 5:1.0
11.03:2
11:2 0/31
019 2:1.0
0.76-1 01/3
(1.88
0177% E13)
1:95 2:100
1358: 1U:3
(1:33
(1.7 tu]l
699 2:0.0
01:5-22 5.22
01002 0/01
08
0/02
0191 -1000
0.88-00 0133
(880 0101
0.50 - 0/53
019 0:00
0.00--000 0.31
09::200 61]
0.04:3 D022
0.30-
0.80-
0.1
0.602
0100 2-1.0
0 0102::0000 0.00
1.00 0000000 0.00
0.00 02 0.00
3.00 2.00
11:0:2 (130
080-2000 0.00
0153 0 (19)
1:00:00:00
1.00 0.00
1.00 0.00
100 0.00
0:N 2:10.5
0180
0.55 E/11
5012
1017 2:1.0
0130-11 0122
0.50
0.55 [/ll
0:55 2:10
0.88 1124
0.50
0.7.2
SDX nau
BBM
BU00 1-100
01:0022000 0U3
0150 E/90
11:0
1086 2:10
0.00 [131
0858 (03)
115-2:00
0.20 0634
088.1100 1651
Putoon
018
D181 11.00
01.70:2
1154 0J83
016 - 2.04
0.37
0.70 0212
1001 1.00
1157
015.33
600 2:00
01:1 00/
0.5 0/33
01:0
0/07
8850 0035
0166 003
0.53 0/5
000--000
0.00:31 0 0.72
066 0012
0 001
les Drod.
0-01 4100
(106-1 D/0
0150 0/91
0100
1486 6:00
0374 (/3)
0.94 6/1
01563 0 (V9)
0110-000
040-8 6/33
0.0 661
Table 1: False-Negative Invariant Earth Mover's Distance Accuracy (FNI-EMDA) shown as mean + standard deviation values. The best performing method per row is emboldened, and our proposed importance metrics are denoted by (*).
4.2 COMPAS Recidivism Risk
COMPAS is a score developed to predict recidivism risk, used to inform bail, sentencing and parole
decisions, and is notorious for the presence of racial bias (Angwin al.) 2016 Dressel & Farid 2018
Tan et al. 2018). Here we investigate the importance of six features, age, race, prior's s count, length of
stay, sex, and charge degree, on recidivism risk using a publicly available dataset (ProPublica 2016).
Figure 3 shows the learned shape functions from an ensemble of 100 NAMs. Sensitive features like
Lenph
Figure 3: NAM shape functions for six predictors of recidivism risk as estimated by an ensemble of 100 NAMs. Red shading highlights the data density. Blue bar plots mark global importance according to four metrics: NAM scale, SD(FGAM), PatternGAM scale, SD(FRGAM) the product of both (PROD), and univariate discriminability (DISCR). In this example, all six features show (marginal but) significant dependencies to recidivism risk, while the fitted multivariate NAM primarily focuses age, length of stay, and priors count.
race and sex, as well as charge degree are of low importance for the multivariate NAM prediction
8

---

1374
4010
1.51
wetoo 5.500
Secler
Figure 4: Learned shape functions for ensemble of 100 NAMs trained on MIMIC-IV data to predict mortality within the first 24 hours of hospital admission, versus PatternGAM shape functions. Green-shaded plots show features with highest univariate informativeness with respect mortality relative to their use in the NAM. Red-shaded plots depict features entering the model with a polarity that opposes their association with mortality, suggesting negative suppression effects. Yellow-shaded features are features used by the NAM for variance reduction without being predictive for mortality. These classical suppressors are nullified by PattemGAM.
evidenced by low values of the (GMM metric, despite displaying modest but significant correlation to the target according to the DISCR( (GAM) metric and univariate PattemGGAM analysis. We may speculate that these latter weak statistical associations could be due to confounding effects such as selection biases affecting the dataset's 's composition rather than these features being truly causal for recidivism. Consistent with this two-sided interpretation, we observe that the PROD metric, quantifying to what extent features are simultaneously informative and used by the model, attains near- zero values for race, sex, and charge degree. These results suggest that the multivariate NAM can achieve its predictive accuracy largely without relying on these sensitive attributes, which could be considered a desirable outcome. This may allow for more nuanced bias mitigating schemes than simply removing features based on raw correlations or a-priori assumptions.
4.3 MIMIC-IV Mortality Prediction
Finally, we evaluate additive models on in-hospital mortality prediction using the MIMIC-IV v2.0 dataset (Johnson et al. 2023a|b) b) hosted on the Physionet platform. The task is formulated as binary classification, where the target indicates whether patient died during the hospital admission within the first 7 = 24 hours of admission. The full details of which features are used and the steps for data preprocessing are shown in Appendix Section F
Figure 4 shows the learned NAM versus PatternGAM-scaeer shape functions for subset of features studied, with the full results shown in Appendix Figure [0] The green, yellow, and red shaded features are selected according to the strongest positive, strongest negative, and smallest absolute 8PGAM coefficients respectively, fit through the post-hoc univariate PatternGAM models f PGAM, The ensemble of 100 NAMs achieves an AUROC of 0.821 with zero feature pairs, which has a minor increase in performance (to AUROC = 0.824) with 64 pairs chosen by the FAST algorithm as such, we choose the former, simpler, model for further investigation.
MAP and Bilirubin mean play a minor role in predicting mortality within the multivariate model, however when modelled univariately, PattermGA finds these features be strongly informative. This could indicate that these features contain less direct information about the target, presumably through unobserved confounding or indirect causation, than other features, which are then preferred by the model. It is also noted that the scale of PatternGAM shape functions is generally elevated compared to GAM functions, as the latter distributes importance over a maximum of (here) 34 features, while the former must use single features for approximating the same target. Interestingly. Dias BP mean shows a negative suppression effect (Darlington) 1968). While intermediate and high Dias BP mean values are negatively associated with mortality. these values enter the NAM's prediction positively, likely for the purpose of removing BP-related variance from other informative features. Finally, the yellow-shader Hematocrit mean and insurance features, while playing role in the NAM, are nullified in the PatternGAM shape function fPGAM, resembling classical suppressors.
5 Discussion
Additive models such as GAMs and NAMs perform on par with deep neural networks in large range of tasks. The purported gain in interpretability of these 'glassbox' methods is, however, elusive. The common interpretation of shape functions 'highlighting features that are associated with the

---

predicted target' is often unjustified for dependent data in the presence of suppressors. Likewise, the strengths and signs of these shape functions cannot be unambiguously related to the prediction target. Here we introduce two general approaches, PatternQL R and PatternGAM, which rely on univariate associations with the target when assigning importance to features and shape functions, thus providing information that is typically desired by developers and users of XAI, and that can facilitate correct decision making in downstream tasks. Conversely, we exposed misinterpretations of conventional additive models both using quantitative ground-truth experiments and real-world analyses. It is important to note, though, that PatternQLR and PatternGAM function as a post-hoc adjustments for explanations they are not designed to be predictive models in themselves for altering a model's 's underlying predictive mechanisms.
PatternGAM effectively ignores suppressor variables, avoiding their misinterpretatton as being target-informative. Combined with classical GAM shape function analysis, PatternGAM can enable users to identify suppressors as features used by the model to reduce predictive variance rather than enabling the prediction itself. In the present MIMIC analyses, this has led to the discovery of negative suppression, where the correlation of certain clinical features with mortality is the opposite of what classical GAM analysis would suggest.
Misinterpretations of suppression effects are widespread in the XAI community and can often be attributed lack of understanding of how multivariate models operate, neglecting that ML models depend on the causal structure of their training data, and may need to assign weight to non informaatte features remove variance. Notably, such misinterpretations can lead to ineffective or even harmful actions downstream. Agaawal et al.] state that the inclusion of sensitive features such as race and sex may lead to less fair models for making bail decisions; thus advocating for exclusion. Similarly, Wang et al. (2021) present GAM analysis suggesting that patients with asthma are prone to have lower risk of dying from pneumonia. This interpretation prompts the authors to suggest manual adjustments to the GAM shape function to 'correct' this effect.
In contrast, we promote two-step approach, in which a GAM is first used learn potentially complex non-linear features, after which PatternGAM is applied post-hoc to assess the contextual importance of these feature through their statistical relationship to the target. In the recidivism example, this approach would flag several sensitive attributes as important, thus potentially biased, but would not necessarily mandate exclusion of these attributes due to their insígnificance for the model. In the pneumonia example, PatternGAM would offer a straightforward way to assess whether the role of asthma lies indeed in contributing target-related information or whether asthma is merely used as suppressor by the model. In the latter case, manual adjustment would not be suitable, likely leading to performance degradation of a potentially optimal model, or worse.
Several modelling choices and technical intricacies of PattemnLR or PatternGAM warrant deeper discussion. First, univariate shape functions can be fitted to either predicted or observed labels using either linear rescaling or complete non-linear re-estimation of the GAM shape functions. These choices offer different solutions within the spectrum between purely data- and purely model-centric explanations. The proposed approach of linearly rescaling GAM functions to fit true labels thereby offers interpretability in terms of the actual building blocks used by the model, while possessing the SAP. Second, statistical guarantees such as for the SAP are provided here only for univariate PatternGAM and PatternQLR attributions. These results can be extended to bivariate features in the sense that feature pairs acting jointly as suppressors are correctly rejected by all proposed methods. Consider the structural model x1 = y + 12,72, x22 = 111, T3 = 112, where 11/2 ~ NO 1). The non-zero shape functions of the optimal GAM are fGAM (x1) = I1 and ( (x2, = - A O despite the fact that 22 and I33 neither individually nor jointly have any statistical association to y. In contrast, all proposed pattern-based methods would assign no significant importance to (12,2z)) For the structural model 2 w/n, I2
n, (O, 1). though, the original QLR and GAM models /QLRIGAM (x1 .22 = x2 for which the PatternQL R and PatternGAM consist of the single term S1,2
shape function coefficients evaluate to b by.2 PQLLR/GAMM 1: thus, indirectly assigning importance to the suppressor ø2. Importantly, though, the IMP AGG rule prevents that this bivariate importance leaks into the aggregated score for 22. Future work will define strict data-driven notions of (higher order) feature interactions, and propose appropriate metrics to disentangle different types of suppression effects from true interactions within the framework of additive models.
10

---

Acknowledgments and Disclosure of Funding
This result is part of project that has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (Grant No. 758985), and the German Federal Ministry for Economic Affairs and Climate Action (BMWK) within the "Metrology for Artificial Intelligence in Medicine (M4AIM)" program of the "QI-Digital" initiative.
References
Agarwal, R., Melnick, L.., Frosst, N., Zhang, X., Lengerich, B., Caruana, R., and Hinton, G. Neural additive models: interpretable machine learning with neural nets. In Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS '21, Red Hook, NY, USA, 2021. Curran Associates Inc. ISBN 9781713845393,
Angwin, J., Larson, J., Kirchner, L., and Mattu, S. Machine bias: There's soft- ware used across the country predict future criminals. and it's biased against blacks. ProPublica, 2016. URL https://wwy http:://www.propublicaa propublica org/article/ machine- bias-biis--aas reias-risk-aae assessments in- crrminal-seeteeeeee.. 04.
Accessed: 2025-05-
Arras, L.. Osman, A., and Samek, W. Clevr-xai: A benchmark dataset for the ground truth evaluation of neural network explanations. Information Fusion, 81:14-40, 2022. ISSN 1566-2535.
Bakir, G., Weston, J., and Schölkopf, B. Learning to find pre-images. In Thrun, S., Saul, L., and Schölkopf, B. (eds.), Advancess in Neural Information Processing Systems, volume 16. MIT Press, 2003. URL http:: httpss:/pproceeddnnggg www.eeddiinnnnnnnn....e......... nwwrins....... wwtpppccpppppecpeeeeeeee forrr_fille files/ wtt.e////paaeee22////2/222
pdf
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., and Elhadad, N. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 2/th ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1721-1730, 2015.
Clark, B., Wilming, R., Dox, A., Eschenbach, P., Hached. S., Wodke, D. J., Zewdie, M. T., Bruila, U., Oliveira, M., Schulz, H., Cornils, L. M., Panknin, D., Boubekki, A., and Haufe, S. Exact: Towards platform for empirically benchmarking machine learning model explanation methods, 2024a.
Clark, B., Wilming, R., and Haufe, S. Xai-tris: non-linear image benchmarks to quantify false positive post-hoe attribution of feature importance. Machine Learning, 1 113())6871-6910, 2024b.
Conger, A. A Revised Definition for Suppressor Variables: Guide To Their Identification and Interpretation, A Revised Definition for Suppressor Variables: A Guide To Their Identification and Interpretation. Educational and Psychological Measurement, 34(1):35 46, 1974.
Darlington, R. B. Multiple regression in psychological research and practice. Psychological bulletin, 69(3):161, 1968.
Dressel, J. and Farid, H. The accuracy, fairness, and limits of predicting recidivism. Science Advances, 4(1):eaa05580, 2018. doi: 10.1126/sciadv.ao05555
Friedman, L. and Wall, M. Graphical Views of Suppression and Multicollinearit in Multiple Linear Regression. The American Statistician, 59(2):127-136, 2005.
Gjølbye, A., Haufe, S., and Hansen, L. K. Minimizing false-positive attributions in explanations of non-linear models. arXiv preprint arXiv:2505. 1/20, 2025.
Golomb, S. W. Polyominoes: puzzles, palterns, problems, and packings, volume 111. Princeton University Press, 1996.
Hastie, T. and Tibshirani, R. Generalized additive models. Statistical science, 1(3):297-310, 1986.
Haufe, S., Meinecke, F., Görgen, K., Dähne, S., Haynes, J.-D., Blankertz, B., and Bießmann, F. On the interpretation of weight vectors of linear models multivariate neuroimaging. Neurolmage, 87:96-110, 2014.
11

---

Haufe, S., Wilming, R., Clark, B., Zhumagambetov, R., Panknin, D., and Boubekki. A. Position: Xai needs formal notions of explanation correctness. In Interpretable Al: Past, Present and Future, 2024.
Honeine, P. and Richard, C. Solving the pre-image problem in kernel machines: A direct method. In 2009 IEEE International Workshop on Machine Learning for Signal Processing, pp. 1-6. IEEE, 2009.
Johnson, A., Bulgarelli, L.,, Pollard, T., Homn, S., Celi, L. A., and Mark, R. MIMIC-IV (version 2.2). https: //dtps://do.... org/ 10. 13026 6k67 2023a. Accessed: 2025-05-06.
Johnson, A. E., Bulgarelli, L., Shen, L., Gayles, A., Shammout, A., Horng. S., Pollard, T. J., Hao, S., Moody, B., Gow, B., et al. Mimic-iv, a freely accessible electronic health record dataset. Scientific
data, (11))11202200
Kindermans, P.-J,, Schütt, K. T., Alber, M., Müller, K.-R., Erhan, D., Kim, B., and Dähne, S. Learning how to explain neural networks: Patternnet and patternatribution, In International Conference on Learning Representations, 2018.
Kindermans, P.-.., Hooker, S., Adebayo, J., Alber, M.. Schütt, K. T., Dähne, S., Erhan, D., and Kim, B. The (un) reliability saliency methods. In Explainable Al: Interpreting, Explaining and Visualizing Deep Learning, 267-280. Springer, 2019.
Lou, Y., Caruana, R., and Gehrke, Intelligible models for classification and regression. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 150-158, 2012.
Lou, Y., Caruana, R., Gehrke, J., and Hooker, G. Accurate intelligible models with pairwise interactions. In Proceedings of the 19th ACM SIGKDD International Conference Knowledge Discovery and Data Mining, KDD '13, pp. 623-631. New York, NY, USA, 2013. Association for Computing Machinery. ISBN 978 150332117777
doi: 10.1 00.114524877555248777777 URL
httpps:/ //doi http///oiooooooo//////oo///////e 10. 1145/ 2487 575. 2487579
Lundberg. S. M. and Lee, S.-I. A Unified Approach to Interpreting Model Predictions. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 30, pp. 4765 Curran Associates, Inc., 2017.
Lundberg, S. M., Erion, G., Chen, H., DeGrave, A., Prutkin, J. M., Nair, B., Katz, R., Himmelfarb, J., Bansal, N., and Lee, S.-I. From local explanations global understanding with explainable AI for trees. 2(1):56-6)) 67, 2020-01.
Nori, H., Jenkins, S., Koch, P., and Caruana, R. Interpretml: A unified framework for machine learning interpretability, arXiv preprint arXiv:1909.0922 09223, 2019.
Oliveira, M., Wilming, R., Clark, B., Budding, C., Eitel, F., Ritter, K., and Haufe, S. Benchmarking the influence of pre-training on explanation performance in mr image classification. Frontiers 200.399/faaa..22.3333333999
in Artificial Intelligence, 7, 2024. ISSN 2624-8212. doi: 2024. 1330919
URL
https:// wwwntiorrrrreee iirsin org/ http/arttcleeeee.... wttp://ceeee...39//frrr.....
ProPublica. COMPAS Data and Analysis for 'Machine Bias', bttps://giite https://github.com http://ittub.ccm/ppppubiiai
compas- 2016.
Rudin, C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206-215, 2019.
Sundararajan, M., Taly, A., and Yan, Q. Axiomatic Attribution for Deep Networks. In ICML, 2017.
Tan, S., Caruana, R., Hooker, G., and Lou, Y. Distill-add-compare Auditing black-box models using transparent model distillation. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pp. 303-310 310, 2018. doi: 10.1 (0.11445377872113277777
Wang, Z. J., Kale, A., Nori, H., Stella, P., Nunnally, M., Chau, D. H., Vorvoreanu, M., Vaughan, J. W., and Caruana, R. Gam changer: Editing generalized additive models with interactive visualization. arXiv preprint arXiv:2112.0322 2021.
12

---

Weichwald, S., Meyer, T., Ózdenizci, O., Schölkopf, B., Ball, T., and Grosse- Grosse-Wentrup, M. Causal interpretation rules for encoding and decoding models in neuroimaging. Neuroimage, 110:48-59, 2015.
Wilming, R., Budding, C., Müller, K.-R., and Haufe, S. Scrutinizing XAI using linear ground-truth data with suppressor variables. Machine Learning. 2022.
Wilming, R., Kieslich, L.., Clark, B., and Haufe, S. Theoretical behavior of XAI methods in the presence of suppressor variables. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 37091-37107. PMLR, 23-29 Jul 2023. URL httpttt://poccceeiinggee
mlr
html
Wilming, R., Dox, A., Schulz, H., Oliveira, M., Clark, B., and Haufe, S. Gecobench: A gender- controlled text dataset and benchmark for quantifying biases in explanations. arXiv preprint 11544 2024.
Zhang, M., Treder, M.. Marshall, D., and Li, Y. Fast explanation of rbf-kernel svm models using activation patterns. In 2024 International Joint Conference on Neural Networks (IJCNN), pp. 1-8, 2024. doi: 10. WAW.UUCCCNN0009992220000006
13

---

Technical Appendices and Supplementary Material
A Theoretical properties of PatternQLR and PatternGAM
Proof (Theorem 1) We show that žj 11. - 8j implies no importance.
1. From x; IL follows that the optimal univariate LLR models g g(E(y)x)) = =b B.OLRPGAM d.ORRPGAM with = 3. and
2, 2.AM 5z( are constant functions of zi. Therefore, the
fitted b, (oOLRPGAM + for N oc.
2. a) Similarly, the original PatternQLR and PatternGAM models 0; POIRPGGM, +. PQLRIGAA
2/ QLRGGAM are constant functions of if the true labels rather than g(ğ) are regressed onto
ze using OLS. Also in this case, 0 POLRPGAM x Covlz, (LR[ZAL + 0. N oc.
b) For specific models. the predicted labels ÿ are linear functions of the input data (that is the identity function) and be shown to be uncorrelated 2z. This is the case for OLS regression models and unregularised linear discriminant analysis acting on non-linear features (Haufe al. 2014). In these cases, Pattern coefficients estimated via a POURPGGAM Covl2 QERGGAM
.0) 0.
3. From UPOLRIPGAM - 0 it follows that f
P.OLRPGAMM + 0.
4. Discriminability metrics such as DISCR measure nothing but the statistical association between y and fg fzzz) and therefore possess the SAP by construction.
Proof 2 (Theorem 2) Consider the linear model Eq. ①1. The optimal GAM shape function for I2 is given by fr everywhere for N + 0o c
FGAM W232 with w2 = -(1 (c51/52)2) 1/2 - which is non-zero almost
B Code, data availability, and computational resources
B.1 Code implementation and availability
Anonymised code is available at httepp//ggthao..cmmmmriraaaaaalt gan with gith the GPL-3.0 license.
The nam subfolder is adapted from the official Pytorch implementation of Agarwal et al. 202 available at httttss://github cwm/lemelmllle nam
The vai_tris subfolder is adapted from the XAI-TRIS implementation available at https: // github. www.booaenaaateeeeeeeeeooeaaeaee comm/braiiddaallaabaaa-aaaaaaaaa with the GPL-3.0 license.
B.2 Data availability
The XAI-TRIS datasets (Clark et al., 2024b) are available to generate hhttps://gi com/ braindatalab/xxii-trze with the GPL-3. 0 license, with fixed random seeds used for reproducibil- ity.
The COMPAS Recidivism data PrroPubicaa 2016) are available at https://github. com/ hatpcc/c//copsoaee compas- analysis and we provide the specific data file used recid. data in the anonymised GitHub repository.
propublica/
The MIMIC-IV dataset (Johnson et al. b023b) is available via PhysioNet (Johnson al. 2023a at feihttps:/// //tptt:://hhssnoeee org/ contentot///ocooccooo htt//mmmiia mwviiiii/2.0/// where training is required on subject han- dling before access can be granted. We use the v2. version of MIMIC-IV.
B.3 Computational resources
All experiments are able to be computed on personal devices, where we have used an M4 Pro MacBook Pro laptop for lot of the prototyping and processing work involved. Due to the nature of
14

---

the data and the lightweight models, no individual model takes more than a minute or two to train on such a laptop. Preprocessing the MIMIC-IV data takes slightly longer, however this is within the order of 30 to 60 minutes. For the XAI-TRIS model training line search and explanation calculation, we distribute jobs across maximum of four NVIDIA A40 GPUs to parallelise the process of using multiple seeds and datasets, however these scripts would likely take around 6 to 18 hours if run locally and sequentially.
C False-Negative Invariant Earth Mover's Distance Accuracy (FNI-EMDA)
HLL L
115
BEVAK Releeat
18 aevoaa A04220
Inbu 00 Nebovait
Brbikid Pe MI-C0R
Greei niDi Raaeaam No
100
t
Figure 5: Examples of different types of explanations and their corresponding Earth Mover's Distance (EMD), Dellse-eggaivee Invariant EMD (FNI-EMD), and Importance Mass Accuracy (IMA) scores when compared the ground truth. This shows the intuition of why FNI-EMD is beneficial over EMD and IMA, where we can achieve false- negative invariance while preserving a continuous and distance- based measure of explanation correctness. With EMD and FNI-EMD, explanation outlining the ground truth is not penalised compared to IMA. One thing to note is the greater resolution of scores for this formulation of the EMD--based metrics, due the ground truth-aware denominator normalisation of the transport cost compared to the original &max formulation proposed (Clark et al. 2024b). This is visible in the 'Corners' explanation, which represents near the lower bound of performance due to attribution as far from the ground truth as possible. Here, our new formulation of the EMD-based metries has lower bound 0.108, compared the original formulation's 's lower bound of 0.473.
Appendix Figure shows examples of different types of explanations and their corresponding Earth Mover's Distance Accuracy (EMDA), False-Negative Invariant EMD Accuracy (FNI-EMDA), and Importance Mass Accuracy (IMA) scores when compared to the ground truth. This shows the intuition of why FNI-EMDA is beneficial over EMDA and IMA, where we achieve false-negative invariance while preserving a continuous and distance-based measure of explanation correctness. With EMDA and FNI-EMDA. explanation outlining the ground truth is penalised compared to IMA. In resolutiono imaging tasks, the discrepancy between EMD-based metrics and IMA for outline-styl style explanations may grow larger, the entire contiguous mass of signal would have to be highlighted as important for IMA to not penalise the explanation. One thing to note is the narrower range of scores for the original formulation of the EMMD-based metries, where the 'Corners' explanation represents near the lower bound of performance at 0.473 due to attribution as far from the ground truth as possible. With our ground truth-aware formulation, this lower bound is reduced to 0. 0.108, presenting higher resolution of values possible.
15

---

D XAI-TRIS Implementation
D.1 Data Generation
Following from Clark et (2024b), we use the 8 8 variant, where each dataset consists of images of size, formulated D (x!m) ) NY This contains independent and identically distributed observations (z(*) E R P, mn) E {0, 1))m= with N 10000 and dimensionality D 64. The entities z(() and y(") represent instances of the stochastic variables X and Y. governed by a joint (x,1), v). In each defined scenario, the instance 2²(*) is synthesised probability density function px, yx,y (x,
by integrating a signal pattern (") E RP, encapsulating the ground truth features for explanations, with background noise 17(*)) E IRD. The noise component 17("), indicative of non-informative background, is drawn from a multivariate normal distribution (0, ID), resulting zero mean and an identity covariance ID white Gaussian noise. This setup is designated as the WHITE scenario. In each classification task, an alternate background, CORR, is specified where a two-dimensional Gaussian spatial smoothing filter G RDD + RD modifies the noise element 17(*), with the smoothing parameter (spatial standard deviation of the Gaussian) set to smooth = 3. As discussed later in Section D.1.3 the correlated noise backgrounds are where suppressors emerge, where the strength of smooth determines the strength of suppression in the 'support' distance of correlation between pixels. Here, we also propose an explicit form of suppressor in the optional distractor pattern d E IR D where this form of background component contains no class-related information, but provides more explicit overlap with ground truth features a!1) than the correlated noise background. When this explicit distractor is used, we designate the specify the scenario as DIST WHITE or DIST CORR depending on the other background component used.
In Clark et a analysis, a scenario is also considered where the signal pattern (*) undergoes a random spatial rigid body transformation (involving translation and rotation the tetromino; shortened RIGID) R(a), E. + PR.". We do not consider this scenario here, we are looking to provide global explanations, and the non-fixed nature of the signal in this case means that there no single global explanation. In all other scenarios, the identity transformation is utilised, such that R2(1) (n)) a'"), The transformed signal, distractor, and noise components, (R(a), a!")) and (G)n)()), are horizontally concatenated into matrices A = {(R()) a
(R(M) (((A),,..(((())) ()))).
D = {dd))).... ). and = {GG o oy ,y())),... ..., (i))) The three components are then normalised by the Frobenius norms of A, D and E: (R(m) a a(m)) +- (R(m) / (d(m)) + (d'))) //| and (G o (1)) + (G o (1) E|Le, where the Frobenius norm of matrix M is defined as ||M][e == (EN= Em (2)
1/2
Thee weighted sum of the signal, distractor,
and background components is computed, where the scalar parameters oea Old; Oiyy E [0, 1] determines the signal-to-noiss ratio (SNR), such that a, + aa = 1. This forms two generative models, which combine these components either additively or multiplicatively. For data generated through either process, each sample (n()) R&D is scaled to the range [-1, 111 such that that.x (()) + x((1) max (*))
where max [x)) denotes the maximum absolute value of the sample z(n).
D.1.1 Additive Generation
The data generation process for the n-th sample is defined as
x an (Ie(()) ())) + and + a. (G,(Gon
where the signal pattern a!1) E D varies, embodying tetromino shapes based on the binary class
Bernoulli process with success probability of 0.5. When the distractor pattern is used, aa, ag (1 aa)/2 for simplicity of parameterisation, however an imbalanced weighting of the two components can be used long as ag + au + a, = 1. If the distractor pattern is not used, d(x) 0. The formulation with an explicit distractor is akin to the data
label (()) which is distributed according
generation process seen in Wilming al. (2022).
16

---

D.1.2 Multiplicative Generation
The sample-wise data generation process is defined as
(n) (1 (R(")od("))
(R()) o alti)) (1-
and (Gon(ni).
(8)
where a (*)), d(n), 17 (n) R(m), and G are defined as previously stated. the distractor pattern is not used in the final scenario, d(n) is again
D.1.3 Emergence of suppressors
In the scenarios where background noise is correlated, the explicit distractor pattern is used, or both, the presence of suppressor variables is induced in both the additive and the multiplicative data generation cases. A suppressor identified as a pixel not part of the foreground R(n) a a("), but is correlated with a foreground pixel through the application of the smoothing operator G or spatial overlap of ("") with d(n), Drawing on characteristics of suppressor variables previously reported (Conger 1974 Friedman & Wall 2005 HHaufe et al.) 2014). has been hypothesised and since shown that XAI methods erroneously attribute importance to suppressor features in both linear and non-linear settings (Wilming et al. 2022, Oliveira al., 2024| Clark et al., 2024b). This misatrribution can lead to decreased explanation performance when compared to scenarios involving just white noise backgrounds with no distractors.
D.1.4 Classification scenarios
We study three distinct classification scenarios that are introduced using tetrominoes (Golomb, 1996), which are geometric shapes consisting four features. They are then utilised to define each signal pattern (n)) 8 and distractor pattern d!n) E R8 when used. These tetrominos are used to induce statistical associations between the features and the target in the following classification scenarios.
Linear (LIN) In the linear case, the additive generation model from equation is employed, where R(n) represents the identity transformation, combining the pure signal pattern and the Gaussian white noise background additively. T-shaped tetromino patterns ar and L-shaped tetromino patterns ao are utilised for signal patterns, positioned near the top-left corner if = O and near the bottom- right corner if y 1, respectively, thus constituting the binary classification problem. Each four-pixel pattern is encoded such that for each pixel in the tetromino pattern, positioned at the i-th row and j-th column, d..
T/L 1, and zero otherwise.
Multiplicative (MULT) The multiplicative generation process ( with signal patterns ar, az is defined with the same tetrominoes as in the linear case, while transformation Rin)) remains the identity transform. In this scenario, a degree of non-linearity introduced as the foreground tetromino pattern, when overlaying the background noise, is reduced towards zero. Therefore, values either increase or decrease depending on their original sign. The complexity introduced by the non- con-iincrrtyy renders linear classifiers unable solve this classification problem effectively (Clark et al. (024b). This configuration is meant to evaluate how different machine learning methods can adjust to and manage intricate, interconnected data presentations that are not linear.
Exclusive or (XOR) In the XOR configuration, an additive challenge is presented where both tetromino variants, denoted as T/LL, are utilised in each sample, with the transformation g(") maintaining its role as the identity transform. Within this setup, the class membership is defined such that for the first class (where = 0), a combination of both tetromino shapes is superimposed on the image background, either in positive or negative overlay, expressed as a xon++ a" + at and a XOR -aT a.. Conversely, for the second class (where y = 1), the tetromino shapes are displayed in contrasting manner; shape is overlaid positively, and the other negatively, denoted as a XOR+ = aT " and a XOR- -aa" T + a.. This ensures that all four XOR configurations represented with equal frequency within the dataset.
In all cases, if the distractor pattern d is used, this represents four 2 x 2-px squares in the results of Figure 8 as well as Tables
3 and In the demonstrative example of Figure [ this instead
17

---

represents four 2 x 3-px blocks so that the distractors fully overlap the signal. In both cases, each distractor tetromino takes a random sign +/- in each sample, however the strength of suppression could be increased by correlating the signs between each distractor tetromino (i.e., the two left distractor tetrominoes have opposite signs, and the two on the right have opposite signs). Examples of the generated data can be seen in Appendix Figure 6 using the four 2 x 2-px formulation of distractor patterns.
Signal
Distractor
Sample
Sample 2
Sample Sample 4
Sample
Sample
Sample
Sample 8 Sample 9 Sample 10
WHITE
conn
IN DI
IN DS
MUITE
XOR DEST
Figure 6: Examples of the data generated for the XAI-TRIS datasets Clark et al. 2024b), with the adaptation of static distractor patterns as an alternative and explicit form of suppressor variable compared to the correlated noise backgrounds used by the original authors.
D.2 Defining Ground- Cround-Truth Feature Importance
Following Clark et al. (2024a), we can formalise the ground truth sets for significant pixels as follows.
For the LIN and MULT scenarios, all pixels belonging the (smoothed) T and L shaped tetrominoes have a direct statistical association to the target, whereas there are no bivariate feature interactions by construction; therefore
At MULT = fill (R(m) a #0, j (1)...,64), IN MuL. = 0.
(9)
18

---

Note that in line with the definition of importance through the SAP, pixels of both tetrominoes form the ground-truth, even if in the LIN and MULT cases, only one the tetrominoes is present each individual image.
In the XOR scenario, no individual pixel has a marginal dependence on the target; however, all pairs of pixels where one pixel is part of the L-shaped tetromino and the other one is part of the T-shaped tetromino form an interaction with the target. Therefore we have
AXOR- , To = {(i (R()) o al("") #0A (R o all(")) #0, j. E fl...... 64]]
(10)
The aggregated input-space ground truth is identical in all scenarios:
THiN, MULT, XOR = ALiN MULT = Li 3k (j. k) }
MULT
(11)
D.3 Classifiers
Here we outline the parameterisation of each classifier used for subsequent training. The XAI-TRIS library outputs the data pre-split three-fold into training, validation, and test data, defaulting to 90/5/5 split respectively. However, all models except the MLP do not take the validation data as input, with the NAM and EBM implementations performing validation split internally. As such, concatenate the training and validation data as an input to all models other than the MLP. For the XOR scenarios with the NAM and EBM models, pairwise interactions are first identified using the FAST algorithm (Lou et al. 2013) with 128 interactions.
Neural Additive Model (NAM) We utilise the official Py implementation of NAM We instantiate each feature network for both main effects fi and interaction subnets fiy as Multi-Layer Perceptrons (MLPs) with hidden units [16, 16, 16). taking either one two features as inputs, respectively. The Adam optimiser is used with the default learning rate of 0.02082 and with a binary cross-entropy (BCE) loss function, training the model over a maximum of 100 epochs with patience of 50 epochs. Output penalisation with the default value of 0.2078 is used to regularise smaller outputs of each subnetwork, similar ridge regression.
Quadratic Logistic Regression (QLR) We implement a Logistic Regression model from scikit-learn with the quadratic feature expansion mentioned in Section 3 No penalty ap- plied and the intercept is not fitted.
Multi-Layer Perceptron (MLP) A standard MLP is trained using PyToorc. The network archi- tecture consists of an input layer, followed by three ReLU-activval hidden layers of size (32,16.8) and an output layer. The model trained for a maximum of 100 epochs with a batch size of 64, using the Adam optimiser with a learning rate of le and BCE loss function. Early stopping is implemented with a patience of 50 epochs based on the validation loss. The model with the best validation loss is stored and used for evaluation.
Kernel Support Vector Machine (kSVM) We use a Support Vector Classifier from scikit-leane with a precomputed Radial Basis Function (RBF) kernel, calculated between all training samples. The parameter y = 1/(d d x V Var(X)) defines the distance of influence of a single training example, for d the number of features and Var(X) the sample variance. This can be seen as the inverse of the radius of influence of samples selected by the model as support vectors. For prediction on the test set, the kernel is computed between each test sample and all training samples K iest, train-
Explainable Boosting Machine (EBM) Finally, we use the from the interpret library. All parameters are kept at their default values.
D.4 Training
To find the optimal signal-to-noke ratio (SNR) parameterisation, take line search across 10 values between zero and one, where the results can be seen in Appendix Figure] 7 After this, we
htttpp://giihuub.cc///eeeess
19

---

Scenario
aa
NAM
QLR
MLP
Kernel SVM
EBM
LIN WHITE
0.20
0.87 0.02
0.88
0.91 #.01
0,92 0.01
0.92 # 0.01
LIN CORR
0.10
0.90
1.00
1.00
1.00
LIN DIST WHITE
0.20
0.05
0.98
0.00
0.99
0.00
LIN DIST CORR
0.05
0.07
1.00
1.00
1.00
1.00
MULT WHITE
0.60
0.79
0.90
0.03
0.84
0.05
MULT CORR
0.40
0.01
0.99
1.00
0.96
1.00
MULT DIST WHITE
0.60
0.79
0.89
0.01
0.03
0.01
MULT DIST CORR
0.40
0.76
0.09
1.00
0.07
1.00
XOR WHITE
0.30
0.05
0.98
0.00
0.00
0.98
XOR CORR
0.30
0.93
1.00
1.00
1.00
0.99
XOR DIST WHITE
0.20
0.91
0.94
0.96
0,95
0.95 0.01
XOR DIST CORR
0.20
0.91 0.03
1.00
+
1.00 1.00 0.00
0.98 = 0.01
Table 2: XAI-TRIS model accuracy (mean + standard deviation) across scenarios and backgrounds for the chosen signal-to-noiss ratios (SNRs), shown by the signal parameter Oag
select SNR values for each such that the classification accuracy of all models the test dataset is or above 80%. The final model accuracies and chosen SNRs (parametrised by the signal strength weighting (ia) are shown in Appendix Table 2 As mentioned in the main text, noise strength is set to Ong
Oa when no distractor is present, and Od, Ogy 0n)/2 when distractors are present.
101
24
24
3.1
Figure XAI-TRIS model accuracy results for line search of signal-to-noise ratios (SNRs) for each classification scenario, separated by scenario and model type. Data with suppressor variables present (either as explicit distractor patterns, correlated background noise, both) results performant models at significantly lower SNR values.
D.5 Results
20

---

Figure 8: Global heatmaps for XAI-TRIS scenarios across all XAI methods tested. While most methods show at least some attribution to the signal tetrominos, the EBM and Kernel Pattern (Zhang)
ct al. 2024) struggle, particularly with correlated noise scenarios. PatternNet and PatternAttribunt (Kindermans et al. 2019), SHAP (Lundberg al. 2020-01), and Integrated Gradients (Sundararajan et al. (2017). are all implemented for the Multi- Layer Perceptron (MLP) model, and tend to also attribute strong importance distractor features directly neighbouring the ground truth signal. All methods proposed in this paper perform well, as also evidenced by the quantitative results of Section with no single 'best' method being immediately noticeable qualitatively.
MILLY
XLX
PATT
E:81
0.38.
01:5
0.30-1111
(1.11
S0 moser
600 2.000
0.55-2
0.55 = 601
6/97
099 2:005
0.542 007
0.94.2 0/33
0.55 6/01
091 2:0.0
0.00:200 0.34
0.00 0/3
0.51 101:2000 0/3
50
800-2:0.0
01:5-22 6/9
0.28 0/90
08:08 6/01
0:00 -.000
0.28-2 0130
0188 0/00
0.50 8/90
8::02..00
0.04 2
0.502 0/30
0.01 002
DECaL pon
8.21 2:100
11:13
01:7 6111
E112
0.21 8.1.0
11:01
0170
01:4 E130
0.26 1:100
0.33
(170
RUD
3.00 1-0.0
0805 0.00
1.00 0.00
0.9 0.00
0.50 0.83
0.03 0.03
0.99 0.00
0.05 0.00
0.000:0.33
0.07 1624
0.09 000
039 0.00
polay
000
Cry
0:28 0.000
0.48-2
0.3 6/11
0/00
0:1 0.00
0.55-200 0130
0.31-3
0:31- 0/00
0:2--900
0.66-200 0031
050 603
0.6-2 000
015 6J11
1009 1.100
0106
0.86 E151
8-101-110
0.3.1.00
011.110 10S
022 2.0.1
1840-2000 0.0
0:8-2 0.11
0.33 0.14
0.22 2010
0.11 0335
034 0/0
0113- 008
0/2-0..1
033-21.3 0.30
016-2001 92)
SHAA
0:28 1.06
08:5 D09
011 D/9
11:3
0.39 0.10
0861 EVT
0.38 Eu5
B/51 1.10
0.77 6038
0.00 667
0328 bd5
Table 3: Importance Mass Accuracy (IMA) quantitative evaluation results. Values are mean + standard deviation. Best result per row is emboldened.
E COMPAS Recidivism risk
21

---

Nervel
WISTH
CORS
WIITS
DIST
WIITE
CORS
DIST
WIITE
DIST
WIITTI
CORR
DIST
WHITE
DIST
8546 21100
11.33:2
18.2 0/12
D001 2:0.4
0190 0021
18.0 6015
01112:100
00:00:100 1111
11.3 1u3
PATT
688 2:0.0
08:3-:00 0/33
0:58 0/00
0:8
0:11-200 01.0
088-01 000
0:115000 0.90
031 -.000
0:80--00 0:30
0:00-10 019
0.:0- 6133
sns yngay
6011
6:9
0.35-
0:00
0.5
0.57
DOCCL, Dacc
808 2:0.0
01.33 6033
0155 6/22
11:0
1:01 2011
0.20-110 6604
0880 6/22
0153 6/11
[031 2-100
0300 1622
030: 1012
01:022 1v12
2:1.0
0183
(/31
0.70-1100 1031
018
1030
0:0021
Pacis maaay
08 0-0.1
0.07 007-0000 0.00
0.55 001
0.53 - 607
8:50 -080
0:5-11 0:30
0.08-1 0.00
0:55 0:00
0:00 0:00
0.9--000
051 601
0.53110 000
-0.00
01:8--
0:44 6:01
8111
8133
081-11 6011
0:40
0. 010
RRM
0190 00000
-1 60)
0.0--00 6:1
0.75 - 601
679 -000
0.8-000 0/30
0.65 600
0.5 0.0
B65 965--966
0.28 6:31
065-100 601
061 008
608 21100
0152 2
0.00
0100
E07 2111
081 1034
0
013 1003
B88 2:10
012 0634
013
0088 2-105
0112 608
011 - E/7
1173 EIT
877 2:330
0.h32 RU3
037 002
037 E/8
0322-2 1621
03T RNT
0822 t0s
Crac.
000 0.04
0.94 =
0.54 005
001
870 0016
0.88 001
0.8
0.5 602
0.9 =0
0.88 038
0.54 005
0.56 005
Table 4: Earth Mover's Distance (EMD) Accuracy quantitative evaluation results. Values are mean + standard deviation. Best result per row is emboldened.
Feature Feeture Comelation
X:2
0.00
0.2
pron
6:23
312
Feature
Corr
MI
HSIC
Age
-0.1928
0.0161
13.6641
Race
-0.1222
0.0042
8.5912
400
4.43
111
860
4.06
-130
Prior Counts
0.2908
0.0592
32.0686
Length of Stay
0.1067
0.0187
7.8228
Sex
0,1020
0.0053
3.1343
Charge Degree
-0.1112
0.0062
5,5309
campp
Table 5: Dependence Scores between COMPAS fea- Figure 9: Pearson correlation scores between tures and Recidivism risk, shown for Pearson relation, mutual information (MI) and the Hilbert- Schmidt independence criterion (HSIC).
COMPAS features used to predict recidivism risk.
F MIMIC-IV Mortality prediction
We construct a cohort for hospital mortality prediction using the MIMIC-IV v2.0 database. Structured tables from the hosp and icu modules are loaded using pandas, with data filtered to include only hospital admissions with available outcomes.
field in the admissions table. Patient age at admission is computed by combining anchor_ag and anchor_yesr _ear from the patients table with the year of hospital admission. The resulting raw age is capped at 90 to respect MIMIC-IV de-identification procedures. Admissions with implausible missing age information are excluded.
Feature extraction. Outcome labels are derived from the nospital_expires
We extract vital signs and laboratory values from chartevents and labevents, respectively:
Vital signs include heart rate, arterial pressure (MAP), respiratory rate, SpO2, temper- ature, and components the Glasgow Coma Scale (GCS). GCS components (eye, verbal, motor) were summed create a total GCS score.
Laboratory variables include sodium, potassium, chloride, bicarbonate, blood urea nitrogen (BUN), creatinine, glucose, white blood cells (WBC), hemoglobin, hematocrit, platelets, total bilirubin, ALT, AST, lactate, pH, and anion gap.
Each event is mapped to a clinical variable using predefined itemid dictionaries. For each variable, compute the mean value over the first T' hours after admission, with T (1,24, 48, 72).
We show the results for the first 24 hours after admission. Glasgow Coma Scale (GCS) components summed to yield total GCS score.
22

---

In addition to time-series features, we include static variables: age, gender, race, insurance type, marital status, primary language, admission type, and source. Race, admission location, and admission type are mapped to coarse categories:
Race: Grouped into White, Black, Hispanic, Asian, and Other/Unknownn.
Admission Location: Categorised into Emergency, Transfer In, Referral, Inter- nal/Procedural, and Unknown.
Admission Type: Simplified into Emergency, Urgent, Elective, Observation, and Unknown.
We also compute two aggregate features: the number of diagnoses (diagnoses. icd) and the number of preseriptions initiated within the first hours pprescriptions per hospital admission:
Diagnoses count: Number of unique ICD diagnoses.
Prescription count: Number of prescriptions started within the first T hours.
Handling missing data and outliers. We employ a two-stage strategy for missing data. First, rows with missing values in critical features are dropped These features age, heart rate, MAP, respiratory rate, SpO2, temperature, GCS total, lactate, creatinine. Second, remaining missing values handled via median imputation for numerical variables and constant-value imputation ('Unknown) for categoricals. All categorical variables are then ordinal-encoded with mapping dictionaries saved for interpretability.
Outliers are systematically identified and removed using interquartile range (IQR) filtering. Numerical
variables assessed individually, and observations falling beyond the lower bound (Q1 5 x IQR) or upper bound (Q3 + 5 x IQR) are excluded. This ensures the robustness statistical estimates and model stability.
Cohort finalisation. After feature construction, we filter to a minimal set of patients with complete outcome and critical data. A stratified group-aware split is applied using a group shuffle split, ensuring that no subject (subject_id) appears both training and test sets.
After preprocessing, the final sample size is N 23190 with 19616 unique patients and 3214 samples of deceased patients, split into rainiin/validdaiion/essii data, leading typical class imbalance for medical tasks. The full final list of studied clinical variables and their correlation to the
prediction target can be seen in Table 6
23

---

Feature
Corr Orig.
p-val Orig.
Corr Transf.f(a)
p-val Transf. f(x)
pH
-0.077
<0.001
0.092
<0,001
Marital Status
0.057
<0,001
0,015
0.074
Potassium
0.062
<0.001
0.057
<0.001
WBC
0,078
<0.001
0.087
<0.001
Heart Rate
0.080
<0.001
0,074
<0.001
Creatinine
0.090
<0,001
0.118
<0.001
Bilirubin Total
0.019
0,026
0,063
<0.001
Anion Gap
0.115
<0.001
0.104
<0.001
Sodium
0.025
0.003
0.078
<0.001
Resp Rate
0.127
<0.001
0.127
<0.001
Hematocrit
-0.040
<0.001
0.030
<0.001
prese count 24h
0.113
<0.001
0.105
<0.001
Hemoglobin
-0.064
<0.001
0.054
<0.001
Age at Admission
0.161
<0.001
0.161
<0.001
SpO2
-0,069
<0,001
0,062
<0.001
Lactate
0.152
<0.001
0.120
<0.001
Admission Type
0,062
<0,001
0,092
<0.001
Dias BP
-0.068
<0.001
0.053
<0.001
Platelets
0.010
0.238
0.030
<0.001
Glucose
0.033
<0.001
0.038
<0.001
Sys BP
-0.071
<0,001
0.072
<0.001
Admission Location
0.002
0.844
0.020
0.016
MAP
-0.766
<0.001
0.065
<0.001
ALT
0.002
0,850
0,063
<0.001
Bicarbonate
-0.043
<0.001
0,079
<0.001
Chloride
-0.004
0.638
0,033
<0.001
Temperature
-0.027
0.001
0,064
<0.001
Diagnosis count
0.148
<0.001
0.136
<0.001
Race
0.020
0.016
0.004
0.615
Gender
-0.020
0.017
0.020
0.017
Insurance
-0.044
<0.001
0.033
<0.001
BUN
0.157
<0.001
0.166
<0.001
AST
0.043
<0.001
0.070
<0.001
GCS Total
-0.235
<0.001
0.189
<0.001
Table 6: MIMIC-IV feature-wise correlations and p-values with the target (Hospital Expire Flag) for mortality prediction T = 24 hours after admission. This is shown for the original data, and then for the feature transformed by the learned shape function fx,)) for the given feature. Features like platelets, gender, and marital status have non-zero shape functions but near-zero correlation with the target, and when nullified by PatternGAM, this shows that they may have suppressive effect on the data. By contrast, features like GCS, age, and BUN have the scale of their importance preserved by PatternGAM, and here we can see that they have stronger correlation with mortality.
24

---

032
444
011
3.420
610
1.00
441
022
C81
521
4.4
Figure 10: All learned shape functions for an ensemble of 100 NAMs trained on MIMIC-IV mortality data within the first hours of ICU admission, versus PatternGAM shape functions. Data density shown above each plot, where regions of low density present higher uncertainty in the learned shape functions.
G Additional Plots
This section shows additional plots referenced in the appendix above that have been added here to not disturb the referencing order given in the submitted main text.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The Abstract and Introduction sections reflect the papers contributions and scope, and the introduction states the assumptions and limitations of current work, ending with dedicated block to state our contributions and limitations of our findings.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The Introduction briefly states the limitations of the work, ending with dedicated block state our contributions and limitations of our findings. The Background and Methodology sections list the assumptions made by prior and related work, and how these relate to our contribution. The Discussion section limitations listed alongside avenues for future work that would overcome these limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specifiiattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
26

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: In the Background and Methodology sections, we provide the full set of instructions and assumptions calculating activation patterns for additive models and benchmarking the resulting explanations. The Appendices provide additional details (For example, Appendix Section C provides more details on the benefits and properties of the FNI-EMD metric).
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Thcorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Where possible, we state the key experimental setup details in the main paper, with the Appendices providing the specific and exact instructions. We state the availability of all code to achieve the experimental results in the main text in the form of an anonymised GitHub repository. and point Appendix Section which provides additional details on code usage, data availability, computational resources required and the limits thereof. For code, we have used fixed random seeds where possible.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
27

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Similar the prior question, state the availability of all code achieve the experimental results in the main text in the form of an anonymised GitHub repository, and point to Appendix Section B which provides additional details code usage, data availability, computational resources required and the limits thereof. For code, we have used fixed random seeds where possible. All data is openly available, with only the MIMIC-IV data (Johnson et al. 2023b) requiring the completion of a data usage agreement and required training course available via Physionet (Johnson et al. 2023a).
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.c ec/
Polycy) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips. fattps.cc/pubbll wwtppcccccppppececccccccccc wwwpic//giise/eeee htt////ooeSubieeee /oso/e/Suudd//o issimmissiooi onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Where possible, we have provided details in the main text, with the complete details available in the Appendices. The full code to run all experiments is available with link stated at the start of the Results section and additional details in Appendix Section B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail
that is necessary to appreciate the results and make sense of them.
28

---

The full details can be provided either with the code, in appendix, or as supplemental
material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide error bars for the quantitative results of Table in the main text, and for training results that are elaborated the Appendices where appropriate. Details on the number of experiments/modellllaaass along with the full instructions on parameterisation. Guidelines:
run for each are provided in the Appendices
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the details compute resources in Appendix Section B
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We have reviewed the NeurlPS Code of Ethies and confirm that the research complies with these guidelines in every respect.
29

---

Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In the introduction we justify why misinterpretation (particularly with respect to suppressor variables being highlighted as important in explanations) is harmful, and why it is necessary avoid this. Throughout the paper we discuss real world examples in the COMPAS recidivism risk (ProPublica) (016) and MIMIC-IV datasets (Johnson et al.) 2023b). In the Discussion section we revisit the argumentation by other authors in the field related to our work, and the broader impacts of our work.
Guidelines:
The answer NA means that there no socictal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We diseuss in the results for the COMPAS dataset (ProPublica) 2016) that is very important to tread carefully in the analysis of biases (there specifically biases against race and sex) and the notion of what 'correcting' such biases entails. Our approach aims to prevent misuse and promote safe and fair usage of machine learning models in the context interpretability.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring
30

---

that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We state the ownership of code and data in Appendix Section B with subse- quent sections for individual datasets and experiments giving the details on specific licenses and terms of use.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, patersssppwssiiii theode. com/ com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: New XAI-TRIS datasets (Clark et al. 2024b) are introduced in the main text with the full generation and usage details available the Appendix Section D Data preprocessing and usage instructions for the real world examples COMPAS (ProPublical (016) and MIMIC-IV (Johnson et al. (2023)) are stated in their respective results with the full instructions given in the appendices. All generation and preprocessing assets available with the code provided.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
31

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper uses existing datasets COMPAS (ProPublica 2016) and MIMIC-IV (Johnson et al., 2023b) which themselves make use of data human subjects. All details can be seen in the original source materials.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: For the MIMIC-IV dataset, given the de-identified nature of the data, the Beth Israel Deaconess Medical Center's ss ethical committee waived the requirement for informed consent (Johnson et al. 2023b). For the COMPAS data, there are known risks and biases including racial biases (Angwin et al.] 2016 Dressel & Farid 2018 Tan et al., 2018)) which we discuss in the main text including the risk of unfair treatment as a result. However, we do not provide new data experiments of our own that involve human subjects, we have answered NA to this and the above questions.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification: We have stated our usage LLMs at the start of the Results section, and have ticked the relevant checkboxes on OpenReview to disclose this, including: Editing (e.g., grammar, spelling, word choice), Data processing/filteringg Visualizing results for submission, Facilitating or running experiments, Implementing standard methods.
Guidelines:
32

---

The answer NA means that the core method development in this research does not involve LLMs as any important. original, non non-standard components.
Please refer to our LLM policy (https://neurip:: /ttp://pp.ccc//offfeeeee//////// for what should or should not be described.
33


Reviews:
[{"review_id": "VxN6j7pvBm", "reviewer": "Reviewer_HbQD", "summary": "This paper introduces PatternGAM and PatternQLR, two new explanation methods for additive models that correct for misinterpretations caused by suppressor variables—features that are not causally linked to the target but appear important due to correlations. By extending the activation pattern framework (originally for linear models) to additive models, the authors enable more faithful feature attributions in settings where standard methods fail. The proposed techniques are evaluated on both synthetic (XAI-TRIS) and real-world datasets (COMPAS and MIMIC-IV), demonstrating improved robustness and interpretability over popular XAI methods.", "questions": "Please check the weakness outlined above.\n\n1. Does PatternGAM generalize to models beyond additive structures (e.g., deep networks, GNNs)?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength**\n\n1. **Timely and Important Topic:**  Tackles a critical issue in XAI—false-positive attributions caused by suppressors, especially relevant in high-stakes domains like healthcare and criminal justice.\n\n2. **Novelty**: The extension of activation patterns to additive models is well-motivated. PatternGAM and PatternQLR provide a principled, covariance-aware explanation mechanism.\n\n3. **Thorough Evaluation:** Strong empirical results across synthetic and real-world datasets. The introduction of False-Negative Invariant Earth Mover’s Distance (FNI-EMD) is a meaningful addition to the XAI evaluation toolkit.\n\n**Weakness**\n\n1. **Computational Overhead:**  Computing full covariance-based activation patterns, especially with interaction terms, may not scale well to very high-dimensional or real-time applications.\n\n 2. PatternGAM tends to produce \"subset explanations\"—highlighting only some relevant features. While this is justified, it could limit applicability in settings requiring complete ground-truth attribution.", "was_revised": true, "final_justification": "I have read the rebuttal. I have already given positive score to this paper.  I would like to keep my score same.", "rebuttal": "Thank you for the accurate and thoughtful summary. Please note that we deliberately avoid framing our method in terms of faithfulness, as this concept (as commonly used in XAI) can systematically assign importance to suppressor features — a concern raised in recent work (e.g., Wilming et al., 2023). This is also discussed by Haufe et al. (2024). Instead, we focus on producing explanations that better reflect statistical signal attribution, disentangling predictive utility from spurious correlations introduced by suppressors.\n\n\n**Weakness**\n\n **1. Computational Overhead: Computing full covariance-based activation patterns, especially with interaction terms, may not scale well to very high-dimensional or real-time applications.**\n\nThank you for raising this. While computational overhead is a valid concern for explanation methods, PatternGAM is designed to be lightweight in practice. PatternGAM reduces to computing the covariance between each scalar-valued feature (or interaction) output and the target, rather than requiring a full covariance matrix or matrix-vector product. This can be computed efficiently using standard statistics over the dataset. Moreover, unlike methods such as LIME or SHAP, PatternGAM does not involve training local surrogate models or computing Shapley value approximations. The method is purely post-hoc and closed-form. \n\nIf needed, approximate strategies such as diagonal/shrinkage covariance estimation, mini-batch statistics, or limiting the number of interaction terms (as we do via the FAST interaction selection algorithm) offer further scalability. We will make this more explicit by adding the above points to the discussion section in the revised version and provide bounds on the underlying computational complexity.\n\n**2. PatternGAM tends to produce \"subset explanations\"—highlighting only some relevant features. While this is justified, it could limit applicability in settings requiring complete ground-truth attribution.**\n\nThank you for raising this point. We agree that PatternGAM can produce sparse or subset-style explanations, where only a subset of the relevant features receives non-zero attribution. This may reflect property of additive models like NAMs, where the model may shut down some redundant informative features through the function $f_i$, i.e. a feature’s shape function $f_i(x_i)$ collapses to a constant. Then, an informative feature $x_i$ would become an uninformative transformed feature $z_i$. Here, PatternGAM would correctly assign it zero attribution, even if that feature was relevant in the original data distribution. In this sense, sparsity in the explanation may not indicate incompleteness, but rather that the model has learned to ignore some redundant or weakly informative variables.\n\n**Questions:**\n\n**Please check the weakness outlined above.**\n\n**1. Does PatternGAM generalize to models beyond additive structures (e.g., deep networks, GNNs)?**\n\nThank you for the question. PatternGAM, in its current form, is designed specifically for additive models. We see this as a first step toward bringing the soundness of the activation pattern framework to non-linear architectures. We chose (neural) additive models as a target because they are actively promoted as “intrinsically interpretable” models - yet, as we show, they are still vulnerable to suppressor-induced misinterpretations. PatternGAM addresses this by correcting how importance is attributed post-hoc, without requiring inverse mapping assumptions.\nWhile each subnet in a NAM can be made arbitrarily deep, and thus has some architectural flexibility, generalising PatternGAM to  deep models or graph neural networks would require a principled way to extract structured, linearly separable representations suitable for pattern-based transformations - a challenge we see as promising future work.\n\n\nReferences:\n\nWilming, Rick, et al. \"Theoretical behavior of XAI methods in the presence of suppressor variables.\" International Conference on Machine Learning. PMLR, 2023.\n\nHaufe, Stefan, et al. \"Position: XAI needs formal notions of explanation correctness.\" Interpretable AI: Past, Present and Future, 2024."}, {"review_id": "CAu4ZqskO4", "reviewer": "Reviewer_Xc1z", "summary": "This paper identifies and addresses a critical shortcoming in how additive models—commonly perceived as inherently interpretable—are typically interpreted. It shows that such models, including generalized additive models (GAMs), neural additive models (NAMs), and explainable boosting machines (EBMs), can misattribute importance to suppressor variables, leading to misleading global explanations.\n\nTo address this, the authors introduce: 1. PatternQLR, which applies activation pattern correction to quadratic logistic regression; 2. PatternGAM, a generalization of the activation pattern framework that corrects global explanations in complex additive models via covariance-aware, post-hoc linearization of model outputs.\n\nThey further propose False-Negative Invariant Earth Mover’s Distance (FNI-EMD) as a metric better suited to assessing the quality of global explanations, especially in scenarios where explaining only a subset of ground truth features should not be penalized.\n\nExperiments on synthetic (XAI-TRIS) and real-world datasets (COMPAS and MIMIC-IV) validate that PatternGAM outperforms both raw model weight inspection and post-hoc attribution methods (e.g., SHAP, Integrated Gradients), particularly in scenarios involving suppressors or correlated noise.", "questions": "1. The current formulation and evaluation of PatternGAM focus entirely on global explanations—i.e., producing a single explanation per class or model, rather than per input. Have you considered extending PatternGAM to the local explanation setting (e.g., similar in spirit to LIME or SHAP)? Specifically, could a covariance-aware linear projection be performed around a local neighborhood of an instance to produce a corrected explanation?\n\n2. PatternGAM uses a post-hoc linear regression over shape function outputs to recover corrected attributions. While this is conceptually elegant and efficient, it may be sensitive to model complexity. How do you assess or control the fidelity of the linear approximation to the underlying model? Could this lead to under- or over-attribution in highly non-linear regions?\n\n3. Your work extends naturally to models with pairwise interactions (e.g., EBM, NAM with pairwise terms). However, it is unclear whether the method generalizes beyond this. Could the covariance correction be generalized to additive models with three-way or higher-order interaction terms? If so, how would the linear regression stage be modified to accommodate this structure?\n\n4. PatternGAM does not merely provide attributions—it offers corrected estimates that may be more robust to confounding. This opens up downstream opportunities. Could PatternGAM be used to guide feature selection (e.g., pruning suppressors), or to identify potential fairness issues in model behavior? Have you explored this in practice?\n\n5. FNI-EMD is a compelling metric that addresses a real weakness in common evaluation metrics (e.g., IMA, EMD) by ignoring false negatives. However, its formulation and examples are tightly coupled to spatial data (e.g., 2D tetromino patterns). Could FNI-EMD be extended to structured tabular data, categorical features, or time series? Or is it specific to grid-based visual domains?", "limitations": "Partially addressed. The authors note that their method is restricted to additive models and focuses on correcting statistical rather than causal misinterpretation. However, there is little discussion of: 1. Deployment risk: In real-world settings (e.g., COMPAS), even corrected attributions can be misunderstood by users. 2. Interpretability trade-offs: No analysis is provided on how users interpret PatternGAM explanations compared to other baselines. 3. Generality: The paper does not explore how to generalize PatternGAM to more complex architectures beyond NAMs and EBMs.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n\nThe paper surfaces an underappreciated issue in additive model interpretation—spurious attribution due to suppressor variables—and offers a principled correction.\n\nPatternGAM is built upon solid mathematical foundations, extending the activation pattern correction to non-linear models in a consistent and interpretable manner.\n\nAcross a range of scenarios, especially under correlated or confounded inputs, PatternGAM produces explanations that more accurately reflect known ground truth relevance.\n\nThe experimental results span synthetic, criminal justice, and medical domains, showing consistent advantages of the proposed method in both performance metrics and qualitative heatmaps.\n\nFNI-EMD is a well-motivated addition to the XAI evaluation toolkit, avoiding false penalties for partial but correct explanations.\n\nWeaknesses\n\nThe method is tailored to additive model families (GAMs, NAMs, EBMs) and cannot directly be applied to more expressive or end-to-end deep models (e.g., CNNs, transformers).\n\nWhile PatternGAM adjusts explanation attribution, it doesn’t alter model training or prediction. The approach could miss interpretability issues that arise during optimization or from model biases.\n\nPatternGAM is positioned as a global explanation tool, but no discussion is provided on whether or how it might be adapted to instance-level interpretability.\n\nLike most attribution methods, PatternGAM is not causal. In high-stakes domains like recidivism or healthcare, users could potentially overinterpret covariance-aware corrections as causal insight.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the accurate and thoughtful summary of our work. We have had to remove a lot of explanation of our responses and have combined responses on repeated points/questions (for example, generality of PatternGAM, a local version of PatternGAM) due to rebuttal length constraints. Please feel free to ask for further clarification over the next week!\n\n**The method is tailored to additive model families (GAMs, NAMs, EBMs) and cannot directly be applied to more expressive or end-to-end deep models (e.g., CNNs, transformers).**\n\nWe see PatternGAM as a first step toward bringing the soundness of the activation pattern framework to non-linear architectures. We chose (neural) additive models as a target because they are actively promoted as “intrinsically interpretable” models - yet, as we show, they are still vulnerable to suppressor-induced misinterpretations. PatternGAM addresses this by correcting how importance is attributed post-hoc, without requiring inverse mapping assumptions.\nGeneralising PatternGAM to end-to-end deep models like CNNs and transformer architectures would require a principled way to extract structured, linearly separable representations suitable for pattern-based transformations - a challenge we see as promising future work.\n\n**While PatternGAM adjusts explanation attribution, it doesn’t alter model training or prediction. The approach could miss interpretability issues that arise during optimization or from model biases.**\n\nWe agree that PatternGAM, by design, does not alter model training or predictions - it operates entirely post-hoc to analyse learned representations. While it's unclear which specific interpretability issues the reviewer refers to, we would like to emphasise that PatternGAM possesses the Statistical Association Property (SAP), as formalised by Wilming et al. (2023). This property ensures that features with zero statistical dependency on the label (even nonlinearly) receive zero attribution. In contrast, methods like LIME, SHAP, LRP, or Integrated Gradients can attribute importance to suppressor variables. This makes PatternGAM particularly well-suited for detecting issues like confounding or the \"Clever Hans\" effect (Lapuschkin et al., 2019) where a model exploits spurious correlations in the data.\n\n**Like most attribution methods, PatternGAM is not causal. In high-stakes domains like recidivism or healthcare, users could potentially overinterpret covariance-aware corrections as causal insight.**\n      \nThis is also true. However, again due to the statistical association property, PatternGAM is better suited as a screening tool to study causal associations than any existing feature attribution method known to us (e.g., LIME, SHAP, LRP, IG). While the dependencies uncovered by PatternGAM do not dissociate causal, anticausal, or confounded relationships, they provide a principled starting point for downstream causal analysis or auditing. We emphasise this distinction in the revised paper to help guide appropriate interpretation in high-stakes domains.\n\n\n**Questions:**\n\n**1. The current formulation and evaluation of PatternGAM focus entirely on global explanations—i.e., producing a single explanation per class or model, rather than per input. Have you considered extending PatternGAM to the local explanation setting (e.g., similar in spirit to LIME or SHAP)? Specifically, could a covariance-aware linear projection be performed around a local neighborhood of an instance to produce a corrected explanation?**\n       \nThank you for this very thoughtful suggestion. In fact, we are exploring exactly that idea in a separate publication, for which we already published a preprint. We believe that the complexities that arise in the local setting due to the various design choices with respect to defining local neighbourhoods and covariances and the different possible applications and experiments (as the resulting methods are not restricted to tabular data but applicable to image classification tasks, for example), warrant a separate manuscript. Nevertheless, we will be happy to disclose our preprint here after the review process is completed for the reviewer’s interest (provided this is possible).\n\n**2. PatternGAM uses a post-hoc linear regression over shape function outputs to recover corrected attributions. While this is conceptually elegant and efficient, it may be sensitive to model complexity. How do you assess or control the fidelity of the linear approximation to the underlying model? Could this lead to under- or over-attribution in highly non-linear regions?**\n\nPatternGAM assumes, consistent with the GAM and NAM architectures, that the model prediction is a linear combination of nonlinearly transformed (per-feature) shape function outputs. This structural constraint ensures that a post-hoc linear regression over these outputs is not an approximation in the usual sense, but rather reflects the actual composition of the model. For more complex, non-additive interactions that cannot be expressed through individual or pairwise shape functions, the additive assumption may become limiting. In such cases, model performance may be degraded. We advocate that explanations should only be applied to well-performing models, where the output representation is then linearly separable.\n\n**3. Your work extends naturally to models with pairwise interactions (e.g., EBM, NAM with pairwise terms). However, it is unclear whether the method generalizes beyond this. Could the covariance correction be generalized to additive models with three-way or higher-order interaction terms? If so, how would the linear regression stage be modified to accommodate this structure?**\n\nExtending PatternGAM to handle higher-order interactions (e.g., three-way or more) is conceptually straightforward. These higher-order terms can be added as explicit components in the GAM or NAM architecture - either as handcrafted terms or learned sub-networks - and treated as additional features in both the model and the post-hoc PatternGAM attribution step. Importantly, PatternGAM does not require explicit construction of a high-dimensional covariance matrix. The attribution reduces to computing the covariance between each shape function output (whether a main effect or interaction term) and the target.\n\n**4. PatternGAM does not merely provide attributions—it offers corrected estimates that may be more robust to confounding. This opens up downstream opportunities. Could PatternGAM be used to guide feature selection (e.g., pruning suppressors), or to identify potential fairness issues in model behavior? Have you explored this in practice?**\n\nWe agree that PatternGAM provides corrected explanations by disambiguating suppressors from informative features, and we appreciate the suggestion to explore its use for downstream tasks like feature selection or fairness auditing.\nHowever, we would like to clarify that by “correction” we refer strictly to correcting explanations, not altering the model itself. This distinction is crucial: Bayes-optimal models may rely on suppressor variables to improve predictive performance (as discussed by Wilming et al., 2023), and such variables should not necessarily be removed from the model. \n\n**5. FNI-EMD is a compelling metric that addresses a real weakness in common evaluation metrics (e.g., IMA, EMD) by ignoring false negatives. However, its formulation and examples are tightly coupled to spatial data (e.g., 2D tetromino patterns). Could FNI-EMD be extended to structured tabular data, categorical features, or time series? Or is it specific to grid-based visual domains?**\n\nWhile our empirical focus was on 2D spatial domains (e.g., XAI-TRIS), the FNI-EMD metric is not inherently limited to grid-based visual data. The core requirement for FNI-EMD is the ability to define a ground cost (distance) between features, which allows it to be extended to other domains, such as temporal distance for time series, Mahalanobis distance in feature space for structured tabular data, or Hamming distance for categorical features. We will clarify this distinction in the discussion section of the revised text, and will also mention the alternative ground metrics stated above for their respective problem domains in Section 3.2.\n\n**2. Interpretability trade-offs: No analysis is provided on how users interpret PatternGAM explanations compared to other baselines.**\n\nThank you for the comment. We agree that understanding how users interpret explanations is important. While a formal user study is out of scope for this paper, we view it as a valuable direction for follow-up work and are actively exploring it. The standard NAM explanations (via shape functions) serve as a natural baseline, and our results demonstrate how PatternGAM corrects key misattributions caused by suppressor variables. We acknowledge that additional baselines (e.g., gradient-based or heuristic methods) may further contextualise our results, so we will include representative comparisons to such baselines in the final manuscript.\n\n**Paper Formatting Concerns:** \n\nThank you for the suggestions. We will adopt these idea in the revised manuscript. For example, Figure 2 will outline the ground truth tetromino subplots, which are the same respective positions as in Figure 1. We will expand Section 3.1 to improve clarity by providing brief intuitive explanations alongside the formal derivations - particularly in the PatternGAM paragraph. \n\nReferences:\n\nClark, Benedict, Rick Wilming, and Stefan Haufe. \"XAI-TRIS: non-linear image benchmarks to quantify false positive post-hoc attribution of feature importance.\" Machine Learning 113.9 (2024): 6871-6910.\n\nHaufe, Stefan, et al. \"Position: XAI needs formal notions of explanation correctness.\" Interpretable AI: Past, Present and Future, 2024.\n\nWilming, Rick, et al. \"Theoretical behavior of XAI methods in the presence of suppressor variables.\" International Conference on Machine Learning. PMLR, 2023."}, {"review_id": "XslH6y9bX9", "reviewer": "Reviewer_Rw6c", "summary": "The paper extends the “activation-pattern” correction developed for linear models to the broader family of additive models.  It introduces a covariance-aware explanation framework that yields two new attribution methods—PatternGAM (for generalised / neural additive models with pairwise interactions) and PatternQLR (for quadratic logistic regression)—and proposes a variant of Earth-Mover’s-Distance (FNI-EMD) that is invariant to false negatives.  Experiments on the XAI-TRIS synthetic benchmark, COMPAS recidivism scores, and MIMIC-IV mortality prediction show that PatternGAM/PatternQLR reduce spurious attributions to suppressor variables and sometimes nullify sensitive features while preserving accuracy.", "questions": "Please refer to Weaknesses.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. Sound formulation of a covariance-aware correction for additive models; clear empirical protocol; publicly released code & data.\n\n2. Paper is generally well written, with thorough appendix and checklist.\n\n3. Addresses the practical problem of false-positive attributions in “interpretable” additive models; may inform fairness audits where NAMs are popular.\n\n4. Adapts linear activation-pattern idea to a new model family; introduces FNI-EMD metric.\n\nWeaknesses \n\n1. The operation of premultiplying the weights by the covariance matrix requires a formal proof to guarantee its validity.\n\n2. The computational complexity increases explosively with dimensionality, rendering the method hard to use in practice; the paper should explore approximate strategies that can alleviate this issue.\n\n3. The empirical evaluation omits comparisons with several mainstream explanation techniques—such as SHAP, LIME, and Integrated Gradients—whose inclusion is necessary for a fair assessment.\n\n4. Purely additive model structures are uncommon in real-world applications, which inherently restricts the practical reach of the proposed approach.\n\n5. The method cannot remove dependencies stemming from nonlinear correlations; although the authors acknowledge this limitation, it remains important to analyse how influential such dependencies become in more complex settings.", "was_revised": true, "final_justification": "The authors’ response has addressed my concerns, and I appreciate the clarifications provided. I have therefore increased my score accordingly.", "rebuttal": "**Weaknesses**\n\n**1. The operation of premultiplying the weights by the covariance matrix requires a formal proof to guarantee its validity.**\n\nThis is straightforward to show. In the revised paper, we will explicitly define the Statistical Association Property (SAP), as formalised by Wilming et al. (2023). This property ensures that features with zero statistical dependency on the label (even nonlinearly) receive zero attribution. In the revised paper, we will provide a formal proof that the covariance adjusted shape functions and weights have the SAP property. We will show this for main effect (non-interaction) features, however extending the SAP to interaction features is non-trivial and will be an interesting direction for future work.\n\nWilming, Rick, et al. \"Theoretical behavior of XAI methods in the presence of suppressor variables.\" International Conference on Machine Learning. PMLR, 2023.\n\n **2. The computational complexity increases explosively with dimensionality, rendering the method hard to use in practice; the paper should explore approximate strategies that can alleviate this issue.**\n\nWe appreciate the reviewer’s concern regarding scalability. While the presentation of PatternGAM using covariance-based correction may give the impression of high computational complexity, the method is efficient in practice. Specifically, PatternGAM reduces to computing the covariance between each (individual) nonlinearly transformed feature (or interaction term) and the target, which can be done without explicitly forming the full covariance matrix or computing matrix-vector products. We will include a formal derivation of this in the revised version.\n\n**3. The empirical evaluation omits comparisons with several mainstream explanation techniques—such as SHAP, LIME, and Integrated Gradients—whose inclusion is necessary for a fair assessment.**\n\nWe thank the reviewer for the feedback. However, we would like to clarify that both SHAP and Integrated Gradients are included in our empirical evaluation. These methods are reported in the final two columns of Table 1 (as well as Tables 2 and 3 in the appendix), and correspond to columns 7 and 8 in Figure 6.\n       \nWhile we did not include LIME in our evaluation, this was a deliberate choice. Prior work (e.g., Wilming et al., 2022, 2023; Clark et al., 2024) has highlighted that LIME often fails to recover informative features in synthetic benchmarks with structured ground truth, such as ours. We therefore prioritised attribution methods with stronger theoretical foundations or empirical performance in comparable settings. We will clarify this decision and cite supporting work in the revised manuscript.\n\nClark, Benedict, Rick Wilming, and Stefan Haufe. \"XAI-TRIS: non-linear image benchmarks to quantify false positive post-hoc attribution of feature importance.\" Machine Learning 113.9 (2024): 6871-6910.\n\n**4. Purely additive model structures are uncommon in real-world applications, which inherently restricts the practical reach of the proposed approach.**\n\nWe appreciate the reviewer’s point that purely additive model structures are less common in unconstrained real-world modeling. However, additive models are  highly relevant and widely used in recent practice ,particularly in domains that require interpretable or auditable decision-making. For example, Caruana et al. (2015) showed that an intelligible GAM with pairwise interactions could match black-box model accuracy in predicting pneumonia risk while surfacing and correcting clinically implausible patterns. Ravindra et al., (2019) studied multiple applications of GAMs to link air pollution and climatic variability with adverse health outcomes. Yang et al. (2025) recently showed that GAMs can be effective for predicting energy savings based on measurement and verification of efficiency measures in commercial buildings, which is a key requirement of sustainability goals. In such contexts, avoiding false positive attributions to suppressor variables is essential, making PatternGAM’s contribution directly relevant to real-world deployment of interpretable models.\n\nIn our paper, we demonstrate the applicability of PatternGAM on two real-world datasets: COMPAS, a widely used fairness benchmark involving criminal recidivism prediction, and MIMIC-IV, a large-scale clinical dataset representative of real-world ICU decision-making. These reflect genuine use cases where interpretability is critical, and where additive models like NAMs and EBMs are increasingly adopted as presumed \"glassbox\" alternatives to black-box predictors.\nin the Discussion section of the revised manuscript (around lines 323-342), we will make it more explicit that there has been a recent increased interest in tabular data and tabular foundation models, and link this to existing writing on the popularity of using additive models for tabular data.\n\nCaruana, Rich, et al. \"Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission.\" Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining. 2015.\n\nRavindra, Khaiwal, et al. \"Generalized additive models: Building evidence of air pollution, climate change and human health.\" Environment international 132 (2019): 104987.\n\nYang, Jian, et al. \"Climate adaptive energy efficiency modeling using a generalized additive approach to optimize building performance across Chinese climate zones.\" Scientific Reports 15.1 (2025): 20088.\n\n\n**5. The method cannot remove dependencies stemming from nonlinear correlations; although the authors acknowledge this limitation, it remains important to analyse how influential such dependencies become in more complex settings.**\n\nWe thank the reviewer for raising this point. However, we would like to clarify that PatternGAM is explicitly designed to detect and attribute non-linear dependencies between features and the target, as long as these are captured within the model structure -such as univariate nonlinearities or explicit interaction terms. This aligns with the assumptions of the underlying GAM or NAM model: if the model captures a given nonlinear dependency, PatternGAM will reflect it in the attribution.\nThe method is not restricted to linear correlations. In fact, in our evaluations (e.g., on the XAI-TRIS datasets, which include structured noise and nonlinear distractors), PatternGAM performs robustly even in the presence of nonlinear  correlations, outperforming other methods in identifying true feature relevance. We also include the non-linear dependence measures Mutual Information and the Hilbert-Schmidt Independence Criterion (HSIC) in Table 4 in the appendix to show that dependencies captured are not merely linear.\nTo clarify, we do not claim in the paper that PatternGAM is limited to linear correlations, nor does the method filter them out - it highlights statistical associations as learned by the model, including nonlinear ones. If a GAM is not expressive enough to capture complex dependencies (e.g., beyond included pairwise or higher-order terms), then both model performance and explanation quality may degrade. In such cases, we recommend that explanations only be interpreted for well-performing models, where the learned representation is likely faithful."}, {"review_id": "6FG0Je2w8V", "reviewer": "Reviewer_3mR9", "summary": "This work proposes a novel post-processing method for generalized additive models (GAMs) to account for data covariance and thereby lessen the influence of suppressor variables on explanations derived from the GAM. This is achieved by extending the idea of activation patterns by Haufe et al. from linear models to polynomial feature spaces (PatternQLR) and GAMs (PatternGAM). In PatternQLR, this work maps (invertibly) the data into a non-redundant quadratic feature space (similar in nature to a degree-2 polynomial kernel, only with redundant terms removed) then trains a linear classifier. Similarly, in PatternGAM, this work maps the data into a feature space determined by the component functions (both marginal and interaction components) of a GAM. In the experiments in this work, these component functions are neural additive models with either one or two-input subnetworks, for marginal and interaction components, resp. This work then trains a logistic regression model over the mapped features, computes activation pattern vectors, and thereby obtains pattern shape functions by inverting the model output and applying the activation pattern vector. These pattern shape functions incorporate information about the covariance in either quadratic or GAM feature spaces, and can thereby identify and ignore suppressor variables. Experiments, on both synthetic and real data, are provided to verify the algorithms' efficacy, along with a novel evaluation metric.", "questions": "1. In the real-world case studies, I was surprised by the use of an ensemble of 100 NAMs, where this was not mentioned before. Why was an ensemble used, and not a single NAM? Also, is it possible to represent an ensemble of 100 NAMs, each using the same features, as a single NAM via a weighted feature-wise combination of the individual NAMs in the ensemble?\n\n2. I found it interesting that, in both real-world case studies, interaction terms did not meaningfully improve the predictive power of the models. I have several subquestions in this direction. Does the underlying NAM ensemble, before applying PatternGAM, achieve comparable generalization performance to the NAM with interactions, or does PatternGAM improve the generalization performance to the level of an interaction NAM? Have you tested this phenomenon on other datasets? If so, does it seem to generally hold that interaction terms aren't useful on so-called 'real-world' tabular data?\n\n3. Could the use of LASSO regularization in the logistic regression problem solved in creating pattern activation vectors help to consolidate predictive power into the smallest possible number of variables, and further eliminate the influence of sensitive features in the case studies? Or would this be somehow counterproductive?\n\n4. Related to the previous question, how does this method compare in ignoring the influence of suppressor variables to sparse GAM methods, such as FastSparse? Intuitively, sparsity regularization during training would achieve a similar objective by forcing the optimization algorithm to consolidate predictive power into as few features/splits as possible.\n\n5. In the definition of FNI-EMD, what motivated the choice to identify a ground-truth interaction as one where both marginal features are present? It seems like this over-represents the set of ground-truth interactions. Also, does FNI-EMD prefer subset-style explanations over complete ones, or does it merely refrain from additionally penalizing subsets?\n\nBonus question out of curiosity: On the synthetic vision problem, can the results obtained by this method be compared to an interpretable case-based reasoning neural network, such as ProtoPNet?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n1. The problem statement is very well-motivated, and the proposed solution is demonstrated to be, via comparison to related work, a novel and interesting way to address the detrimental effect of suppressor variables on GAM interpretability. The allure of a kernel-based method to solve this problem was misleading, as the authors identified, because it was not easily invertible. The proposed solution seems just as powerful (in the quadratic case, it only eliminates redundant/symmetric terms) and is easily invertible to recover precise pattern shape functions.\n\n2. I appreciated that this paper did not claim to solve all problems with GAM shape function-based explanations, such as confounder variables and data artifacts like default missing values. The focus of the paper on suppressor variables helped to justify the methodological choices and led to very interesting experiments which demonstrate clear practical utility and significance of the proposed method.\n\n3. The synthetic experiments, along with the proposed false-negative invariance distance metric, are convincing and interesting in their own right. Figure 1 clearly demonstrates that the pattern-based method successfully ignores the incorrect influence of many irrelevant features to the prediction task, and Figure 2 gives a striking visual which clearly shows the very intuitive interaction patterns used by PatternGAM to solve the XOR problem with tetrominos. These interaction patterns seem to be a) clearly the correct way for a model to reason about the problem, b) unobtainable by previous GAM methods, and c) unexplainable by a non-interpretable vision model. \n\n4. The COMPAS and MIMIC-IV examples in the text were convincing and demonstrated that this approach can give novel insights about real data which were not possible via previous methods. \n\n## Weaknesses\n\n### Significant Weaknesses\n\n1. I felt that a coarse theoretical argument detailing why pattern-based methods are able to ignore suppressor variables would have improved the paper. I recognize that suppressor variables have yet to be clearly defined by the literature, and this is clearly identified by the authors. However, I feel as though a more clear working definition, backed up by references to theory on linear models from prior work, could have helped to ground the intuitive arguments provided by this work for the proposed methods. The example provided in the background is helpful, but lacks a more general claim to justify the activation pattern approach as fixing the suppressor variable problem. As it stands, the reader must trust that incorporating data covariance into the model solves the problem (as it does in the linear case), which is intuitive to me but not rigorous.\n\n2. Table 1 only weakly supports the claim that PatternGAM significantly outperforms past methods on the FNI-EMD metric. Both PatternAttribution and Int. Grads. achieve comparable performance to PatternGAM on most scenarios, and both of them beat PatternGAM on every XOR scenario. I respect that the authors did not cherry-pick results, and I do not think that this invalidates the novelty or significance of the paper -- however, I do think that this weakens the claim that PatternGAM is uniquely suited to generating faithful subset explanations and avoiding false positives compared to other state of the art methods. \n\n### Minor Weaknesses\n\n1. Without having read the appendix to learn more about the data generation process, the orange boxes in Figure 1 were very hard to understand. Also, incorrect explanations happen outside of the orange boxes in Figure 1, which I found to be confusing if the purpose of this method is to eliminate distractors (it's eliminating more than just distractors). After reading the appendix, the figure made a lot more sense.\n\n2. The analysis of the COMPAS task in section 4.2 seemed to slightly overstate the proposed method's capabilities to ignore influence of sensitive features. This approach only applies if the sensitive features happen to be suppressor variables.", "was_revised": true, "final_justification": "My discussion with the authors revealed the following issues, all of which were resolved:\n1. It was not clear whether PatternGAM was a predictive method or an explanation method.\n2. There was not theoretical justification in the paper itself for why the proposed method indeed fixes the influence of suppressor variables.\n3. The figures in the paper were not incorrect, but led me to make incorrect conclusions about the paper.\n\nThe authors have proposed good changes and additions to the camera ready version of the paper that\n- will not substantively change the paper, but\n- will make the message much clearer and the contribution easier to understand.\n\nWith that in mind, I am keeping my recommendation to accept the paper, and increasing my confidence in acceptance.", "rebuttal": "**Significant Weakness 1:**\n\nWe appreciate the reviewer’s thoughtful suggestion. We agree that a more explicit theoretical argument and working definition for suppressor variables would strengthen the paper.\n\nIn the revised version, we will:\n\n1. Provide a clear working definition of suppressor variables in the context of additive models, grounded in prior literature (e.g., Conger, 1974;  Haufe et al., 2014; Wilming et al., 2023). In our context, we define a suppressor as a variable that improves predictive performance by carrying information about other features, but has no statistical dependency with the target variable. This contrasts with informative variables, which are directly associated with the label.\n    \n2. Formally demonstrate that PatternGAM avoids attributing importance to such suppressor variables, based on the Statistical Association Property (SAP) introduced by Wilming et al., (2023). Specifically, we will show that the pattern value - which reduces to the covariance between the nonlinearly transformed feature and the target - is zero when no statistical dependency exists. This allows PatternGAM to disambiguate suppressors and noise from confounders and informative features.\n    \n3. Expand our intuitive explanation with a more general argument: when the NAM learns a linearly separable representation from potentially nonlinear input space, PatternGAM projects the learned decision boundary back through the data distribution.\n\nConger, A. J. (1974). A Revised Definition for Suppressor Variables: a Guide To Their Identification and Interpretation. Educational and Psychological Measurement, 34(1), 35-46\n\nHaufe, Stefan, et al. \"On the interpretation of weight vectors of linear models in multivariate neuroimaging.\" Neuroimage 87 (2014): 96-110.\n\nWilming, Rick, et al. \"Theoretical behavior of XAI methods in the presence of suppressor variables.\" International Conference on Machine Learning. PMLR, 2023.\n\n\n**Significant Weakness 2:**\n\n\nPatternGAM is designed to explain additive models such as NAMs or EBMs. In the XOR scenarios, for example, the NAM architecture itself exhibits limited predictive performance compared to the MLP (see Figure 5), reflecting a mismatch between model capacity and data complexity. SHAP, PatternAttribution, and Integrated Gradients are applied to the MLP that perform better on these tasks. Even with near perfect classification performance, methods like SHAP and Integrated Gradients have been shown analytically to assign importance to suppressor features (Wilming et al., 2023), and PatternAttribution has shown suboptimal performance in non-linear tasks where suppressors are present (Clark et al., 2024). \n\nThus, while attribution methods on MLPs may appear to outperform PatternGAM when evaluated the FNI-EMD metric, this is in part a result of comparing explanations across fundamentally different model types. When restricted to additive model architectures, PatternGAM offers significant advantages in not attributing false positive importance to suppressors, as we demonstrate in both synthetic and real-world settings.\n\n\n**Minor weakness 1:**\n\nThank you for pointing this out. We agree that the current version of Figure 1 may be unclear without the context provided in the appendix, particularly regarding the meaning of the orange boxes. To clarify:\n\n1. The orange boxes indicate distractor regions - i.e., suppressor features that carry no statistical association with the target but influence the model’s decision through correlations with informative features.\n\n2. We will update the figure caption and lines 230–234 to explicitly define the orange regions and distinguish them from noise or overlapping regions. Additionally, we will clarify that PatternGAM suppresses not only distractors (suppressors), but also any features with no statistical association to the target.\n\n**Minor weakness 2**\n\nThank you for raising this point. Our intent was not to overstate this capability but to highlight an interesting outcome in the COMPAS task: PatternGAM neutralises the attribution to race and sex, suggesting that, in this model, these features behave more like suppressors than confounders or true predictors of recidivism. We will revise the text in Section 4.2 to more clearly distinguish between suppressors and irrelevant features, which PatternGAM is explicitly designed to identify and suppress; versus confounders, which may still receive attribution if they are statistically associated with the label.\n\n**Question 1:**\n       \nWe can see in Figures 3 and 4 that the many lighter shaded lines are individual models from the ensemble, showing the variation and stochastic nature of the NAM optimisation process. The emboldened line can be seen as the ‘mean function’, or the reviewer’s idea of a weighted feature-wise combination. Taking an ensemble of many NAMs allows us to capture this average effect and account for such variation.\n      \n**Question 2**\n\nThank you for raising this interesting observation. First, we would like to clarify that PatternGAM is a post-hoc interpretability method, not a model architecture. It does not affect model performance - it interprets the predictions of an already trained additive model (NAM or EBM), including when interaction terms are present. We will revise the terminology in the final version to better reflect this distinction.\n\nIn both COMPAS and MIMIC-IV, we observed that adding pairwise interaction terms yielded only minor improvements in AUROC (e.g., 0.821 without interactions and  0.824 with interactions for the MIMIC-IV setting). This suggests that the majority of predictive signal in these tasks is already captured by univariate nonlinear effects, and the added value of interactions is limited due to redundancy or diminishing returns. For instance, in MIMIC-IV, many features (e.g., age, prior admissions, lab values) are already strong individual predictors.\n\n**Question 3**\n\nAs discussed in Haufe et al. (2014), sparsity-inducing regularisation does not eliminate suppressors; in fact, suppressors can persist in even the sparsest optimal models if they help cancel noise in correlated predictors. PatternGAM’s goal is not to remove suppressors from the model, but to avoid assigning significant importance in explanatory importance. While LASSO could produce more compact explanations, it would not distinguish suppressors from true predictors, and may bias the pattern attribution undesirably.\n\n\n**Question 4**\n\nWhile sparse GAM methods such as FastSparse aim to improve interpretability by concentrating predictive power into a minimal subset of features, they do not eliminate suppressor variables. As shown in Haufe et al. (2014), suppressors can remain essential for optimal predictive performance and may be retained even under heavy sparsity constraints. PatternGAM addresses this and will not attribute importance to suppressors.\n\nIn that sense, PatternGAM complements sparse GAMs: it can be applied to any trained additive model, including sparsified ones, to produce corrected attributions that distinguish true signal from suppressors.\n\n\n**Question 5:**\n\nRegarding Equation (7): the matrices $\\gamma_{uv}$​ and $C_{uv}$​ do not refer to interactions between pixels $u$ and $v$. Instead, they define the mass flow and cost of transporting explanation values between individual pixels in the FNI-EMD metric. In this formulation, the ground truth consists of individual important pixels, and the transport cost is set to zero between any pair of ground-truth pixels, regardless of spatial distance. This allows the metric to ignore false negatives, as intended.\n\nRegarding the ground cost matrix C′ for interactions: you're correct that the current draft does not specify this in full detail, which we will update. For interaction terms (e.g., $Z_{ij}$​), the ground distance is defined as the sum of Euclidean distances between both constituent pixels.\n\n**Also, does FNI-EMD prefer subset-style explanations over complete ones, or does it merely refrain from additionally penalizing subsets?**\n\nFNI-EMD does not prefer subset-style explanations over complete ones - it simply refrains from penalising explanations that are subsets of the ground truth. FNI-EMD transforms any explanation into the closest subset of the ground-truth support (effectively projecting the explanation onto the ground-truth region), and then computes the transport cost needed to move the attribution mass to that subset. This projection incurs zero cost, so any explanation that is a subset of the ground truth - whether partial or complete - receives the same (perfect) FNI-EMD score.\n\n**Bonus question out of curiosity: On the synthetic vision problem, can the results obtained by this method be compared to an interpretable case-based reasoning neural network, such as ProtoPNet?**\n\nThank you for the interesting question. We believe that PatternGAM and ProtoPNet serve fundamentally different explanatory goals. PatternGAM provides feature-level attributions, identifying which parts of the input contribute to the model’s decision, with respect to the underlying data distribution. In contrast, ProtoPNet offers a case-based reasoning approach, explaining predictions by referencing similar training examples or learned prototypes."}]

Author rebuttal:

Meta-review: The paper develops model explanation methods intended to disentangle features with statistical associations, and in particular to show dependence of additive models on features while controlling for "suppressor variables" (a type of / similar to confounder variables, depending on ones definition of confounding)

It builds off an approach called activation patterns, this pre-multiplies the vector of regression coefficients with the covariance matrix of predictor variables. The main contribution appears to be extending this approach to additive models in two ways, one by including interaction terms, and one by using GAM (generalized additive model) basis functions.

Reviewers were in agreement that the paper addressed an important problem and provided interesting empirical demonstrations. The discussion surfaced many questions which authors were able to answer sufficiently. Authors have committed to changes and additions that will clarify and strengthen the paper. These are all important, including more clearly defining suppressor variables (perhaps by using a formal causal framework like structural causal models)

I might add a suggestion for clarity: notation should clearly distinguish between (true) data generation models, predictive models, and explanation models. For example, a simple convention that could be helpful is to notice the difference between explaining $Y = f(X) +$ noise and explaining $\hat Y = \hat f(X)$.
Decision: Accept (poster)