Exploring the cloud of feature interaction scores in a Rashomon set
Authors: Sichao Li
Rong Wang
Quanling Deng
Amanda S Barnard
Venue: ICLR 2024 poster (2024)
Keywords: Feature interaction
variable importance
Rashomon set
model interpretability
Primary area: visualization or interpretation of learned representations
TL;DR: A well-trained predictive model may not preserve the true feature interactions and there exist multiple well-performing predictive models that differ in feature interaction strengths

Abstract: Interactions among features are central to understanding the behavior of machine learning models. Recent research has made significant strides in detecting and quantifying feature interactions in single predictive models. However, we argue that the feature interactions extracted from a single pre-specified model may not be trustworthy since: *a well-trained predictive model may not preserve the true feature interactions and there exist multiple well-performing predictive models that differ in feature interaction strengths*. Thus, we recommend exploring feature interaction strengths in a model class of approximately equally accurate predictive models. In this work, we introduce the feature interaction score (FIS) in the context of a Rashomon set, representing a collection of models that achieve similar accuracy on a given task. We propose a general and practical algorithm to calculate the FIS in the model class. We demonstrate the properties of the FIS via synthetic data and draw connections to other areas of statistics. Additionally, we introduce a Halo plot for visualizing the feature interaction variance in high-dimensional space and a swarm plot for analyzing FIS in a Rashomon set. Experiments with recidivism prediction and image classification illustrate how feature interactions can vary dramatically in importance for similarly accurate predictive models. Our results suggest that the proposed FIS can provide valuable insights into the nature of feature interactions in machine learning models.

Published as a conference paper at ICLR 2024
EXPLORING THE CLOUD OF FEATURE INTERACTION SCORES IN A RASHOMON SET
Sichao Li, Rong Wang, Quanling Deng & Amanda S. -Baraadd School Computing Australian National University (s (sichao. cccholl1, 11, rong. wang, quan deng, amanda. s. wtw....aaarrraee........... Canu. edu. au
ABSTRACT
Interactions among features are central understanding the behavior of machine learning models. Recent research has made significant strides in detecting and quantifying feature interactions in single predictive models. However, we argue that the feature interactions extracted from single pre-specified model may not be trustworthy since: a well-trained predictive model may not preserve the true true feature interactions and there exist multiple well-pet rell-performing predictive models that differ in feature interaction strengths. Thus, we recommend exploring feature interaction strengths in a model class of approximately equally accurate predictive models. this work, we introduce the feature interaction score (FIS) in the context of a Rashomon set, representing collection of models that achieve similar accuracy on a given task. We propose a general and practical algorithm to calculate the FIS in the model class. We demonstrate the properties of the FIS via synthetic data and draw connections to other areas of statistics. Additionally, we introduce Ha l plot for visualizing the feature interaction variance in high-dimensional space and a swarm plot for analyzing FIS in a Rashomon set. Experiments with recidivism prediction and image classification illustrate how feature interactions can vary dramatically in importance for similarly accurate predictive models. Our results suggest that the proposed FIS can provide valuable insights into the nature of feature interactions in machine learning models.
MOTIVATION AND PRIOR WORK
Understanding the behavior of machine learning models receiving considerable attention. Many researchers seek to identify what features are important and to describe their effects. Recently, several works have taken one step further from explaining feature attribution towards interaction attribution. This shift aims to explore the interplay between features and uncover how their interactions shape the behavior of the model. Such interaction occurs when the influence of one feature, denoted as æ; on the model's prediction can not be decomposed into sum of subfunctions that do not involve the corresponding feature (Soookina et al. (2008); Friedman & Popescu (2008); Lou et al. (2013); Tsang al. (2021). denoted X,,, statistically defined as: * Eier fi v.). For example. there exists feature interaction effects in sin(-) function between 1 and æ2, where sin(2x + x22 o (1 + f2 Higher-order interactions among features are defined similarly. The feature interaction can provide more comprehensive and accurate understanding of the underlying factors driving model predictions, leading to robust and improved models; prominent applications include recommendation systems Guo al. (2017): Tsang et al. (2020a), DNA sequence analysis and modeling Greenside et al. (2018), and text-image information retrieval Wang et al. (2019).
Existing approaches that detect and explain the feature interactions mainly include (a) traditional statistical approaches such analysis of variance (ANOVA) and H-statistics Fisher ( (1992); Man del (1961): Friedman & Popescul (2008): Greenwell et al. (2018) and (b) more recent machine learning model-based methods such Additive Groves (AG) Sorokina et al. (2008). AG is non-parametric method of identifying interactions by imposing structural constraints on an additive model of regression trees that compare the performance of unrestricted and restricted prediction models. LASSO-based methods Bien et al. (2013) select interactions by shrinking the coefficients of insignificant terms zero. Specifically, neural network-based models, e.g. feed-forward neural

---

Published as a conference paper at ICLR 2024
Model
Rashoman set (R(E, f
B[L()Z))
2
E(U(')) [Z)
+1
110
FIS Fl..
R5,00)am
P1S,
(a) linear model y > w, x. shows feature interaction (left); see e.g. [Sorokina et al. 2008), A neural network with activation layer y = (<2) Wi x.) shows feature interaction (right).
(b) The a-axis shows the FIS the feature set each model f while the y-axis shows the expected loss of the model in the model class. The FISC range is highlighted in blue.
Figure 1: Two accurate models with different feature interactions and an illustration of FIS in hypothetical Rashomon set R(c, f F)).
networks, Bayesian neural networks and convolutional neural networks, are utilized to examine the weight matrices T'ang et (202T): Singh et al. (2019); Cui al, (2020). Several more advanced explainable frameworks have been proposed to capture feature interactions from well-defined models, e.g. Shapley Taylor Interaction Index (STI) Grabisch & Roubens [1999]; Sundararajan al. (2020),
Integrated Hessians (IH) Janizek al. (202 and Archipelage Tsang et al. (2020b).
Limitations of previous work Previous works can be summarized as detecting and explaining feature interaction effects in single trained model Here, we argue that the high accuracy a well- trained model can not guarantee the preservation of true feature interactions. As a simple example, 4ac
consider the two error-free models fi (a, b, c) -b+ /ba---a4ac finding root of the quadratic equation ax.² bu + 0; the input variables a, b, and exhibit different feature interactions in these two models as there is a sign difference in the numerator. See Fig. a for another general example where well-performing linear model with feature interaction might be modeled using deep neural network with the same model performance but with feature interaction. A detailed proof of concept is provided in Appendix C
and f2(a,
-b- -b-e--4ac for
2a
In general, for a given task, there exists a class of equally accurate models that lead to different feature interactions. To address the identified limitations, one potential solution is to analyze the feature interaction in set of well-performing models, Emerging research Rudin (2019); Fisher et al. (Dong) & Rudin (2020): | & Barnard (2022a) has demonstrated the necessity examining feature importance not just within single well-trained model but across a set of high-performing models, but none of these works delved into a comprehensive analysis of feature interactions
Our contributions In this paper, study feature interaction and demonstrate that investigating feature interactions a single model may fall short of accurately identifying the complex relationships between features. A more meaningful approach involves exploring feature interactions within class of models. Our primary contributions can be summarized as:
We introduce feature interaction score (FIS) as means to quantify the feature interaction strength in a pre-specified predictive model. In Sec. [2. and Sec. 4.1 we establish and validate the connection between FIS and the existing literature.
We consider FIS in a model set and introduce FIS cloud (FISC) in Sec. 2.1 in Sec. how feature interactions contribute predictions and how feature interactions can vary across models with reasonable and explainable accurate predictions in terms of FIS.
We demonstrate
We present a non-linear example in multilayer perceptron (MLP) to mathematically char- acterize the Rashomon set and FISC. For the general case, propose greedy search algorithm to explore FIS and to characterize FISC in a Rashomon set with two novel visualization tools, namely Halo and swarm plots.
More related works discussed in the Appendix
2

---

Published as a conference paper at ICLR 2024
2 FEATURE INTERACTION SCORE AND THE CLOUD OF SCORES
The terms "statistical interaction", "feature interaction", and "variable interaction" in the literature are highly related (Tsang et al., 2021 Greenwell et al. 2018 Sorokina et al., 2008 Tsang et al. 2020b). Here, we use the term "feature interaction" and "feature interaction score" " to indicate a statistical non-additive interaction of features in model. In general, we use bold lowercase letters such as u to represent a vector and v denotes its i-th element. Let the bold uppercase letters such as A denote matrix with [[-j] being its i-th row and j-th column entry. The vectors a(t,) and ((-i) are its i-th row and j-th column, respectively. Let (X; y) E Rnx(pt1) denote the dataset where X [-1],, ( .... [-) is a p covariate input matrix and is rt-length output vector. Let I be a subset of feature indices: I C ...,, p} and its cardinality is denoted by [Z]. All possible subsets are referred as (I I C p)] In the context of no ambiguity, we drop the square brackets on the vector and simply use F, to denote the feature. Xie is the input matrix when the feature of interest (denoted
here) is replaced by an independent variable. Let f guxp - IR" be a predictive model and (f(X), ), y) + R be the loss function. The expected loss and empirical loss are defined Leep E [[[ x), y)] and Lemp Ei= [(f(x)))) (.))).)))..).....).))))). y:), respectively.
2.1 DEFINITIONS
Mask-based Rashomon set Our goal is to explore feature interactions in a set of well-performing models, also known Rashomon set (Fisher et al. 2019). For a given pre-specified class of predictive models (c = =()
y). the Rashomon set is defined as
([)(f(X), y) s (X), +e),
(I)
where I* is trained model (or reference model), and > is a pre-defined acceptable loss tolerance. Given the existence of equivalent simple and complex models discussed in Semeenov et al. (2022), one way to characterize a model is to concatenate a layer, denoted as m E R, into a backbone model. That is,
Y F with E(L(S(X); y)), Im s.t. R[L(S o m ())),, y)] S EIL( y))]
y)] +
With this "input pre-processing layer, we refer to the resulting Rashonmon set as the Mask-based Rashomon sei and claim that there always exists an alternative for any model in the Rashomon set, logically expressed as: VS E F (B(L([(X), y)] (Em (B[L(S* o mm(X) y)] = EIL(J(X),))) See Appendix B for the detailed natural deduction.
Feature interaction score (FIS) The common way to quantify feature interaction based on the statistical non-additivity Lou et al. | (2013) is to measure the performance change of feature importance from the baseline (Sundararajan et al. 2017: Janizek et al. 2021). The performance change can be decomposed into main feature effects and feature interaction effects. Several feature attribution methods have identified the necessity of baseline value representing lack of information when generating explanations. We refer to individual feature effects which do not interact with other features as the main effect. Inspired by model reliance and similar methods Fisher et al. (2019): Zhuu et al.) (2015); Gregorutti et al. (2017) that measure the change in the loss by replacing the variable of interest with new random independent variable, we denote by (f( ([ (), 3) - (xX), y)l and (Pz(f) [[ y) ([L(f(X), y)] the main effect and joint effect respectively. Herein, [[L(J(X), y) is the baseline effect that provides interpretability. In practice, we usually permute the features of interest multiple times to achieve a similar measurement (Datta et al. 2016).
With this in mind, the FIS is defined as the difference between the loss change of replacing multiple features simultancously and the sum of the loss change of replacing multiple features individually:
FISz(f) = px(l)) - Σ p..().
(3)
VET
Feature interaction score cloud (FISC) We can generate the set of FISs as: FIS((f) = (FIST, (f), FIST,,
Thus, the FIS of certain feature set I in the Rashomon set R(c, f*.F) given by (ISC2(R) = (F) f E R). illustrated in Fig. (b). We refer to the set of FISs with respect to feature set in a model class as FISC. This cloud of scores has the range (VISCz(R)min) FISCz(RR)n.n = [minseR F F1Sz(f), maxjER F1Sz(f))
(4)
The complete set of FISs in Rashomon set is (TS = (FTS(()) : E R).
3

---

Published as a conference paper at ICLR 2024
2.2 RELATION TO EXISTING FEATURE INTERACTION ATTRIBUTION APPROACH
Since FISC focuses on explaining feature interactions across set of models rather than a single model, existing explainable methods for a single model should be viewed special case of our framework. We demonstrate that the state-of-the-art (SOTA) ArchDetect method Tsang et al. (2020b) can be derived from our approach: see Appendix D
COMPUTATIONAL CHALLENGES OF FISC
The first challenge is establishing the theoretical upper and lower bound of FISC for non-linear models characterized by uncertain, potentially non-convex loss functions. The second challenge arises from the NP-hard nature of searching the Rashomon set defined by hypothesis space Semenoval al. 2022); Hsu & Calmon 2022). We first show that is possible to find the boundary of FISC under certain assumptions and conditions in Sec. 3.1 aligning with previous work in this area (Fisher Dong & Rudin 2020). Then we present a sampling algorithm for more general scenarios
al. 2019
in Sec. 3.2
where model structures and loss functions might be uncertain. This approach enables us
to empirically approximate the FISC.
3.1 AN EXAMPLE APPLICATION OF FISC IN MULTILAYER PERCEPTRON (MIP)
Here, we consider a non-trivial MLP with a sigmoid activation function as an example. Let + be predictive model defined as
f(X) = T 1 1+e- +
(5)
The expected root mean squared error (RMSE) loss is E[L(f y) = 9)) mathematically equivalent to Ely f(X)] or (X) y). For simplicity, we consider the case Ely - f(X) to derive the condition on the mask m to characterize the Rashomon set. Without loss of generality, we assume a > 0 and a scaling on c by a. We further introduce an assumption that
f(X))² that is
E e e-m TTxxx
c S min
Ite Ite-m" T x ],E
1Te-mT xxx ]}. Invoking Eq. 2 this assumption allows us to derive
the following inequality
1- 1----e -e-
1 1 +e 3
E In
[ X)m" |
e- c- ce_8"xx) x
(6)
13- ce 8x B"
which gives the condition on m that characterises the mask-based Rashomon set. We refer to Appendix for its derivation.
3.2 GREEDY SEARCH FOR EMPIRICAL FISC LOWER AND UPPER BOUNDS
Unfortunately, due to the nonlinear nature of FIS, it is difficult to get a closed-form expression of the exact hypothetical bounds in general, especially when f and functions vary. Here we propose a general way to sample from the Rashomon set and approximate the range of FISs. The RMSE loss function and a pairwise feature interaction I = {i.j} are chosen to illustrate the idea, as RMSE will show the connection between method and ArchDetect in Archipelage.
3.2.1 SAMPLING FROM A MASK-BASHE RASHOMON SET
Instead of searching for all possible models, we sample models from the mask-based Rashomon set by concatenating multi-valued mask m [m,,7y, ... _mp} into a backbone f, resulting in new model Mm f o m. We can sample the Rashomon set by exploring different m such that
R( f. (M E F JE (X), v) < (XL(F(X),)) +e).
All models are characterized by and only masks that potentially affect the FIS are of interest. Redundant models such as the composition of identity functions can be ignored to save resources.
AA smaller means smaller Rashomon set with models closer the reference well- model f*.
4

---

Published as a conference paper at ICLR 2024
We illustrate this through one-layer mask, denoted by my with the component of feature z;:
(mr); : = m; i i I,
(8)
The FIS defined in Eq. 3 with the above setting can be rewritten as: (I5Z( (m(
TT
In practice, instead of calculating all possible I, we can calculate all main effects in the feature space (Mm; F (e((M,;)) first and then inversely calculate any order of interaction by decomposing my = lliez m..
3.2.2 GREEDY SEARCH ALGORITHM
To search possible (m.)... for features' main effects within the Rashomon set, we propose greedy search algorithm. For each feature's main effect sp: (M,,), set two p-length vectors of ones m,+ and m,- for upper bound search and for lower bound search, as both actions can increase loss values if the model f is well-trained. During the search process, learning rate to regulate the number of models generated for all dimensions. Higher learning rates lead reduced number of models. The searching process continues until the loss condition is not satisfied Mm, & F. This condition is imposed on the loss difference, denoted as O; = E[L(Mm, (X), (LL(f(X), y)).. Here, each mask corresponds to concatenated model, resulting in an associated loss value. Any mask during training meets Mm, E F and is one of the target models for feature æ, in R(c, FF). Finally, we can obtain model class with main effects [Mm, F (p((Mmm)) } 11=1- and calculate any order of feature interaction scores:
uniform
(RS:(R) = (FISz(Mmz) Mmzz E R] with Eq. 9 see the pseudocode in Algorithm
3.2.3 THE INFLUENCE OF JOINT EFFECTS ON THE BOUNDARY OF THE RASHOMON SET
When searching for all main effects of features in the Rashomon set through the loss difference, isolate the feature of interest and record the loss difference to ensure the Rashomon set condition. In theory, the joint effects of features should be the feature interaction. the sum of the main effects is less than c, then the joint effect would also be less than However, in reality, the presence of feature interaction causes the joint effects to surpass the boundary. This motivates us to study the influences of interaction and the extent to which it varies.
of main effects of features when there is
Algorithm Greedy search algorithm
Require: learning rate > 0 0
Ensure: L( m(X), ). #) L(J" +
m. 1 1 m 1"" 1 counter
VA
VR
loss + L(f (X))
while counter < 4 do if searching for the upper bound of m then m + Copy learning rate
Figure 2: Exploring feature interaction of function f = 24 + 2. + I; æÿ and f I, u,t +-r * in Rashomon set. The data points are sampled from both functions. Exploring Φ; and Φ) separately enables us to draw red circles representing Σie7 Φi where the radius, e.g. 0.1 can be poten- tially expressed by di = 0.01 and o,- 0.09: a detailed example is provided From inner to outer, the radii are 0. 0.2, and 0.3, respectively and the corresponding joint effects can be calculated by øt, and &i,jk from which we plot the blue curves. This can be achieved in 2D (left) and higher dimensions, e.g. 3D (right).
m' +
m 6. + L(f" f o m + (X.).)) lossref
if Φ. S then
m4 = m'
else
learning rate + learning rate x 0.1 counter + counter +
else searching for the lower bound of
m' + Copy(m.)
Appendix G
m' [s] < m
learning rate
6. L( r m -X.). lossref
if Φ. then
m m'
else
learning rate + learning rate x 0.1 counter counter +
5

---

Published as a conference paper at ICLR 2024
wow.oooeddddee drectan 455
1::0
26. 45.8. -5..0
43/6
103/6
(a) aoo and 21
(b) 2o and D22
(c) Zo and 223
(d) # and 222
Figure 3: Pairwise Halo plot from FISC applied to MLP trained from NID setting. The ground truth interaction defined function f(x) = Aa; ( U24 + ((;; (=t) +Z= 2y from left right are: true, true, false, true, respectively.
Halo plot A Halo plot fixes the sum of main effects within the boundary of the Rashomon set and visualizes the change in the joint effect. The fixed loss change formulated as: Σvez 6 (MM.. f) where & (0, é) to ensure the loss condition. Then map n-sphere with radius i and plot corresponding empirical loss Φt, (..j(Mma (: f) E[L(Mm.., (X), ), y)] [f y)]. from which we can directly visualize how the joint effect varies and affects the Rashomon set boundary. Any order of interaction Halo plot can be similarly derived and two simple examples are used to illustrate
the idea when [ 2 (2D) and [ 3 (3D) in Fig. 10 For higher order interaction visualizations, we discussed in Appendix F
Our framework effectively addresses the computational expense associated with calculating interactions, eliminating the need to calculate all pairs, especially in scenarios involving higher-order interactions. By performing a single-time search for all main effects and subsequently calculating the desired interactions, the proposed algorithm significantly reduces computational complexity. Specifically, a subset of FISC range (FISCr(R) bounded by can be approximated using representative points Σiez (MM J) = (. Collecting ordered pairs S {(o,z, o,y) (x, y) R, 0.1 z, y 0.9, z = 1} is sufficient to characterize pairwise interactions, which requires |S| x 2 calculations. Similarly, any order of interaction requires |S] 2|Z| model predictions.
Computational time
FISCr (RRSCCrr)aaaaa man]
4 EXPERIMENTS
In order to showcase the practical value of FISC, we have set two primary objectives and they are: (a) demonstrate the concept of FISC using a well-trained model that overlooks feature interactions, and (b) to examine the impact of feature interactions and explore the variation in FIS on accurate model predictions from our greedy sampling method. In the experiments below, we identify the main feature effects within the Rashomon set and calculate the FIS and FISC using representative models, followed by visualizations using Halo and swarm plots. Note that the meaning of axis Halo plots refers to Sec. 3.2.3
4.1 QUANTITATIVE COMPARISON AND EXTENSION
Synthetic Validation In order to validate the performance of FIS, we use SOTA from Tsang al. (2020b) generate ground truth feature interactions and apply our framework to the synthetic functions in Table in the same context. The interaction detection area under the curve (AUC) on these functions from different baseline methods provided Tables I and 2 As the relation to ArchDetect provided in Appendix D. our FIS is designed under certain conditions to detect the interactions of their proposed contexts and the results are as expected.
Exploring FISC from accurate MLP relying on "inaccurate" interactions The above accuracy is based on the knowledge of given functions, but in practice, we usually have a dataset with unknown functional relationships. This prompts to consider how account for the performance of a well-traines model that achieves high accuracy in making predictions, yer fails to preserve ground truth interactions with FISC. In this study, conduct general black box explanation task.

---

Published as a conference paper at ICLR 2024
Specifically, utilize a dataset generated by model F1 (assuming unknown to us) and train an MLP using the same neural interaction detection (NID) setting.
Table I: Functions with ground truth interactions, where x" [1, 1, L..., 1] R x' = [-1, -1.-.,.. -1] 24 is a key-value pair function. A(2; z) is defined as if index of belongs any keys of function 2, otherwise 11
IR40 and 2
T0 (x) E وド=
10
Ei=1 A(= + A(;
20
30
R, 40
j 2k=1
2
F2(x) (x) = A(:;
{at 1
30
;=
Fa(z) Aa; [2)
2ド= j= Tj
F((x) = A(a: (2f, 2 + A(x; {2t be
Table 2: Comparison of pairwise feature interaction detection
AUC. FIS is set in the same context to detect feature interactions.
Method
F1
F2
F3
F4
Two-way ANOVA
1.0
0.51
0.51
0.55
Neural Interaction Detection
0.94
0.52
0.48
0.56
Figure 4: Pairwise Halo: 'prior' and 'juvenile' (top); and triplewise Halo: 'prior', 'juve- nile' and 'charge' (bottom).
Shapley Interaction Index
1.0
0,50
0.50
0.51
Shapley Taylor Interaction Index
1.0
0.50
0.50
0.51
ArchDetect
1.0
1.0
1.0
1.0
FIS in the context (this work)
1.0
1.0
1.0
1.0
Table 3: MCR_ comparison between results from VIC (right column under each feature) and results in bold from greedy search algorithm (left column under each feature).
MCR
Age
Race
Prior
Gender
Juvenile
Charge
AVG(MCR)
1.113
1.021
0.969
1.013
1,330
1.048
0.951
1.009
1.367
1.038
1.065
1.004
MAX(MCR+)
1.141
1.060
1.007
1.037
1.372
1.116
0.986
1.032
1.390
1.097
1.095
1.021
MIN(MCR-)
1.072
0.987
0.919
0.994
1.123
0.988
0.900
0.989
1.278
0.992
1.004
0.990
(MCR+)-(MCR-)
0.068
0.073
0.088
0.043
0.249
0.129
0.086
0.043
0.112
0.105
0.091
0.031
Based on the ranking metrics AUC of the receiver operating characteristic curve (ROC), the MLP achieves an interaction detection accuracy of 56%. The trained MLP set as the reference model and we explore FISC in the Rashomon set with / = [0.2e, 0.4e, 0.6e, 0.8e, respectively, to observe that the range of each pairwise feature interaction exceeds the threshold extracted above; see Fig. 3 and Fig. aa). This provides insights into the role of FISC in bridging the gap between unknown ground truth function f and a well-trained model that fails to preserve true feature interactions. The potential loss of feature interaction preservation in model can be mitigated by other models in the Rashomon set. In this case, one can conclude that there always exists a model in the Rashomon set in which certain features interact.
4.2 RECIDIVISM PREDICTION
The use of the COMPAS score, reported by the ProPublica news organization, has been instrumental in examining its applicability in proprietary recidivism prediction, as evidenced by previous studies (Flores et al.] 2016 Rudin al. 2020). However, it has been demonstrated that interpreting pre-specified model has its limitations, as exemplified by the Rashomon set (Fisher et al. 2019 Dong Rudin, 2020). While the Rashomon set has been studied, the significance of feature interactions within the set has not yet been explored.
4.2.1 COMPARISON WITH EXISTING RELEVANT LITERATURE
Feature importance comparison-variee variable importance cloud (VIC) Given the absence of an identical task in existing literature, adopt the experiment settings utilized in prior research Dong &
"oodel class reliance (MCR) is the range of feature importance values (main effect in our definition) in the Rashomon set, used by Dong & Rudin (2020) and Fisher al. (2019)
7

---

Published as a conference paper at ICLR 2024
age va prior
1.0
roce geleee
1
Intraction Dart
the Samtones
1
1
praor vs jiveriecrmee
Fdmero-
a1
prlor currentchorge
28:4-20 20
0.000 0.001 0.002 0.003
D.0
0.000 0.000
0.007
0.000
0/002
0.004
0,005
0.005
0.020
FIR
(a) Swarm plot of FISC from MLP
(b) Swarm plot of FISC from logistic classifier
Figure 5: Illustration of FISC using swarm plots, where the black vertical line is the threshold and the yellow star is the reference FIS. Each point in the plot coloured based on loss value.
Rudin (2020) that aligns with our research focus. We use the dataset of 7,214 defendants in Broward County, Florida Washington (2018), the same logistic model, and set 0.05. To guarantee fair and unbiased comparison, we explore the Rashomon set using our greedy search algorithm and evaluate the feature importance using MCR. The resulting outcomes are presented in Table and it demonstrates the promising range of feature importance in general case. Although the feature importance not the focus of our study, it's still worthy of further work. Dong Dong & Rudin (2020) proposed the information compensation structure and stated that information loss in "prior" was most efficiently compensated by 'juvenile', but the underlying reason was missing. We find that VIC can not capture the higher-order information compensation structure.
Table 4: Comparison of FISC calculated from different sampling methods (partial). We use vs to denote the interaction pairs and best results are highlighted in bold.
AWP sampling
Greedy sampling (ours)
Random sampling
Interaction pairs
FISmin
FISmax
FISmin
FISmax
FISmin
F1Smaz
age vS race
-0.00026
0.000113
-0.00089
0.002069
-8.04E006
6.32E-06
age va prior
-0.00017
0.00014
-0.00122
0.001975
-1.02E-05
1.19E-05
age va gender
-0.00011
0.000135
-0,00178
0.001341
-1.99E-05
1.68E-05
age juvenilecrime
-0,00019
0.000139
-0,00167
0.002496
-1.31E-05
8.15E-06
age currentcharge
-0.0003
0.000351
-0.00257
0.00268
-1.24E-05
1,61E-05
FISC calculated from different sampling methods-AWP and random sampling Due to the limited availability of relevant sampling benchmarks in the current literature, established an evaluation framework by considering two accepacee methods as baselines: sampling from adversarial weight perturbations (AWP) Wu et al. (2020); Tsai et al. (202I) and sampling with different weight initialization seeds Li & Barnard (2022a) SSmenooa et al. (2019). Together with our introduced greedy algorithm, utilized these methods to sample models from the Rashomon set. We adopt an MLP as the reference model and keep it consistent among all methods. With set e = 0.05, we constructed the Rashomon set and calculated the corresponding FISC, partial results presented in Table. 4 It is clear that our approach explored broader range of FISC in comparison to the alternatives. Complete results are provided in Appendix [ with separate swarm plot visualizations.
Going beyond the literature The logistic model from Dong & Rudin (2200) is not optimal so we train and improve the model as in Appendix H We apply FISC to this model with settings t= [0.2e, 0.4e, 0.6e, 0.8e, e). and the results are shown in Fig. (). From the Fig. 5)b) we observe that the interaction of 'prior' and jjuvnile' or 'prior' and 'charge' can be extended to a similar extent. However, the compensation structure proposed by VIC fails to acknowledge the potential of the latter interaction. Additionally, we provide confirmation and explanation for their statement, as the latter case necessitates greater loss change (indicated by darker red color) compared to the
8

---

Published as a conference paper at ICLR 2024
Figure The utility of FISC an image classification task. From left to right: the original segmented image, top 5 main effect segmentations for accurate dog predictions by the transformer (coloured in blue), top 5 important interacted segmentations for accurate dog predictions by the transformer (coloured in red), and top 5 important interacted segmentations for accurate cat predictions by the transformer (coloured in purple).
former, thereby facilitating easier compensation. We present the pairwise interaction and triplewise interaction in Fig. 4, allowing us to observe the reliance on higher-order interactions. For instance, we observe that the decrease in reliance the feature interaction 'prior' and 'juvenile' (x-y plane) can be compensated by the interaction between "juvenile" and 'charge' (y-z plane) or "prior' and 'charge' (x-z plane). Likewise, the color scheme depicted in the figure conveys equivalent information.
4.3 FEATURE INTER ACTION IN IMAGE CLASSIFICATION WITH FISC
Over the past decade, large and pre-trained transformer architectures have achieved SOTA perfor- mance on a wide variety of tasks (Sarzynska- Wawer et al., 2021 Dosovitskiy et al. (021I. Here we aim to study how large-scal: model relies on feature interactions and how reasonably accurate predictions can differ in terms of their variances in FIS. We adopt the pre-trained SOTA vision
transformer ViT-B-16 developed in Dosovitskiy et al. (202T); Paszke et al.) (2019) with a top-5 accuracy of 95.32% two objects, a cat and a dog, sharing similar visual characteristicc et al. (2014). The primary predictions from the model classifying the image ranks as: Pekinese 0.49, Japanese spaniel 0.16, and Tiger car 0.06. The model identifies the image as resembling a dog more than a cat.
ImageNet and chose an image from the COCO dataset that encompasses
The image is segmented features by the quickshift method Vedaldi & Soatto (2008), following the protocol Tsang et al. (2020b) shown in Fig. a). We use FISC to explain the interaction effects. The top 5 important segmentations are coloured blue in Fig. [])), where importance increases according to colour lightness. The most important part of the image is the forehead of the dog. A similar observation is reported in (Tang al., (2020b). Considering the joint effect resulting in the dog classification, the top 5 crucial pairwise interactions are given in Fig. [(c). Notably, the lightest component interacts with other components more than once. Results shows that the model relies heavily on single forehead segmentation to predict a dog, but this segmentation is disappearing in the top 5 features for predicting a dog relying on interactions. If our objective is to classify the image as a cat, our findings suggest in Fig. d) that the model would rely on both the dog and cat segmentations.
5 DISCUSSION
The motivation behind the development of FIS and FISC is to explain feature interaction in a model class, rather than focusing specific model. In this study, we highlight that feature interaction can vary significantly across models with similar performance. This work initializes the exploration of feature interaction in Rashomon set. By leveraging the FISC, we approach the true feature interaction from set of well-trained models. The variation of FIS allows for the explanation of diverse models that rely on distinct feature interactions and this property holds significant potential for selecting models that exhibit the desired feature interaction dependency, illustrated in synthetic validation, recidivism prediction and image classification with promising results. Further exploration of feature interaction with other ways of characterizing the Rashomon set is subject to future work. Additionally, we have developed a visualization tool that effectively illustrates the joint effect on the Rashomon exploration and adopt swarm plots to depict the variability of FISs in model class. These tools has the potential to inspire future investigations in various domains. Our code is available at fihtpps: / hub. com/ chicchooiioo I.i Letoooiii/geee / zed wwwmon_setttteeeeeeee

---

Published as a conference paper at ICLR 2024
REFERENCES
Jacob Bien, Jonathan Taylor, and Robert Tibshirani. A lasso for hicrarchical interactions. Annals of statistics, 1113:11111, 2013.
Tianyu Cui, Pekka Marttinen, and Samuel Kaski. Learning global pairwise interactions with Bayesian neural networks. ECAI, 2020.
Anupam Datta, Shayak Sen, and Yair Zick. Algorithmic transparency via quantitative input influence: Theory and experiments with learning systems. In 2016 IEEE symposium on security and privacy (SP), pp. 598-617. IEEE, 2016.
Jiayun Dong and Cynthia Rudin. Exploring the cloud of variable importance for the set of all good models. Nature Machine Intelligence, (112)880-8222 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition Scale. ICLR, 2021.
Christopher Eames and M Saiful Islam. Ion intercalation into wo-ddieensional transition- carbides: global screening for new high-capacity battery materials. Journal of the American Chemical Society, 136(46):16270 16286-)66627--627666 2014. doi: 10.1021/ja508144..
Aaron Fisher, Cynthia Rudin, and Francesca Dominici. All models are Wrong, but Many are Useful: Learning Variable's Importance by Studying an Entire Class of Prediction Models Simultancously. Mach. Learn. Res., 20(177):1-81, 2019.
Ronald Aylmer Fisher. Statistical methods for research workers. Springer, 1992.
Anthony W Flores, Kristin Bechtel, and Christopher T Lowenkamp. False positives, False negatives, and False analyses: A rejoinder to "Machine Bias: There's 's software used across the country to predict future criminals. and it's biased against blacks". Fed. Probation, 80:38, 2016.
Jerome H Friedman and Bogdan Popescu. Predictive learning via rule ensembles. The annals of applied statistics, pp. 916-954, 2008.
Yury Gogotsi and Babak Anasori. The rise of MXenes. ACN Nano, 23(8):8491-849) 2019. doi: 10.1021/acsmano.9600
Yury Gogotsi and Qing Huang. Mxenes: two-dimensional building blocks for future materials and devices, 2021.
Michel Grabisch and Mare Roubens. An axiomatic approach to the concept interaction among players in cooperative games. International Journal of game theory, 28:547-565, 1999.
Peyton Greenside, Tyler Shimko, Polly Fordyce, and Anshul Kundaje. Discovering epistatic feature interactions from neural network models of regulatory DNA sequences. Bioinformatics, 34(17): i629-i637, 2018.
Brandon M Greenwell, Bradley C Boehmke, and Andrew J McCarthy. A simple and effective model-based variable importance measure. arXiv preprint arXiv:1805.0441 04755, 2018.
Baptiste Gregorutti, Bertrand Michel, and Philippe Saint-Pierre. Correlation and variable importance in random forests. Statistics and Computing. 27:659-677 678, 2017.
Huifeng Guo, Ruiming TANG, Yunming Ye, Zhenguo Li, and Xiuqiang He. DeepFM: Factorization- Machine based Neural Network for CTR prediction. In Proceedings the Twenty-Sixth In- ternational Joint Conference on Artificial Intelligence, IJCAI-17, pp- 1725-1731, 2017. doi: 10.24963/jcca.221117733 uttpstts:///oo: org 1t..g///...000 wttp://.............22..22
Hsiang Hsu and Flavio Calmon. Rashomon capacity: A metric for predictive multiplicity in classifi- cation. Advances in Neural Information Processing Systems, 15:28988-29001, 2022.
10

---

Published as a conference paper at ICLR 2024
Joseph D Janizek, Pascal Sturmfels, and Su-In Lee. Explaining explanations: Axiomatic feature interactions for deep networks. The Journal of Machine Learning Research, 22(1)):687747700 2021.
Sichao Li and Amanda Barnard. Variance tolerance factors for interpreting neural networks, arXiv preprint arXiv:2209. 138888, 2022a.
Sichao Li and Amanda S Barnard. Inverse design of MXenes for high-capacity energy storage materials using multi-target machine learning. Chemistry of Materials, 34(11):4964 4974, 2022b.
1o0: wttpssccheemmatee22000000000
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings. Parl V 13, pp. 740-755. Springer, 2014.
Yin Lou, Rich Caruana, Johannes Gehrke, and Giles Hooker. Accurate intelligible models with pairwise interactions. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 623-631, 2013.
John Mandel. Non-additivity two-way analysis of variance. Journal of the American Statistical Association, 56(296):878 8888888 1961.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lever, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf. Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems 32, pp.
8024-8035. Curran Associates, Inc., 2019. URL http:://paeers..nuriiss...oo hec.://00..LLLtpppppaeeeeseeeee 9015 an- ar ott.apaawa.aaaa....reorrrorreorr hww.aaannnnnnnnaairr.aaaa Library.
style
pdf
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence, 1(5):206 22120,6111,11 2019.
Cynthia Rudin, Caroline Wang, and Beau Coker. The age of secrecy and unfairness in recidivism prediction. Harvard Data Science Review, 2(1):1, 2020.
Justyna Sarzynska-Wave Wawer, Aleksander Wawer, Aleksandra Pawlak, Julia Szymanowska, Izabela Stefaniak, Michal Jarkiewicz, and Lukasz Okruszek. Detecting formal thought disorder by deep contextualized word representations. Psychiatry Research, 304:1141355 2021.
Lesia Semenova, Cynthia Rudin, and Ronald Parr. A study in rashomon curves and volumes: A new perspective on generalization and model simplicity in machine learning. arXiv preprint arXiv: 017550000015,,,.... 4, 2019.
Lesia Semenova, Cynthia Rudin, and Ronald Parr. On the existence of simpler machine learning models. In 2022 ACM Conference on Fairness, Accountability, and Transparency, pp. 1827-1858, 2022.
Chandan Singh, W. James Murdoch, and Bin Yu. Hierarchical interpretations for neural network predictions. In International Conference on Learning Representations, 2019. URL https
//openreview net./ www.:nne/fofommmeeeee feeee/formmmmddsseorooettt
Daria Sorokina, Rich Caruana, Mirek Riedewald, and Daniel Fink. Detecting statistical interactions with additive groves of trees. In Proceedings of the 25th international conference Machine learning, pp. 1000-1007, 2008
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pp. 3319-3328, PMLR, 2017.
Mukund Sundararajan, Kedar Dhamdhere, and Ashish Agarwal. The Shapley Taylor interaction index. In International conference machine learning, pp.
PMLR, 2020.
11

---

Published as a conference paper at ICLR 2024
Yu-Lin Tsai, Chia- Yi Hsu, Chia-Mu Yu, and Pin- Yu Chen. Formalizing generalization and adversarial robustness of neural networks to weight perturbations. Advances in Neural Information Processing Systems, 14:19692-197044 2021.
Michael Tsang, Dehua Cheng. Hanpeng Liu, Xue Feng, Eric Zhou, and Yan Liu. Feature interaction
interpretability: A case for explaining ad-recommendatioo systems via neural interaction detection. In Internationaal Conference on Learning Representations, 2020a. URL /
net Ltte/Eoounn.1d-Bkghhhe
Michael Tsang, Sirisha Rambhatla, and Yan Liu. How does this interaction affect me? Interpretable attribution for feature interactions. Advances neural information processing systems, 33:6147-
2020b.
Michael Tsang, Dehua Cheng, and Yan Liu. Detecting Statistical Interactions from Neural Network Weights. ICLR, 2021.
Andrea Vedaldi and Stefano Soatto. Quick shift and kernel methods for mode seeking, In Computer EccionEECCV 2008: 10th European Conference on Computer Vision, Marseille, France, October 12-18, 2008, Proceedings, Part IV 10, pp. 705-718. Springer, 2008
Zihao Wang, Xihui Liu, Hongsheng Li, Lu Sheng, Junjie Yan, Xiaogang Wang, and Jing Shao. Camp: Cross-Modal Adaptive Message Passing for Text-Image Retrieval. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), October 2019.
Anne I. Washington. How argue with algorithm: Lessons from the compas debate.
Colo. Tech. LJ 17:131, 2018.
Dongxian Wu, Shu-Tar Xia, and Yisen Wang. Adversarial weight perturbation helps robust general- ization. Advances in Neural Information Processing Systems, 33:2958-2969, 2020.
Ruoqing Zhu, Donglin Zeng, and Michael Kosorok. Reinforcement learning trees. Journal of the American Statistical Association, 1106512):1770-1777 2015.
12

---

Published as a conference paper at ICLR 2024
A MORE RELATED WORKS
An analysis of variance (ANOVA) test is employed evaluate the significance of interaction terms in an additive model that includes all possible pairwise interactions Fisher (1992); ; Mandel (1911). The test examines the variance within each group compared the variance between the groups and determines whether any differences observed in the means of the groups are likely due to chance or to significant factor. H-statistics measure the strength of pairwise interactions based on the concept of Partial Dependence Function (PDF), which represents the relationship between a target variable and a specific feature while holding other features constant. The H-statistics can be calculated by comparing the PDF of the target feature with the PDF of the target feature conditioned on the presence of another feature. The method has been extended to feature importance and feature interaction detection Friedman & Popescu (2008), ; GGrenwwel et al. (2018). Sorokina et al. (Sorokina et al. (2008) proposed grove-based method called Additive Groves (AG) that extends the concept of decision tree ensembles to incorporate additive structure in the model. The method involves training individual trees to capture additive effects of the features on the target variable and then combining them to create powerful ensemble model. By incorporating additivity into the model, Additive Groves can capture more complex relationships among features. In interaction detection, lasso-based methods are also widely used due that they can quickly select interactions. One can construct an additive model with many different interaction terms and let lasso shrink the coefficients of unimportant terms to zero Bien et al. (2013).
Recently, more neural network- -based methods are proposed to detect feature interactions. Bayesian Neural Networks (BNN et al. (2020) is to evaluate pairs of features with significant second-order derivatives at the input. By analyzing the posterior distribution of the model parameters, BNN can identify which pairs of features have significant interaction effect. Specifically, the method evaluates the pairwise interaction by computing the difference between the posterior distributions of the model predictions with and without the interaction term. The Neural Interaction Detection (NID) method detects statistical interactions between features by examining the weight matrices of feed-forward neural networks Taang et al. (02T). Specifically, interactions are determined by finding a cutoff' on the ranking using special form of the generalized additive model. Deep Feature Interaction Maps detect interactions between two features by calculating the change in the attribution of one feature incurred by changing the value of the second Greenside et al. (2018). It calculates the interaction between pairs of features by comparing the attribution maps generated when each feature is varied individually to the maps generated when both features are varied simultaneously. Singh et. al. [Singh] et al. (219)) generalized Contextual Decomposition Greenwell et al. (2018) to explain interactions for forarr and convolutional architectures.
Most recently. several works have been proposed to attribute predictions to feature interactions The Chapley-Taylor Interaction Index measures the contribution of pairwise feature interactions in machine learning models by combining the Shapley value and the Taylor expansion to estimate the interaction effects. The Shapley value quantifies the marginal contribution of each feature, while the Taylor expansion approximates the prediction function considering main effects and pairwise interactions Grabisch & Roubens| [199);; Sundararajan et al. (020). Integrated Hessians (IH) Janizek] et al. 12221) is method used assess feature interactions in deep neural networks. It involves computing the integrated Hessian matrix and integrating the Hessian matrix over the input data, IH captures both the local and global curvature information of the loss landscape by analyzing the eigenvalues and eigenvectors of the integrated Hessian matrix. The Archipelago [Tsang et al. (0206)) architecture specifically designed for attributing feature interactions in machine learning models. It offers a framework based on mixed partial derivatives for identifying and quantifying the contributions of interactions between features. which consists of an interaction attribution method, ArchAttri,ute, and an interaction detector, ArchDetect.
These above methods show excellent accuracy in feature interaction predictions from pre-specified model. However, as more researchers advocate exploring a set of equally good models, it is worthy discovering feature interactions in a model class, defined as the Rashomon set. The Rashomon set is named after Akira Kurosawa's 's film "Rashomon", and it refers to a collection of diverse but equally plausible models that achieve comparable performance on a given task or dataset. Fisher Fisher et al. (2019) introduces the concept of model class reliance (MCR) the variability of Variable Importance (VI) values across a well perfforrnon model class known as the Rashomon set. MCR provides a more comprehensive and nuanced understanding of feature
measure that captures captures
13

---

Published as a conference paper at ICLR 2024
importance by considering the range of VI values obtained from multiple prediction models within the class. Following their work, Dong Dong & Rudin (2020) explores the cloud of variable importance, referred as VIC for the set of all good models and provides concrete examples in linear regression and logistic regression. Another paper] Li & Barnard (2022a) proposed a post-hoc method, variance tolerance factor (VTF) to interpret set of neural networks by greedy searching all possible neural networks with certain conditions.
B UNVEILING THE MASK-BASEL SAMPLING RASHOMON SET
As stated that it's always easier to find an accurate but complex model, it refers to a model with more complex structures. It's safely claimed that concatenating layers into backbone model increases the complexity and therefore we logically demonstrated that such a model with more mask layers can achieve similar or better performance. Then we applied propositional logic to prove, for any model in the Rashomon set, we can find an alternative by adding mask layers into the reference model (same as the backhone model). Our aim can be mathematically represented as: vf E F (B(L(f(X) y)] (3m (E(L(f" m X)) y) = [[[(((X) (X). y))) with details below:
1
A E E +
EL( y) +)) v (BFL(LL())) 3m (I (E
- -
2
E (LL(
3
BEL( +
4
(XIL("(X)))) -m v))
(EI m()) v)1 (L b)))
5
E[L(F(X))
6
([L(F((X) f)1 (X), cc)
--E, 3, 5
7
3m (f ml X)) v)) < ([
+ E, 4,5
8
m'
E| ml (X)
9
E[)(f( v) B[ o x)) +e
R, 6,8
10
E[)(f(XX), v) ([ +
Logic,
11
EmE[L(f (x) (X))
3-1, 9, 10
12
3m (E[L(f* ml x), BL(f
3-E, 7, 8-11
13
[f v)) 3m (EL(F" o mm ()),y) 2 =(
--1, 5-12
14
Wf E F (B (3m (EiL(f -
V-I, 3, 13
C ADDITIONAL PROOF OF CONCEPT
C.1 PROBLEM SETTING
To illustrate the idea, designed additional experiment in the synthetic dataset and solved the quadratic problem. The problem is to train a model to solve the quadratic equation ax + br + c = where the variables b, and inputs and ω and 22 the outputs. Based on mathematical
principles, two crror-free models can be found as I b w0
2a
Given the function, we randomly sampled 12,000 data points from the uniform distribution a, b, (4(0, 1) with fixed seed as input and calculated outputs accordingly. It's noted that the outputs might be complex numbers. The trainttsst/alidattinn set split 1.0.. The regression problem can be fit using different types models. Here we applied a traditional artificial neural network and achieved MSE 0.3353 in the training set.
14

---

Published as a conference paper at ICLR 2024
predict ons
ground truth
8000
4000
-4-
B0000
-200
-250
-200
-150
-100
-50
Figure 7: Left: Learning curve. Right: Prediction VS Ground Truth.
C.2 GROUND TRUTH INTERACTIONS
The given function f - -hvi-taar foe can be seen as an ideal model with (f = 0. which
2a
corresponds to a true feature interaction. The expected FIS of a, can be calculated as follows:
FISa,b(f)) =Pab(f") =R[TU/ (X (ax)))) - EIT(( (a))) (X
+
(10)
"
( -x))))))) -y) (x (o))) y)² (x) - y)²
Where is the number of samples and interaction between b, c and a, e can be calculated in the same way. The interaction values are absolute to indicate the strength. The interaction values are absolute indicate the strength, summarized in the Table.
Table 5: Ground truth of feature importance and interaction
Pa(l5)
(p((f*)
(p([())
(aat(()))
(fa)((")
(p*)(((*))
116.4602
24.2023
0.2924
21.2866
1.3813
0.5945
C.3 APPROXIMATED FISC
Now, assume that do have an optimal model in the real-world and our objective is train a model to fit the data, the regression problem is fitted using a simple neural network and achieved The learning curve and prediction versus ground truth are illustrated in Fig. 7 After training the model, we applied the FISC to it and plotted the swarm plots, which are Each point the swarm plot represents an FIS value from a specific model.
MSE=0.33 in the test
provided in Fig. g
when we consider the lefumost point among the feature pair a, b, it implies that there exists model with lower FIS for a, b compared to a, c. This means that there was model trained that happened to have lower FIS for b than b, using the same dataset with promising loss. This observation contradicts the truth interaction calculated previously. However, through the analysis of FISC, we found that the range of FIS values for each feature pair covers the truth interaction. By examining the statistics of the results, we can approach true feature interactions, which will be the focus of our further work.
15

---

Published as a conference paper at ICLR 2024
632
Intrrctton Pars
Featume
10
x1
120
Feature importance in the Rashomon
(h) FIS the Rashomon
Figure 8: Results from a set of models sampled from the well-trained neural network.
D RELATION TO ARCHDETECT
The framework ArchDetect Tsang et al. (202bb) uses h,hy
to detect the interaction existence
defined as
8 = fa ((.s)) ((d(ss)) a(tis))
- -((1() + + + - f(a'y) + F()) + xx()))
(11)
based on mixed partial derivatives, where f is a black box model, x² represents the sample we want to explain and x' represents neutral reference sample, which often zero vectors. If > then interaction exists, otherwise not. The author claimed 100% accuracy in synthetic dataset and defined the feature interaction strength as the square of kch,, Here we show that for any model Mm in the model class, the FIS value is equivalent to å under conditions that the loss is set as RMSE. In our setting, for any model Mm. the expected RMSE loss E L (X), y)] E B[V - Mm, (X) For simplicity, we consider F Mm (X) (meaning assuming the model approximate the true solutions from either below above). For I {i,j} and V.M.,,, R(e,g, following the definition, we can derive
FTSr(Mmz) (R(M..., (X ((6))) + B[Mm.., (X)
E[Mm., (X (Xx1)) ElMm., (X))))).
(12)
which is mathematically equivalent to in view of the decomposition of the interaction into four terms.
E DERIVATION OF THE UPPER AND LOWER BOUND OF FISC IN MLP
E.0.1 CHARACTERIZING THE MASK-BASED RASHOMON SET
We first apply the mask m [m,, m2, ..... mp) to the reference model Mm = = m. To characterize the mask-based Rashomon set, the goal is to find the conditions for m such that:
(L < E(L(((X)) +
(13)
After applying the above settings, we can rewrite Eq. as:
E y
1+e X'X + 6) - e E a T 1 1 m
x + 8'x
Ely - aT
1 1+e-3"x + +e.
te.
16

---

Published as a conference paper at ICLR 2024
Without loss of generality, we assume
and scaling by This can be simplified to 1
1
E
E
E
+c.
(14)
+ e ST
e m n"" x
+ e
S" x
With further assumption that S min lec x²xxx duced to
EI
m xxx l}, this incquality is re-
live
E In (e- - + ce -0-
1-c- cc (
x xlm In e- - ce 3 x )} (15)
1+ + cc (
which gives the condition m that characterises the mask-based Rashomon set.
E.0.2 CALCULATING THE EXPECTED FISC LOWER AND UPPER BOUNDS
With the above setting, we calculate the FIS:
1
FMS)(MM) = Ely- (a²
+ b)] Fy (a
1te mm 3' + +b]]-
1 e m m 8''''''X
X
1
(Ely (aT
+ +b] Flyy (aT
1te m - p' + +b]]+
+ m""B"X, X
Ely y - (aT 1te m X, b)] Ely (a T
1 1te m x + b]]). (16) T-emmn
To find the minimum and maximum values of FIS, apply the condition (domain) Eq. 6 and the critical point (when m = m, such that F1Sr(Mm! 0) of the function (M.) We denote
3m
m2 (
1 --c-ce-$0 -0- x
1 10-1xx+tee 3 x e- -c-ee-8xx) (17)
1+6 +
m1
3 In x + + (xxx)
In
Now, one can denote
FISmin == inf FIFIF
min min(F FISI FIFIIIIMMM... F1I11FFSMM (Mmm)
(18)
FIS := sup FISt (Mm) = (F FIS FIS2(M (Mm.)).
m
The range of FISC then characterized as (FISCI(R)... min: FISCI(R) max1 [FIS FISmus)]
Remark. The exact critical point m = m, such that FIS²(Mm) 0 is difficult to obtain as requires solutions involving polynomial of degree seven and exponeniaa//ogariinnn fanctions. However, this can be approximately by root-finding algorithms such as Newton's 's method. Another approximation is to first-order Taylor expansion of FISz(Mm at 1. The analytical expression is still extremely complicated, posing difficulties in finding extreme values of FIS, so we obtain this critical point by a root-finding algorithm. We present a generic method in Sec. 3.2
F HIGHER ORDER INTERACTION VISUALIZATION DISCUSSION
In our paper, we provide visualizations of pairwise interaction in halo plot and swarm plot, and triplewise interaction in halo plot. These two-level interactions are the most studied in the literature and higher-order interactions are sometimes considered redundant in terms of attribution (Bien et al. 2013 Dong Rudin 2020). Theoretically, halo plot and swarm plot can visualize any order of interactions and swarm plots can visualize any order of interactions empirically. However, visualizing higher-dimensional spaces (>3) can be challenging due to the limitations of human perception, which makes directly visualizing higher-order interactions using halo plots not feasible.
We acknowledge this limitation and offer suggestions for users who insist visualizing higher-order interactions (>3) using halo plots. To visualize higher-order interactions (>3) in halo plots, we can apply some commonly used higher-order visualization methods, e.g., encoding color information as 4th dimension and applying dimensionality reduction. To demonstrate this, we provided an example of visualizing 4-way interactions. The experiment based on the recidivism prediction in Sec. 4 On the existence of 3-way interaction visualization, we encoded color as 4th feature, where lighter colors indicate the larger interactions, as shown in Fig. [
'A smaller means smaller Rashomon set with models closer the reference well-trained model fref-
17

---

Published as a conference paper at ICLR 2024
0.12
0.10
0.05
0.00
0.00
0.08
0.05
-0.05
0.06
-0.0
-0.10
0.10
0.10
0.04
0.05
0.00
-0.10
-0.05
-0.05
-0.05
0.02
0.00
0.00
0.05
0.05
0.10
0.10
Figure 9: Left: 3-D halo plot. Right: 4-D halo plot
G ILLUSTRATION OF HALO PLOT WITH CONCRETE EXAMPLE
Following the setting in the main text that function is f + Iy 22 # and 0.1. We choose a scenario when the main effect of feature x; is ob = 0.01 and the main effect of feature is Φb 0.09. Given 6i 0.01, can derive the following with Mean Squared Error (MSH) loss function
Φ = L(f o m. (X) L(f(X))
= (y - (mar + #; ma3 * xz)))² - (y- (x. 23 + v4 xz)))
0.01.
(19)
Similarly,
Φj = L(f aom -m,(X)-L(fXX)
(y (x; + m,J + myx;))² -I (Ii + +x;
0.09.
(20)
Considering variables sampling from the normal distribution N(v, o²) ;ı ~ Nuu, o²) and y is ground truth from the function. We can determine two possible values for m,, namely 0.95
and 1.05, as well as two possible values for mj. namely 0.85 and 1.15. From these options, we can identify four potential sets that guarantee the sum of main effects be 0.1: (m; my) (0.95, 0.85), (0.95, 1.15), (1.05, 0.85), (1.05, 1.15). Next, we proceed to compute Φi.j (y (m, + +m, + m, * my (y- (z, + x. + * +z² using the aforementioned sets. These calculations allow to generate four values of ē, which can be plotted along eircle with radius of Following the above procedure, we collected ordered pairs [(0 * I, Oj y) (I,y) E R. 0.1 S
ar y 0.9, x + y = 1} and plotted 36 points, shown in Fig. 10
H OPTIMIZING THE REGRESSOR FROM VIC
We downloaded the public code from https:/ htt:://zendo..o www.:oo..org/reccr///5555555 IIIIHVHZB2HI and extracted model parameters from the logistic model in VIC. We realized that the model can be optimized further and showed the comparison between two models loss vs individual masks of each feature in Fig. 11
I COMPLETE RESULTS FROM THREE SAMPLING METHODS
Due to the page limitation, here provided the complete results from three sampling methods: sampling from adversarial weight perturbations (AWP), sampling with different weight initialization seeds and greedy search approach, summarized in Table 6 To provide more comprehensive
18

---

Published as a conference paper at ICLR 2024
0.50
0.25
0.25
-0.50-
0.75
-1.00
-1.0
-0.5
0.0
0.5
Figure 10: Exploring feature interaction of function f = x, + 2j + 24 in the Rashomon set. From inner to outer, the radii 0.1, 0.2, 0.3, 0.4, and 0.5, respectively.
race
0.00
0.52
cumerchh
2.0
Jacce Anceleeiine damentcherge
0.3
088
2.00
-041
2.02
+0.02
-0.23
-0.44
Figure 11: Left: greedy search algorithm applied in a logistic classifier extracted from VIC Dong & Rudin (2020). Right: greedy algorithm applied in another optimal logistic classifier, The x-axis refers individual masks while the y-axis represents the loss change from the base model.
understanding of FISC, swarm plots are displayed below. In these plots, each data point represents a FIS derived from the sampled models, without absolute value transformation, as this comparison more directly illustrates the distribution of FISC across different sampling methods.
J AN APPLICATION IN REAL-WORLD DATASET
To illustrate the usage of FISC, we applied our method to MXenes, an early transition metal carbides with two-dimensional (2D) structures exhibiting metallic conductivity and hydrophilicity. These materials are denoted by 1,, with M representing an early transition metal such as Sc or Ti, X signifying either Carbon (C) Nitrogen (N) and functional groups T (e.g., O, F, OH) for surface termination Gogotsi & Anasori 2019); Gogotsi & Huang (202I). By intercalating different ions and molecules like L and K MXenes electrochemical attributes such as voltage, induced charge and capacity can be modulated, serving as potential battery material.
We used dataset that contains 360 compounds intercalated with Li+ Na+, K+, and Mg ions from Eames & Islam (2044):L. Li & Barnard (2022b). We represented the dataset by their categories, for instance, category encoded by 0-3 to describe Li, Na, K, and Mg, respectively, aiming to discover the interactions between structures. The dataset is split 90/10 for training/testing. We trained 3 MLPs for predicting their 3 properties: voltage, induced charge and capacity, individually. The R² for each reference model testing is reported in Figs. [5 16 17 We searched the Rashomon set based on the well-trained reference models with e = 0.05 and corresponding swarm plots are provided in
Figs. 15
16 17
19

---

Published as a conference paper at ICLR 2024
Table Comparison of FISC calculated from different sampling methods (complete). We use vs to denote the interaction pairs and best results are highlighted in bold.
AWP sampling
Greedy sampling
Random sampling
Interaction pairs
FISmin
F15maz
PISman
FISmaz
FISmin
FISmaz
age race
-0.00026
0.000113 13
-0.00099
0,002069
-8.4E-006
6.32E-06
age prior
-0.00017
0.00014
-0.00122
0.001975
-.02E-05
1.19E-05
age gender
-0.00011
0.000135
-0.00178
0.001341
-1.99--0
1.68E-05
age vs juvenilecrime
+0.00199
0.000139
-0.00167
0.002496
-..11E05
8.15E-06
currentcharge
+0.003
0.000351
-0.00257
0.00268
-1.24E-05
1.61E-05
race prior
+0,00299
0.000225
-0.00151
0.006728
-1.23E-05
1.811-05
gender
-0.0002
0.000158
-0.00224
0.000606
-9,33-006
1.34F-05
race vs juvenilecrime
-0.00019
0.000189
-0.00851
0.001587
2.13F-00
1.13F-05
race va curentcharge prior vs gender prior juvenilecrime prior vs currentcharge gender vs juvenilecrime
-0.00034
0.00027
-0,00719
0.002191
1.87E-05
1.4EE-05
-0,00018
0.000191
-0,0006
0.002526
1.58E-05
2.50E-05
-0.00019
0.00022
-0,0068
0.002592
2.06E-05
1.91E-05
-0.00029
0.00022
-0.00423
0.002219
2.233-05
2.16E-05
-0.00012
0.000148
-0.00828
0.000754
1.244-05
1.04E-05
gender currentcharge
-0.00035
0.000211
-0,01362
0.003708
22.11-05
1.61E-05
juvenilecrime currentcharge
-0.00022
0.000498
-0.00917
0.000693
-2.14E-05
2.28E-05
age
age vs prlor
gurdder
age vo curreiicaarge
prior
race vs jivenlecttme
rooe oureenchhare
gaider
pricr vn javeniecrime
4rm M- Rpe..eg flaggt-e
prtor currentcharge
pavorioorimoo
pender camenictunge
jvent acrmme curenicharge
F18
Figure 12: FISC from our greedy sampling method
20

---

Published as a conference paper at ICLR 2024
Age
Ago
age ve gerster
age vo cureettcaarge
FacE prior
race
tace piveritecmme
surenttsharge
prior vos ganater
priar javerilccrine
pror sumentcharge
gender javer alouine
jonerlosiim curantehurge
300000
0,0002
Figure 13: FISC from AWP sampling method
Age
agn
Age vo gendter
juremtecrimee
age vs cumentichrrge
race prior
FaOE javerilectime
surenttcher
prior gendir
prior Javenilocrme
prir enbrentgorggg
gender vs juvenilecrime
surentchaarg
joveileaiin surenichagg
TerE
Figure 14: FISC from random sampling method
21

---

Published as a conference paper at ICLR 2024
VS
-0.0010
VS
0..0008
Loss the Rashomon set
Interaction Pairs
M vs Z
-..0006
Tvs
0.0004
T vs Z
f Expoigg
0.0002
0.000
0.001
0 0002
000
000
0.005
0. 0.006
FIS
Figure 15: FISC from charge prediction. The R² of the reference model (depicted by the yellow star) achieves 0.98 on the testing set.
The results reveal noteworthy observation: FIS exhibits a broad range when category Z is involved in interactions during the prediction of induced charge. This aligns with our common knowledge, given that ions, which form the category Z, are the cause of induced charge. These insightful results offer more comprehensive understanding of feature interactions within the Rashomon set and potentially guide researchers in making informed decisions for future research.
22

---

Published as a conference paper at ICLR 2024
M
0.0035
M vs X
0.0030
Loss the Rashomon
Interaction Pairs
M
0.0025
0.0020
vs X
NBr fp
0.0015
0.0010
X vs Z
0.0005
0.000
0.001
0. 0.002
0. 0.003
0.004
0.005
0.006
FIS
Figure 16: FISC from voltage prediction. The R2 of the reference model (depicted by the yellow star) achieves 0.71 on the testing set.
M
#
0.0035
M vs
0.0030
Interaction Pairs
0.0025
Loss the Rashomon
M
0.0020
Tvs X
0.0015
0.0010
X vs Z
0.0005
0.000
0.002
0.004
0.006
0.008
0.010
FIS
Figure 17: FISC from capacity prediction. The R2 of the reference model (depicted by the yellow star) achieves 0.90 on the testing set.
23


Reviews:
[{"review_id": "Bc7tafAilj", "reviewer": "Reviewer_CnMm", "summary": "The paper proposes feature interaction scores (FIS), where it characterizes feature interactions based on a model class (which achieve similar performance for a task \"Rashomon Set\") instead of a single model only. Complementary to this, the paper introduces an algorithm that can be used to compute FIS based on Rashomon sets, and a \"Halo\" plot to visualize the said feature interactions.\n\nOverall, the paper is well written and makes a significant contribution. I have a list of clarifying questions/comments. I would be able to make better analysis of the results based on the discussion. I look forward to author's responses.", "questions": "1. In the example where the paper considers roots of a quadractic equation, it is unclear what \"the input variables a, b, and c exhibit different feature interactions in these two models\" means. Are the \"two models\" the two roots?  Further explanation is required, it is confusing what the readers are asked to infer.\n\n2. Consider adding a sentence to connect eq (6) to (2), currently it is a sentence fragment. \n\n3. \\mathcal{M} is not defined in (7) (its first use).\n\n4. How is eq (6) connected to (7)? It seems that the write-up connecting these two was skipped.\n\n5. Is there a reason to define and call the function in (5) as g(.)? Since the write-up so far uses f(.) for the predictive fuction, can the paper use f(.) instead of g(.)? The rationale is that in (8) the authors define the Rashomon set using g, while (1) uses f. Further, this change means that \\mathcal{M} can be defined in (1), and can improve the flow of the paper.\n\n6. What does the paper mean by \"then inversely calculate any order of interaction by decomposing...\" at the end of 3.2.1.\n\n7. Notation \\mathbbm{1}{i=1}^{p} will be more appropriate than $(1)_{i=1}^{p}$ in Algorithm 1. But more importantly, it is unclear how m+ is different from m-. They seem like a vector of size p of ones, i.e. both are the same. If so, then what is the difference between m_i+ and m_i-?\n\n8. Algorithm 1 is somewhat difficult to follow. Since this is the key contribution, can the authors explain the role of various terms such as \"learning rate\", \\phi_s etc. are?\nEditing the algorithm to have comments, or updating 3.2.2 to reflect the terms in Alg. 1, can help. For instance, I am still unclear how the computations are taking place over multiple models, and if this is being influenced by \"learning rate\" somehow. \n\n9. Can the authors explain what they mean by \"In theory, the joint effects of features should not exceed the boundary when there is no feature interaction.\"? Mathematical formulation can help. \n\n10. Which functions is depicted in Fig. 3? We see x_0, x_1, and x_2, but it is unclear what was the original functional form.\n\n11. What does \"*\" refer to in computational time section?\n\n12. Fig. 4 would benefit from axis labels, and needs to be referred to in the write-up.\n\n13. Halo plots -- the blue curves for a fixed \\epsilon account for all \\phi_{i,j} (in the 2D case), but what is the x and y axis supposed to denote? Do the negative and positive values on the y axis carry meaning? Why does the x axis not have any axis values?\n\n14. The term \"MCR\" is undefined.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The paper is very well-motivated\n2. The proposed method is creative\n2. The paper is very clear and easy to read (a few parts need some clarifications; see \"Questions\"), I congratulate the authors for their clarity of presentation.\n3. Makes an important contribution.", "weaknesses": "1. The greedy algorithm (one of the main contributions) is unclear and difficult to follow (but can be clarified during discussion)\n2. Halo plots are new, and need more discussion/explanation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L3FP559IRt", "reviewer": "Reviewer_gxVk", "summary": "This paper presents to explain the feature interactions in a model class. Two visualization tools are developed to analyze the feature interactions.", "questions": "N.A.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Proposed to look for multiple feature interaction sets based on shapley values.\n2. Two visualization methods are proposed for analyzing and visualizing the FIS.", "weaknesses": "1. I  am not fully convinced by the motivation of this paper. i.e., why do we need to explain feature interactions in a model class?\n2. The novelty of Shapley value calculation is not well presented. The focus is totally on the Rashomon set side.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EYD0HApwgs", "reviewer": "Reviewer_UrRA", "summary": "This paper argues that a well-trained predictive model may not accurately preserve the true feature interactions, and multiple well-performing predictive models can exhibit variations in feature interaction strengths. Therefore, they recommend exploring feature interaction strengths within a model class consisting of approximately equally accurate predictive models. The authors suggest exploring feature interaction strengths within a model class comprising models that are approximately equally accurate. The authors introduce the concept of Feature Interaction Score (FIS) within the framework of a Rashomon set. To facilitate the calculation of the FIS within this model class, they present a practical algorithm to calculate FIS. FIS is a straightforward and heuristic method, but it is still novel to me.", "questions": "1. Is your work the first one to propose to use the loss change to measure the strength of feature interactions?\n2. From my perspective,  feature interactions typically don't have a definitive \"ground truth.\" Therefore, how can you prove the superiority of your Feature Interaction Score (FIS) compared to other baselines without directly comparing it to a ground truth?\n3. What is MCR in Table 3? \n4. What is the range of mask $m_i$? Mask values are used to be 0-1. However, in your algorithm, they seem to be $m_i\\in \\mathbb{R}$.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The originality of this paper is great. The problem is clearly defined.\n\n2. The proposed method is technically sound to me.\n\n3. The definition of FIS is a novel and reasonable tool to analyze feature interactions intuitively.", "weaknesses": "1. The overall presentations need to be improved. Many figures do not have the axes' labels. \n2. The experiments should be conducted on a broader range of datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)