Interpretability with full complexity by constraining feature information
Authors: Kieran A Murphy
Danielle Bassett
Venue: ICLR 2023 poster (2023)

Abstract: Interpretability is a pressing issue for machine learning. Common approaches to interpretable machine learning constrain interactions between features of the input, sacrificing model complexity in order to render more comprehensible the effects of those features on the model's output. We approach interpretability from a new angle: constrain the information about the features without restricting the complexity of the model. We use the Distributed Information Bottleneck to optimally compress each feature so as to maximally preserve information about the output. The learned information allocation, by feature and by feature value, provides rich opportunities for interpretation, particularly in problems with many features and complex feature interactions. The central object of analysis is not a single trained model, but rather a spectrum of models serving as approximations that leverage variable amounts of information about the inputs. Information is allocated to features by their relevance to the output, thereby solving the problem of feature selection by constructing a learned continuum of feature inclusion-to-exclusion. The optimal compression of each feature---at every stage of approximation---allows fine-grained inspection of the distinctions among feature values that are most impactful for prediction. We develop a framework for extracting insight from the spectrum of approximate models and demonstrate its utility on a range of tabular datasets. 

Published as a conference paper at ICLR 2023
INTERPRETABILITY WITH FULL COMPLEXITY BY CONSTRAINING FEATURE INFORMATION
Kieran A. Murphy' and Dani S. Bassett Bassett 1,2,3,55,6677
Dept. of Bioengineering. School of Engineering & Applied Science,
U. of Pennsylvania, Philadelphia. PA 19104, USA
Dept. of Electrical & Systems Engineering, School of Engineering & &Applied Science,
U. of Pennsylvania, Philadelphia. PA 19104, USA
2 Deett. of Neurology, Perelman School of Medicine, U. of Pennsylvania, Philadelphia. PA 19104, USA * *Dept, of Psychiatry, Perelman School of Medicine, U. of Pennsylvania, Philadelphia, PA 19104, USA Dept. of Physics & Astronomy, College of Arts & Sciences, U. of Pennsylvania, Philadelphia, PA 19104, USA "The Santa Fe Institute, Santa Fe, NM 87501, USA
To whom correspondence should be addressed: dsb @ seas.upenn.edu
ABSTRACT
Interpretability is pressing issue for machine learning. Common approaches to interpretable machine learning constrain interactions between features of the input, sacrificing model complexity in order render more comprehensible the effects of those features on the model's output. We approach interpretability from new angle: constrain the information about the features without restricting the complexity of the model. We use the Distributed Information Bottleneck to optimally compress each feature as maximally preserve information about the output. The learned information allocation, by feature and by feature value, provides rich opportunities for interpretation, particularly in problems with many features and complex feature interactions. The central object of analysis is not single trained model, but rather spectrum of models serving as approximations that leverage variable amounts of information about the inputs. Information is allocated to features by their relevance to the output, thereby solving the problem of feature selection by constructing a The optimal compression
learned continuum of feature of each feature-at at every stage of approximation- alloows fine-grained inspection of the distinctions among feature values that most impactful for prediction. We develop a framework for extracting insight from the spectrum of approximate models and demonstrate its utility on range of tabular datasets.
INTRODUCTION
Interpretability a pressing issue for machine learning (ML) (Doshi Velez & Kim, 2017; Fan et al., 2021; Rudin et al., 2022). As models continue grow in complexity, machine learning is increasingly integrated into fields where flawed decisions can have serious ramifications (Caruana et al., 2015; Rudin et al., 2018: Rudin, 2019). Interpretability is not binary property that machine learning methods have or do not: rather, it is the degree to which learning system can be probed and comprehended (Doshi-Velez & Kim, 2017). Importantly, interpretability can be attained along many distinct routes (Lipton, 2018). Various constraints on the learning system can be incorporated, such as force feature effects combine models in exchange for degree of comprehensibility (Molnar et al., 2020; Molnar, 2022). In contrast to explainable Al that happens post-hoc after a black-box model is trained, interpretable methods engineer the constraints into the model from the outset (Rudin et al., 2022).
simple (e.g.. linear) manner, restricting the space of possible
In this work, we introduce a novel route to interpretability that places no restrictions on model complexity, and instead tracks how much and what information is most important for prediction. By identifying optimal information from features of the input, the method grants a measure of salience to each feature, produces spectrum of models utilizing different amounts of optimal information about the input, and provides a learned compression scheme for each feature that highlights from where the information is coming in fine-grained detail. The central object of interpretation not single

---

Published as a conference paper at ICLR 2023
b
C
X1
X X2
X
U1
Full complexity
X2
X2
U2
Feature effects
X3
Y
X₃
Σ
X3
U3
Continueus- feature inclusion
Xx x
Xw
XN Un
Figure I: Restricting information flow from features grants interpretability with full complexity. (a) Feature selection methods optimize a binary mask on features to uncover the most informative ones. (b) Interpretable ML methods sacrifice model complexity in exchange for comprehensible feature interactions (shown is a generalized additive model (GAM) with a neural network to transform each feature). (c) Our method places an information bottleneck penalty on each feature, with constraints on complexity before or after the bottlenecks. The features X; are compressed to optimal representations U4. which communicate the most relevant information Compressing the features in this way allows feature inclusion/cxclusion to exist on a continuum. The learned compression scheme for each feature grants insight into feature effects on the model output.
the rest of the model.
optimized model, but rather a family of models that reveals how predictive information is distributed across features every level of fidelity.
The information constraint that we incorporate is a variant of the Information Bottleneck (IB) (Tishby et al., 2000; Asoodeh & Calmon, 2020). The IB extracts relevant information in relationship, though it generally lacks interpretability because the extracted information is free to be any black-box function of the input. Here, Bassett, 2022), which siloes the information from each feature and applies a sum-rate constraint appropriate level in the model's s processing: after every feature is processed and before any interaction effects between the features can be included. Before and after the bottlenecks, arbitrarily complex processing can occur, thereby allowing the Distributed IB to be used in scenarios with many features and complex feature interactions.
the Distributed IB (Estella Aguerri & Zaidi, 2018; Murphy &
In this paper, we develop framework for extracting insight from optimization with the Distributed IB. Through experiments on variety of tabular datasets and comparisons to other approaches in interpretable ML and feature selection, we demonstrate the following strengths of the approach:
Capacity to find the features and feature values that are most important for prediction. The Distributed IB optimally allocates information across the features for every degree of approximation. Each feature is compressed during optimization, and the compression schemes reveal the distinctions among feature values that are most important for prediction.
2. Full complexity of feature processing and interactions. Compatible with arbitrary neural network architectures before and after the bottlenecks, the method gains interpretability without sacrificing predictive accuracy.
3. Intuitive implement, minimal additional overhead. The method a probabilistic encoder for each feature, employing the same loss penalty 3-VAE (Higgins et al., 2017). A single model is trained once while annealing the magnitude the information bottleneck.
RELATED WORK
We propose a method that analyzes the information content of the features of an input with respect to an output. The method's generality bears relevance to several broad areas of research.
Feature selection and variable importance. Feature selection methods isolate the most important features in a relationship (Fig. la), both to increase interpretability and to reduce computational burden (Chandrashekar & Sahin, 2014; Cai et al., 2018). When higher order feature interactions are present, the large combinatorial search space presents an NP-hard optimization problem (Chan- drashekar & Sahin, 2014). Modern feature selection methods navigate the search space with feature inclusion weights, or masks, that can be trained as part of the full stack with the machine learning model (Fong& & Vedaldi, 2017; Balm al., 2019; Lemhadri et al., 2021; Kolek al., 2022).
2

---

Published as a conference paper at ICLR 2023
Many classic methods of statistical analysis attempt to identify the most important variation in data. Canonical correlation analysis (CCA) (Hotelling, 1936) finds linear transformations of random variables that maximize linear correlations in the transformed space. The class of transformations can be extended to include kernel methods (Hardoon et al., 2004) and deep learning (Andrew et al., 2013; Wang al., 2016). The desired linear correlation can be generalized to mutual information instead (Painsky et al., 2018). Relatedly, analysis of variance (ANOVA) finds a decomposition of variance by features (Fisher, 1936; Stone, 1994), and recent extensions incorporate ML-based transforms (Märtens & Yau, 2020). A similar set of methods known as variable importance (VI)
quantify the effect of different features on an outcome (Breiman, 2001; Fisher et al., 2019).
Interpretable machine learning, Interpretability is often categorized along various binaries. One of those binaries is as either intrinsic or post hoc (Molnar et al., 2020). Intrinsic interpretability occurs when the route to comprehensibility is engineered into the learning model from the outset, as in this work. By contrast, post-hoe interpretability constructed around an already-traines model (Rudin et al., 2022). Another distinction is local versus global. Local interpretability is comprehensibility built up piecemeal by inspecting individual examples (e.g., LIME (Ribeiro et al., 2016)), whereas for global interpretability comprehensibility applies to the whole space of inputs at once, as in this work.
Intrinsic interpretabilty is generally achieved through constraints that have been incorporated to reduce the complexity of the system. For example, generalized additive models (GAMs) require feature effects to combine linearly (Fig. lb), though each effect can be an arbitrary function of a feature (Chang et al., 2021; Molnar, 2022). By using neural networks learn the transformations from raw feature values to effects, neural additive models (NAMs) maintain the interpretability of GAMs with more expressive effects (Agarwal al., 2021). Pairwise interactions between effects can be included for more complex models (Yang et al., 2021: Chang et al., 2022), though complications arise from degeneracy and the growing number of effect terms to be inspected. Regularization can be used to encourage sparsity in the ML model so that fewer components weights in the network (Ng, 2004) or even features (Lemhadri et al., 2021)
to produce the final prediction.
Information Bottleneck. Our proposed method can be seen as regularization to encourage sparsity of information. is based upon the Information Bottleneck, a method to extract the most relevant information in a relationship (Tishby et al., 2000) that has proven useful in representation learning (Alemi et al., 2016; Wang et al., 2019; Asoodeh & Calmon, 2020). As a form of post-hoc interpretability. Bang et al. (2021) and Schulz et al. (2020) employ the IB to explain trained model's s predictions. The IB's potential for intrinsic interpretability is hindered by the fact that it has only one bottleneck for the entire input, obscuring from where relevant information originated. By contrast, our method based on multiple IBs affects the training of a model itself to provide intrinsic interpretability.
Distributing IBs to multiple sources of information has been of interest in information theory solution to multiterminal source coding, or the "CEO problem" (Berger et al., 1996; Estella Aguerri & Zaidi, 2018; Aguerri & Zaidi, 2021; Steiner & Kuehn, 2021). The problem is the optimal compression of multiple signals independently before transmission to a "CEO" for prediction of a target variable. In Murphy& & Bassett (2022), features of data were used as the distributed sources of information so that the Distributed IB revealed the relative importance of the features. Here we expand the framework of interpretability by analyzing the learned compression schemes for insight to feature value importance, and by benchmarking with related interpretability and feature selection methods.
3 METHODS
Conceptually, the Distributed IB introduces a penalty on the amount of information used by black- box model about each feature of the input. Independently of all others, features are compressed and communicated through learned noisy channel; the messages are then combined for input into prediction network. The information allocation and compression schemes-for for every feature and every magnitude of the penalty- serve windows into feature importance and into feature effects.
3.1 DISTRIBUTED VARIATIONAL INFORMATION BOTTLENECK
Let X.Y ~ p(z, y) be random variables representing the input and output of a relationship to be modeled from data. Mutual information is a measure of the statistical dependence between two random variables; it is the reduction of entropy of one variable after learning the value of the other:
((x:Y) = H(Y) - XY)x)),
(I)
where H(Z) Egsp(ee) log p(=)] is Shannon's 's entropy (Shannon, 1948).
3

---

Published as a conference paper at ICLR 2023
a
b
I(XY)
Informatioon ont
Dist. IB
I(U: ((UY)
Possible models
Error
Possible
information
allocations
Dist. IB
E
Σ DEL
Information in
The tradeoff between information into model and predictive power out. (a) The distributed information plane displays the sum total information from all features, encoded indepen- dently of one another, against the information about the output Y. While there are many possible allocations of feature information (gray region), the Distributed IB objective allows us to find a continuum of allocations (black curve) that contain the most information about the output for the least information about the input. Increasing the bottleneck strength / traverses the optimal trajectory from the green point-all bottlenecks fully open, and all information is available for prediction to the red point-where predictions are independent of the input. (h) A pragmatic alternative to (a) displays the sum of the KL-ddivergences Σ DKL. (roughly corresponding the information utilized by the model about the features) against the particular error metric being optimized. Optimizing the Distributed IB objective finds a continuum of models (black curve) that utilize the least information about the features for the least error. In the large Σ DxL regime, the effect of the bottlenecks are negligible and the models are essentially unconstrained machine learning. A Distributed IB optimization gradually increases B, the strength of the bottlenecks, shedding information.
Figure 2:
The Information Bottleneck (Tishby et al., 2000) is a learning objective to extract the most relevant information from about Y. It optimizes a representation U that conveys information about X in the form of a stochastic channel of communication U ~ p(u[x). The representation is found by minimizing a loss with competing mutual information terms balanced by a scalar parameter (3,
LB XI)
The first term is the bottleneck, penalty on information passing into U, whose strength is controlled by the parameter B. Optimization with different values of / produces different models of the relationship between X and Y; single sweep over 3 produces continuum of models that utilize different amounts of information about the input X. In the limit where 3 + O, the bottleneck is fully open and all information from X may be freely conveyed into U in order for it to be maximally informative about Y. As B increases, only the most relevant information in X about becomes worth conveying into U, until eventually an optimal U is uninformative random noise.
One challenge in using the IB is that mutual information is generally difficult to estimate from data (Saxe et al., 2019; McAllester & Stratos, 2020). Accordingly, variants of IB replace the mutual information terms of Eqn. 2 with bounds amenable to deep learning (Alemi et al., 201 6; Achille & Soatto, 2018; Kolchinsky et al., 2019). Here we follow the variational information bottleneck (VIB) (Alemi et al., 2016), whose framework resembles that of 3- 3-VAEs (Higgins et al., 2017).
sample is encoded as distribution in representation space p(u[z) [xx o,e) with neural network parameterized by weights & and a source of noise e ~ A( 1) to facilitate the "reparam- eterization trick" (Kingma Welling, 2014). The bottleneck restricting the information from X takes the form of the Kullback-Leibler (KI.) divergence between the encoded distribution p(u[a: and prior distribution r(u) = (/00 1). As the KI. divergence tends to zero, all representations are indistinguishable from the prior and from each other, and therefore uninformative. A representation u is sampled from p(u[x) and then decoded to distribution over the output Y by a second neural network parameterized by weights th g(y/u) (u) g(u, v. In place of Eqn. 2, the following may be minimized with standard gradient descent methods:
Lvm (p)))(((u)>))ll)
Ellog e(ylu))
4

---

Published as a conference paper at ICLR 2023
U~
p(ulz
b
ab
c
Prior r(u) X)) bits
--0-01
IU:X) bit X(U:x)
Bileshare, weekday
Figure 3: Confusion matrices to visualize compression schemes. (a) An input value is encoded to a distribution over u. The Bhattacharyya coefficient quantifies how distinguishable the distributions are for different encodings, which we express matrix. The diagonal (white) values are indistin- guishable and everything else (blue) is distinguishable, yielding 2 bits information.
(b) bit of information allows hard clustering of the values, such that the ab cluster is distinguishable from the ed cluster, but no more than that. (c) bit also allows a soft clustering, where some pairs of values are partially distinguishable. (d) When trained with wwo-dimensional embedding space per feature the Bikeshare dataset, we recover a soft clustering the day of the week. Weekend distinguishable from weekday, with intermediate values of distinguishability otherwise.
The Distributed IB (Estella Aguerri & Zaidi, 2018) 8) concerns the optimal scheme of compressing multiple sources of information as they relate to relevance variable Y. Let (X;} be the features of X; we use the Distributed IB to encode each feature X; independently of the rest, and then integrate the information from all features to predict Y. A bottleneck is installed after each feature the form of compressed representation U, (Fig. Ic), and the set of representations Ux = {U,} is used as input to a model to predict Y. The same mutual information bounds the Variational IB (Alemi et al., 2016) can be applied in the Distributed IB setting (Aguerri & Zaidi, 2021),
LDu 3
>
Ellog ((v/ux)]-
(4)
Arbitrary error metrics with feeature information regularization. Often it may be more natural to quantify predictive power with error metric other than cross entropy; the information penalty on each feature remains unchanged. For instance, mean squared error (MSE) may be used instead,
-= 3 Σ DKL + lly - 0ll
(5)
where y and 9 are the prediction and ground truth values, respectively. B is no longer unitless ratio of information about Y to the sum total information about the features of X, and the optimal range must be found anew for each problem. Optimization will nevertheless yield a spectrum of approximations that prioritizes the most relevant information about the input features.
Boosting expressivity for low-dimensional features. Consistent with results reported by Agarwal et al. (2021), we found that the feature encoders struggled with expressivity due to the low dimension- ality the features. We positionally encode feature values (Tancik et al., 2020) before they pass to the neural network, taking each value to (sin(w)=), sin(w==)),,. with the encoding frequencies.
3.2 VISUALIZING THE OPTIMALLY PREDICTIVE INFORMATION
Distributed information plane. Tishby & Zaslavsky (2015) popularized the visualization of learning dynamics the so-called information plane, a means display how successfully information conveyed about the input of a relationship is converted to information about the output. Here we employ the distributed information plane (Fig. 2a) to compare the total information utilized about the features Σ [(U4; X,) to information about the output / [{U,); Y). In the space of possible allocations of information to the input features, the Distributed IB finds those that minimize Σ (Ug (G) while maximizing I((U,); Y) (black curve, Fig. 2a).
pragmatic alternative to the distributed information plane replaces Σ I(U; X2) with its upper bound used during optimization: the sum of KL divergences across the features. Additionally, to be compatible with arbitrary error metrics, we replace the vertical axis [([U,]; Y) with the error used for training.
5

---

Published as a conference paper at ICLR 2023
Table 1: Performance on various tabular datasets. NAM and NODE-GAM allow no feature interac-
tions; NODE-GA contain pairwise feature interactions; the Neural Oblivious Decision Ensemble (NODE) LassoNet, and Distributed IB methods are full complexity models (allowing arbitrary feature
interactions). tResults from Chang et al. (2022).
Bikeshare (RMSE 4)
Wine (RMSE.. L)
MIMIC-II (AUC % t)
MIMIC-III (AUC % 1)
Microsoft (MSE 4
NAM (Agarwal et al., 2021)
99.9=1.3
0.71+0002
83.0=o.a
81.7+0.2
0.5908
NODE-GAM (Chang et al., 2022)
100.7
0.71+00.0
83.2=1.11
81.4
0.5821 00000
NODE-GA (Chang et al., 2022)
49.8
0.67+10.02
84.6=1.1
82.2
0.5618::0.0 0003
LassoNet (Lemhadri et al., 2021)
46.0=2.0
0.68
83.4=0.8
79.4
0.5667 0012
NODEŞ (Popov et al., 2019)
36.2=119
0.64++0.01
84.3
82.8
0.5570::0.000
Distributed IB (Ours)
38.0=1.1
0.62++0.01
84.9=1.2
80.2
0.5790
The entire curve of optimal information allocations in Fig. 2b may be found a single training run, by increasing logarithmically over the course of training as Eqn. 4 is optimized. Training begins with negligible information restrictions (3 ~ 0) so that the relationship may be found without obstruction (Wu et al., 2020; Lemhadri et al., 2021), and ends once all information is removed. Each value of B corresponds to a different approximation to the relationship between X and Y, and we can use the KL divergence for each feature as a measure of its overall importance in the approximation.
Confusion matrices to visualize feature compression schemes. Transmitting information about feature is equivalent to conveying distinctions between the feature's possible values. In the course of optimizing the Distributed IB, the spectrum of learned models range from all input values perfectly distinguishable (8 + O, unconstrained machine learning) to all inputs perfectly indistinguishable (the model utilizes zero information about the input). Along the way, the distinctions most important for predicting Y are selected, and can be inspected via the encoding of each feature value.
Each feature value 2; is encoded to a probability distribution p(u,(z:;) in channel space, allowing the compression scheme to be visualized with a confusion matrix showing the similarity of feature values (Fig. 3a). The Bhattacharyya coefficient (Kailath, 1967) is symmetric measure of the similarity between probability distributions p(2) and q(=), and is straightforward to compute for normal distributions. The coefficient is 0 when the distributions are perfectly disjoint (i.e., zero probability is assigned by p(z) everywhere that ((2) has nonzero probability, and vice versa), and I when the distributions are identical. In other words, the Bhattacharyya coefficient between encoded distributions p(u|æ a) and p( u/x b) is 1, the feature values = and x b are indistinguishable to the prediction network and the output independent of whether the actual feature value was or b. On the other extreme where the Bhattacharyya coefficient is O, the feature values are perfectly distinguishable and the prediction network can yield distinct outputs for x = a and x = b.
We display in Fig. 3b,c possible compression schemes containing one bit of information about hypothetical æ with four equally probable values. The compression scheme can be a hard clustering of the values into two sets where = a and x = b are indistinguishable, and similarly for x == (Fig. 3b). Another scheme with one bit of information is the soft clustering shown in Fig. 3c, where each feature value is partially 'confused" with other feature values. In the case of soft clustering, a learned compression cannot be represented by decision tree; rather, the confusion matrices are the fundamental distillations of the information preserved about each feature. An example in Fig. 3d shows the learned compression scheme of the weekday feature in the Bikeshare dataset (details below). The soft clustering distinguishes between the days of the week to different degrees, with the primary distinction between weekend days and weekdays.
and
4 EXPERIMENTS
We analyzed a variety of tabular datasets with the Distributed IB and compared the interpretability signal to what is obtainable through other methods. Our emphasis was the unique capabilities of the Distributed IB to produce insight about the relationship in the data modelled by the neural network. Performance results are included in Tab. 1 for completeness. Note that the Distributed IB places no constraints on the machine learning models used and in the 3 - 0 regime could hypothetically be as good as any finely tuned black-box model. The benefit of freedom from complexity constraints

---

Published as a conference paper at ICLR 2023
Distributed IB
LassoNet (Lemhadri et al., 2021)
a
b
Bikeshare
Itoportant features:
Bikeshare
hour a &emp year
warkingday
weathersit
RMSE
sesson
DKL (bitss)
RMSE
Important foatures:
50-
30-
temp
workingday atemp
hour
year
12
Σ Dkt. (bits)
Number of features
d
2.0
MiceProtein
Important features:
2.0-
MiceProtein
Closs eutropy (bite)
Cross eutropy (bits)
32
1.5
46
30
49
51
DRL (bits)
1.5-
1.81
1.0-
Imppor fant features:
32
3.
76
30
2. 10
4.
65
10
15
Σ
DKL. (bitss
Number of features
Cruss eutropy (bits) @D
0.38
MIMIC-II
Impertant features:
0.388
MIMIC-II
Cross eutxopy (bite)
PFratio Bilarubin
Age
CCS
0.31
Renal
DEL (bats)
0.34
0.30-
Important features:
AdmissionType 3. Renal
5. Na.
2. Bilirubin
4. MetastatieCaneer
0.26
0.26
0
10
15
20
12
16
Σ DKL (bits)
Number features
Figure 4: Distributed information plane trajectories with information allocation by feature. (a) For the Bikeshare dataset, the optimal RMSE eπror versus Σ DKL (black) decays more information is used by the model about the features. The shifting allocation of information to the features tells of their relevance to the prediction at each level of approximation. Only a few bits about handful of features is enough for most of the performance. We label and color the first features to contribute non-negligibly the approximations. (b) The LassoNet trajectory for the Bikeshare dataset yields only a discrete series of error values and an ordering of the features. Note the LassoNet trajectory extends the point of all features included, whereas the Distributed IB is truncated at 20 bits of DRI. (c, d) Same as (a, b) for the MiceProtein dataset, with expression levels of 77 proteins measured to classify mice one of eight categories related to Down syndrome. (e, f) Same as (a, b) for the MIMIC-II dataset, binary classification task related to ICU mortality, with 17 input features.
was particularly clear in the smaller regression datasets, whereas performance was only middling on several other datasets (refer to App. B for extended results and App. C for implementation details).
4.1 THE FEATURES WITH THE MOST PREDICTIVE INFORMATION
We used the Distributed IB to identify informative features for three datasets in Fig. 4. Bikeshare (Dua & Graff, 2017) is regression dataset to predict the number of bikes rented given 12 descriptors of the date, time, and weather. MiceProtein (Higuera et al., 2015) is a classification dataset relating the expression of 77 genes to normal and trisomic mice exposed to various experimental conditions. MIMIC-II (Johnson et al., 2016) is binary classification dataset to predict hospital ICU mortality given 17 patient measurements. We compared our results using the Distributed IB to those obtained using LassoNet (Lemhadri et al., 2021), feature selection method that attaches an LI regularization penalty to each feature and introduces a proximal algorithm into the training loop to enforce a constraint weights By gradually increasing the regularization, LassoNet finds a series of models that utilize fewer features while maximizing predictive accuracy, providing a signal about what features necessary for different levels of accuracy.
7

---

Published as a conference paper at ICLR 2023
hour
atemp
workingday
seas0n
Dataset: Bikesbare
Prob
20C
Prob
Prob
0C
0:00
6:00
12:00
18:00
20c
100
False
Troe
Win
Spx
Sum
Fall
0:00
Σ Dxt: 1.9 bits
6:00
False
indistugussabb
All valtes
Win
indistingrssabll
All vilues
RMSE: 127.0
Spr
12:00
20C
Sum
18:00
True
Fall
40C
b Σ Dxui 4.3 bits
0:00
0C
Win
Indistingushaab
All values
6:00
False
RMSE: 105.2
Spr
12:00
20C
Sum
18:00
True
Fall
40C
0:00
Σ Dxu' bits
Win
6:00
False
RMSE: 75.6
Spr
12:00
20C
Sun
88:00
True
Fall
40C
0:00
Σ Dxu: 16 bits
IC
Win
6:00
False
RMSE: 55.2
Spr
12:00
20C
Sun
18:00
True
Fall
40C
e
Neural Additive Models (Agarwal et al., 2021)
Ellect
Effect
Effect
Elllect
RMSE: 99.9
0:00
6:00
12:00
18:00
0C
20C
10C
False
Wmn
Spr
Sum
Fall
Figure 5: Important information within each feature via inspection of the learned compression
schemes. (a-d) For the Bikeshare dataset at different levels of approximation, confusion matrices display the learned compression schemes for four the top features found in Fig. 4. Following the scheme in Fig. 3, the pairwise distinguishability of feature values visualizes the information preserved about each feature. For the categorical features (hour, work i Indday, and season), each row and column of the confusion matrices corresponds to unique value, and the probability distributions the top of the figure are aligned with the columns the matrices. For the continuous variable at emp (apparent temperature), 1000 values the inspected points; the rank order plot at the top shows their order in the matrices. As the utilized information Σ DKL increases, too does the performance; with only four bits the Distributed IB can perform about as well as NAM, highlighting the value of using full-complexity model. (e) The effects of these features on the output can directly visualized with NAM; the horizontal coordinates match those of the matrices in (a-d) and the black traces display effects for an ensemble of 25 NAMs.
sampled randomly from the dataset and sorted for as
Like LassoNet, the Distributed IB probes the tradeoff between information about the features and predictive performance, though with important differences. LassoNet can only vary the information used for prediction through discrete-valuee proxy: the number of features. Binary inclusion or exclusion of features is common limitation for feature selection methods (Bang et al., 2021; Kolek et al., 2022). By contrast, the Distributed IB compresses each feature along a continuum, selecting the most informative information from across all input features. At every point along the continuum an allocation of information to each of the features. For example, in Fig. 4a found that only few bits of information (Σ DKL), largely deriving from the hour feature, was necessary to achieve most of the performance of the full-information model, and to outperform the linear models (Tab. 1). The capacity to extract partial information from features is notable in comparison to the LassoNet performance (Fig. 4b), which improves only after multiple complete features have been incorporated.
8

---

Published as a conference paper at ICLR 2023
In contrast to Bikeshare, where decent approximations required only a handful of features, the MiceProtein (Fig. 4c,d) and MIMIC-II (Fig. 4e,f) datasets were found to require information from many features simultancously. The information allocation is dynamie, with some features saturating quickly, and others growing in information usage at different rates. From these observations, we see that the signal obtained by the Distributed IB can usefully decompose datasets with large numher of features and complex interaction effects, all within a single training run.
4.2 FINE-GRAINED INSPECTION OF FEATURE EFFECTS
In addition measure of feature importance in the form of information allocation, the Distributed IB offers a second source of interpretability: the relative importance of particular feature values, revealed through the learned compression schemes of each feature. One of the primary strengths of a GAM is that the effects of the feature values are straightforwardly obtained, though at the expense of interactions between the features. By contrast, the Distributed IB permits arbitrarily complex feature interactions and still maintains window into the role played by specific feature values in the ultimate prediction. In Fig. 5 we compared the compression schemes found by the Distributed IB to the feature effects obtained with NAM (Agarwal et al., 2021) on the Bikeshare dataset.
The Distributed IB allows one to build up to the maximally predictive model by approximations that gradually incorporate information from the input features. To first approximation (Fig. 5a), with only two bits of information about all features, the scant distinguishing power was allocated to discerning certain times of day-comug periods, the middle of the night, and everything else-in the hour feature. Some information discerned hot from cold from the apparent temperature (atempp, and no information about the season or whether it was a working day was used for prediction. We are unable to say how this information was used for prediction the price of allowing the model to be more complex-but we can infer much from where the discerning power was allocated. At four bits, and performing about well as the linear models in Tab. 1, the Boolean worki i iidda was incorporated for prediction, and the evening commuting time was distinguished from the morning commuting time. With eight and sixteen bits of information about the inputs, nearly all of the information was used from the hour feature, the apparent temperature became better resolved, and in season, winter was distinguished from the warmer seasons.
The feature effects found with NAM (Fig. 5e) partially corroborated the distinguishability found by the Distributed IB, particularly in the hour feature (e.g., commuting times, the middle of the night, and the rest form characteristic divisions of the day in terms of the NAM effect). However, we found that the interpretability of the NAM-effects tend to vary significantly across datasets (see App. B for more examples). For example, the apparent temperature (atemp) and temperature (t (tmm)) features were found have contradictory effects (Fig. 10) that were difficult to comprehend, presumably arising from feature interactions that NAM could not model. Thus while the Distributed IB sacrifices a direct window to feature effects, tracking the information on the input side of a black- box model may serve as one of the only sources of interpretability in cases with complex feature interactions.
DISCUSSION
Interpretability can come in many forms (Lipton, 2018); the Distributed IB provides novel source of interpretability by illuminating where information resides in a relationship with regards to features of the input. Without sacrificing complexity, the Distributed IB is particularly useful in scenarios with large numbers of features and complex interaction effects. The capacity for successive approximation, allowing detail to be incorporated gradually so that humans' limited cognitive capacity (Cowan, 2010- Rudin et al., 2022) does not prevent comprehension of the full relationship, plays a central role in the Distributed IB's 's interpretability.
The signal from the Distributed IB -aaaticularll the compression schemes of feature values (c.g.. Fig. 5)- vulnerable to inerritterrttttionn. This vulnerability is partly due to a difficulty surrounding intuition about compression schemes (i.e., it is perhaps most natural to imagine 1 bit of information as a single yes/no question, or a hard clustering (Fig. 3b), rather than as a confusion matrix (Fig. 3c)). It is also partly due to the inability to make direct claims about how a particular feature value changes the output. Rather than supplanting other interpretability methods, the Distributed IB should be seen as new tool for interpretable machine learning, supplementing other methods. A thorough analysis could incorporate methods where the effects are comprehensible, possibly using a reduced feature set found by the Distributed IB.

---

Published as a conference paper at ICLR 2023
REFERENCES
Alessandro Achille and Stefano Soatto. Information dropout: through noisy computation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40 (12):2897-2905, 2018. 4
Learning optimal representations
Rishabh Agarwal, Levi Melnick, Nicholas Frosst, Xuezhou Zhang, Ben Lengerich, Rich Caruana, and Geoffrey E Hinton. Neural additive models: Interpretable machine learning with neural nets. Advances in Neural Information Processing Systems, 34:4699 47111 2021. 3, 5, 6, 9, 18
Iñaki Estella Aguerri and Abdellatif Zaidi. Distributed variational representation learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(1):120-138, 2021. doi: 10.1109/ TPAMI.2019.2928806 3,
Alexander A. Alemi, lan Fischer, Joshua V. Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410 2016. 3, 4,
Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. Deep canonical correlation analysis. In International conference on machine learning, pp. 1247-1255. PMLR, 2013.
Shahab Asoodeh and Flavio P. Calmon. Bottleneck problems: An information and estimation- theoretic view. Entropy, 22(1 22(11)::32) 2020. 2,
Muhammed Fatih Balm, Abubakar Abid, and James Zou. Concrete autoencoders: Differentiable feature selection and reconstruction. Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 444-453. PMLR, 09-15 Jun 2019.
Seojin Bang, Pengtao Xie, Heewook Lee, Wei Wu, and Eric Xing. Explaining black-box by using deep variational information bottleneck approach. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11396-11400 1404, 2021. 3,
Toby Berger, Zhen Zhang, and Harish Viswanathan. The CEO problem [multiterminal source coding).
IEEE Transactions on Information Theory, 42(3))8877-00 1996. 3
Leo Breiman. Random forests. Machine learning, 45(1):5-32, 2001. 3
Zoe Budrikis. Growing citation gender gap. Nature Reviews Physics, 2(7):346-346, 2020. 20
Jie Cai, Jiawei Luo, Shulin Wang, and Sheng Yang. Feature selection in machine learning: A new
perspective. Neurocomputing, 300:70-79, 2018. 2
Neven Caplar, Sandro Tacchella, and Simon Birrer. Quantitative evaluation of gender bias in astronomical publications from citation counts. Nature Astronomy, 1(6):1-5, 2017. 20
Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Mare Sturm, and Noémie Elhadad. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 2 ACM SIGKDD international conference on knowledge discovery and data mining, pp- 1721-1730, 2015.
Paula Chakravartty, Rachel Kuo, Victoria Grubbs, and Charlton Mcllwain. #CommumicationSoWhicc Journal of Communication, 68(2):254-266. 2018. 20
Girish Chandrashekar and Ferat Sahin. A survey on feature selection methods. Computers & Electrical Engineering. 40(1):16-22 2014. 2
Chun-Hao Chang, Sarah Tan, Ben Lengerich, Anna Goldenberg, and Rich Caruana. How interpretable and trustworthy are gams? In Proceedings of the 27th ACM SIGKDD Conference Knowledge
Discovery & Data Mining, pp. 95-105, 2021. 3
Chang, Rich Caruana, and Anna Goldenberg. Node-gam gade-gam::aa Neural generalized additive model for interpretable deep learning. In International Conference on Learning Representations, 2022. 3, 6, 14
10

---

Published as a conference paper at ICLR 2023
Nelson Cowan. The magical mystery four: How is working memory capacity limited, and why? Current Directions in Psychological Science, 19(1):51-57, 2010. 9
Michelle L. Dion, Jane Lawrence Sumner, and Sara Mc Mauaugl Mitchell. Gendered citation patterns across political science and social science methodology fields. Political Analysis, 26(3):112-327, 2018. 20
Finale Doshi-Velez and Been Kim. Towards rigorous science of interpretable machine learning. arXiv preprint arXiv: orXiv:/702.0800 1702.0868, 2017.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL WRR.htt:://acchiv
ics.uci uci. edw/ml. 7, 16
Jordan Dworkin, Perry Zur,, and Danielle S. Bassett. (In)eiting action to realize equitable future. Neuron, 106(6):890-894, 2020a. 20
Jordan D. Dworkin, Kristin A. Linn, Erin G. Teich, Perry Zurn, Russell T. Shinohara, and Danielle S. Bassett. The extent and drivers of gender imbalance in neuroscience reference lists. Nature
Neuroscience, 23(8):918-99) 926, 2020b. 20
Iñaki Estella Aguerri and Abdellatif Zaidi. Distributed information bottleneck method for discrete and gaussian sources. In International Zurich Seminar on Information and Communication (IZS 2018). Proceedings, pp. 35-39. ETH Zurich, 2018. 2, 3.5
Feig Fan, Jinjun Xiong, Mengzhou Li, and Ge Wang. On interpretability of artificial neural networks: A survey. IEEE Transactions on Radiation and Plasma Medical Sciences, 5(6):741-760, 2021. 1
Aaron Fisher, Cynthia Rudin, and Francesca Dominici. All models wrong, but many useful: Learning a variable's ' importance by studying an entire class of prediction models simultancously. Mach. Learn. Res., 20(177):1-8) 2019. 3
Ronald A. Fisher. The use of multiple measurements in taxonomic problems. Annals of Eugenics, (2):179-188, 1936. 3
Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturba- tion. In Proceedings of the IEEE international conference on computer vision, pp. 3429-3437,
2017. 2
David R. Hardoon, Sandor Szedmak, and John Shawe-Taylor. Canonical correlation analysis: An overview with application to learning methods. Neural Computation, 16(12):2639-2664 2004.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 3-vae: Learning basic visual concepts with con- strained variational framework. International Conference on Learning Representations (ICLR), 2017. 2, 4
Clara Higuera, Katheleen Gardiner, and Krzysztof J Cios. Self-organizing feature maps identify proteins critical to learning in mouse model of down syndrome. PloS one, 10(6):e0122126, 2015.
Harold Hotelling. Relations between two sets of variates. Biometrika, 3374 1936. ISSN 0006-3444. doi: 10. 0o.1093biomeee28.34.332 URL https: /ttp//////oo.oo/0110//999 biomet/ / 328.3-8.888 4.2.21..
Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii,
freely accessible critical care database. Scientific data, 3(1):1-9, 2016. 7, 1616
Thomas Kailath. The divergence and bhattacharyya distance measures in signal selection. IEEE transactions on communication technology, 15(1):52-60, 1967.
Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes. International Conference
Learning Representations (ICLR), 2014. 4
11

---

Published as a conference paper at ICLR 2023
Artemy Kolchinsky, Brendan D. Tracey, and David H. Wolpert. Nonlinear information bottleneck. Entropy, 21()2), 2019. ISSN 1099-4300. doi: 10.3390/e211211811 URL httpps:// /wwww//wwww././////////// http:/wwwppppp..///////t//. com/1099-440000 1109994300221112///111
4
Stefan Kolek, Due Anh Nguyen, Ron Levie, Joan Bruna, and Gitta Kutyniok. A rate-distortion framework for explaining black-box model decisions. In International Workshop on Extending
Explainable Al Beyond Deep Models and Classifiers, pp. 91-115. Springer, 2022. 2,8
Ismael Lemhadri, Feng Ruan, Louis Abraham, and Robert Tibshirani. Lassonet: A neural network with feature sparsity. Journal of Machine Learning Research, (2(127) 2021. URL http: // //jmlr orgg hwtp://appss//eeee / -0- html. 3, 6, 7,
Zachary C Lipton. The mythos of model interpretabilityy In machine learning, the concept of interpretability is both important and slippery. Quene, 16(3):31-57, 2018. 1,9
Daniel Maliniak, Ryan Powers, and Barbara F. Walter. The gender citation gap in international relations. International Organization, (7(4):889-922 2013. 20
Kaspar Märtens and Christopher Yau. Neural decomposition: Functional ANOVA with variational autoencoders. In International Conference on Artificial Intelligence and Statistics, pp. 2917-2922
PMLR, 2020. 3
David McAllester and Karl Stratos. Formal limitations on the measurement of mutual information. In International Conference on Artificial Intelligence and Statistics, pp. 875-884. PMLR, 2020.
Christoph Molnar. Interpretable Machine Learning: A Guide for Making Black Box Models Explain- able. 2022. 1,3
Christoph Molnar, Giuseppe Casalicchio, and Bernd Bischl. Interpretable machine learning brief history, state-of-the-at art and challenges. Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 417-431, Springer, 2020. 1,3
Kieran A Murphy and Dani S Bassett. The distributed information bottleneck reveals the explanatory structure of complex systems. arXiv preprint arXiv:2204.07576. 2022. 2,
Andrew Y. Ng. Feature selection, LI vs. L2 regularization, and rotational invariance. In Proceedings of the twenty-first international conference on Machine learning, pp. 78, 2004.
Amichai Painsky, Meir Feder, and Naftali Tishby. An information-theorett framework for non-linean canonical correlation analysis. CoRR, 2018.
Sergei Popov, Stanislav Morozov, and Artem Babenko. Neural oblivious decision ensembles for deep
learning tabular data. arXiv preprint arXiv: 2019. 6
Tao Qin and Tie- Yan Liu. Introducing LETOR 4.0 datasets. CoRR, http: / /arxiv.iivvvvv////////a////aaav/ org/ /bbs 001.3366.597.... 2597. 16
2013. URL
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-11444 2016. 3
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206-215 2019.
Cynthia Rudin, Caroline Wang, and Beau Coker. The age of secrecy and unfairness in recidivism prediction. arXiv preprint arXiv: 111001811.00711, 2018.
Cynthia Rudin, Chaofan Chen, Zhi Chen, Haiyang Huang, Lesia Semenova, and Chudi Zhong. Interpretable machine learning: Fundamental principles and 10 grand challenges. Statistics Surveys, 16:1-85, 2022. 1,
Andrew M. Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan D. Tracey, and David D. Cox. On the information bottleneck theory of deep learning. Journal of Statistical Mechanics: Theory and Experiment, 1742- 1442-5468/ab3985 URL https:tt/////.... //oo org/10..000..........oo 1088808828888
dec 2019. doi: 10.1088/ 68 F48882Fa3398..
12

---

Published as a conference paper at ICLR 2023
Karl Schulz, Leon Sixt, Federico Tombari, and Tim Landgraf. Restricting the flow: Information bottlenecks for attribution. arXiv preprint Stt1:200100039966 2020.
Claude Elwood Shannon. A mathematical theory of communication. The Bell System Technical Journal, 27(3):37--423, 1948. 3
Steffen Steiner and Volker Kuehn. Distributed compression using the information bottleneck principle.
In ICC 2021 IEEE International Conference on Communications, pp. 1 1-6, 2021. doi: 10. 109/ (CC42927:2021.550032
3
Charles J. Stone. The use of polynomial splines and their tensor products in multivariate function estimation. The Annals of Statistics, 22(1):118-4) 171, 1994. 3
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems, 33:7537-7547, 2020. 5
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEFE Information Theory Workshop (ITW), pp. 1-5. IEFE, 2015.
Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method. arXiv
preprint physics/00040557, 2000. 3, 4
Qi Wang, Claire Boudreau, Qixing Luo, Pang-Ning Tan, and Jiayu Zhou. Deep multi-viww information bottleneck. In Proceedings of the 2019 SIAM International Conference on Data Mining, pp. 37-45. SIAM, 2019.
Weiran Wang. Xinchen Yan, Honglak Lee, and Karen Livescu. Deep variational canonical correlation analysis. arXiv preprint 1410044610.03444, 2016.
Tailin Wu, Ian Fischer, Isaac L. Chuang, and Max Tegmark. Learnability for the information bottleneck. In Uncertainty in Artificial Intelligence, pp. 1050-1060. PMLR, 2020.
Zebin Yang, Aijun Zhang, and Agus Sudjianto. Gami-net: An explainable neural network based on generalized additive models with structured interactions. Pattern Recognition, 120:108192, 2021.
Dale Zhou, Eli J. Cornblath, Jennifer Stiso, Erin G. Teich, Jordan D. Dworkin, Ann S. Blevins, and Danielle Bassett. Gender diversity statement and code notebook vl. Zenodo, 2020. 20
Perry Zur,, Danielle S. Bassett, and Nicole C. Rust. The citation diversity statement: a practice of transparency, way of life. Trends in Cognitive Sciences, 24(9):669 672. 2020. 20
13

---

Published as a conference paper at ICLR 2023
Appendices for Interpretability with full complexity by constraining feature information
A SYNTHETIC EXAMPLE
We work through a synthetic example with the Distributed IB in Fig. 6. There are two binary-valued
interventions A, B which we take to the features X1, X2 of the input X, affecting a binary outcome Y with joint probability distribution shown in Fig. 6a. Using 10,000 samples from the joint distribution, we train the Distributed IB and recover the distributed information plane (Fig. 6b) revealing that intervention A is
important for predicting the outcome Y.
Each feature is embedded to a two-dimensional Gaussian so that they may be directly visualized (Fig. 6c), alongside the computed confusion matrix. Note that the feature values of the interventions split along the cardinal axes because the embeddings are diagonal Gaussians. That intervention A split horizontally while B split vertically is due to chance.
B EXTENDED VISUALIZATIONS
We show in Fig. Distributed IB optimizations for the remaining three datasets mentioned in the manuscript, alongside their LassoNet counterparts. In Figs. 8 and we show the compression schemes for some of the most important features in the Wine and MIMIC-II datasets. Finally, in Fig. 10, we display the compression schemes for all features of the Bikeshare dataset over dense sampling of information rates.
C IMPLEMENTATION DETAILS
Code and additional examples may be found through the project page, cecrbbtted-ifformaaiin bottleneck.githubbin
All Distributed IB experiments GPU; 100k steps on the Bikeshare dataset takes a couple minutes, and on Microsoft about 2 hours.
single computer with 12 GB GeForce RTX 3060
The data loading and preprocessing is heavily based on the publicly released code from NODE- GAM (Chang et al., 2022).
We subclassed Kensorfloe ow. keras Model that training can leverage the keras high level API, and is only one step more difficult than the model. compile (); model. fit flow. The additional step is the creation of keras Callback control the 8 ramp over the course of training. All of the information allocations can be recovered from the tensorflow keras. cal lallsbcks... History object returned the end of training.
Training hyperparameters and architecture details are shown in Tab. (common to all datasets) and Tab. (dataset-speciffe). In the first stage of hyperparameter tuning, we explored parameter space in an unstructured manner and found that most parameters affected all datasets similarly. After some tuning, we settled on the values listed in Tab. 2. We found the effects of dropout rate and the number of annealing steps to differ by dataset, and selected optimal values out of (0, 0.1, 0.3, 0.5) and (2.5 5 x 10º, x 10a 105), respectively (Tab. 3).
C. DATASETS
used the dataset preprocessing code released with NODE-GAM (Chang et al., 2022) on Github with minor modification to one-hot encoding" for categorical variables instead of leave-one-out encoding when input to the Distributed IB models, unless there were more than 100 categories.
14

---

Published as a conference paper at ICLR 2023
b
App Approcination
Intervention
111
(X2)
2.0
False
Closs eatropy (bits)
Intervention
by (bits)
False
0.7
0.9
Intervention
(X,)
0.8
Intervention
True
0.25
0.05
0.7
P(Y X,)
H(Y
0.0
0.5
1.0
1.5
20
2.5
Σ DKL about features (bite)
Approxitaation:
iv
vii
DKL.
bits
0.3 bits
0.6 bitt
1 bit
1.4 bits
1.9 bits
2.5 bits
Intervention
emphddiings
compression
scheeme
False
confusiin matrix
True
Intervention B
embeddinggs
comprression scheme
False
confiusiou matrix
True
Figure 6: Synthetic example: two interventions binary-valued outcome. We visualize information allocation on simple example. (a) Consider a hypothetical situatio with two possible interventions, and B, applied uniformly at random and affecting binary outcome Y. The effects of applying the interventions can be visualized by 2x2 table displaying P(Y = 1 (b) By training the Distributed IB with 10,000 samples from the joint distribution represented by the table in (a), we obtain trajectory in the distributed information plane (black) showing the cross entropy error as function of the amount information incorporated about the interventions. Intervention A is found to be more important in predicting the outcome of Y. When information about intervention B incorporated, the error eventually bottoms out the conditional entropy H(Y (X). (c) With each feature value embedded to two-dimensional Gaussian distribution, alongside the computed confusion matrices. For the seven approximations highlighted in (b), we display the learned compression schemes for interventions A (top) and B (bottom). As noted in the information allocation in (b), the feature values of intervention A are distinguished first, and then the values of intervention B around approximation and beyond.
can visualize the embeddings
15

---

Published as a conference paper at ICLR 2023
wine
wine
6315
0.852
-30
6.500
Dxt. (bite)
0.22
125
6.25
-10
0.75
45
675
0.50
6700
Σ
Dxt. (bits)
Number of features microsoft
C
microsoft
d
0.70
3.00
966-
MSE
0.44
0.75
0.0
0.99
0.5
034
32.5
Σ
DxL (bits)
Number of features mimic3
mimic3
Cruss autropy (bita)
555
Cross enrrroyy (bite))
0.32
2.5
(bits)
0.21
2.1
0.24
0.28
0:28
0.5
028
Σ Dyt (bits)
Number of features
Figure 7: Information plane optimizations for the remaining datasets. We display Distributed IB optimizations alongside LassoNet results. Note the Σ DxL range is not uniform across datasets. (a, b) Wine, regress wine quality with 12 features (Dua & Graff, 2017); (c, d) Microsoft Learn to Rank, 136 features (Qin & Liu, 2013); (e, f) MIMIC-III, similar to MIMIC-II but with 57 features (Johnson al., 2016).
Parameter
Value
Positional encoding frequencies
[1, 2, 8]
Nonlinear activation
Leaky ReLU (a = 0.2)
Feature encoder MLP (one per feature)
[128, 128]
Bottleneck embedding space dimension
Joint encoder MLP
[256, 256]
Batch size
128
Optimizer
Adam
Learning rate
3 x 10-4
Smitill
2 x 10-5
Binal
Table 2: Training parameters for the Distributed IB experiments.
16

---

Published as a conference paper at ICLR 2023
Wine
alcohol
volatile acidity
free 802
residual sugar
10
100
0.5
50
10-
Σ Dia 1 bit RMSE: 0.81
b
1011 1.0
100
Σ Diat 3.5 bits RMSE: 0.76
201
C
Σ Dia:: 28 bits RMSE: 0.71
d
10-
1.0
0.30
201-
Σ Dia' RMSE: 0.67
11H
1.
0.0
1200
1000
50-
201 320 10
Noural Additive Models (Agarwal al., 2021)
Elfect
Effect
Effect
Effeet
RMSE: 0.71
Figure Feature effects: Wine. (a-d) For the Wine dataset, we display distinguishability matrices different levels of approximation for four features. To the left (top) of the distinguishability matrices are the feature values corresponding to the row (column), which have been sorted. The features (alcohol, vol aolaailee acidity, free S02, and residual sugar) were the first four to contribute to the approximations in the low Σ DKL regime of Fig. 7a. As Σ DKL increases, the RMSE improves. (e) The effects of these four features can be directly visualized with NAM; the horizontal coordinates match those of the matrices (a-d) and the black traces display effects for an ensemble of 25 NAMs.
C.2 COMPARISON METHODS
LassoNet (Lemhadri et al., 2021) We used the publicly released Pytorch code" with only minor modifications: (i) fixed typo in inter facefaces. pyy (delete '='), (ii) changed the objective outputs during training to solely the error metric (excluding the regularization terms) for the plots of Fig.
We used the LassoNetClaalee Classi fier and LassoNet Regressor classes, instead of the cross- validation versions that released after the original publication, and performed our cross- validation with the same splits for the other methods. For each dataset we tuned the following hyperparameters with a grid search: L2-regularization weight y E (0, 0.01 0.1, 1} and the dropout
lassonet/interraceee py#
hhttpe///githbb.cmmm//as-s- lassonet/interracees 1251
8eca51 (tt.1l1166777700011110001 00000
17

---

Published as a conference paper at ICLR 2023
a
MIMICIII
PFratio
Renal
Bilirubin
1500
100
30-
20-
30-
10-
20-
Σ Dia' bit
Error (X-ent): 0.36 bits
AUC: 0.65
1800
b
Σ DKL: 4 bits
Error (X-ent): 0.34
AUC: 0.74
100
C
Σ DEL: 10 bits
Error (X-ent): 0.32 bits
AUC:
100
00
d
Σ DEL: bits
Error (X-ent): 0.30 bits
AUC: 0.82
100 500
100
831
30 201
e
Noural Additive Modols (Agarwal et al., 2021)
Elfeet
Difoct
Inffect
Effect
AUC: 0.83
Figure 9: Feature effects: MIMIC-II. (a-d) For the MIMIC-II dataset, we display distinguishability matrices at different levels of approximation for four features. To the left (top) of the distinguisha- bility matrices are the feature values corresponding to the row (column), which have been sorted. The features (P FFrrati io, Age, Renal, and B irubin) were the first four to contribute to the approximations in the low Σ DxL regime of Fig. 4. For the categorical variable Renal the barchart displays the frequency of the particular values. As Σ DKL increases, the cross entropy error decreases and the ROC AUC (area under eurve) increases. (e) The effects of these four features can directly visualized with NAM; the horizontal coordinates match those of the matrices in (a-d) and the black traces display effects for an ensemble of NAMs.
fraction in (0, 0.1, 0.5). We selected the parameters with the best integrated performance on the validation set (metric X number of features), and used the same hidden dimensions as the joint encoder for the Distributed IB method (2 layers of 256 units).
Neural additive models (Agarwal et al., 2021) We used the publicly released Tensorflow code?. We modified the size of the feature encoders: we encountered memory issues with the default feature encoder sizes of [M, 64, 32], with M a function of the number of unique values in the training set for that feature Instead we made each feature encoder size [256, 256, 256] for the datasets with under 20 features, and [128, 128, 128] for the rest.
additive_moden com
wttp:lu...ooooooo///eeeee/// http:///s..tttooooooo/////e////s builder.py 1.283
s/grap ivv_rodde graphh
18

---

Published as a conference paper at ICLR 2023
Bikeshare
weathersit
windapeed
holiday
yF
hun
bett
Σ DKL. RMSE
0.9 bits.
147.2
1.9 bits, 127.0
120.9
4.3 bits, 105.2
5.3 bits. 89.8
8.1 bita. 75.6
11 bits. 65.2
16 bits, 55.2
25 bits. 47.5 3
40 bits, 42.2
59 bits,
38.9
bits
38.0
NAM Effect
Figure 10: Feature compression schemes: Bikeshare, extended. We show the compression schemes for all 12 features of the Bikeshare dataset, for many values of Σ DRt., and compare in the bottom row to the feature effects found by NAM. The features were ordered left to right by the order of crossing threshold value Figure 4. Note the gradual addition of features and of distinguishability as the information rate increases. Eventually most of the feature value confusion matrices tend toward diagonal matrices, signifying that each unique feature value is distinguishable from all others in the compression scheme.
For each dataset we tuned the following hyperparameters with a grid search: hidden activation in {rel lellu exu), dropout in (0, 0.1,).3), weight decay (1.2 regularization) in (0, 10-5 10-4).
19

---

Published as a conference paper at ICLR 2023
Parameter
Dataset
Value
Annealing steps
Bikeshare
10
Wine
10
MIMIC 2
10*
MIMIC 3
5 x 10*
Mice protein
5 x 10*
Microsoft
10
Dropout fraction
Bikeshare
0
Wine
0.1
MIMIC 2
0.3
MIMIC 3
0.5
Mice protein
0.1
Microsoft
0.1
Table 3: Dataset-specific parameters for the Distributed IB experiments.
D CITATION DIVERSITY STATEMENT
Science is a human endeavour and consequently vulnerable to many forms of bias; the responsible scientist identifies and mitigates such bias wherever possible. Meta-analyses of research multiple fields have measured significant bias in how research works cited, to the detriment of scholars in minority groups (Maliniak et al., 2013; Caplar et al., 2017; Chakravartty et al., 2018; Dion et al., 2018; Dworkin et al., 2020b). We use this space to amplify studies, perspectives, and tools that we found influential during the execution of this research (Zurn et al., 2020; Dworkin et al., 2020a; Zhou et al., 2020; Budrikis, 2020). We sought proactively consider choosing references that reflect the diversity of the field in thought, form of contribution, gender, race, ethnicity, and other factors. The gender balance of papers cited within this work was quantified using combination of automated gender-api.com estimation and manual gender determination from authors' publicly available pronouns. By this measure (and excluding self-citations our current paper), our references contain 5% (omman(frstt/oommndd(oo woman/man, and 69% man/man. This method limited that a) names, pronouns, and social media profiles used to construct the databases may not, in every case, be indicative of gender identity and b) cannot account for intersex, -inarry or transgender people. We look forward to future work that could help us to better understand how to support equitable practices in science.
the first and last authors of 12% man/woman, 14%
20


Reviews:
[{"review_id": "em9ZHQpHTSp", "reviewer": "Reviewer_sKPb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new class of interpretable models, instead of limiting model complexity (like using simple interpretable models) or limiting the feature interaction (like in GAM), the paper constrains the amount of information in the model by using Distributed Information Bottleneck IB. IB introduces a constraint after every feature is processed and before any interaction effects between the features can be incorporated. The features are compressed to optimal representations which communicate the most relevant information to the rest of the model This can be viewed as a penalty for the amount of information used by a model. ", "summary_of_the_review": "The overall idea is very interesting. However, I am not completely convinced that the proposed models are in fact interpretable since we do not know how features actually interact. Also, the interpretations provided by the paper were very hard to understand which again questions the overall interpretability of the method.  Interpretations are generally created for humans to have a better understanding of the models and how it uses different features. I am not sure this model achieves this.", "strength_and_weaknesses": "Strength:\n---\n\n- The idea is quite novel, constraining the amount of information going into model for interpretability is very interesting.\n- The idea is well motivated.\n- Unlike GAMs the approach allows feature interaction since there are no restrictions on how the features can interact after IB.\n- Using such an approach on tabular data does not cause model accuracy degradation.\n\n\nWeakness:\n---\n- I am concerned about the interpretability from the following aspects:\n     - Is this model actually interpretable since we are not sure how features interact?\n     - What does it mean (in terms of interpretability) to need part of a feature? Like using some information about the age of humans important?\n    - Explanations produced by the model are very hard to interpret, I read the paper multiple times and I don't fully understand the interpretations produced in figure 5.\n\n- The paper is well written but the figures are very difficult to understand, better interpretations should be included,\n- Given that this paper is mainly about interpretability more interpretability results are needed, for example, the use of a synthetic dataset showing that the optimal amount of feature information is actually extracted by IB. A user study can also be useful.\n- The proposed model only applies to tabular datasets and can not be used in other domains like images, language and time series (although this is the case for NAM).\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "JPi6Mfajt-", "reviewer": "Reviewer_qhwf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a framework for interpretability based on a feature-wise Information Bottleneck. It allows training classifiers that use optimal feature transformations maintaining only the required information. Two forms of visualizations of this information content are presented as global model explanations: Quasi-information plane trajectories with feature-wise information curves and confusion matrices for a single feature to show which information is used by the model. The interpretations are discussed for three tabular datasets with categorical features.", "summary_of_the_review": "The research questions addressed in the paper are timely and interesting to the XAI community. The work presents some promising results and interesting visualizations for the information accessed by a model. However, its computational overhead, the close relation to other works that use IB, and unclear benefits over these works limit the possible impact, which is why I am currently leaning towards rejection.", "strength_and_weaknesses": "Strengths:\n\n1. Overall, I found the presentation well-structured and clear. \n2. Interpretability of AI algorithms is an important and timely topic. Specifically, the tradeoff between model complexity and interpretability is an important subject of study.\n3. The presented techniques can provide reasonable insights into feature usage and feature selection. I think the visualizations, as shown in Figures 4 and 5, provide some useful insights into feature importance and can be useful for feature selection.\n4. Many information-theoretic notions, such as entropy or mutual information, are sometimes hard to grasp. The distinguishability matrices used in this method help to better understand which aspects are important.\n\nWeaknsses:\n\n1. For the proposed method, an additional compression model needs to be trained, and this combination of models also needs to be trained for a sufficiently high number of betas. This is a lot more than required by other post-hoc techniques and may limit the usability of the method in practice.\n2. While the method can quantify the information represented by the features, it does not give insights into how the information is used. For some applications, it might also be beneficial to see the effect of the information on the prediction. It would also be useful to analyze, whether there could be a faulty model that uses the right information but leads to wrong conclusions. Such behavior could not be discerned by the proposed method. Also, interactions are not visualized at all.\n3. The motivation and the advantages of the proposed framework over other black-box explanation methods need to be improved. I am not exactly sure what the advantage of the proposed framework over other feature selection / attribution frameworks is. There are other methods that compute attributions, which also allow for more complex dependencies than GAMs or linear models. An advantage of GAMs / linear models is that these models are white box models. However, the proposed method is a black-box method. In this class a variety of methods including general constructs such as SHAP or the information bottleneck by Schulz et al. can also be applied with interactions between features.\n4. So far, the method is restricted to categorical tabular data. In the paper, only categorical features are discussed. It would be nice to generalize and deploy the framework for continuous features and other data types, such as image or text data.\n\n\nMinor comments:\n\n1. Section 2 (Related work): The paragraph that discusses CCA, ANOVA and how linear correlation can be replaced by MI for feature selection seems very far away from the present work. Instead, the space could be used better to discuss works (e.g., those mentioned in the next box) that deploy the IB for interpretability.\n2. Section 3 is very long. I would have appreciated subsections to better organize the content.\n3. Sum H(X) and Sum I(X,U) (in Figure 2a) only denote the total (mutual) information in all features when the features are independent. Otherwise, it is only an upper bound. As far as I see independence was never assumed, so I was a bit confused here.\n\nQuestions:\n\n1. Is the sparsity enforced (that the KLD for some features is 0 bits for some features, BikeShare in particular) or is it a side-product of the IB approach?\n2. How long is the computational time of the method compared to other approaches? How many values for the KLD were used to compute the curves shown in Fig. 4?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Evsr9BUShnn", "reviewer": "Reviewer_zAhj", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors develop an interpretability framework based on information theory. The advantage/novelty of the approach being that it works directly on the 'informativeness' of the individual features and their joint interactions w.r.t. a given target under potentially a variety of losses and a variety of neural network architectures as predictive models.  ", "summary_of_the_review": "The ideas presented are of interest but more effort should be put in demonstrating the capacity of the framework to offer useful (and understandable) insights.  \n\nAfter the authors' modifications I feel the paper has improved and is ready for publication. ", "strength_and_weaknesses": "Good:\nThe approach goes beyond selecting a (discrete) number of active features but rather continuously varies the degree of compression. \nAn in depth analysis can be performed by looking at a confusion matrix type of representation that is in principle richer.\n\nBad:\nThe claim that the approach improves the interpretability of complex models needs to be better defended: the type of analysis offered in the end is often unclear or complicated: \n- capacity to extract relevant features: the method allows to quantify <<how much information is required - total, and from each feature - for a given level of predictability>>, ok how is this different from the information that we can get from the Lasso Path along the regularization parameter using the LARS algorithm? How does one use this information in practice?\n- as an example of analysis, the author report a confusion matrix type of representation that allows to conclude that the model is first paying attention to discriminating specific ranges of values for some variables. How different would this be from partitioning in a few bins each variable and then run a recursive feature elimination selection technique to identify which (interacting) ranges of which variables are important? \n- <<The capacity for successive approximation, allowing detail to be incorporated gradually >> how can this in practice be used to offer insights? is it possible to identify some prototypical scenarios, i.e. if this happens than this can be concluded?\n- the experimental section would benefit from including some artificial cases where the known important interactions are known a-priori and it can be shown that they are recoverable by the proposed approach but not by other baselines (such as Lasso or recursive feature elimination selection techniques).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "71ju2nGp5H0", "reviewer": "Reviewer_rdFY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a technique to improve interpretability of models by compressing the information in features and feature values using Distributed Information Bottleneck. The redundant and useless features/feature values are intended to be removed by the bottleneck layer so that only the remaining relevant features/values are used for model prediction. This promotes model sparsity which makes the predictions more comprehensible.", "summary_of_the_review": "While the application of the information bottleneck is novel, relevant literature has been missed out and comparison with existing techniques for interpretability needs to be covered in the experiments.", "strength_and_weaknesses": "1. Overall, the paper needs to make the language clearer and improve the presentation of the algorithm.\n\nExamples of text that are unclear:\n\n    a. Abstract: \"The central object of analysis is not a single trained model, but rather a spectrum of models serving as approximations that leverage variable amounts of information and range from the uninformative to the most performant model obtainable.\"\n    \n    b. Abstract: \"tackling the problem of feature selection with inclusion/exclusion existing on a learned continuum.\"\n\n    c. The paper keeps referring to 'a spectrum of models'. But it is not clear what that means in this context. Please provide a clear definition of it in the beginning.\n    \n    d. Section 4.1: \"The signal obtained by the Distributed IB can help decompose through datasets with a large number of features and complex interaction effects.\"\n    \n    e. Section 3: \"The Distributed IB (Estella Aguerri & Zaidi, 2018) concerns...\" -- there is some confusion. Distributed IB is a reference to a previous work as well as being used as the name of the proposed algorithm (Table 1). Should disambiguate in text.\n\n\n2. Two of current most popular techniques for interpretability LIME [r1] and SHAP [r2] are missing in discussions and experiments.\n\n\n3. In the experiments section, there is no comparison with existing techniques for interpretability.\n\n\n4. Section 3 should present the techniques for training and information plane visualization in Algorithm formats. A schema of the overall architecture of the algorithm should also be illustrated as a picture for clarity.\n\n\n5. It is hard to understand whether the proposed technique is a 'global' interpretability technique or 'local'. As per my understanding from statements like in Section 3 \"...The penalty has the effect ... information that is least helpful ... is discarded.\", the technique is 'local'.\n\nA 'global' (e.g., LASSO) technique allows us to identify the most important features/feature values across all training data.\n\nA 'local' (e.g., LIME [r1]) technique allows us to identify the features/feature values that lead to a specific prediction for a specific input data.\n\n\nReferences\n\n[r1] Ribeiro, Marco Tulio, Sameer Singh, and Carlos Guestrin. \"Why should I trust you?: Explaining the predictions of any classifier.\" Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. ACM (2016).\n\n[r2] Lundberg, Scott M., and Su-In Lee. \"A unified approach to interpreting model predictions.\" Advances in Neural Information Processing Systems (2017).\n\n>>>>>>>>>>> Update after author response\n\nI thank the authors for responding to my comments. At this time I do not have sufficient reasons to change my scores. Some comments:\n\n1. \"...from the uninformative to the most performant model obtainable...\" -- The range 'from uninformative .... to ... performant' makes little sense (like saying 'from brightest ... to hottest'). It is not clear what the two ends of the scale 'uninformative' and 'performant' mean i.e., are these in terms of interpretability or accuracy?\n\n2. Thanks for pointing out that the method is 'global'. Global methods are not as interesting (w.r.t research and application) as 'local' methods that explain model response on specific examples, especially when the number of training examples is large and the number of features is small. When the number of features is large (few hundreds), the global feature selection methods are useful, but usually the objective is then to clean the data rather than to enhance interpretability.\n\n3. Figure 4 needs additional labels/captions on the figures to improve readability. E.g., Distributed IB/LassoNet needs to be displayed on the plots, else it is difficult to follow the dense caption text.\n\n4. I am still of the opinion that the language needs to be made clearer and straight-forward, figures should be better presented, and the 'interpretability' aspect is not convincing -- for a 'global' interpretable algorithm, it outputs too much information and requires too much effort from an end user.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster