Parameter Averaging for Feature Ranking
Authors: Talip Ucar
Ehsan Hajiramezanali
Venue: Submitted to ICLR 2023 (2023)
Keywords: Parameter averaging
feature ranking
feature importance
robustness
interpretability
tabular data

Abstract: Neural Networks are known to be sensitive to initialisation. The methods that rely on neural networks for feature ranking are not robust since they can have variations in their ranking when the model is initialized and trained with different random seeds. In this work, we introduce a novel method based on parameter averaging to estimate accurate and robust feature importance in tabular data setting, referred as XTab. We first initialize and train multiple instances of a shallow network (referred as local masks) with "different random seeds" for a downstream task. We then obtain a global mask model by "averaging the parameters" of local masks. We show that although the parameter averaging might result in a global model with higher loss, it still leads to the discovery of the ground-truth feature importance more consistently than an individual model does. We conduct extensive experiments on a variety of synthetic and real-world data, demonstrating that the XTab can be used to obtain the global feature importance that is not sensitive to sub-optimal model initialisation.

Under review as a conference paper at ICLR 2023
PARAMETER AVERAGING FOR FEATURE RANKING
Anonymous authors
Paper under double-blind review
ABSTRACT
Neural Networks are known to be sensitive to initialisation. The methods that rely on neural networks for feature ranking are not robust since they can have variations in their ranking when the model is initialized and trained with different random seeds. In this work, we introduce a novel method based on parameter averaging to estimate accurate and robust feature importance in tabular data setting, referred as XTab. We first initialize and train multiple instances of a shallow network (referred as local masks) with different random seeds for downstream task. We then obtain a global mask model by averaging the parameters of local masks. We show that although the parameter averaging might result in global model with higher loss, it still leads to the discovery of the ground-truu feature importance more consistently than an individual model does. We conduct extensive experiments synthetic and real-world data, demonstrating that the XTab can be used to obtain the global feature importance that is not sensitive to sub-optimal model initialisation.
variety of
INTRODUCTION
Neural networks (NNs) have gained wide adaption across many fields and applications. However, one of the major drawback of NNs is their sensitivity to weight initialisation (McMahan et al., 2017). This drawback is not critical for most classification and regression tasks, and is less obvious in applications such as explainability in most computer vision (CV) tasks. The problem is more obvious in settings, in which we pay attention individual features (e.g., a feature in tabular data, pixel in the image) rather than group of features (e.g., a region in the image). And it becomes critical in settings, in which we might need to make costly decisions based the explanation that the model gives for its outcomes. Few such applications include disease diagnosis in clinical setting, drug repurposing in drug discovery, and sub- nub-population discovery for clinical trials, in all of which the discovery of important features is critical. In this work, we investigate the robustness of neural networks to model initialisation in the context of feature ranking, and conduct our experiments tabular data setting.
The methods developed to explain predictions should ideally be robust to model initialisation. This especially important to build trust with stakeholders in fields such as healthcare. In this work, we define the "robustness" as one, in which the feature ranking from the model is not sensitive to sub-optimal model initialisation. Some examples of robust models are seen in tree-based approaches such the random forest (Breiman, 2001) and XGBoost (Chen & Guestrin, 2016), especially when they are used together with methods such as permutation importance. In these methods, each tree is grown by splitting samples on each decision point by using impurity metric such Gini index for the classification task. The importance of a feature in single tree is typically computed by how much splitting on a particular feature reduces the impurity, which is also weighted by the number of samples the node is responsible for. The importance scores of the features are then averaged across all of the trees within the model get their final scores. It is this averaging that might be one of the reasons why these models are robust and consistent when used for feature ranking. However, we should make a distinction between the robustness of a method and the correctness of its feature ranking as tree-based methods are known to have their shortcomings (Strobl et al., 2007; Li et al., 2019; Zhou & Hooker, 2021). To get robust explanation using neural networks, we could use an ensemble approach by training multiple neural network-based models to get feature importance, and use the majority rule to rank them. However, the ranking of features by using the ensemble of models may still not be easy in cases where the same feature(s) get ranked equally likely across different positions by the models. Moreover, the ensemble approach requires us to store all models so that we can use them to explain prediction at test time, which is not ideal. Instead, in this work, we propose a novel

---

Under review as a conference paper at ICLR 2023
Framework
Data
Training Instance (TI)
(Xtratn )train)
tosst 2 teest)
m
m
When training K-models
m Scale(my+
During final training
Train models K times with different random seed
Xu = m² ox
Final Training
M
ml
+
M
Mg
x
1
Mg
K
M
Mg
m
Noise
h
(a)
(b)
(c)
(d)
(f)
Figure I: Left: Framework; a) Train the models K-times with different random seeds, b) Obtain the
global mask, c) Final training by using global mask (frozen weights) and a new local mask (trained). Details of each training instance; d) Generating mask from input, e) Feature bagging using masked input, f) Aggregating the embeddings of the subsets. E: Encoder, M: Mask, C: Classifier.
Right:
method, in which we obtain single global mask model that is based on averaging the parameters of multiple instances (local masks) of the same model. We take advantage of the sensitivity of NNs to initialization by initializing and training each local mask with a different random seed. We show that although the global model might have a higher loss than an individual model, it ranks features more correctly and consistently, and hence can used to extract the feature importance.
Our primary contributions in this work are the following: We obtain global model by averaging the parameters of multiple instances of a shallow neural network trained with different random inirialisation and use it to extract feature importance. The global model obtained in this manner might have higher loss than any of the individual models (McMahan al., 2017). We show that although this is true, the global model is still able to discover the ground-truth feature importance more consistently than an individual model does. We also demonstrate that weight regularization such as dropout and weight-clipping can improve the robustness and consistency of the global model. We show that the existing the state of the art (SOTA) methods proposed for feature ranking or selection not robust model initialisation. Finally, provide insights via extensive empirical study of parameter averaging using both synthetic and real tabular datasets.
METHOD
Parameter averaging is extensively studied the context Federated Learning (McMahan et al., 2017), in which individual models trained on datasets stored in different devices, and global model is obtained by averaging individual models in various ways. For example, the naive parameter averaging is shown to give lower loss full training set than any individual model trained on a different subset of the data when the individual models are initialized with same random seed (McMahan al., 2017). It is well known that the loss surface for typical neural networks is non-convex (McMahan et al., 2017) and, hence, averaging parameters of models could result in sub-optimal global model, especially when their parameters are initialised differently. However, the loss surfaces of weerpparameterized NNs are shown to be well behaved and less prone to bad local minima in practice (Choromanska al., 2015; Dauphin et al., 2014; Goodfellow et al., 2014). In light of these observations, we investigate settings, in which we can combine multiple models that are initialized and trained with different random seeds to obtain a global model that is less sensitive to sub- sub-optimal initialisation of any individual model. So, in this work, propose framework to obtain such a global model that can be used for both feature ranking and selection. We show that global model is able extract feature importance correctly and consistently especially when the network architecture is shallow. We also show that this behaviour breaks down for deep architectures although regularizing their weights still helps improve them.
2.1 TRAINING
Figure shows our framework, in which we use a shallow neural network as mask generator that in turn is used to learn important features and their weights for downstream task. The hidden layer
2

---

Under review as a conference paper at ICLR 2023
of the mask generator has the same number of hidden units as the number of input features. We use a sigmoid activation at its output to have values in [0. 1] range. In this work, without the loss of generality, we use the classification task for the experiments as shown in Figure I (right). We also adopt the SubTab framework (Ucar et al., 2021), in which we use a shallow, overcomplete encoder architecture followed by classifier and train them together with the mask generator on subsets of the features. Please note that one can train the mask generator with a simple classifier in a standard way instead of integrating it into the SubTab framework. We conduct an ablation study to show that our method is agnostic to whether one uses SubTab or not in Section E of the Appendix. The summary of model architectures and syper-parametes is in Section C.1 and Table Al in the Appendix.
High-level overview: A mask generator, an encoder and a classifier are trained K times using the same training set. K training runs can be parallelized in a distributed setting, or can be run in series on the same machine. At the beginning of each run, we change the random seed before initialising all models (i.e. mask, encoder, and classifier) using Kaiming He uniform initialization (He al., 2015) with the gain of /5 for linear layers. At the end of each training run, we keep the learned weights of the mask model referred as the local mask. So we have K different set of weights for the same mask model at the end of K runs. Then, we average the parameters of K local mask models to obtain the weights of the global mask model. In Section 3.4, we show that the global mask is good at extracting feature importance, but it can be sub-optimal for the classification task since it has a higher loss than an individual model as shown in Figure 6. Thus, we initialise and train the models one final time, during which we combine the output of global mask model (weights frozen) with the one from a local mask (trained). The local mask is trained to gain back any potential loss in classification performance. We should note that one can also choose to fine-tune the global mask, but we prefer to use it as a reference improve on in the final training.
Training to obtain a local mask: We train a local mask generator, a classifier and an encoder for a downstream task. Mask generator, M, gets data X, and generates a mask m my. We then mask the input X by using an entry-wise multiplication with m to generate a masked input XM. We use m² instead of m to push low values in m towards zero. In our experiments, we observed that using m 2 works better than m. m = (X)). and XM = 2 O X
(1)
Inspired by the proposal for subsetting features in SubTab (Ucar et al., 2021), we then generate subsets of data by dividing the features of XM: (z. Zk. }. Learning from subsets of features is shown to be effective in learning good representations for downstream tasks such as classification while enabling parameter sharing between the features of the tabular data (Ucar et al., 2021). We also add noise to randomly selected features in each subset since we observe that adding noise improves classification performance and the robustness of feature ranking as discussed in Section L of the Appendix. To add noise, we first generate a binomial mask, B, and a noise matrix, both of which have the same shape as the subsets, and are samplee for each subset. The entries of the mask are assigned to 1 with probability p. and to 0 otherwise. As example, the corrupted version, Tie of subset i; is generated as following:
Tie = (1 3) o x B
where i E {1, 2....
(2)
Please note that different noise types can be used to generate e. In this paper, we mainly experiment with Gaussian noise, (0, o²), except for SynRank 100 dataset, for which we use swap noise (Ucar et al., 2021). The encoder takes each of the corrupted subsets {Fie, Zje; ...). and projects them up to generate corresponding embeddings, {h4, hy, )..) As in SubTab (Ucar et al., 2021), we aggregate the embeddings by using mean aggregation to get the joint embedding, h, as shown in Figure! 1. Finally, the classifier makes prediction using the joint embedding h. We minimize the total loss by using the objective function in Equation 3 that consists of two loss functions; i) Cross entropy for the classification task (Equation 4), ii) Mask loss consisting of Gini index and an extra term taking the mean over the entries of the generated mask to induce sparsity (Equation 5):
Liotal = Lrask + Lmask
(3)
Least = N EN
yi * log(d)) - (1 yi) * log (6)
(4)
Lmask N/ N 1 EP + N EN 1. Ep Mij- where g = m2 (1 m)² (5)
We update the parameters of the local mask encoder, and classifier using the total loss (Equation 3). At the end of each t training run, we collect the parameters of the local mask, M,.-
3

---

Under review as a conference paper at ICLR 2023
Final training: Once the K training runs completed, obtain global mask model by averaging the parameters K individual local masks as shown in Equation
K K
Mg
M
(6)
1=1
where Mi, is the local mask generator collected at hthh run, and Mg is the global mask. Mg might give sub-optimal performance in downstream task since it is shown to result in higher loss than a local mask model (Section 3,4). But, do not want to lose the benefits that come with averaging the parameters in M, by fine-tuning it. So, to avoid the potential degradation in performance, we do one final training. We change the random seed, and initialize all the models. In this step, we don't 't train the global mask, but rather train a new local mask together with the encoder and classifier as before. However, the difference from the previous training instances is that the mask used for masking the input data is obtained by summing the output of global mask (with frozen weights) and local mask (being trained), followed by scaling this output to make sure that the maximum entry in the mask is as shown in Equations 7, and 8:
my = (X) and my = Mi(X)
m = (m, + m2) /C where C' = max (mg + mı) and XM = -
(8)
We should note that C scaler, i.e. maximum entry in my + my sum. We use the same loss functions described in equations 3, 4 and 5 to update the parameters of the local mask, encoder and classifier. We should note that my can be computed various ways such as using gating mechanism similar to input and forget gates in LSTMs (Hochreiter & Schmidhuber, 1997). We can also choose to keep updating M, in a sequential manner rather than averaging parameters of multiple models all at once. We leave these ideas as future work. Our method is summarized in the Algorithms and 2 in the Appendix.
2.2
TEST TIME
At test time, we use mg shown in Equations 7 to infer the feature importance. my shown to give a robust global ranking of features in our experiments. In XTab, the importance score for a feature the mask weight in the final generated mask. The mask weight indicates the feature's 's relative importance, and we rank the features based on their mask weights. We extract the global feature importances for test set by getting mask values for all samples and computing the mean values over the samples for each feature:
N
1 N
f. = Mg and F.
where J E Rd represents mask weights (i.e. feature importance) for the d number of features in sample zι E Rd, hence f gives an instance- wise feature importance for 7th sample. F E Rd gives the mean of mask weights over N samples and we use when computing the global feature importance. Please note that "global" is an overloaded term in the sense that "global" refers to feature's 's overall importance across all samples in dataset in the context of feature ranking i.e. it is not related the term "global" in the global mask model. Finally, when ranking the categorical features, we can rank individual onc-hot encoded features to show importance of each sub- sabecoregory. We can also sum the weights of each one-hot encoded feature to get the overall weight for the parent category. We use both when comparing our method to other methods Sections L.4 and L.5 the Appendix.
3 EXPERIMENTS
We conduct extensive experiments on diverse set tabular datasets including six syntetic datasets as well as real world datasets such as UCI Adult Income (Income) (Kohavi, 1996). and UCI BlogFeed- back (Blog) (Buza, 2014). We conduct our initial experiments on synthetic datasets since their ground-truth important features are known. We also compare global feature rankings obtained by
4

---

Under review as a conference paper at ICLR 2023
the proposed method for symthetic datasets to those given by recently published neural network- based methods such as Invase (Yoon et al., 2018), L2X (Chen et al., 2018), TabNet (Anik & Pfister, 2021), and Saliency Maps (Simonyan et al., 2013). We report the detailed results Income and Blog datasets in Sections I. and M while additional experiments using synthetic datasets from L,2X (Chen et al., 2018) are in Sections J of the Appendix respectively. For completeness, we also make comparisons with methods such as permutation feature importance used together with random forest (Breiman, 2001) and gradient boosting classifier (Pedregosa al., 201 in Section F of the Appendix.
3.1 DATA
SynRank dataset: We generate a synthetic dataset, referred as SynRank, consisting of training and test sets with 10k samples each for a binary classification to evaluate whether our method can rank important features in correct order. We first generate data X from 10-dimensional standard Gaussian with no correlations across the features (O I). We then shift the sixth feature, fe- to be centered around 100 for the first 45% of samples. For the next 35% of the samples, we shift the first feature Si to be centered around 10. The remaining 20% of the samples are kept same as is. We generate the label Y by sampling it as a Bernoulli random variable with P(Y = 1 = 1/(1 + 9X)). In this case, ( is defined samples respectively. So the first 45% and 35% of the samples will be labeled as and with a high probability, respectively. For the remaining 20% samples, we can expect the proportions of class labels to be similar since /2 is from standard Gaussian with μ = 0. Based on this dataset, we expect that our method discovers the global feature importance ranking as fo J f2.
ezp(fa), exp(fi) and exp(f22 f2) for the 45%. 35% and 20% of the
Syn Rank 100 dataset: This dataset is same as the SynRank, but it has 100 features instead of 10. The features f100- S1, and f7s are the equivalents of fe. /1. f2 in SynRank respectively, and hence the feature ranking is f100 > fi > frs while the remaining 97 features are uninformative.
Synthetic datasets from L2X: We run experiments on four synthetic datasets used for binary classification in L2X (Chen et al., 2018). For each dataset, we have 10k training and 10k test set. In first three datasets, we generate data X from 10-dimensional standard Gaussian and assign labels using P(Y = 1 = 1/(1 + g(X)) in each dataset, where g(X) is defined in the following way: i) XOR: exp(fi fi f2), ii) Orange Skin: exp( Ei f.² 4), and iii) Nonlinear Additive: exp(-1100 s2n(2 * fi) + 2 [fzl + fa + exp(-
In the fourth dataset, iv) Switch: We generate f10 from mixture of two Gaussians centered at +3 respectively with equal probability. If f1o is from the (3,, 1), then we use (fi, /2, f3, ʃa} to generate Y from the Orange Skin model. Otherwise, we use (/s, S6, JT. [a] to generate from the Nonlinear Additive model. ʃo is not used when generating labels. We include some results for L2X Switch in the main paper while the results on other datasets can be found in Section H. I and J of the Appendix.
Income:
Income is public dataset based on the 1994 Census database (Kohavi, 1996). It is used for a classification task of predicting whether the income of a person exceeds S50K/yr by using heterogeneous features such as age, gender, education level and so on. It contains 32.5k and 16k samples for training and test sets respectively. The dataset has 14 attributes consisting of 8 categorical and 6 continuous features. We dropped the rows with missing values, and encoded categorical features using one-hot encoding. Once we encode the categorical features as one-hot, we end up with 105 features in total.
BlogFeedback Referred as Blog in this work, it is a UCI dataset (Dua & Graff, 2017) and contains the number of comments in the upcoming 24 hours for blog posts. It includes 281 variables consisting of 280 integer and real valued features and 1 target variable indicating the number of comments a blog post received in the next 24 hours relative to the basetime. We converted the target to binary variable to use the data for a classification task of predicting whether there is a comment for a post.
The more details Income and Blog datasets are added in Section B of the Appendix.
3.2 COMPARING GLOBAL MODEL. TO LOCAL MODELS
We start our experiments with the classification task on SynRank dataset to get insights into how parameter averaging works for extracting feature importance' as shown in Figure 2. The results for
Unless specified otherwise, when we say feature importance, we refer global feature importance.
5

---

Under review as a conference paper at ICLR 2023
16
Festures
True top featura
Ranking
12
F2
15
14
33
97
0
50
13
16
25
12
110
01 0202 91 SoM,
0.30
Predicted top feature
(c)
(d)
Figure 2: SynRank dataset: a) Feature rankings from each of 10 local masks M, referred as 4 in the figure, obtained at a particular training run for 10 separate runs. b) Feature rankings from the global model, obtained by averaging the parameters of individual models up to a specific run i.e. cumulative average (CA). For example, 9s corresponds to the global model obtained by averaging the parameters of first 3 local masks (11 22 1z). с) The feature importance weights from Mg obtained by averaging the parameters of all local masks. d) Instance-wise feature importance: Comparing ground-truth top feature predicted top feature from Mg for each sample (shown in percentages).
L2X datasets (Chen et al., 2018) can be found in Section H while the details hyperparameters such as p used for generating the binomial mask, 3, and the variance of Gaussian noise, e ~ (O0 o²). can be found in Section C.1 of the Appendix. We train our models the whole training set for the downstream task 10 times, each time with a different random seed. We store the parameters of the trained masks, referred local masks, from each training and denote them as (M,, M,2,-... Mi }- We examine the feature importance obtained from each of 10 local masks for the test set (Figure 2a). We observe that each local mask gives a slightly different ranking. More specifically, we can have different ranking, depending on which seed is used when training the models. The main reason for this variation is the model initialisation since everything else are kept same across different trainings. We then evaluate the effect of averaging over the parameters of the local masks on feature ranking in progressive way. To do this, we obtain a global mask Mg, as cumulative average (CA) over the k local masks, i.e. Mg. = 1/k Ei= ML, as shown in Figure 2b. For example, Mgs corresponds to averaging the parameters of the first three local masks i.e., M₁, M Mr, shown in Figure 2a. We refer to Ma as Mg for simplicity in the rest of the paper. We can see that the feature ranking becomes more stable as we use more local masks in the parameter averaging to obtain the global mask model (Figure 2b).
3.3 EVALUATING THE CORRECTNESS OF FEATURE RANKING
Figure 2c shows the feature weights obtained from M, Mano- We first note that the weights of the features are correlated with the frequency and position of their ranks across all local masks. Specifically, fo is ranked little higher on average than fi is (since occasionally takes 3rd rank). Hence, Mg correctly gives fe a little more weight than fi and suggests fz as the 3rd most important feature as shown in Figure 2c. Moreover, we investigate the instance-wise feature importance for XTTa.. Since we optimize our models using global objective function (Equation 3), we expect Mg to be biased towards globally important features when estimating the feature importance for an individual sample as confirmed in Figure 2d. We see that M, is biased towards the globally important features, and fails to rank f2 as the most important feature for 20% of the samples the test set, for which we generate labels using P(Y = 1 = 1/(1 + cxp(fz)) and f2 is sampled from N(0 1).
3.4 THE ROBUSTNESS AND CONSISTENCY PARAMETER AVERAGING
We compare the robustness and consistency of the global feature importance extracted from various methods by running each method 10 times with different random seeds on SynRank and L2X Switch datasets in Figure 3. XTab discovers the top three features as fo > fi f2" for SynRank consistently while the rankings from TabNet, Invase, L2X and Saliency Maps are not robust to initialisation (Figure 3a). For example, TabNet has wide variation in its rankings whereas L2X gets the ranking completely wrong. Please note that Invase and L2X are originally proposed as feature selection methods. So, we use the feature-selection probabilities for Invase and the frequency of

---

Under review as a conference paper at ICLR 2023
uo N
Ranking
w 2
Ranking
bow
5
8.1.12
XTab TabNet 12X
Invase
SM
XTab
TabNet
L2X
Invase
SM
(a) Rankings of fo. Ji, and fa from SynRank
(b) Ranking of fio from L2X Switch
Figure 3: Comparing different models for the consistency and correctness of their feature rankings by using SynRank and L2X Switch datasets. fo > fi > fz is the correct feature ranking for SynRank while fio is the most important feature for L2X Switch. Abbreviations; SM: Saliency Maps.
feature being selected for L2X when computing the feature rankings. Moreover, in Figure 3b, we repeat the same experiment for L2X Switch dataset and show the ranking of the most important feature f10 by the same models. XTTab ranks f10 as the most important feature consistently while others fail. Please note that this is the hardest synthetic dataset in a way that the effect of f10 on the sample labels is not direct, rather it influences the labels indirectly by deciding which features to be used for label generation. The details of the training for the models as well as the ranking results for all 1,2X datasets are in Sections C and J of the Appendix, respectively.
The effect of weight regularization on the robustness. We run additional experiments on SynRank under three conditions: We apply i) No weight regularization to the weights of the mask model i.e., our original setting so far, ii) Dropout with p = 0.2 for the layers with leaky ReLU activation, iii) Weight-clipping 0.2, 0.2)) limit the magnitude of the weights in each layer. For each of the three cases, we train 20 separate models and compare two different settings. In the first setting, we compute the variation in the feature rankings given by 20 local mask models (top second setting, we obtain 100 global models, each of which is obtained by averaging the parameters of 10 local models bootstrapped from 20 models. We compare the variation in feature rankings given by global models (second setting- bottom row in Figure 4a-c) to that of 20 local models (first setting top row). We observe that: i) Regularization methods such dropout and weight-clipping has a little effect in improving the variation across 20 local models (e.g., comparing f1, f2 and fs across three cases at the top row). ii) They do not improve the robustness of the parameter averaging significantly (e.g., comparing same features across three cases at the bottom row). iii) Parameter averaging results in more robust estimation of feature rankings (comparing top and bottom rows). Overall, the global models are able to discover important features in the correct order (e.g.. f6 > fl > f2). and the variation in feature ranking across global models is small (almost zero) for ground truth important features (e.g., fi, f2 and fe at the bottom row in Figure 4c). We should note that we conduct the same experiment for SynRank 100 (Figure 4d), L2X Switch, Income, and Blog datasets, for the latter three of which the weight regularization helps improve robustness of parameter averaging (Figures 5, A8, A18, and A22 respectively). Thus, the weight regularization can help improve the robustness and weight-clipping works better than dropout in our experiments. We also observe that the parameter averaging itself pushes the magnitude of the weights towards a tight range around zero as shown in Figure A9 (Section I.I of the Appendix), indicating a potential relationship between robustness and a tighter weight distribution.
in Figure 4a-c). In the
Exploring the mask generator with deeper architecture. We re-run the robustness experiments by replacing the shallow mask model with a deeper model (5 hidden layers) and show that the weight regularization also helps with the robustness of parameter averaging in deeper networks although the parameter averaging does not work as well as the shallow networks especially if weight regularization not used (see Figure A1f(f) for Income dataset the Appendix). Additional results for SynRank, L2X Switch, and Income can be found in Section F.2, I.2 and L.3 of the Appendix respectively.
Comparing the loss and solution space of local and global models. We consider two sets of parameters: O, for a local model and 0,, for the global model. We can compare the possible loss and solution space by interpolating from local to the global model: 0 = (1-a) ft + a * 8g. where is swept from 0 to in 50 steps. Figure 6a-b shows two separate examples of such interpolation done using SynRank dataset. In Figure 6a, the local model (a = 0) has the wrong feature ranking of f1 f6 f2. As move from local model (a 0) the global model (a 1), the estimate
7

---

Under review as a conference paper at ICLR 2023
Bee Bom-gunasn
Ranking
Renking
Gew
Rankinn B
Rankiing
n09 Apooes
2000000 5
Features
Features
Features
Features
Ranking
S Dowwwa
Ranking
manking
Bes u
Ranking
000000000
00220000
Features
Features
Features
Features
(a) No regularization
(b) Dropout (p=0.2)
(c) Weight-clipping (WC)
(d) WC MCCC-ynRan1100
Figure 4: SynRank and SynRank 100 datasets: The important features are fe > f1 f2 and f100 fi > S75 for SynRank and SynRank 100 respectively while the rest of the features is not informative. Top row: Variations in feature rankings across 20 local mask models when we apply a) no weight regularization, b) Dropout (p 0.2), and c) Weight-clipping ((-0.2, 0.2]) for SynRank dataset. Bottom row: Variations in feature rankings 100 global models, each of which obtained by averaging the parameters of 10 models bootstrapped from 20 local models for the same three cases. d) Same as (c) i.e. weight-clipping (WC), but the experiment is repeated for SynRank 100 dataset.
Hanking DeSNDutn
Ranking
(a) Income-Loccs
(b) Income-Global
(c) Blog-Local
(d) Blog-Global
Figure 5: Income and Blog datasets: Repeat of the experiment in Figure 4c, showing variations in feature rankings across 20 local mask models in (a) and (c) as well as variations across 100 global models in (b) and (d) (Please features in Income are; ms: marital status, en: education-nume num, cg: capital-gain hpw: hours-pea week, occ: occupation, rel: relationship, cl: capital-loss, native-country.
Section L.3 and M.1 for more results). Abbreviations used for
of feature ranking gets better. The global model estimates the ranking correctly (/6 fl f2).
Similarly, in Figure 6b, different local model has the wrong feature ranking of f6 > f2 > fl while the global model again gives the correct ranking. Moreover, for SynRank, the expected global feature importance weights are F [Fi, F2, F6, ..., Fio] [0.35, 0.2, O..., 0.45, 0]. So, can compute the loss of the mask model by using mean squared error between expected weights and To Σd=
the model's estimate: L 1 -10 (Fa F2)², where F is defined in Equation 9. Thus, we plot how this loss changes as we interpolate from local to global model in Figure 6c, which corresponds to the interpolations in Figure 6a-b. It indicates that the loss increases model, which is mainly due to the fact that the noise floor (nf) increases both cases as shown in Figure 6a-b and that the weight of f6 moves away from 0.45 in the case of Figure 6b. Averaging parameters of the multiple models with different initialisation is known to give a global model that might have higher loss than any of the local models (McMahan al., 2017). However, we show that the global model is still better at estimating feature ranking and more robust than the local models.
move towards the global
4 RELATED WORKS
Parameter averaging Averaging parameters to get a global model has been extensively studied in the Federated Learning setting under different assumptions; The convex optimisation under ITD)
8

---

Under review as a conference paper at ICLR 2023
Feature weight
0.4
Feature weight
0.4
0.018
f1
0.015
0.3
0.3
f6
0.013
0.2
0.2
nf
0.010
0.007
0.1
0.0
0.5
1.0
0.0
0.5 alpha
1.0
0.0
0.5
1.0
alpha
alpha
(a)
(h)
(c)
Figure 6: SynRank: (a-b): Two examples showing the changes in the weights of features estimated by model that is obtained by interpolating between single local model and the global model: 8-= (1 a) 0r + a * fg. As we move from local model (a = 0) to the global model (a = 1), the estimate of feature ranking gets better (i.e., f6 > fl > f2) even though the loss of the model gets worse as shown in (c). nf: Noise floor, referring to average weight of the rest of the uninformative features i.e., /3 15, /7 f10. The plots show the mean and 95% confidence intervals computed by repeating the experiments with 100 global models, each of which is obtained by bootstrapping 10 out of 20 local models. In (a-b), global models are interpolated with two different local models.
data assumption, in which it is shown that the global model is no better than single model in the worst-case (Arjevani & Shamir, 2015; Zinkevich et al., 2010; Zhang et al., 2012). ii) The non-convex optimisation under IID and non-IlD data assumptions, in which individual models are initialized from the same random initialization to avoid bad local minima before training each independently (McMahan al., 2017). Parameter averaging using models with same initialisation is studied under different contexts as well (Wortsman et al., 2022; Polyak & Juditsky, 1992; Izmailov et al., 201 2018). In Wortsman al. (2022), the authors average the parameters of multiple models, each of which pre-trained model by using different cyper-parameters In this case,
is obtained by fine-tuning the fine-tunig process starts from the same initial model i.e., pre-trained model. The parameter averaging in (Polyak& & Juditsky, 1992; Izmailov et al., 2018) is done by averaging the parameters of the same model along the trajectory stochastic optimisation during the training. Moreover, dropout method is previously shown approximate model averaging implicitly (Srivastava et al., 2014; Goodfellow et al., 2013). However, we show that the dropout alone is not enough to achieve robustness in Section 3.4. In this work, study non-convex setting under the IID assumption, and consider averaging model parameters obtained across multiple training runs to produce the final global model. What differentiates our method from aforementioned works is that initialise the models with different random seeds. Although averaging the parameters of the models trained with different random seeds is shown lead bad local minima (MeMahan et al., 2017), we show that the global model obtained in this way gives robust estimate of the feature importance and can be used for feature ranking. We review other related works in Section D of the Appendix.
CONCLUSION
In this work, we show that global model obtained by averaging the parameters multiple instances of a shallow network trained with different random seeds can be used to estimate global feature importance and that its estimates are not sensitive sub-optimal initialisation of individual models. Furthermore, regularization methods can enhance the robustness of parameter averaging. We give insights into how parameter averaging can be useful for feature ranking through extensive experiments using synthetic and real tabular datasets. Our method can also be extended to other modalities such as images, graph etc. and we leave as a future work. Finally, the following are some of the shortcomings of our approach; i) The global model is biased towards globally important features and hence instance-win wise feature importance will be biased, ii) We still need to do hyper parameter search hyper-parameter
for feature bagging, noise etc., iii) The features in the real world datasets can have more intricate relationships such as multicollinearity. making the ranking of the features difficult, in which case our method can be used for feature selection rather than feature ranking, and iv) Our method needs additional compute during training, but this can be climinated by integrating our method into K-fold cross validation, assuming K 10.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Sercan O Anık and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. In AAAI, volume 35, pp. 6679-6687, 2021.
Yossi Arjevani and Ohad Shamir. Communication complexity of distributed convex learning and optimization. Advances in neural information processing systems, 28, 2015.
Leo Breiman. Random forests. Machine learning. 45(1):5-32, 2001.
Krisztian Buza. Feedback prediction for blogs, In Data analysis, machine learning and knowledge discovery, pp. 145-152. Springer, 2014.
Jianbo Chen, Le Song, Martin Wainwright, and Michael Jordan. Learning explain: An information- theoretic perspective on model interpretation. In International Conference on Machine Learning, pp. 883 892. PMLR, 2018.
Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. Proceedings of the 22nd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, KDD 16, pp. 785-794, New York, NY, USA, 2016. ACM. ISBN 978- 42324----0- doi: 10.1 N0.1145//339772.2939778 URL httt:://///tt /doi. acm. org /10 110.1144 1145/ 2939 39672.667222 2939789
Anna Choromanska, Mikael Henaff, Michael Mathieu, Gérard Ben Arous, and Yann LeCun. The loss surfaces of multilayer networks. In Artificial intelligence and statistics, pp. 192-204. PMLR, 2015.
Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in digen-imnniionll non-convex optimization. Advances in neural information processing systems, 27, 2014.
Dheeru Dua and Casey Graff. Uci machine learning repository, 2017. URL http::htt/chhhiiee ics. wwu.ucueee mediieddmmm.
Jerome H Friedman. Greedy function approximation: gradient boosting machine. Annals of statistics, pp. 1189-1232, 2001.
lan Goodfellow, David Farlee Mehdi Mirza, Aaron Courville, and Yoshua Bengio. Maxout networks. In International conference on machine learning, pp. 1319-1327. PMLR, 2013.
Ian J Goodfellow, Oriol Vinyals, and Andrew M Saxe. Qualitatively characterizing neural network optimization problems. arXiv preprint arXiv:1412.6544 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1786 1997.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Av- eraging weights leads to wider optima and better generalization. arXiv preprint 1rXxv:8803.05407 2018.
Ron Kohavi. Scaling up the accuracy of naive-bayes classifiers: A decision-tree hybrid. In Kdd, volume 96, pp. 202-207, 1996.
Xiao Li, Yu Wang, Sumanta Basu, Karl Kumbier, and Bin Yu. A debiased mdi feature importance measure for random forests. Advances in Neural Information Processing Systems, 32, 2019.
Pantelis Linardatos, Vasilis Papastefanopoulos, and Sotiris Kotsiantis. Explainable ai: A review of machine learning interpretabilltyy methods. Entroyy 23(1):18, 2020.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
arXiv: 2017.
10

---

Under review as a conference paper at ICLR 2023
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Commumication efficiient learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1282. PMLR, 2017.
Ian E Nielsen, Dimah Dera, Ghulam Rasool, Nidhal Bouaynaya, and Ravi P Ramachandran. Robust explainability: tutorial on gradient-based attribution methods for deep neural networks. arXiv preprint ar arXiv:2107. 14000, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché- -ucé-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019. URL http: wttppapppssssss///ee neuri wctpccc/paae wccpappr//////eeeceeeeeeeeeeeape 1115--pttrchaa---te ivee- -e- -ersstttieeeyylllllll weeeeee-hhhheeee forerformancee -eeaaancccdeeeee-eeee deep- library. pdf.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830 2011.
Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal control and optimization, 30(4):838-855, 1992.
Ali Raza, Kim Phuc Tran, Ludovic Koehl, and Shujun Li. Designing ecg monitoring healthcare system with federated transfer learning and explainable ai. Knowledge-Bil Based Systems, 236:107763, 2022.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference computer vision, pp. 618-626. 2017.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint urXiv:1312.6033. 2013.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-198, 2014.
Carolin Strobl, Anne-Laure Boulesteix, Achim Zeileis, and Torsten Hothorn. Bias in random forest variable importance measures: Illustrations, sources and solution. BMC bioinformatics, 8(1): 1-21, 2007.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pp. 3319-3328. PMLR, 2017.
Erico Tjoa and Cuntai Guan. A survey on explainable artificial intelligence (xai): Toward medical xai. IEEE transactions on neural networks and learning systems, 32(1 (((1))4793--8133 2020.
Talip Ucar, Ehsan Hajiramezanali, and Lindsay Edwards. Subtab: Subsetting features tabular data for self-supervised representation learning. Advances in Neural Information Processing Systems, 34, 2021.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roclofs, Raphael Gontijo-Lpess, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Komblith, al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, pp. 23965-23998. PMLR, 2022.
Jinsung Yoon, James Jordon, and Mihaela van der Schaar. Invase: Instance wise variable selection using neural networks. In International Conference Learning Representations, 2018.
11

---

Under review as a conference paper at ICLR 2023
Yuchen Zhang, Martin Wainwright, and John C Duchi. Communication--cfiitii algorithms for statistical optimization. Advances in neural information processing systems, 25, 2012.
Zhengze Zhou and Giles Hooker. Unbiased measurement of feature importance in tree-based methods. ACM Transactions on Knowledge Discovery from Data (TKDD), 15(2)11211,00)
Martin Zinkevich, Markus Weimer, Lihong Li, and Alex Smola. Parallelized stochastic gradient descent. Advances in neural information processing systems, 23, 2010.
A ALGORITHM
Algorithm Main learning algorithm
input: batch size N, the structure of encoder (c), classifier (c), mask (m);
Set the seed and do train-test split seed = 57;
Training and test sets.
X Xtrain Xtstt get data(random_seed = seed);
Initialize list to hold mask models from each of K-training instances.
masks lisk = =
Train models K-times, each time with a different random seed. for in {K} do
Change random seed and initialize models seed seed + 17;
initialize all models (e, c, m) with new seed;
Train mask, classifier encoder and return trained mask model.
-, - frain-for train_for domrrrraarrtrrmnnnnnnnnn (Xtrains Mtrain))
Collect parameters of the local mask model. masks list.append( (M,);;
end
Average the parameters the masks collected to obtain global mask
M, k Ek, Mi:
Change the random seed and initialize models
seed = seed + 17;
initialize all models (e, c, m) with new seed;
Train mask, classi fer and encoder while using global mask (frozen)
e, Mr, Mg troin-for fr donnstreemm(rraiinng (Xtrain Vtrain), Mg M,); Return Encoder (e), Classifier (c), Local and Global masks return e, c, M₁, Mg:
12

---

Under review as a conference paper at ICLR 2023
Algorithm 2: Training routine used for the downstream task
def train for downstreammraiinnng, val=None, Mo=None)
X troin Utrain = training foreach epoch Epochs do
foreach batch (Xa. E (Xtrains Mtrain) do Get the output from the local mask m = M(
If the global mask is provided, get its output
if Mg + None then
with gradient:
my = (x)(x))
Final mask is the sum local and global masks, scaled by the max entry in the sum.
m (m + mgg/mar(mm)) + mg)
Get the masked input using the square of the mask
XM = m² Ø X
# Generate subsets of data (i.e. feature bagging) ceneeratesuuscts((XN) # Obtain embeddings
T4, 24, T'k.
ha hy. hg.... (encoder(z) for I in [g.
# Aggregate embeddings to get the joint embedding aggregate(h,; h2, ...)..)
Get predi rredictions using the joint embedding
Upred = classi fiersifier(h))
Compute losses
Ltaek = CroosEntropy(@ppedd.u
Tmmask Gini(m)
L Trask + T-mask
# Update models. Note that the global mask is not trained.
backprop and update local mask, encoder and classifier
If validation set is provided, run validation using the steps above (no loss computation).
Return Encoder, Classifier, Local and Global masks return encoder, classife,, Mj, M,
13

---

Under review as a conference paper at ICLR 2023
B DATA
B.1
ADULT INCOME DATASET
Adult Income (Income) is public dataset based the 1994 Census database (Kohavi, 1996). is used for classification task of predicting whether the income of person exceeds S50K/yr by using heterogeneous features such as age, gender, education level and so on. contains 32.5k and 16k samples for training and test sets respectively.
Train- Validation- Test Split: Training and test sets are provided separately (Kohavi, 1996). We split the training set into training and validation sets using 80-20% split to search for hyper-parameters. Once hyper-parameters was fixed, we trained the model on the whole training set.
The dataset has 14 attributes consisting of categorical and 6 continuous features. We dropped the rows with missing values, and encoded categorical features using one-hot encoding. Once we encode the categorical features as one-hot, end up with 105 features in total. Features are normalized by subtracting the mean and dividing by the standard deviation, both which are computed using training set.
Features:
Class imbalance: It an imbalanced dataset, with only 25% of the samples being positive.
B.2 UCI BLOGGFEDBACKK DATASET
Referred Blog in this work, it contains the number of comments the upcoming 24 hours for blog posts. Although the dataset can be used for regression, we turn it to a binary classification task to predict whether there comment for a post or not.
Train- Siiiin-Vlidation-TTe Test Split: UCI (Dua & Graff, 2017) provides one training set, and 60 small test sets. We combined all the test sets into one test set. We split training set training and validation using 80-20% split to search for hyper-parameters. We trained the final model using all of the training set.
It includes 281 variables consisting of 280 integer and real valued features and target variable indicating the number of comments a blog post received in the next 24 hours relative to the basetime. We converted the target (the last column in the dataset) to binary variable, in which 0/1 indicates whether the blog post received any comments. Similarly to Income dataset, we used standard scaling normalize the features.
Features:
Class imbalance: ~ 36% of the samples are positive training set while is 30% in the test set.
B.3 SYNTHETIC DATASETS FROM L2X:
We run experiments four synthetic datasets used for binary classification in L2X (Chen al., 2018). For each dataset, we have 10k training and 10k test set. In first three datasets, we generate data X from (0-dimensional standard Gaussian and assign labels using P(Y 1|X) 1/(1 g in each dataset, where g(X) ) is defined in the following way: i) XOR: exp(S [ f2), ii) Orange Skin: erp(E== f:² -22-4, 4), and iii) Nonlinear Additive: ezp(-100 100 sin(2 fi) 22 |fzz ferpsfs-pp----- exp( In the fourth dataset, iv) Switch: We generate f10 from a mixture of two Gaussians centered at +3 respectively with equal probability. If Sio from the (\3,, 1). then generate from the Orange Skin model. Otherwise, we use [Js, J7, S8} to generate from the Nonlinear Additive model. fg is not used when generating labels.
{hi, f2, f2, J4] to
B.4 DATA LICENSE
Aduld Income and BlogFeedback are under Open Data Commons Public Domain Dedication and License (PDDL).
14

---

Under review as a conference paper at ICLR 2023
C DETAILS OF THE EXPERIMENTS IN THE MAIN PAPER
C.I MODEL ARCHITECTURES AND HYPER-PARAMETER FOR XTAB
The classifier has three linear layers, two of which are followed by a leakyReLU and dropout (p=0.2). For the mask generator, use two architectures; i) shallow: A linear layer followed by leakyReLU and a final linear layer, ii) deep: five linear layers, each followed by leakyReLU, and a final linear layer. The last layer for both mask generator and classifier uses sigmoid activation. The number of hidden units in each layer in the mask generator is same as the number of features in the input while we use 1024 units in each hidden layers of classifier. During training, a learning rate of 0.001 is used for all experiments and we optimize the batch size and total number of epochs.
Table A1: Architectures & are; a: Standard deviation used for Gaussian noise, MR: Mask ratio p.
used in framework for each dataset. Abbreviations
Dataset
Mask
Encoder
Classifier
Subsets / Overlap
a/MR (p)
Noise
Batch/Eppoch
SynRunk
110, 10]
[1024]
[0224, 1024, 1024]
37
0.5A0.5
Gaussian
1024, 40
SynRankI00
[100, 100]
[1024]
[1024. 1024, 1024]
2/75%
NA/0.5
Swap
1024, 40
Income
[105, 105]
[1024]
[1024, 1024, 1024]
//25%
0.3/0.2
Gaussian
1024,
Blog
[280, 280]
[1024]
[1024, 1024. 1024]
7/75%
0.3/0.2
Gaussian
256, 20
1.2X XOR
[10. 10]
[1024]
[1024, 1024, 1024]
2/75%
0.05/0.2
Gaussian
1024, 40
L2X Orange
[10. 10]
[1024]
[1024, 1024, 1024]
2/75%
0,05/0.2
Gaussian
1024. 40
L2X N. Additive
[10. 10]
[1024]
[1024, 1024, 1024]
2/75%
0.01/0.2
Gaussian
1024,
L2X Switch
[10.
[1024]
[1024, 1024. 1024]
2/75%
0.05/0.3
Gaussian
1024,
C.2 IMPLEMENTATION AND RESOURCES
We implemented our work using Py Torch (Paszke et al., 2019). AdamW optimizer (Loshchilov & Hutter, 2017) with betos = (0.9, 0.999) and eps 07 is used for all of our experiments, We used a compute cluster consisting of Volta GPUs throughout this work.
C.3 DETAILS FOR TRAINING 1,2,, INVASE, TABNET, SALIENCY MAPS AND INTEGRATED GRADIENTS (IG)
L2X: We used the official implementation of L2X2. We set all of hyperparameters, following the instruction in L2X paper (Chen et al., 2018). For each data set, we trained neural network model with three hidden layers. The explainer a neural network composed of two hidden layers. The variational family is based on three hidden layers. All layers linear with dimension 200. The number of desired features is set to the number of true features. We fixed the step size to be 0.001 experiments. The temperature for Gumbel--offtma approximation is fixed to be 0.1. Since the model is proposed for feature selection, we used the average number of selected features for each sample the test rank them.
Invase: We followed the hyperparameter selection as instructed in Invase paper (Yoon et al., 2018) and all the experiments based on the official Keras implementation'. We fixed the learning rate and À as 0.0001 and 0.1, respectively. The actor and critic models are three layer neural networks with hidden state dimensions 100 and 200, respectively. L2 regularization is set to be 0.001 and activation function is ReLU, We used the feature selection probability, which is the output of the actor model, to rank the features.
We used the well established Py implementation of TabNet We set the hyper- parameters as N₂ = N₁, 8, Aeparse = 0.001, B 1024, B, = 128, 7 = 0.3, and learning rate 0.02. For all experiments, we used sparsemax as the masking function and OneCycleLR as the learning rate scheduler. The other parameters are set to be same as the default choices.
TabNet:
Saliency Maps and Integrated Gradients (IG): For Saliency Maps (Simonyan et al., 2013) and IG (Sundararajan et al., 2017), we used the same architecture as XTab and trained the models using SGD
hhtts//gitthbbcco///nnbbLLLLL1L1 hhang://ggtiaaaccmm/eeemmaaaaaaa
15

---

Under review as a conference paper at ICLR 2023
with learning rate of 0.01. For Saliency Map, we considered absolute value of each sample gradient for ranking. For IG, we used Captum Py PyTochh library".
D MORE ON RELATED WORKS
Explainability The literature in explainability and model interpretation is extensive and we refer the reader to the survey papers (Linardatos et al., 2020; Nielsen et al., 2021: Tjoa & Guan, 2020) for a more complete review. In this work, we compare our method to the commonly used methods (Random Forest (Breiman, 2001), Gradient Boosting Classifier (Friedman, 2001; Pedregosa et al., 2011)), to those based on the gradients and/or activations (Saliency Maps (Simonyan et al., 2013) and Integrated Gradients (IG) (Sundararajan et al., 2017)). to the ones that rely on the learnable masks (TabNet (Ank & Pfister. 2021), Invase (Yoon et al.. 2018)) and to some of the recently published feature selection methods (L2X (Chen et al.. 2018), Invase (Yoon al., 2018)). What distinguishes our work from the aforementioned works is that we focus on the sensitivity of the feature rankings to model initialisation neural networks. Our goal is to achieve the robustness of tree-based methods such as Random Forest (Breiman, 2001) in neural network setting. In this regard, we compare our results to neural trrtook--bseee methods both in the main paper as well as in the Appendix.
Explainability in Federated Learning There is some recent work in the intersection of explainable AI (XAI) and Federated Learning (FL) such as the application of Gradient Gradient-weighted Class Activation Mapping (Grad-CAM) (Selvaraju et al., 2017) to explain the classification results in clectrocardiogra- phy (ECG) monitoring healthcare system (Raza et al., 2022). However, it still remains to be an open problem. Lastly, although our method is not proposed for Federated Learning setting, we believe that it can still be used in this area.
E ABLATION STUDY ON ARCHITECTURE
Training Instance (TI)
181
When training K-models During final training
my mmale(mp+m.))
XN m² xX
Mj
X
+
X
Mg
mm
Notse
(b)
Figure A1: Standard architecture that uses only mask generator and a simple MLP-based classifier.
Ramking
Bew vara
0.00000000000 &
$ 0000E6 @
Features
Features
Figure A2: Repeating the experiment in Figure 4a, using a standard architecture shown in Figure Al. The important features are fe fi f2 for SynRank while the rest of the features is not informative. Left: Variations in feature rankings across 20 local mask models Right: Variations in feature rankings in 100 global models, each of which is obtained by averaging the parameters of 10 models bootstrapped from 20 local models for the same three cases. Comparing it to our original architecture based on SubTab (Ucar et al., 2021), we see that the robustness of feature ranking achieved via parameter-averaging is agnostic to the choice of architecture used for the downstream task.
16

---

Under review as a conference paper at ICLR 2023
F ADDITIONAL RESULTS FOR SYNRANK DATASET
F.I RANKING COMPARISON OF DIFFERENT MODELS IN TABLE FORMAT
Table A2: Comparison of feature ranking obtained from 10 runs of different methods for SynRank. Expected ranking is f6 f1> f2, and incorrect rankings are shown in bold. This is the same result shown as a figure in Figure 3a in the main paper.
SynRank dataset
Ranking
XTab weight regularization
Runs
5
9
10
Test Acc.
16
11
13
13
[10
0:5971
2
f6
11
F2
19
13
18
14
f7
F10
0.9965
f6
f1
15
E3
f7
f10
0.9951
f6
f1
13
13
FLO
E7
0.9955
7
f6
11
F2
19
15
F4
IS
F1O
f7
0.9964
f6
P9
F3
f7
F10
0.9964
9
f6
11
19
13
I7
f10
0.9971
10
f6
F1
F2
FS
F4
18
T7
F10
0.9973
Rankings
GBCP
16
1I
12
L1O
19
1N
17
14
E3
0.9996
Rankings From TabiNe
fl
f6
18
19
f10
14
f3
0.9990
9.9988
f6
f1
F2
110
19
15
18
14
0. 0.9980 17
ff
F7
F9
F3
18
0.9978
f6
FI
[3
19
15
0.9984 47
6
f6
II
13
12
14
110
ES
19
0.9988 E7
fI
16
19
F10
0.9963 F4
9
16
LS
110
0.9995 I7
Rankings Irom Invisee
f10
0.9988
19
F10
17
08
0. 0.9989
fI
16
12
18
F5
f7
0. 9.9999 f10
f6
Fl
ES
f10
19
0. 0.9989 f3
17
IR
F1O
ES
0.9992
15
9.895 18
8
fI
16
12
F4
IS
F3
F7
f10
0. 0.9993
F3
F7
16
II
I3
I8
f10
0.9991 E7
Runkings from
L2X
2
fI
12
16
13
f10
F4
I8
19
0. 0.9983
f10
19
0.9969
12
13
IS
0.9975
5
FI
12
16
13
F4
19
F1O
0.9976
II
12 16
19
F10
0.9985
8
FA
19
F10
0.9976
10
FI
12
E6
IS
FA
I7
19
F10
0.9979
Rankings Saliency Maps
56
Fl
14
13
F1O
I8 0.9817
fI
16
13
17
F4
0.9816 1
0.9820
16
I1
F2
E10
IN
15
I3
17
0.9824 1
I8
f10
F7
0.9812 E4
15
0.9817
7
fl
F6
12
17
19
f3
f1Q
18
0.9814
0.9817
f6
fl
15
f10
f7
f
f4 0.9821
f6
1I
R8
15
f10
14
19
0.9813
17

---

Under review as a conference paper at ICLR 2023
F.2
RESULTS FOR DEEP NETWORK
Ranking
Renking
Ranking
10
11 12 13 14 15 16 17 18
19 110
11. 12 13 14 IS 16 17 гa 19 110
11 f2
f3
14 15 16 Features
f7 1B 19 110
Features
Features
Ranking
Ranking
Ranking
f2
19
Features
Features
(a) No regularization
(b) Dropout (p=0.2)
(c) Weight-clipping ([-0.2, 0.2])
Figure A3; Measuring the robustness of parameter averaging for deep networks (5 layers). We use SynRank dataset, in which the important features are fo > fi > fz while the rest not informative. Top row: Variations in feature rankings across 20 local mask models when apply a) no weight regularization, b) Dropout (p = 0.2), and c) Weight-clipping ((-02, 0.2)). Bottom row: Variations in feature rankings in 100 global models, each of which obtained by averaging the parameters of 10 models bootstrapped from 20 local models for the same three cases. Weight-clipping helps global models discover important features in the correct order consistently (see the last column, bottom row)
80
Frequency
80
Frequency
Frequency
60
60
200
40
40
100
20
-0.5
0.0 Weights
0.5
-0.5
0.0
0.5
-0.2
0.0 Weights
0.2
Weights
80
80
Frequency
Frequency
60
Frequency
60
40
40
40
20
20
-0.25
0.25
-0.25
0.25
-0.2
Weights
Weights
Weights
(a) No regularization
(b) Dropout (p=0.2)
(c) Weight-clipping
Figure A4: Measuring the weight distribution for deep networks (5 layers) when we apply a) no
weight regularization, b) Dropout (p 0.2), and e) Weight-clipping (0-0.2, 0.2)). Top row: The weight distribution of a single local mask model selected at random. Bottom row: The weight distribution of global mask model obtained by averaging the parameters of 10 models bootstrapped from 20 local models.
18

---

Under review as a conference paper at ICLR 2023
F.3
RESULTS FOR SHALLOW NETWORK
40
100
Frequency
30
Frequency
Frequency
10
-0.5
0.0 Welghts
0.5
-0.5
-0.2
0.0 Weights
Weights
50
Frequency
40
Frequency
40
Frequency
30
30
20
20
10
15
-0.5
0.0
-0.5
0.0
-0.2
0.2
Weights
Weights
Weights
(a) regularization
(b) Dropout (p=0.2)
(c) Weight-clipping
Figure A5: Measuring the weight distribution for shallow networks when apply a) no weight regularization, b) Dropout (p = 0.2), and e) Weight-clipping ((-0.2, 0.2)). Top row: The weight distribution of single local mask model selected at random. Bottom row: The weight distribution of a global mask model obtained by averaging the parameters models bootstrapped from 20 local models.
G ADDITIONAL RESULTS FOR SYNRANK
Ranking
Ranking
Ranking
10
f1091
f109]
f1001
Features
Features
Features
be wo en WN
Ranking
Ranking
Ranking
f1092
142
f109g
f109
Features
Features
Features
(a) No regulaiizatin
(b) Dropout (p=0.2)
(c) Weight-clipping (l-0.2, 0.2))
Figure A6: SynRank100 dataset: The important features are f100 > fi > frs while the rest of the features is not informative. Top row: Variations in feature rankings across 20 local mask models when we apply a) no weight regularization, b) Dropout (p = 0.2), and c) Weight-clipping (1-0.2, 0.2)). Bottom row: Variations in feature rankings in 100 global models, each of which is obtained by averaging the parameters of 10 models bootstrapped from 20 local models for the same three cases. We shallow network for the mask model.
19

---

Under review as a conference paper at ICLR 2023
H
FEATURE IMPORTANCE RESULTS FOR L2X DATASETS
Ranking
B
10
1 12 la
12 Ro 1
1. 12 la lo lo Iy lo Ise
81 1o
to
Features
1... WNN
Ranking
91 92 9oM,
02
01.92 G4M,
01 97
Ranking
11
11
110
f2
13
92
14
1
19
(7
110
f10
0
98
16
16
11.0
L2X XOR
L2X Orange
(c) L2X Additive
(d) L.XX Switch
Figure A7: Top row: Feature rankings from each of 10 local masks Md referred as l in the figure, obtained at particular training run for 10 separate runs. Middle row: Feature rankings from the global model, obtained by averaging the parameters of individual models up to a specific run i.e. cumulative average (CA). For example, 93 corresponds to the global model obtained by averaging the parameters of first local masks (4, 12, lz). Bottom row: The feature importance weights from Mg obtained by averaging the parameters of all local masks.
We note that the weights of the features are correlated with the frequency and position of their ranks across local masks. Specifically, fi and f2 in L2X XOR dataset keep switching positions between 1st and 2nd ranks across all ten runs (top row in Figure A7(a)). Therefore, M, computes their importance weights to be similar, giving a slight edge to fz since it is ranked as # 11 by six out ten local masks (the first and third rows in Figure A7(a)).
In the L2X Switch dataset, f1o is used as the switch feature to change whether the label is determined f4 by fs fs and is discovered the most important global feature by Mg
by the features (the bottom row in Figure A7()).. Please note that this is the hardest synthetic dataset in a way that the effect of fio on the sample labels which features to be used for label generation. This might be the main reason why commonly used method such permutation feature importance fails, ranking f1 the most important feature in our experiments with random forest and gradient boosting classifier used together with permutation feature importance (please see the results Section J. of the Appendix). Consistent with the ground truth, Mg also discovers fg an uninformative feature (bottom row in Figure A7(d)).
direct, rather it influences the labels indirectly by deciding
In L.2X Orange dataset, our method correctly discovers the first four features as the most important ones with almost equal weights while it indicates fi and fa as the most important features in L2X Nonlinear Additive.
20

---

Under review as a conference paper at ICLR 2023
1 ADDITIONAL RESULTS FOR L2X SWITCH
I.1 RESULTS FOR SHALLOW NETWORK
We run additional experiments on L2X Switch under three conditions: We apply i) No weight regularization to the weights of the mask model. ii) Dropout with p= 0.2 0.2 for the layers with leaky ReLU activation, iii) Weight-clipping (l-0.2, 0.2)) to limit the magnitude of the weights each layer. For each of the three cases, train 20 separate models and compare two different settings. In the first setting, we compute the variation in feature rankings given by 20 local mask models (top row in Figure A8). In the second setting, obtain 100 global models, each of which is obtained by averaging the parameters of 10 local models bootstrapped from 20 models. We compare the variation in feature rankings given by global models (sccond setting bottom row in Figure A8) to that 20 local models (first setting top row). We observe that: i) Regularization methods such dropout and weight-clipping help improve the variation across 20 local models, but the improvement is not substantial (e.g., comparing fe and fio across three cases at the top row). ii) However, they help improve the robustness the parameter averaging significantly (e.g., comparing fe and fiu across three cases at the bottom row). We also observe that the parameter averaging itself pushes the magnitude of the weights towards a tight range around zero as shown in Figure A9 (Section I.1 of the Appendix). indicating potential relationship between robustness and tighter weight distribution. Overall, global models are able discover important features the correct order, and the variation in feature ranking across global models is small (almost zero) for ground truth important features when we apply weight regularization (please see fg and fio the bottom row in Figure A8b and Figure A8c). iii) Weight-clipping works better than dropout in our experiments, but dropout results could perhaps be improved by
search on the variable.
Ranking
Renking
Ranking
10
F1 R2 B fd 15 16 (7 18 19 F20 Features
11 12 13 14 15 r6 77 Ba 19 r10 Features
f1 f2 f3 64 f5 16 f7 18 F9 Features
Ranking N
Ranking
Ranking
Features
Features
Features
(a) No regularization
(b) Dropout (p=0.2)
(c) Weight-clipping ([-0.2, 0.2])
Measuring the robustness of parameter averaging for shallow networks. We use L,X Switch dataset, in which the most and the least important features are f1o and fy respectively. Top row: Variations in feature rankings across 20 local mask models when apply regularization, b) Dropout (p = 0.2), and e) Weight-clipping ([ (.2, 0.2)). Bottom row: Variations in feature rankings in 100 global models, each of which obtained by averaging the parameters of 10 models bootstrapped from 20 local models for the same three cases.
Figure A8:
weight
21

---

Under review as a conference paper at ICLR 2023
40
40
50
Frequency
30
Frequency
30
Frequency
30
20
20
10
10
-0.5
0.0 Weights
0.5
-0.5
-0.2
Weights
Weights
40
40
50
Frequency
Frequency
Frequency
30
30
20
30
20
20
10
-0.25
0.00
0.00
0.25
-0.2
0.0
Welghts
Weights
Weights
(a) No regularization
(b) Dropout (p=0.2)
(c) Weight-clipping
Figure A9: The weight distribution for shallow networks when we apply a) weight regularization, b) Dropout (p = 0.2), and c) Weight-clipping ((-0.2, 0.2)). Top row: The weight distribution of single local mask model selected random. Bottom row: The weight distribution of global mask model obtained by averaging the parameters of 10 models bootstrapped from 20 local models.
Feature weight
Feature weight
0.3
0.30
0.25
f1
f10
0.2
0.20
rest
0.15
0.0
0.5
1.0
0.0
0.5
1.0
alpha
alpha
Figure A10: Two separate examples showing the weights of the features from model that is obtained by interpolating between a single local model and the global model, which is obtained by bootstrapping 10 local models out of 20: 0" (1-ol a) * 0 a where a swept from 0 to in 50 steps. The figure on the left uses a local model that ranks features f1 > f10 rest > f9 while f10 > f1 rest f9. As we move from local model (ar 0) to the global model (a 1), the estimate of feature ranking gets better (Expected feature rank: f10 > f1 > rest. f9). The plots show the mean and 95% confidence intervals for the weights of the features obtained by repeating the experiments with 100 bootstrapped global models, each of which is interpolated with the same local model. In both examples, we see that the global model predicts the rankings of the features correctly even when the initial ranking from the local model might be incorrect (e.g., left plot). "rest" refers to average weight the rest of the informative features i.e., f2 f8.
the one on the right uses local model that ranks features
22

---

Under review as a conference paper at ICLR 2023
1.2
RESULTS FOR DEEPER NETWORK
Ranking
Ranking
Ranking
10
11 12 ra 1S 16
19 120
11 12 1a 14 Is 16
19 F10
11 12
13
14 Is 16 17 ra 1O EIO
Features
Features
Features
T
T
T
Ranking
Ranking G
Rankingg
No regularization
Dropout (p=0.2)
(c) Weight-clipping (I-0.2, 0.2))
Figure A1l: Measuring the robustness of parameter averaging for deeper networks (5 layers). We use L2X Switch dataset, in which the most and the least important features are f1o and f9 respectively. Top row: Variations in feature rankings across 20 local mask models when we apply a) no weight regularization, b) Dropout (p=00 0.2), and c) Weight-clipping ((-0.2, 0.2)). Bottom row: Variations in feature rankings in 100 global models, each of which is obtained by averaging the parameters of 10 models bootstrapped from 20 local models for the same three cases. Although the weight regularization helps improve the robustness, still variations for feature f1o when the model deep (see last two columns at the bottom row). Also, weight-clipping works better than dropout again.
300
80
Frequency
60
Frequency
Frequency
200
40
40
100
20
20
-0.5
0.0
0.5
-0.5
0.0
0.5
-0.2
0.0 Weights
0.2
Weights
Weights
80
80
Frequency
Frequency
60
Frequency
60
60
40
40
40
20
20
-0.25
0.00
0.25
-0.55
0.00 Weights
0.25
0.0 Weights
0.2
Weights
(a) No regularization
Dropout (p=0.2)
Weight-clipping
Figure A12: Measuring the weight distribution for deep networks (5 layers) when we apply a) no
weight regularization, b) Dropout (p 0.2), and e) Weight-clipping ((-0.2, 0.2)). Top row: The weight distribution of a single local mask model selected at random. Bottom row: The weight distribution of a global mask model obtained by averaging the parameters of 10 models bootstrapped from 20 local models.
23

---

Under review as a conference paper at ICLR 2023
J MORE RESULTS FOR SYNTHETIC DATASETS FROM L2X
In this section, we show the robustness of our method, XTab, by listing the global feature impor- tance from Mg using test set for L2X (Chen al., 2018) datasets across 10 separate runs of our method. Please note that we don't It use weight regularization for XTab in any of these experiments. We also list global feature importance obtained by using permutation importance together with random forest (RFP), and gradient boosting classifier (GBCP). For L2X Nonlinear Additive and 1.2X Switch datasets, we also compare XTa other approaches such as TabNet( Pfister, 2021), Invase( Yoon et al., 2018), 1.2X(Chen et al., 2018), Saliency Maps(Simonyan et al., 2013), and Integrated ( et al., 2017) for comparison.
J.1 COMPARING XTAB TO OTHER METHODS USING L2X SWITCH DATASET Table A3: Comparing XTab to other methods using L2X Switch dataset by listing global feature importance rankings across 10 separate runs with different set of random seeds.
L2X Switch
Ranking from XTab weight
regularication is used)
10
Test
f10
12
16
EV
0U9768
65
0,975
12
08
01
f10
66
f6
0.9741
f10
12
85
GBCP
66
F6
0.9676
RFP
0,9575
Ranking
14
110
12
13
15
18
19
87
0,9000
15
110
86
11
12
13
19
18
95
0.9733
64
F1
110
F2
13
19
15
96
0.9546
f1
64
F2
F10
15
F3
86
F7
19
0.9723 15
F10
F2
F4
1%
F7
19
16
0.9633 6%
11
E30
23
15
16
18
87
0.9569
FI
E3
110
18
19
0.9725
F1O
84
F3
FI
55
F7
66
19
0.9683
F4
E]
F2
F10
73
F5
f6
F7
F8
0.9895
10
FI
F10
F3
F4
22
FS
F7
16
0.9733 80
Ranking from Iuvase
Runs
6
8
10
Test.Ac.
0.983
Konking
fl
F1
E3
15
F2
F10
F9
86
18
0.9569
f1
12
17
13
64
110
68
66
0.9913 65
f1
12
13
19
65
F10
66
17
0.9943 6%
11
12
18
23
E10
17
16
15
0.9576
11
23
12
64
110
151
15
16
0.9891 19
fl
510
15
F2
54
F3
F7
86
0.9925 19
fl
64
13
15
12
F7
86
110
19
08986 18
fl
16
12
F10
16
17
0.9005 65
12
18
23
17
16
E10
0.9923
10
11
84
12
E3
26
ELO
15
18
0.9906 89
Ranking Trom Salismey Maps
Runs
6
8
10
Test
Integruted Grodients
F2
f1
F4
16
F3
96
f7
F10
0.9334
12
f1
13
24
16
18
E10
19
0.9355
12
11
13
24
18
85
110
17
59
0.9378
12
11
13
26
14
18
E10
19
0.9392
12
F6
F1
13
66
18
110
17
0.9922
12
15
64
F1
16
13
F10
f7
16
0.9388 60
12
f1
13
24
ELU
18
16
19
0.9346
12
f1
E4
23
16
17
E10
89
0.9377
15
12
16
F1
26
13
f7
E10
89
0.9397
10
15
f1
64
23
18
f10
F7
69
09331
24

---

Under review as a conference paper at ICLR 2023
J.2 COMPARING XTAB OTHER METHODS USING L2X NON- ADDITIVE DATASET
Table A4: Comparing XTab to other methods using 1.2X Non-Linear Additive dataset by listing feature importance rankings across 10 separate runs with different set of random seeds.
LAX Nunlinear Additive
Kanking from Xlab (no weight regularization used)
Runs
2
8
9
10
Text Are:
F1
F4
85
17
F2
f10
11
14
13
13
19
1M
12
16
110
0J9K35
F1
F4
F3
F7
15
18
1O
F2
F6
110
0.9822
F1
14
F3
15
19)
F7
T2
16
F10
TX
0.9761
f1
F4
E3
15
17
19
18
F2
f6
f10
0.9835
10
11
14
13
15
131
17
TO
IN
110
0,9775
GRCP
11
F4
12
F10
P5
TO
BX
0.9934
REF
F2
110
19
18
0,9853
Ranking From Tab Nee
Runs
10
Test Acc.
11
14
15
86
18
12
17
13
19
0.9785
f1
14
15
16
F3
18
19
17
12
110
0.9752
11
15
F4
17
19
110
12
13
10
17
0.9751
11
61
15
16
186
13
12
110
F7
0.9785 19
11
15
14
10
IN
T1U
12
X3
19
0.9737 17
F1
15
13
12
F10
P9
0.9758 17
f1
15
F10
19
0,9760
15
IN
F7
0.9758 19
13
Kankingg From Inwaae
2
TI
F4
12
18
T10
T6
15
0:987
F2
f10
18
0.987
16
0.986
0.987
12
17
f10
0,985
FI
14
12
IN
I5
17
16
0,995
10
II
E4
22
ET
IN
19
F1O
16
0
Kunking from L2X
Runs
2
10
'est Acc.
15
18
08
13
11
14
13
11
12
110
17
19
0.9785
F1
15
14
16
1X
110
F3
0.9737 10
f1
15
E4
83
fh
12
F1O
19
17
0.9758
f1
15
64
16
13
12
18
f10
F7
0.9760 19
F1
14
IN
TIU
17
13
0.9728 19
fI
F4
06
FIQ
FI
0.9758 (0)
fI
15
B
f1O
0.9751
fI
13
12
24
110
f7
19
18
16
0.9534
f1
13
12
64
15
110
19
16
17
18
0:9870
11
13
14
12
110
15
10
17
13
0.9876
f1
11
F4
05
17
F10
12
f6
18
19
0.9878
f1
E4
f6
18
Kanking from Integrated Gradients (MG)
Runs
2
10
Test Ace.
T6
f1
12
f10
f1
14
110
12
19
18
17
15
0
f1
14
F3
15
12
16
110
TO
TX
17
0.9899
fl
F4
E3
F2
19
IR
15
f1O
f6
f7
0.9533
10
f1
14
13
12
10
11U
15
17
IN
0.950
25

---

Under review as a conference paper at ICLR 2023
J.3 THE RESULTS FROM L2X XOR AND ORANGE DATASETS FOR XTAB
Table A5: Showing the robustness of our method, X'aa,, by listing the global feature importance from Mg using test set for I.2X XOR and Orange datasets across 10 separate runs of our method. We also listed global feature importance obtained by using permutation importance together with random forest (RFP), and gradient boosting classifier (GBCP) for comparison. In L2X XOR, we expect fi and f2 to be equally likely to be top 1 and features and vice versa. For L2X Orange, expect the first four features to be the top four features in no particular order.
LAN XOR
LEX Orunge
Kanking from XTab (so weight regalariratio used)
Ranking from XTab (so weight regularizaion used)
Runs
2
3
3
6
8
10
Text Ace.
Runs
2
8
10
Texl Ace.
11
0.999
89
010
K
THE EFFECT OF NOISE ON THE TEST ACCURACY AND FEATURE RANKING
1
Feature Ranking
Test Accuracy
0.99
1
0.98
f10
0.97
f1
2
0.96
0.95
3
0.05
0.15
0.3
0
0.05
0.15
0.3
Noise Level,
Noise Level,
(a) Test accuracy
(b) Feature ranking from Mg
Figure A13: L2X Switch dataset: At each noise level, we ran our framework 10 times with different of random seeds to compare how a) the accuracy and b) the rankings of the top two features from Mg. i.e. f10 and /1, changes. Test accuracy improves when Gaussian noise with low variance is added. Compared some other datasets, the feature ranking stable with no, or low noise. Please note that we show 95% confidence interval only for feature f1o for clarity.
26

---

Under review as a conference paper at ICLR 2023
L
EXPERIMENTS ON ADULT INCOME DATASET
L.1
ARCHITECTURE SEARCH VIA
AND FINAL TEST ACCURACY RESULTS
Average CV Accuracy
85.5
86
1xtReLU
Uieer
2xLRoLU
Averaag CV Accuracy
85.5
86
- 1024
256
Accuracy
86
2XLReLU
Linear
85
3xUReLU
85
2048
85
3xLReLU
84.5
84.5
84.5
84
84
84
Run #
Run
Run
(a) i of layers in the Mask
(b) # of hidden units in the Classifier
(c) Final test accuracy
Figure A14: a) Comparing average cross validation accuracy for 10-fold cross validation (CV) by using mask generator architectures with different number of hidden layers. 10-fold CV is repeated ten times with different starting random seeds i.e. each run corresponds to one 10-fold CV run. 'Linear' means that the mask model has single linear layer while '2xLReLU' means two hidden layers with leaky ReLU activation. In all cases, the final layer of the mask is a linear layer with sigmoid activation. b) Same experiment repeated for the LxLReLU mask generator while increasing the number of the hidden units used in the classifier. c) The final test accuracy obtained once we settled on IxLReLu mask and classifier with 1024 units in hidden layers and re-run the experiments using our framework i.c. training models on the full training set multiple times. For comparison, we included the results for other architectural choices for the mask.
We first search for the optimum number of layers for mask generator, keeping the classifier architecture fixed as [1024, 1024, 1024]. The classifer has three linear layers, two of which are followed by leaky ReLU and dropout(p=0.2). The last layer uses sigmoid activation. We compare four choices for mask generator; i) A single linear layer with sigmoid activation, ii) A linear layer followed by leaky ReLU and another linear layer with sigmoid activation (referred IxLReLU) iii) two linear layers, each of which is followed by leaky ReI.U, and one linear layer with sigmoid, i.e. RxLR and iv) 3xLReLU. The number of hidden units in each layer in the mask generator is same as the number of features in the input (i.e. 105 in the case of Income dataset). We modified our framework accommodate K-fold cross validation (CV). We first generated a 10-fold CV dataset from the training set. For each fold, we changed the random seed before initialising and training our models on the training fold. We obtained the validation accuracy using the corresponding validation fold. This is a slight change to our original framework, in which we train models K-times on the same training set. We repeated this experiment with 10-fold CV for 10 times with different set of random seeds. As shown in Figure A14a, 1xl RxLReLU gives the best performance for all 10 repeated experiments. Please note that we also ran experiments on ReL. with wider hidden layer and observed that the standard deviation in validation accuracy increases with wider mask model (not shown). Thus, choose the number hidden units to be same as the number input features for all datasets and experiments throughout the paper.
We then repeated the first experiment. In this case, we used the mask generator with IxLReLU, and varied the number of units for the hidden layers of the classifier. With everything else kept same, wwer-parameterised classifiers with 1024 and 2048 hidden units give the hest performance. We used 1024 for the remainder of our experiments. These choices for the mask generator and the classifier are used for all other datasets and experiments since they work well as shown and discussed in the main and supplementary sections the paper.
27

---

Under review as a conference paper at ICLR 2023
Rankino
menttal statos
Sgn
marttal suatus
ag0
hounsijer per-eesk
hours per
education num
gain
gain
educatoinn momaato
accuputon
Capital invs
occupation
rocattonaby capital losa natiee country
race
10
rebatoonsio
native country
91. 92 94 9s 96 ge 9e
8044
2.08
(a) Local Masks (M2 s)
(b) Global Mask as CA
(c) Mg
(d) My
capital-gain hours per-waek
aducatton-num
occupation
capital-loss
sex
education
marital-status
relationship
netive-country
Figure A15: Using a shallow mask architecture for Income dataset. Feature importance extracted from a) Each local mask model ML, referred 4 in the figure, obtained a particular training run for 10 separate runs, b) Global model obtained by averaging the parameters of individual models up to specific run i.e cumulative average (CA). For example, 9s corresponds averaging the parameters of first 3 local masks (1, 12. 13) to obtain the global mask, from which we obtain the feature importance, c) Feature importance from the global mask model obtained by averaging all individual models, i.e. Mg in (b), d) Feature importance from Mf.
Rankina
mardal 2zaus
captal-gsin
mantal stacus
capiat- gain
bours par moek
hours per
aducaaion son--nom
education rumm
occupation
capiaallss
rane
occupation
capital capitallctss Pehattonship
10
relatienshio notive country
native country
ls b
01 92 94 O5 96 Ge So M,
0.20
(a) Local Masks (Mi, 's)
(b) Global Mask CA
(c) Mg
(d) My
Ranking
nutte--co countyy
education
occupation
worlickess
native country
artucation
accupation
meortal mnurtal-stauu
relattenship
mwritalshaa siaiiiiaaiin relabinsho
rece
1944
eapital cducaticn-nom
hours pjrr weeds capiaal gain
12 Is le Is
lo h Is Is ho
Mg
255
5:00
7.50
20.00
(e) Local Masks (Mr. ss
(l) Global Mask as CA
(g) M,
(h) My
age
capital-gain hours-per-week
occupation
capital-loss
education
marital-statue status
relationship
workcless
race
native-country
Figure A16: First Row: Same experiment in Figure A15, but without noise the input. Removing noise makes the rankings less robust. Second Row: Same experiment as in Figure A15, i.e. using noisy input, but the mask architecture deeper (5 hidden layers). Parameter averaging breaks down with deeper mask model architecture (c-h).
L.2 AVERAGING PARAMETERS OF SHALLOW NETWORKS
We start our experiments with the classification task on Income dataset to get insights into how parameter averaging works on a well-studied, real world dataset for extracting feature importance We used p 0.2 when generating the binomial mask, B, and considered NO σ²) with 0.3.
Unless specified otherwise, when say feature importance, refer global feature importance.
28

---

Under review as a conference paper at ICLR 2023
As shown in Figure 1, we train models the whole training set for the downstream task 10 times, each time with a different random seed. We store the parameters of the trained masks, referred as local masks, from each training and denote them {M MM,,...,,, M., J. We examine the feature importance obtained from each of 10 local masks across all samples for the test set (Figure A15a). We observe that each local mask gives slightly different ranking, especially for lower ranked features. More specifically, we could have different ranking (c.g., "age">" "maritual-statu vs "maritual-status" "tatuaa"-sttts"""""""""""" ""age" as top two features), depending on which seed is used when training the models. The possible reasons of this variation can be both the model initialisation as well as multicollinearity among the features of the Income dataset.
We then evaluate the effect of averaging over the parameters of the local masks on feature importance progressive way. To do this, we obtain a global mask Mg, as cumulative average (CA) over the k local masks, i.e. Mg, = 1/k Σ Mi.- For example, Mgs corresponds to averaging the parameters of the first three local masks (i.e. Mi, MM2 Mi, in Figure A15a). We refer to Mgno as Mg for simplicity in the rest of the paper. Figure A 15b shows the results for the global masks Mg gw- which we can observe that the feature ranking becomes more stable as we use more local masks in the parameter averaging.
Figures A15c and A15d show the feature importance from MG M and the final mask Mf. respectively. We first note that the weights of the features are correlated with the frequency and position of their ranks across all local masks. For example, "age" is ranked at the top more often than "marital-stare statuss in Figure A15a, hence its weight given by Ma is relatively higher than that of "maritual-statur We also observe that M moves both "capital-gain and "capital-los"" up in the ranking.
Effect of noise. We further investigate the effect of removing Gaussian noise from the subsets of the masked input in Equation 2. To this end, set O, and follow the same procedure described above (Figure Al5a-d). Comparing Figures A15b and A A16b, we can conclude that adding noise the input makes the global rankings more robust (a detailed comparison is in Section L.6 of the Appendix).
29

---

Under review as a conference paper at ICLR 2023
L.3 ROBUSTNESS PARAMETER AVERAGING
Ranking SuN
Beasau mun mwNN
Bart Besumurrann
Ranking Ge 1am-
Ranking
tin eduration tootoon ruin nafftal man/tal--aase occupattiO
relationslig
Byre CANP ToR
cactoo edurcts aurcenic run
So Riall-rant mai
Ecupation nat occupatie
relatiorion
Cap Bate Cation maviiall sital-ssatte thn
edu eott TUSTITUI
accupal
relationall
capi
Ranking b Bo Dowaauann
Bew sem--ooo ennonoo m.0
Ranking
Ranking
capital age
Peverion--t eduterionn rien-num mer native accupation relationship aibonniii
hours-yee pprital-staree
capita) age palliass education Diccation-as num
hours, parital-staris macitall dl-statuus
occupation
relationship
tal.aalooo Capital llss
education-nin Ieltaaiet-uuse num hours- per---err-- week perital-stan
age
native occupacion relationship
(a) No regularization
(b) Dropout (p=0.2)
Weight-clipping
Figure A17: Shallow network Top row: Variations in feature rankings across 20 local mask models when we apply a) no weight regularization, b) Dropout (p = 0.2), and c) clippig (10.2,, 0.2)). Bottom row: Variations in feature rankings in 100 global models, each of which is obtained by averaging the parameters of 10 models bootstrapped from 20 local models for the same three cases.
Ranking t Go var BINN DINN
Ranking
Ranking
cays 201 Upattin eetu hiath decupatte
relations Miatoorshig War workelaa
copie Totion ealll eslifcatio
ralatio Lip Mart
capl aure Capiiie
T Tin edu tha
reletiorch TO Ke
WO workettss
Sccupetii
Weot Del maath
o wi on wit
Gear aoor-oonnnauu aou
Ranking
Ranking
Ranking
capital aire
dluttion-- non--uu criiion hours Se H-steeu
rxcupatiin
are relationshi
workdlma nanis
capital ROTH aducation DocOOO--UUR
ma deaaaa-aaaan stauis
WORK
oec oecupatio C-O-----TTTS bon
workelake
capital oare Saain
BlusaUin VoO---UU por StATU
relationann
workslans
relation
(a) No regularization
(b) Dropout (p=0.2)
(c)
Weight-clipping
Figure A18: Deep network- Top row: Variations in feature rankings across 20 local mask models when apply a) weight regularization, b) Dropout (p 0.2), and e) Weight- clipping (-0.2,, 0.2).. Bottom row: Variations in feature rankings in 100 global models, each of which is obtained by averaging the parameters of 10 models bootstrapped from 20 local models for the same three cases. Please note that the global model in the case of deeper networks results feature ranking that is not consistent with local model estimations, indicating that the parameter averaging does not work as well as it does for the shallow networks.
30

---

Under review as a conference paper at ICLR 2023
L.4 VARIATIONS GLOBAL FEATURE IMPORTANCE OBTAINED FROM Mg AND COMPARING IT TO THE ONES FROM TABIE PFISTER, 2021)
Table A6: Comparing variations in global feature importance obtained from Mg across 10 different runs our framework with different sets of random seeds to those obtained from TabNet. The rankings from TabNet not consistent across multiple training runs ours. Abbreviations are; ms: marital status, en: education-num, eg: capital-gain hpw: hours-per per--eek, race: race, oce: occupation, rel: relationship, cl: capital-loss, native-country. Please note that no weight regularization is used for XTab.
Global Feature Importance from Mg XTab
Runs
3
4
5
6
7
9
10
age
hpw en
cg
OCC
race
nc
2
age
mss
hpw
en
cg
OCC
race
rel
cl
nc
3
age
ms
hpw
en
cg
00c
race
nc
4
age ms
hpw
en
cg
000
race
rel cl
nc
age
ms
hpw
en
cg
000
race
nc
6
age
ms
hpw
en
cg
000
race
rel cl nc
age
ms
hpw
en
cg
000
race
nc
8
age
hpw
en
cg
race
cl
rel nc
9
age
ms
hpw
cg
en
race
occ
rel
ne
10
age
hpw cg
OCC
race
rel cl nc
Global Feature Importance from TabNet
Runs
age
cg
rel
hpw
sex
en
ms
cg
age
cl
occ
hpw sex
race
3
cn
m78
cg
SCX
rel
ed
age
ne
hpw
4
m88
ed
en
rel
cg
occ
cl
hpw
age race
5
cl
cg
rel
age
occ
hpw
WC
6
rel
fw
en
cl
7
is
cg
rel
wc
occ
nc
hpw
SCX
8
ms
occ
cg
ed
hpw
en
race nc
9
m18
og
age
rel
300
en
cl
hpw
race
ne
10
118
rel
we
cl
en
ed hpw
Please note that we choose to compare Xtab and TabNet here since we can compute the rankings of the high-level categorical features in both methods. For the remaining methods, is not easy to compute the importance score of a parent category such as "maritual-statu individual categories (e.g., "single" or "married") directly as shown Section L.5 of the Appendix.
so instead rank the
31

---

Under review as a conference paper at ICLR 2023
L.5 COMPARING XTAB TO OTHER METHODS FOR ADULT INCOME DATASET
Please note that, for categorical features, since it is difficult to compute the importance of a parent category from its one-hot encoded features for other methods, we compare the rankings by the individual categories (e.g. showing the importance of "single", or "married" instead of the importance of their parent category "maritual-status", Abbreviations in the tables are; mes: Married civ spouse. en: education-numm num, eg: capital hpw: hours-per-weeke cl: capital-loss, em: Exce-managerial, nm: never-married, oc: own-child, os: other-service, fw: Final-Wer Weigh, mx: Mexico, hn: Holand- Netherlands, unm: unmarried, nm: never-married, phl: Philippines, tt: Trinadad& Tobago, nif: not-in- fanily,, ts: tech-support, seni: self-emp-not-inc.
Global Feature Importance using XTab (No weight regularization is used)
Runs
age
mess
hpw CB
en
em cl
while
hushand
au
2
ago
hpw
<g
white
busband
3
mes
hpw
<g
em
white
bushand
nm
age
mes
hpw
en
eg
em
white hushand
nm
5
age
hpw
eg
white hushand
6
age
mis
hpw
cg
en
em cl
white hushand
am
7
age mis
hpw
cg
en
em
white husband
nm
8
age
hpw
eg
white hushand
9
mes
hpw
cg
en
em white hushand nm
age
mes
hpw
eg
em
em
mm
Global Feature Importance using Saliency Maps
Runs
mics
og
en
age
cl
hpw em
ps
sen1 os
cg
hpw
age
wife
3
age
hpw
nm
0c
mes wife
em
cl
4
cg
age
hpw
am
oc
mes
wife
0s
5
eg
age
hpw
wife
6
cg
hpw
age
nm
mes
oC
wife
08 ps
7
eg
hpw
age
mes
nm
8
age
hpw
mes
wife
hushand
9
cg
age
mes
nm
0s
oc
wife en
cl husband
cg
age
mes
hpw
nm
wife
Global Feature Importance using Integrated Gradient
Runs
1
cg
miss
age
husband hpw
oc male os
2
age
hpw bushand hpw
female
3
cg
mes
age
nm
hpw en
husband husband 0c os male
eg
mes
nm
age
male female
5
age
hpw hushand hushand oc en female os husband
6
eg
mes
age
hpw
nm
7
cg
age
hpw
nm
female
8
age
hpw
husband
female male
9
cg
mes
mm
age
hpw
en
husband OC
femule male
cg
age
husband
hpw
female
Global Feature Importance using L2X
Runs
TO
cg
nm
divorced female mes divorced nm en hs-grad nif
us
2
hpw
age
unm
mes
Greece
3
eg
mes
Columbia en nm
oc unm Italy
phl
Honduras
cg
age
hpw
em nm hpw en South Irun Poland
male
Italy
5
age
female Italy
phi
6
nm
mafs eg female Cambodia em
Italy
phl
7
cg
male en If divorced af
be
8
os
age
nm
Italy
9
cg
age
nm
hpw
en
South mes
female
Italy
phl
cg
oc
age
Doctorate Hong
South
Global Importance using
Runs
10
mcs
Cg
age
hpw
hushand
private
wife
2
age
mos
eg
hpw
husband
private
cl fw em
3
cg
en
age
female wife
divaroed us private male private nif el em female bushand bachelor wife
4
cg
en
age
hpw
5
mes
en
age
hpw
6
cg
mes
en
age
hpw
7
eg
age
hpw
mes
female
hushand husband em seni nm private
8
em
mes
lipw
age
el
private husband
9
en
cg
age
hpw
mes
white
10
age
cg
private hpw female hushand
32

---

nc Under review as a conference paper at ICLR 2023
L.6 COMPARING VARIATIONS GLOBAL FEATURE IMPORTANCE OBTAINED FROM Mg WITH AND WITHOUT NOISE AT THE INPUT
Table A7: Consistency of the ranking across multiple runs our framework using the following two cases; Gaussian noise added the input (top table) and the input without the noise (bottom table).
Abbreviations are; ms: marital status, en: education- num, eg: sapital-gain, hpw: hours-per-weeke race: race, occ: occupation, rel: relationship, cl: capital-loss, ne: native-country
Gaussian noise, = 0.2 & (N0,0 = 0.3), added to the input
Runs
2
5
9
10
1
age
ms hpw cg
race
rel
cl
nc
2
age
ms hpw
cg
OCC
race
cl
nc
3
age ms
hpw
en og
00c
race
rel cl
nc
4
age
ms
hpw
cg
000
race
rel
cl
nc
5
age
ms
hpw
cg
race
rel cl
nc
age
ms
hpw
en
cg
00c
race
cl
nc
7
age ms
hpw
en
cg
occ
race
rel
cl
nc
8
age
ms
hpw
cg
occ
race
cl
rel
nc
9
age hpw
cg
en
occ
rel
cl
nc
10
age ms
hpw
cg
en
oce
race
nc
No noise added the input
Runs
8
10
1
cg
ms
hpw
en
occ
race
cl
rel
2
cg
age
hpw
cl
rc]
race
3
cg
age
ms
hpw
en
race
0Cc
cl
ne
4
ms age
cg
hpw
en
occ
race
rel
nc
cl
5
cg
ms
age
hpw
en
0CC
race
cl
rel
nc
age
cg
hpw
en
occ
race
rel
nc
7
ms
age
cg
hpw
cn
occ
race
rel
nc
8
ms
age
hpw
en
OCc
cl
race
rel nc
9
age
cg
hpw
cl
rel
10
cg
ms
age
hpw
cn
occ
race
rel
ne
L.7 EXAMPLES OF INSTANCE- WISE IMPORTANCE FROM My FOR SIX SAMPLES FROM ADULT INCOME
Capital- Capital-Goin Goin
Age
serriedciv--popsee civsppusse
Married civspouse
Hours-Per Per-Week
Married Married-civ-spie Age
Age
Education- autoaa--n-mmmm Nim
Hours- Per-Week
Rou:ation-Num Num
Married spouse
Education -um -um
Hours-Per Perk
Capital-Gan Gain
Exec- mecc-managena
Exec-maa cerec-maaaggg managerial
exec-managen il
Capital Capital-Gain Gain
Ranking
White
White
White
Husband
Husband
Hushand Capital-Loss Loss
Capital-Loss
Never Nevermarrien married
Never married
Capital-Loss
Never married
0,20
0.40
0.30
0.20
0.30
0.03
0.05
0.08
Age
Educabin-Nun Num
Marrisd-cle Darried-clu-speuu speuse
Age
Married civ-spuuse
Hours Per Week Per-Wiek
Hours Per-WWekk
Hours Per Ver-Week
Education-Ny Num
Marmed-e -emmed-----oo spouse
Education Aun
Exsc cest-managenn managerial
Capital Gain
Evec-manadee meec-mannneriill angeriai
Capitel- Gain
Exec-mannage mnnigaaia
Capital-
Ranking
White
Whlte
White
Rusband
Husband
Husband
Capital-Los
Capirai-Lss Loss
Capital-Loss
Mever-aaried
Nevermarried
Never married
0.10
0.20
0.30
0.04
U.06
0.03
0.05
0.08
Figure A19: Examples of instance-wind feature importance for five random samples from Adult
Income dataset.
33

---

Under review as a conference paper at ICLR 2023
L.8 SHOWING ROBUSTNESS FOR THE INSTANCE- WISE IMPORTANCE FROM My FOR SINGLE SAMPLE FROM ADULT INCOME ACROSS 10 DIFFERENT EXPERIMENTS
Married- dived--cv--eee spouse
Married- serredddw--spee spespe
Education JduuaationNuum Num
Merried-clv spoute
Age
Age
Age
Edixaaton ation--ooum Num
Hours-pe Serrs-PPrr-iie Week
Exec-mane sxec-manager aa
Hours Bours-Per-Weet Per-Week
Hours s-r S-Per Week
Education Num
Exec-man menc-mmnagger geral
Capital- Gasin
Exec-mane Seec-managerin managerial
Capital-Cos Cain
White
White
Capital-Gal Gain
White
Shuband
Husband
Husband
Capital- Loss
Capital-Loss
Capitai-Loss Loss
never-mamedd
Never-married
Nevermaniid
0.05
0.10
0.15
0.25
0.10
0.15
Married-ci--Be cireee-----pouua spouse
Married spouse.
Married Married-ci--ppone siae---c----psuse Education Num
Age
Hours- Hours-Per-yee Per-Week
Education-Ne Num
Education Num
Hours: Per-Woek Week
Hours- Hours-Per-Wed Per-Week
Exec managera ell
Exec srec-manaade seec-manageria aggiia
Exec Exac-managen managerial
Capital-Gal Gain
Capital- Gain
Capital-Gal
White
White
White
Husband
Husband
Husband
Capital-L.oss Loss
Capital- Capltal-Loss Loss
Capitai- Capltai-Loos Loss Never never-mariien led
Never-mariid
Never married
0.05
0.10
0.15
0.05
0.10
0.15
0.05
0.10
0.15
Married-civ coirred-ci--spp -s-spouse
Married ciu spouse
Married civ spouse Education Rum
Age
Education Niuiattio--uum Num
Educattion Nommattoo-uum Num
Ago
Hours-Pe Per-Wek
Hours Per Weak
Hours- Hours-PPariie Par-Weok
Exec managerial
Exec Exec-mana managerial
Exec-man reec-managerie geriaal
Capital- Capitai-Gain Gain
Capital Capital-Gain Gain
Capital Capital-Gain Gain
White
White
White
Husband
Husband
Husband
Capital-Looss
Capitai-Lors Less
Capital Lons
Never Never-mariie mariie
Never marriad Never-mernidd
Never-marrie
305
0.10
0.5
0.10
0.15
0.05
0.15
Married civ-s-ivuss spouse
Age
Education Nur
Hours Hours-Pen Per-Wek
Exec manageria negerial
Capitai- Capitai-Gain Gain
White
Hushand Capital-Loss
Nev Never-aaried
0.05
0.10
0.35
Figure A20: The robustness for the instance-wise importance from My for single sample from Adult Income across 10 different experiments
34

---

Under review as a conference paper at ICLR 2023
M
RESULTS FOR BLOG DATASET.
Features
1 a aaadu DawNk
MWM
Dow 00 newwaoa wwN
f52
f10
154
152
f20
(51
Ranking
Ranking
f20
f15
f15
f55
f51
L
f10
f101
f12
f62
f11
12 l Is 4 la h0
91999:9449s94999e5MMM
155
0.10
0.20
(a) Local Masks
(b) Global Mask CA
(c) Mg
Figure A21: Blog dataset: Repeating the experiment with Income dataset (i.e. using shallow network, noisy input data) in Figure A15 (a-c) for Blog dataset. We also use weight-clipping ([-0.2, 0.2)).
We repeated the experiment that we did for Income dataset (using shallow network and noisy input data) in Figure A15 (a-d) for Blog dataset. Looking the Figure A21 (a-c), features f52 (number of comments in the last 24 hours before the basetime), f54 (number of comments in the first 24 hours after the publication of the blog post, but before basetime), f51 (total number of comments before basetime), and f20 (the median of 154) are discovered to be the most important for classifying whether blog post would receive a comment.
M. ROBUSTNESS OF PARAMETER AGENGGING
Ranking
Ranking
Ranking
L o u
10
Ranking
Ranking
Ranking
10
(a) No regularization
(b) Dropout (p-0.2)
(c) Weight-elipping
Figure A22: Shallow network- Top Variations in feature rankings across 20 local mask models when we apply a) no weight regularization, b) Dropout (p = 0.2), and c) Weight-clipping (l-0.2, 0.2)). Bottom row: Variations feature rankings 100 global models, each of which obtained by averaging the parameters of 10 models bootstrapped from 20 local models for the same three cases.
35

---

Under review as a conference paper at ICLR 2023
N OTHER WAYS OF IMPLEMENTING OUR PROPOSED METHOD
In the final training in our framework, we use Mg as a reference mask while introducing another mask model to compensate for the potential sub-optimality introduced in Mg:
mg = (X) and my = Mi(X)
(10)
my (mg + mt)/C where C max mg mi) and XM m
(11)
However, the final mask, my is only scaled summation of the outputs from Mg and M,. A better way to do this update can be using a gating mechanism similar to input and forget gates LSTM (Hochreiter & Schmidhuber, 1997). This would enable the model to forget the weights of some features in mg while adding more weights to others through my in the following way:
my = o my + i m:)
(12)
where a is the sigmoid function. We leave the idea of gated masks as a future work.
O BROADER IMPACT
The estimation of feature ranking in many areas such as in healthcare, finance and insurance is critical in decision making process. While taking advantage of neural in these applications is important, we should be mindful of consistency and robustness of our methods. Our proposed method makes contribution towards achieving robust estimation of feature ranking. However, we should be aware of the limits and shortcomings of our approach as well as other similar approaches.
36


Reviews:
[{"review_id": "wGz-ruZk0W", "reviewer": "Reviewer_brop", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a new method to rank feature importance in a neural network. The authors propose to train multiple shallow networks and take the average of the weights for the final ranking. The proposed method is evaluated in several datasets to prove its accuracy and stability. ", "summary_of_the_review": "The paper introduces a new method to rank feature importance in a neural network based on weights averaging from multiple shallow networks. Being a purely empirical paper, I feel some choice in the paper is not well motivated and lack supporting evidence. I believe with some revision it may be published in ICLR.", "strength_and_weaknesses": "Strength\n- The paper motivates the problem well, and presents the idea clearly.\n- Extensive numerical studies are provided to support the authors' claims.\n\nWeaknesses \nOverall I feel this paper is too \"empirical\" while it's not clearly supported in the paper why current design could be the best choice. \n- There are several techniques borrowed from previous work (subsetting features, random noise added, etc), but without a clear justification on whether they are needed (did not seem to find in the experiments).  Naively, maybe just to compare with an average of normally trained networks without all the added tweaks? \n- The authors mention that in deeper architecture the weight averaging does not work well. This raises a question on how to choose the side of the shallow networks in practice? Empirically I think there might be a sweet area for the size of shallow networks, but how do we find it and balance between the size of the network and the efficiency? \n\nMinor Issues\n- I found the method description especially the terminology on mask, classifier and encoder are a bit confusing (Maybe it's just because I am not familiar with this type of presentations).\n- On the robustness vs correctness of the feature importance, I agree with the authors that it's indeed hard to know the latter but is out of the scope of this paper. There are recent work on trees related to this which are missed from the citation (e.g., https://proceedings.neurips.cc/paper/2019/hash/702cafa3bb4c9c86e4a3b6834b45aedd-Abstract.html,  https://dl.acm.org/doi/pdf/10.1145/3429445) \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QLwK5F6FHYy", "reviewer": "Reviewer_iffa", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a novel feature ranking algorithm that consists in two parts: first, a mask generator network is used to determine the importance of each feature, in a similar way the Learning to Explain algorithm (L2X) is used. Later, the average of multiple runs (with different weight initializations) over the mask generator are obtained, which are finally introduced in a final training, combining the weights of the average mask generator with a new local one, which will establish the final ranking of each feature. The experimental results show a better consistency on how the important features are ranked, when compared with the state-of-the-art", "summary_of_the_review": "Although I think it is an interesting paper, it is very difficult to read in its actual form. Furthermore, some decisions require an extra explanation.", "strength_and_weaknesses": "Strengths\n* The algorithm is easy to implement.\n* The consistency in the solution is remarkable.\n\nWeaknesses\n* The paper is very difficult to read and to follow. Some of the information included in the appendix should be accessible at the regular paper.\n* The results are not remarkable when compared against the state-of-the-art.\n* Some theoretical decisions need more clarification.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GY3DHOO7cY", "reviewer": "Reviewer_ZHfL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an approach to estimate feature importance. Multiple shallow mask models are trained with different random seeds. These resulting weights and then averaged to produce a global model that can be inferior in accuracy but better identify the global feature importance. ", "summary_of_the_review": "I think the proposed approach is interesting but the paper needs a considerable revision to both justify the goal and each of the proposed components including a detailed ablation study.", "strength_and_weaknesses": "I found the proposed approach interesting but very ad hoc. There are quite a few components such as feature masks (local and global), feature subsets, feature noise, encoder that generates an embedding for each feature subset and finally a classifier. It is unclear which of these components are needed to improve feature importance stability and which are for classification performance. It is also unclear what authors mean by \"global\" feature ranking. In my understanding feature importance is typically model specific. Tabular datasets often contain subsets of highly correlated features and typically multiple such subsets can be used to get accurate models. Artifacts such as random seed influence which subset (or subsets) the model converges to during training. This is commonly observed in xgboost where, when optimized under the \"hist\" setting and random seed, the trees tend to pick a subset of the correlated features. Averaging together models is likely going to spread importance over correlated features and I don't quite see how this \"global\" importance is useful.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7RJFallmyJA", "reviewer": "Reviewer_G9vP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a novel method based on parameter averaging to estimate accurate and robust feature importance in tabular data setting. The proposed method first initializes and trains multiple instances of a shallow network (referred as local masks) with different random seeds for a downstream task and then obtains a global mask model by averaging the parameters of local masks. The experimental results show that although the parameter averaging might result in a global model with higher loss, it still leads to the discovery of the ground-truth feature importance more consistently than an individual model does.", "summary_of_the_review": "The proposed method is novel, but no practical application validation.", "strength_and_weaknesses": "Strength: Experimental results are good.\nWeaknesses: There is no experimental validation on real task.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject