Interpretable Mixture of Experts
Authors: Aya Abdelsalam Ismail
Sercan O Arik
Jinsung Yoon
Ankur Taly
Soheil Feizi
Tomas Pfister
Venue: Accepted by TMLR ()

Abstract: The need for reliable model explanations is prominent for many machine learning applications, particularly for tabular and time-series data as their use cases often involve high-stakes decision making.  Towards this goal, we introduce a novel interpretable modeling framework, Interpretable Mixture of Experts (IME), that yields high accuracy, comparable to `black-box' Deep Neural Networks (DNNs) in many cases, along with useful interpretability capabilities. IME consists of an assignment module and a mixture of experts, with each sample being assigned to a single expert for prediction. We introduce multiple options for IME based on the assignment and experts being interpretable. When the experts are chosen to be interpretable such as linear models, IME yields an inherently-interpretable architecture where the explanations produced by IME are the exact descriptions of how the prediction is computed. In addition to constituting a standalone inherently-interpretable architecture, IME has the premise of being integrated with existing DNNs to offer interpretability to a subset of samples while maintaining the accuracy of the DNNs. Through extensive experiments on 15 tabular and time-series datasets, IME is demonstrated to be more accurate than single interpretable models and perform comparably with existing state-of-the-art DNNs in accuracy. On most datasets, IME even outperforms DNNs, while providing faithful explanations.  Lastly, IME's explanations are compared to commonly-used post-hoc explanations methods through a user study -- participants are able to better predict the model behavior when given IME explanations, while finding IME's explanations more faithful and trustworthy.

Published in Transactions on Machine Learning Research (05/2023)
Interpretable Mixture of Experts
Aya Abdelsalam Ismail", Sercan Ô. Arik?, Jinsung Yoon,, Ankur Taly?; Soheil Feizi", and Tomas Pfister?
GGenentech Google Cloud AT UUniversity of Maryland
Reviewed on h
Abstract
The need for reliable model explanations is prominent for many machine learning applications, particularly for tabular and time-series data as their use cases often involve high-stakes decision making- Towards this goal, we introduce a novel interpretable modeling framework, Interpretable Mixture of Experts (IME), that yields high accuracy, comparable to 'black-box' Deep Neural Networks (DNNs) in many cases, along with useful interpretability capabilities. IME consists of assignment module and mixture of experts, with each sample being assigned to a single expert for prediction. We introduce multiple options for IME based on the assignment and experts being interpretable. When the experts are chosen to be interpretable such as linear models, LME yields an explanations produced by IME are the exact descriptions of how the prediction computed. In addition constituting a standalone premise of being integrated with existing DNNs to offer interpretability to subset of samples while maintaining the accuracy of the DNNs. Through extensive experiments 15 tabular and time-series datasets, IME is demonstrated to be more accurate than single interpretable models and perform comparably with existing state-of-the-art DNNs in accuracy. On most datascts, IME even outperforms DNNs, while providing faithful explanations. Lastly, IME's explanations are compared to commonly-used post-hoc explanations methods through a user study participants are able to better predict the model behavior when given IME explanations, while finding IME's explanations more faithful and trustworthy.
architecture where the
ingrrentyyinneerretabbll architecture, IME has the
1 Introduction
Tabular and time-serics data appear in numerous applications, including healthcare, finance, retail, environ- mental sciences and cybersecurity, and constitute major portion of the addressable artificial intelligence market (Chui et al., 2018). Although simple interpretable models like linear regression (IRR) or decision trees (D's) or ARIMA had dominated the real-world applications with these data types, deep neural networks (DNNs) have recently shown stato-of-the-art performance, often significantly improving the interpretable models (Arik & Pfister, 2021; Lim et al., 2021; Popov et al., 2019; Joseph, 2021; Gorishniy el al., 2021; Wen et al., 2018), and now they are being used more commonly. Yet, one challenge hindering the widespread deployment of DNNs is their black-box nature (Lipton, 2017) humans are unable understand their complex decision- decision-making process. For most tabular and time-series data applications, explainability is crucial (Caruana et al., 2015; Lipton, 2018) physicians need to understand why drug would help, and retail data analysts need to gain insights on the trends in the predicted sales, and bankers need to understand why a certain transaction is categorized as fraudulent.
In an attempt to produce explanations, various interpretable architectures have been proposed. Attention- based DNNs (Arik& Pfister, 2021; Lim al., 2021) constitute one approach with attention weights being
* *Corresponds to: bamailayy@@genneccmm work done while at Google.

---

Published in Transactions on Machine Learning Research (05/2023)
used explanations; however, recent works (Bibal et al., 2022) show the limitations of such explanations. Differentiable neural versions of generalized additive models (Agarwal et al., 2020; Chang et al., 2021) and soft D''s (Luo et al., 2021) are effective for tabular data but they do not generalize to time-series data. Trend-scasonality dccomposition- based architectures (Oreshkin et al., 2019) are proposed for univariate time-series data but can not be used for tabular or multivariate time-series data. Recently, Puri et al. propose DNNs with explanations in the form of continued fractions, whose interpretation complexity increases with the addition of network layers. The above methods have one or more of the following problems: (a) yielding unreliable explanations; (b) yielding complex explanations; (c) can not generalize to different structured data (i.e, both time series and tabular); (d) have varying amounts of accuracy degradation as they trade model interpretability for accuracy. Overall, a systematic interpretable framework while preserving accuracy, remains to be an imporlant direction.
Interprotable experts
DNWinterpretable experts
Inter Interrprraabin pnnorpretabble a a
(e)
(c)
Excor
Gutppl
DNN
Darpur
Experi
Interpretable Models
(e.g. Linear Regression)
Expert Assigement
Fipeet
DNN Asvemmen
Papeer
Expert Assigmment
Excert
(b)
(d)
DNN assignment
Exceri-+
Gutpal
ONN
Durgur
Exceri
Expert Mookue
Eccert
DNN Models
Expert
Me
Assigemeet
Modi. In
Exper
(e.g..
Figure 1: Interpretable Mixture of Experts (LME) framework for time-series and tabular data, with single-level and hierarchical assignment options. Solid lines denote the selected assignments. (a) shows S-IME, a single-level assignment IME where both the assignment module and the experts are interpretable. (b) shows S-IME,, a single-level assignment IME where the assignment module is a DNN, and interpretabilityy is provided only via the experts. For hierarchical assignment, the primary assignment module chooses between a pre-trained black box expert or S-IME,,4; offering partial interpretabilityy by giving insights into which samples benefit from the complexity of black box models. (c) shows H-IME, hierarchical IME where both the assignment and expert modules are interpretable. (d) shows H-IME,, a hierarchical IME where assignment modules are DNNs and experts are interpretable.
We propose a novel framework for the idea of combining multiple models, that can all or partially be interpretable, in a mixture of experts (ME) framework. ME frameworks are composed of multiple "experts" and an assignment module that decides which expert to pick for each sample. Recent works (Shazeer et al., 2017; Fedus et al., 2021) used ME to replace DNN layers for efficient capacity scaling. On the other hand, we propose employing single ME to fit. different subsets by interpretable experts. Although complex distributions cannot be fit by simple interpretable models, we postulate that small subsets of them can be. Building upon this idea, we show that IME can preserve accuracy while providing useful interpretability capabilities by replacing black-box models with multiple interpretable models. Enabled by innovations in its design, TME is a novel framework with interpretable models that can partially/ Sitialllyfullyyl replace or encapsulate DNNs to achieve on-par with or better accuracy, based on the desired explainability-accurae trade-off. Explanations produced by IME are faithful to the model we show through user studies that IME explanations are easier to interpret and users tend to trust the explanations produced by IME over those produced by saliency methods.
modeling for tabular and time-series data, with
To address interpretability needs of different applications, we propose multiple options for IME: Single- level assignment S-IME, & S-IMEa (Fig.1 (a) & (b)) employs an assignment module, either interpretable (yielding both assignment and expert interpretability) or a DNN (yielding only expert interpretability) with interpretable experts. Hierarchical assignment H-IME, & H-IME, (Fig.1 (c) & (d)) first selects between a pre-trained black-box expert and IMF, and then between different interpretable experts. S-IME, (Fig.1
2

---

Published in Transactions on Machine Learning Research (05/2023)
(a)) generates explanations in the form of the exact description of predictions a concise formula, enabling its use in high-stakes applications that require precise explanations. S-IMF (Fig.1 (b)) can be used for local interpretability, a capability to gain insights on the group of predictions in each cluster. H-IME, & H-IMEa (Fig.1 (c) & (d)) offer interpretable decision making for a subset of samples while maintaining accuracy. As an example, for a real-world Retail lask, approxinately 40% of samples can be assigned to an interpretable expert. while preserving the same accuracy. The trade-off can be adjusted by the user based on the application needs.
2
IME Framework
2.1 Overall design
Inputs
Experts
Prediction
Expert1 error
08
Expertz error
44
u.1
Expert2
Prodiction2
03
Expertg error
Expert3
Prediction3
Assignment welghts
Smoothness Loss
Diversity Loss
Targets
W1
Assignment module [Iterpretable/DNN)
W2
Accuracy Loss
W3
Utilization Loss
Assignment module Loss
Prediction
Figure 2: S-IME consists 'interpretable experts' (3 in this example) and an 'assignment module'. The input to IME can be time-series or tabular data. Each expert produces a single prediction. The input to the assignment module is both feature embedding and the past error made by the experts (when the time dimension exists). The assignment module selects single expert during inference to make the final prediction. We define various losses (shown in red boxes) that are used to supervise the assignment module and expert weights.
IME consists of a set of 'interpretable experts' and an 'assigument module', as overviewed in Fig. 2 (Note that this shows S-TME, a detailed overview of H-IME is shown the Appendix A Fig. 9). The experts can be any interpretable differentiable model, each with its own trainable weights. They can be different architectures (assuming they accept inputs and generate outputs of particular sizes), e.g. expert can be linear regression (LR) while the other can be a soft decision tree (DT) (Frosst. & Hinton, 2017: Luo et al., 2021; Irsoy et al., 2012). The only requirement for them is being differentiable (for which (Ghods & Cook, 2021) provides an overview). The assignment. module can be either an interpretable model or a DNN, as shown in Fig. 1. When we have a pre-trained DNN as one the experts (in H-IME, and H-IME,), we employ a hicrarchical assignment structure so that the first assignment module (DNN assignment module) decides whether sample should be assigned to DNN or IME. If the assignment is not a DNN, second (ezpert assignment module) selects the interpretable model to assign. Furthermore, to control the use of the pre-trained DNN, enabling an accuracy-intery intrreeeaiiterrreabbllttyy trade-off (see Sec. 3.3). Hicrarchical assignment is essential with a pre-trained DNN, since adding pre-trained DNN as an expert candidate (along with other interpretable experts) biases the assignment module, which may lead to choosing the DNN over other experts. During training, the assignment module outputs the selection likelihood for each expert and the target prediction. During inference, the most probable expert is selected. IME can be used for tabular data (where a sample consists of a single observation at given timestep) or time-series (where sample consists of multiple observations over time period).
introduce mechanism
3

---

Published in Transactions on Machine Learning Research (05/2023)
Notation. Broadly, consider a regression problem for imput data with S samples {(X, Y)) with X = [#1,...,Tr] E RNNT, where T is the number of timesteps (for tabular data, we have T = and N is the number of features. Outputs are Y = [....... VH] E R" where H is the horizon (for tabular data = 1). The inputs until timestep t are denoted as Xz. For an IME with n experts, each expert is denoted as /s with being the expert index. The predictions made by the expert are given as [((X) = Y; and the corresponding prediction error, for which we use mean squared error (MSE), a given horizon h, is denoted as C = = Ei= (Bh- 0. The errors made by all experts at time t are denoted as E, = = ",, En,r) The assignment module has two functions: (a) Ay outputs prediction YΛ. (b) Aw outputs n-dimensional vector representing the weight and A share the same weights except for the last layer. Note that this can be generalized to classification by simply changing the model outputs.
the probability of choosing particular expert such that > W; 1. Both Ay
2.2 Learning from past errors
IME can be applicable to general tabular data without any time-dependent feature. However, with time- dependent leatures available, IME benefits from incorporating past errors (with the accuracy improvements shown in Sec. 4.5). For most datasets, the input-output relationships between consecutive timesteps are relevant. This is often due to consecutive timesteps having overlapping information, or external conditions not changing rapidly. For data with time information, we propose to use past errors as informative signals for the assignment. Specifically, concatenation of both the feature embedding X, and past. error made by the experts E,- X₁ [X;; Et-i], is used as the input to the assignment module so that it can take into account of the errors while assigning weights for the next timestep. Note that with causal masking, we ensure that the forecasting error never overlaps with the forecasting horizon. For example, if we are forecasting horizon from timesteps t + 1 to H + +t+1, 1, only the expert errors until timestep t are added to the input of the assignment module.
2.3
Training objectives
IME training is based on the following desiderata:
Overall model accuracy should be maximized.
The assignment module should be accurate in selecting the most appropriate experts.
Utilization of individual experts shouldn't be imbalanced, avoiding all samples being assigned to one expert.
Experts should yield diverse predictions, helping the assignment module to better discriminate between them. For inputs with time information, assignments should be smooth for consecutive inputs, for better generalization and interpretability.
Correspondingly, we propose the following objective functions:
Y) =ppred (f, Aw; X,X, Y) + 3Lutil (Aw, + T (f,X)+ + 8Lsmooth (Aw, X) + ALA, (Ay,X,Y)
(1)
where Y, and À are hyperparameters. Epred, Lutil, Edive Lsmooth and LA, are prediction accuracy, expert utilization, expert diversity, assignment smoothness and assignment. accuracy losses, respectively, explained in detail below.
Prediction accuracy. As in (Jacobs et al., 1991), we propose the log likelihood loss under a Mixture of Gaussians assumption:
= log Σ" A. X é I 1x(2)11/2
(2)
This loss can help encourage expert specialization by comparing each expert separately with the target reduce the average of all these discrepancies. Here, we consider regression problem. For classification, the loss in Eq. 2 can be adjusted accordingly, Appendix B includes more details.
4

---

Published in Transactions on Machine Learning Research (05/2023)
Expert utilization. The assignment module can converge to state where it produces large assignment weights for the same few experts, resulting in some experts being trained more rapidly and thus selected more. To circumvent this, (Eigen et al., 2013) uses hard constraints at the beginning of training to avoid local minimum, (Bengio et al., 2015) uses soft constraints on the batch-wise average of cach expert, and (Shazeer et al., 2017) encourages all experts to have equal importance (i.e, uniform expert utilization) by penalizing the coefficient of variation between different expert utilization. For IME, we propose that each expert should focus on a subset of the distribution. These subsets do not have to be equal in size, all experts should be utilized but not necessarily in uniform manner. Given this, we propose the following utilization objective:
Cutal = =1/NN) Σn E kR; -
N
(3)
where U, is the utilization of expert i such that U, = I/N Ey= wij and k is a hyperparameter to encourage utilization without enforcing uniformily across experts.
Expert diversity. Experts should specialize in different dała subspaces (i.e., they would have different expertise), as specialization helps the assignment module in choosing the correct expert for a given data point resulting in better accuracy. Diversity between the outputs of different experts encourages expert specialization. We encourage diversity by employing a contrastive loss:
Ldiv = - EE= log (x)(8((((X)
(4)
1 [eE] exp (X)(((())))
where S(u, v) - x² ///////lllll denote the dot product between la normalized u and v, [[z E {0,1) is the indicator function such that 1 iff k # i, and r is the temperature parameter. We propose it as minimizing the distance between outputs from the same expert and maximizing the distance between outputs from different experts. We add a Gaussian noise with zero mean and unit variance to the inputs, and define positive pairs as outputs coming from the same expert (with and without the noise) and negative pairs as out puts coming from two different experts (both without noise).
Assignment smoothness. For data with time component, consecutive inputs should have mostly overlap- ping information, so one would expect mostly similar assignment for them. To promote smooth transition of assignments over time, we adopt the Bullback-Leible (KL) divergence (Kullback & Leibler, 1951) between the weights output by the assignment module for consecutive timesteps t-1 and t:
Esmootth (Aw, X) = DKL AAw (Au
(X--1) || (x.)..
where DKI || Q) denote the KT divergence between two distributions P and Q defined on the same probability space. Here, we assume forecasting horizon of 1, and this can be modified for longer forecasting horizons by summing over forecasting timesteps. Assignment smoothness can also be helpful for improving the interpretability, as users can build more reliable insights.
Assignment module accuracy. The assignment module is designed to generate predictions (that are only used during training) along with the weights, and we propose to minimize the following error for superior assignment accuracy:
Ca, ())
N (y (1/M
=
Ay (x))²
(6)
In H-IME to avoid all samples being assigned to the pro-trained DNN expert, additional regularization term is added to Eq. 1: a UDNN, where UDNN is the DNN expert utilization.
H-IME regularization.
UDNN = (1/N) Σ j=1 WDNN.J.
(7)
WDNN is the weight assigned to pre-trained DNN by the hierarchical assignment module.
5

---

Published in Transactions on Machine Learning Research (05/2023)
2.4 Training procedure
The training procedure of IME is overviewed in Algorithm 1. First, the model is trained in end-to-end way to minimize the overall loss given in Eq. 1, until convergence. Then, the experts are frozen, and the assignment module is trained independently to minimize the prediction loss given in Eq. 2. This alternating optimization approach first promotes expert specialization, and then improves the assignment module accuracy on trained experts. Since experts and the assignment. module might be based on different architectures, they might. converge different rates (Ismail al., 2020b), thus, different learning rates are employed for them.
Algorithm 1: Train Interpretable Mixture of Experts
Input: Features X, targets Y, expert learning rate т, assignment learning rate hyperparameters 8, 6, Initialize: All experts parameters f, the assignment module Au & Ay. Also set previous error while not converged do
X =[;
L = Eped (f, Aw, X,X,Y) + BLued (Aw, + WLaio (f, X) + &Lsmooth (Aw; + XCa, (Ay, XY):
f - f - VL;
A. = Aw VL;
Ag Ay VL;
Update E;
Freeze experts and train assignment module; while not converged do
L= Lpred (f, Aw, X, YY):
Av Aw VT;;
3
IME interpretability capabilities
3.1 S-IME;
Fach prediction can be expressed as a switch statement with the cases specifying the prediction functions of experts, and the case conditions specifying the assignment predicate corresponding to each expert. Thus, elfectively have single interpretable function defining each prediction globally-interpretabll model. This can be useful for regulation-sensitee applications where the exact input-output relationships are needed, such criminal justice systems.
3.2 S-IME,
When the assignment isn't interpretable, the stability property of IME allows the data to be clustered into n subsets such that the prediction in each subset comes from a single interpretable expert. This can be likened to local interpretability, as a separate interpretable function expresses the prediction in each cluster. Post-hoc interpretability methods can be used to assess feature importance for different assignments. This can be used for model debugging to verify that cach expert's logic correct.
3.3 H-IME, H-IME,
For hierarchical assignment, the DNN assignment module decides whether a sample is easy i.e. it can be accurately predicted by simple interpretable models vs. difficult i.e. it requires complex models, this is shown through additional experiments in the Appendix C.6. Similar to S-IME,, for the easy samples. If explainability for the difficult samples is desired, post-hoc interpretability methods such as (Lundberg & Lee, 2017) may be adapled, however, their fidelity and faithfulness would be worse than the explanations coming from the mnherently-interpeetabb experts. In addition, understanding why particular samples are assigned as easy vs. difficult can give insights into different data distribution modes (e.g. different seasonal climates and clothing sales), significant regime changes over time (c.g. after an ad campaign is launched for product), and data anomalies (e.g. when a pandemic butbreak occurs).
can obtain local interpretations

---

Published in Transactions on Machine Learning Research (05/2023)
Accuraccy-nterpreeabbliyy trade-off with H-IME. The main difference between H-IME and S-IME is using pre-trained DNN as an expert. The number of samples that are assigned to pre-trained DNN expert can be controlled by a UDNN as described in Eq. 7. Increasing a yields less samples being assigned to the DNN expert, and hence constitutes a mechanism to increase the ratio of samples for which we use interpretable decision making. The effect of changing the value a is empirically shown Sec. 4.1. For some real-world datasets, H-IME can be highly valuable in preserving the accuracy (better than fully-interpretablee S-IME) while utilizing interpretable models for a significant majority of samples.
4 Experiments
We evaluate the performance of IME on numerous real-world tabular and time-scries datascts. As tabular data, we examine both the ones with time component, where the assignment module lakes past errors as input, and the ones without past errors. Detailed descriptions of datasets and hyperparameter tuning are available in the Appendix C.
4.1 Tabular tasks with time component
We conduct experiments on the Rossmann Store Sales data (Rossmann), which has 30 features. Note that for this dataset, there time feature, but we do not have sequential encoding of features from many timesteps, thus the problem is treated tabular data prediction task.
Baselines. We compare IME to black-box models including Multi-layer Perceptron (MLP) (Gardner & Dorling, 1998), CatBoost (Dorogush et al., 2018), LightGBM (Ke et al., 2017) and XGBoost (Chen & Guestrin, 2016); and iheeretty-inteerpetaable Neural Additive models (NAM) (Agarwal et al., 2020), linear regression (LR) and shallow DTs. Our goal is to analyze the achievable performance and the accuracy vs. interpretability trade-ofT.
Training and evaluation. We n=20 experts, either LR,, shallow soft DTss (Frosst & Hinton, 2017), mixture of both. For interpretable assignment, LR is used the assignment module.
Results. Table shows the performance of baselines and IME in RMSE. We observe significant performance gap between a single interpretable model and a black-box model like MLP. Using S-IME, with an interpretable assignment module (the first two rows in the IME section in Table 1), we observe significant outperformance compared a single interpretable model, but underperformance compared to black-box models as expected. The performance becomes comparable black-box models with S-IMEs, using DNN assignment module. This underlines the importance of high capacity assignment for some problems. IME with hierarchical-level assignment, H-IME, and H- H-IME,, yields the the best accuracy, while offering partial interpretability, as shown in Fig. 3, and is achieved by changing the penalty parameter for DNN assignment. As expected, the more samples are assigned to interpretable experts, the less accurate the model becomes. We observe that 20% of samples can be assigned to interpretable models with LR experts and 40% with soft DT experts, with almost no loss in accuracy a large portion the data is sufficiently easy to be captured by interpretable models.
Baselines
Model Name
RMSE
IME
Assigner
Expert
Expert type
Interp. samples
RMSE
MLP
457.72
S-IMF;
Interp.
Interp.
Linear
100.00 %
1298.57 96.95
Black-box Model
CatBoost
520.07
S-IME,
Inserp-
Interp.
SDT
100.00 %
820.83 +: 84.50
LightGDM
490.57
S-IME,
DNN
Interp.
Linear
100.00 %
671.46 + 94.34
XGBoost
567.8
S S-MEE
DNN
Interp-
SDT
100.00%
547.72 + 43.06
S-IME,
DNN
Interp.
Linear/SDT
100.00 %
565.20 191.43
H-IME,
Interp.
Interp./DNi
DNN DNN/Linea /Linear
18.10%
453.33 4: 37.53
Interpeetable Model
Linear Soft DT' (SDT)
1499.45
HME,
Interp. Interp./
DNN /SDT
40.01%
486.94 17.84
1181.17
H-IME,
DNN
Interp//DN
DNN DNN/Linca
32.00%
506.53 + 11.93
NAM
1497.47
H-IME,
DNN
Inter./DNN
/SD/T
43.32%
451.93 +14.08
Table 1: Performance of different methods Rossmann dataset in terms of root mean squared error (RMSE) "IME with interpretable assignment module" performs better than a single interpretable model but worse than black box models. "IME with DNN assignment module" performance is comparable with that of black box models. "Tierarchical IME with a DNN expert" outperforms the best model while offering partial interpretability.
7

---

Published in Transactions on Machine Learning Research (05/2023)
H-IME Linnar Experts
H-IME, SDTTs Experts
H-IME, Linnar Experts
H-IME; SDDTs Experts
1500
1250
RMSE
730
100
0
25
50
75
100
25
50
75
100
25
50
75
100
0
25
50
70
100
of Samples assigned to Interprotable Experts
H-IME
Interpretable Model
MLP
IME
Figure 3: Partial interpretability by H-IME, and H-IME Different accuracy values are obtained by sweeping the DNN penalty hyperparameter a. With ~ 40% of samples assigned to soft DT experts, almost no degradation in accuracy is observed.
4.2 Tabular tasks without time component
On standard tabular tasks without time component, we compare the performance of S-IME; (interpretable assignment and interpretable experts) with other interpretable models. For classification, Telecom Churn Prediction (tel, 2018), Breast Cancer Prediction (Dua & Graff, 2017) and Credit Fraud Detection (Dal Pozzolo, 2015) datasots; while for regression, FICO Score Prediction (fic, 2018) datasets are used.
Baselines. For classification tasks, we compare with other inheerenly-interppetaalle such as Neural Additive models (NAMs) (Agarwal et al., 2020), Explainable Boosting Machines (EBMs) (Caruana et al., 2015), Black box models such Deep Neural Networks (DNNs) and Gradient Boosted Trees (XGBoost).
Results. For classification lasks, we observe the outperformance of S-IME; compared to other inherently- interpretable and black-bor models, shown in Table 2. For regression tasks, none of the inherently- interpretable models outperforms DNNs on the FICO datasct, however, S-IME, is observed to be the second best. Results for partially interpretable versions of IME are provided in the Appendix C.3.
AUC
RMSE
Models
Telecom Churn
Breast Cancer
Credit
FICO
Regression
.849
.995
.975
4.344
Simple Interpretable
DTs
.824
.992
.956
4.900
NAMs
.980
3.490
Inherently Interpretable
EBMs
.852
.995
.976
3.512
S-IME;
.860
.999
.980
3.370
XGBoost
.828
.992
.981
3.345
Non-Interpretable (Black-Box)
DNNs
.997
.980
3.324
Table 2: Higher AUCs and lower RMSEs are better. Bold indicates best results, while italic is second best. We report: results on regression dataset (FICO) for understanding credit score predictions, well as three classification datasets: Credit, Telecom Churn, and Breast Cancer. Baseline results are from (Caruana et al., 2015) and (Interpretml, 2020).
4.3 Time-series forecasting tasks
We conduct experiments on multiple real-world time-series datasets for forecasting tasks, including Electric- ity (Electricity), Climate (Climate) and ETT (Zhon et al, 2021). Unlike tabular data, time-series datasets have models that encode information from multiple timesteps for each prediction.
Baselines. We compare to DNNs including LSTM (Hochreiter & Schnidhuber, 1997), Transformer (Vaswani et al., 2017), TCN (Lea et al., 2017) and Informer (Zhou et al., 2021); and interpretable models including LR and autoregressive (AR) models (Triebe et al., 2019).
Training and evaluation. We use n=10 LR experts for IME and LR or LSTM assignment modules for S- TME; and S-IME, We conduct hyperparameter tuning using grid search based on the validation performance.

---

Published in Transactions on Machine Learning Research (05/2023)
Black Bux Models
White Box Models
IME
Features
Datasets
Forecast horizon
LSTM
Informer
Transforer
TCN
AR
LR
S-IME,
S-IME,
24
.178
.159
.163
.172
.173
.167
.158
.158
Electriity
.204
.188
.199
.155
.198
.188
.180
.800
.251
.24
.252
.235
.242
.244
.217
.244
.094
.094
.092
.095
.088
.090
.011
Clinatt
.128
.176
.131
.134
.140
168
.230
313
.212
.209
.228
231
.222
220
Univariate
.055
.075
.077
.084
.030
.034
.027
.027
ETTh1
48
.202
.109
.131
.13
.019
.054
.040
012
168
.293
.288
.140
.225
.089
.009
.071
.071
24
.093
.121
.073
.076
.067
.071
.065
066
ETT
.122
.145
.108
.110
.104
.101
.096
.096
.256
.169
.248
.017
.017
.077
.013
.011
.011
ETTml
.029
.056
.032
.020
.020
168
.161
.171
.137
.110
.060
.049
.047
.044
21
.066
.088
.092
.067
.127
.079
072
.022
Climate
.098
.119
.216
.095
.106
.097
.097
.204
.112
.129
ETTh1
.301
.694
.274
.148
1.167
.085
.072
.070
Multivariate
168
.518
1.027
.303
.172
1.788
.142
.124
.144
.224
.385
.316
.263
.244
.105
.098
.091
ETT12
48
.500
1.557
.582
.772
.738
.300
.260
.263
168
.923
2.110
1.124
.817
.770
.578
.477
.548
24
.034
.070
.029
0355
.024
.022
.017
.017
ETTml
.065
.074
003
.067
.031
.029
168
.212
.371
440
.127
.069
.078
.078
Winning counts
2
0
3
2
16
14
Table 3: The MSE of baselines and S-1ME for various time-series datasets at different forecasting horizons. Note that here S-IME already outperforms DNNs H-IME not considered.
Results. Table 3 shows the MSE for different datasets and forecasting horizons. For univariate forecasting, IME outperforms black-box models, while for multivariate forecasting, IME is comparable with black-box models (second-best accuracy after TCN). We also observe that using an interpretable assignment doesn't degrade the accuracy.
4.4 Interpretability analyses
Global interpretability. In S-IME, with both the assignment module and experts being interpretable, explanations are reduced to concise formulations. Algorithm 2 exemplifies the discovered interpretable model for Electricity dataset with input sequence length / = 3 and the number of experts = 2, with denoting the expert errors. Note that when comparing global interpretability offered by IME with that of a single interpretable model, IME explanations might be harder to interpret since they involve an n-way classification followed by the explanation produced by each expert.
Algorithm 2: The interpretable model discovered by S-IME,
A Aenpert; -05x, 03x, -.r 3 38e, 06ez
Acapert, = =-03r, +
+.05e2
if Aezperti > Aumpert, then
Me+1 = -049::-1 -292: 1.17
else
1h+1 = --081:;-1- -23x,-2 1.13x,-3

---

Published in Transactions on Machine Learning Research (05/2023)
Expert interpretability. All IME options provide local expert interpretability, in which case, explanations can be given as equations for each expert, or can be conveniently visualized. Fig. 4 exemplifies this on Rossmann.
Expert
Feature Weights
0.75
Expert
Expert
0.25
0.00
-0.25
Store
Day of Number Open
Promo
Store
Assort.
Comp.
Comp.
Comp.
Promo2
WeekCustomers
Type
Distance Month
Year
Figure 4: Expert interpretability. Normalized weights for different LR experts on Rossmann, allowing expert behavior to be easily interpreted. "Expert 1" puts high positive weights 'the number of customers', 'open' and 'competition distance'; "Expert 2n on "assortment'; and "Expert on 'the number customers'.
Sample-wise expert interpretability. Fig. 5 shows feature weights of the interpretable LR model for different samples on Rossmann. For the first sample shown in Fig. 5a, the most influential feature is the number of customers entering the store. Whereas for the second sample shown in Fig. 5b, multiple features influence the model output.
0.4
Raaturg Weignno
Feature Wesgett
Wuak Number
home
Sunn
Asort
Corpa
Runuz
Wook Numberr
Prama Spe
Store
Acuart Damp.
Prenez
(a) The most influential feature the number of customers.
(b) Multiple features influence the model output.
Figure 5; Sample-wise feature weights on Rossmann dataset.
Identifying data distribution modes. We construct synthetic dataset (Fig. 6a) to showcase the additional interpretability capabilities. We run S-IME; with LR assignment (Fig. la) and S-IME; with MLP assignment (Fig. 1b). Fig. 6b shows that both S-IME, and S-IME, assign most samples so that Y = 21 are assigned to Expert 1, = X2 to Expert 2 and Y = Ø3 to Expert 3. In this way, IME yields insights into how different subsets can be split based on unique characteristics.
Identifying incorrect model behavior. Another use case for interpretability is model debugging and identifying undesired behaviours. To showcase IME for this, we focus on synthetic data shown in Fig. 6a. Fig. 6e shows the weights for the experts for S-IME with interpretable (top) and DNN assignment, (bottom) respectively. S-IME, yields almost ideal behavior; Expert assigns highest weight to feature 21, Expert 2 to and Expert 3 to 2z. However, Expert in S-IME, incorrectly assigns the highest weights to feature Investigating different weights in this way can help verify whether the expert logic correct. Model builders can benefit from this insights to debug and improve model performance, e.g. by replacing or down-weighing certain experts.
Identifying temporal regime changes. Fig. 7a shows synthetic univariate datasct where the feature distribution changes over time. S-IME, with LR assignment and S-IME, with an LSTM assignment module used, and all interpretable experts simple auto-regressive models. Fig. 7b and 7e show that IME
10

---

Published in Transactions on Machine Learning Research (05/2023)
IME Assignment
IME Expert Weights
Exprr
Expert
Feature X
X
Feature X1
Target Y
MW
Facture
S- MFA
Expert
Iapert
boart 3
(a)
21:
01
(b)
(c)
Figure 6: Identifying different clusters for S-IME, and S-IME,. (a) Syuthetic data with three features. (b) Number of samples from different modes assigned to each expert. (c) Weights given to each feature by different experts.
identify changes over time, using different experts for distribution modes. This capability can be used to get insights into temporal characteristics and major events.
Features
S-IME;
S-IME
Expert
Feature Value
Expert
Expert
200
600
800
200
400
600
800
Time (a)
Time
Time (c)
(b)
Figure 7: Identifying regime changes. (a) Synthetic univariate time-series where feature distribution changing
time. (b) S-IME; identifies changes in time regime and utilize all experts. (c) S-IMEa identifies changes in regime but only experts used.
User study on IME explanations. TME. offers faithful local explanations, as the actual interpretable models behind each prediction are known. This is in contrast post-hoc methods used to explain black-box methods such SmoothGrad (Smilkov et al., 2017), SHAP (Lundberg & Lee, 2017), Integrated Gradients (Sundararajan et al., 2017), and DeepLift (Shrikumar et al., 2017). Such methods may be unreliable (Adebayo et al., 2018; Hooker et al., 2019; Ghorbani et al., 2019), especially for time series (Ismail et al., 2020a). To demonstrate the quality of IME's explanations, we design a user study that focuses on comparisons with the useeonly-used post-hoc method, SHAP (Lundberg Lee, 2017) (we are referring to the quality of the explana- tions from user perspective ic., given an explanation will the user be able understand, interact and modify model based on the explanation?). We base the objective component on suman--rounded metrics (Doshi- Velez & Kim, 2018), where the tasks conducted by users simplified versions of the original task. We use the task of sales prediction Rossmann, and experiment with S-IME; (LR assignment & 20 experts) and MLP the black-box model. First, we consider counterfactual simulation scenario. For each sample, the participants are given an input and an explanation (along with access to training data for analyses) and asked how the model output (sales prediction) would change with the input changes. The participant are able to choose one of the options: no change, increase or decrease. The same sample with explanations from the chosen interpretable experts of IME and SHAP (Lundberg & Lee, 2017) are provided (as shown in Appendix Fig 12 & Fig 13. All examples shown correctly classified by both models. In total, collect. 77 samples from 15 participants.
11

---

Published in Transactions on Machine Learning Research (05/2023)
When provided with IME explanations, participants can predict model behavior with an accuracy of 69% vs. 42% of SHAP. This shows that explanations provided by IME can be easily understood by participants and are more faithful. Next, we also ask the participants which explanations they trust more for each sample: explanation A/B, both, or neither. IME is chosen for 87% of the cases vs. 6.5% of SHAP (and neither gets 6.5%), demonstrating the trust worthness of IME's explanations. The Appendix D.1 includes more details.
4.5 Ablation studies
S-IME;
0.5
Single Linear Model
1500
LSTM
0.4
S-IME,
1400
S-IME,
RMSE
MSE
0.3
1300-
0.2
1200
100
200
25
100
Input Sequence Length
Number of Experts
(b)
Figure 8: (a) Error with increased model capacity for S-IME with LR on Electricity. (b) Number of experts vs. accuracy Rossmann.
Comparison to black-box model performance. Table 3 shows that IME can outperform black- models. To further shed light on this, we investigate the effect of changing the interpretable model's capacity. For LR experts, as we increase the input sequence length (and hence the number of learnable coefficients), the model accuracy increases shown in Fig. Sa. Note that increasing the sequence length doesn't 't affect the number of parameters for an LSTM. TME outperforms single interpretable model for any sequence length, and starts outperforming LSTM when the accuracy gap between LR and LSTM gets smaller.
Effect of the number of experts. Fig. 8b shows the effect of the number of experts. As the number of experts increases, the accuracy increases until a certain optimal (40 for LR experts, 20 for soft DTs). After this, incrcasing the number of experts causes a slight decrcase in accuracy as experts become underutilized. The number of experts can be treated as a hyperparameter and optimized on validation dalaset. We also note that IME with fewer experts can be desirable for improving overall model interpretability.
Contributions of IME components. We perform ablation studies on important IME components, shown in Table 4. Removing each yields worse performance. Note that IME can be used for any labular dataset without a time component by removing the past error from the input to the assignment module, i.e A([X; Ex- + A(X) (as shown in Sec. 4.2). However, the availability of errors over time improves the performance of the assignment module.
5 Related Work
Mixture of experts. Jacobs et al. introduced ME over three decades ago. Since then many expert architectures have been proposed such as SVMs (Collobert et al., 2002) Dirichlet Processes (Shahbaba & Neal, 2009) and Gaussian Processes (Tresp, 2001). (Jordan & Jacobs, 1994) propose hicrarchical assignment for MEs. (Shazeer et al., 2017) propose an effective deep learning method that stacked ME as layer between LSTM layers. (Shazeer et al., 2018; Lepikhin et al., 2020; Fedus et al., 2021) incorporate ME as a layer in Transformers. (Pradier al., 2021) introduce human-ML ME where the assignment module depends on human- masen rules, and the experts themselves are black- box DNNs. IME combines interpretable experts with DNNs to produce inherently-interpredae architectures with similar accuracy, comparable to DNNs.
12

---

Published in Transactions on Machine Learning Research (05/2023)
Loss function
S-IME;
S-IME,
Proposed IME
1298.57
671.46
Without Lutit
1369.54
710.61
Without Ssmooh
1481.87
844.15
Without La,
1342.89
701.50
Without using past errors
1338.57
745.23
Without freezing step (Sec. 2.4)
1315.70
760.25
Table 4: Ablation studies on Rossmann (metrics in RSME).
Interpretable DNNs for structured data. (Agarwal et al., 2020) propose NAM which uses a DNN per feature. NAM does not consider feature-feature interactions and thus is not suitable for high-dimensional data. To address this, NODE-GAM (Chang et al., 2021) is introduced, modifying NODE (Popov et al., 2019) into generalized additive model. Both NAM and NODE-GAM are only applicable to tabular data. (Luo et al., 2021; Irsoy et al., 2012; Frosst & Hinton, 2017) propose the use of a soft DT to make DNNs more interpretable. N-Beats (Oreshkin et al., 2019) uses residual stacks to constrain trend and scasonality functional forms to generate interpretable stacks for univariate time series. (Shulman & Wolf, 2020) create a per-user decision tree for tabular recommendation systems. In contrast to these, IME (1) provides explanations that accurately describe the overall prediction process with minimal loss in accuracy, (2) supports both tabular and time-series data, and (3) can be casily used for complex large-scale real-world datasets.
6 Conclusions
We propose IME, novel inhereetly-inneeppeelbbb framework. IME offers different options, from providing explanations that are the exact description of how prediction is computed via interpretable models, to adjusting what ratio of samples can be predicted with interpretable models. On real-world tabular and time-series data, while achieving useful interpretability capabilitics (demonstrated on synthetic datasets as well as with user studies), the accuracy of IME is on par with, and some cases even better than, state-of-the-art DNNs. We leave extension of LME to unstructured high-dimensional data types, such as image and text, to future work.
References
Fico community.
Nov
2018. 8, 18
Telco customer churn. https://coomunityy ibm. www. http:///aocooot/oo//oooo///////a /tt.aa/naacoo////0///////aooooon chwreerrr-hu----i-e 1113, Feb 2018. 18
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In NeurIPS, 2018. 11
Rishabh Agarwal, Nicholas Frosst, Xuezhou Zhang, Rich Caruana, and Geoffrey E Hinton. Neural additive models: Interpretable machine learning with neural nets. arXiv preprint arXiv:2004.139) 2020. 2, 7, 8, 13
Screan O. Arik and Tomas Pfister. TabNet: Attentive interpretable tabular lcarning. In AAAI, 2021. 1
Emmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup. Conditional computation in neural networks for faster models. arXiv preprint arXiv:1511.06297 2015. 5
Adrien Bibal, Rémi Cardon, David Alfter, Rodrigo Wilkens, Xiaoou Wang, Thomas François, and Patrick Watrin. Is attention explanation? an introduction to the debate. In Proceedings of the 60th Annual Meeling of the Association for Computational Línguistics (Volume 1: Long Papers), 2022. 2
Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Mare Sturm, and Noemie Elhadad. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In KDD, 2015. 1,
13

---

Published in Transactions on Machine Learning Research (05/2023)
Chun-Hao Chang, Rich Caruana, and Anna Goldenberg. Node-gam: Neural generalized additive model for interpretable deep learning. arXiv preprint or arXiv:2106006613, 2021. 2, 13
Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In KDD, 2016.
Michael Chui, James Manyika, Mehdi Miremadi, Nicolaus Henke, Rita Chung, al. Notes from the AI frontier. McKinsey Global Institute, 4 2018.
Climate. Jena-climate dataset. https://www kaggk/// comm/massrrb//eeaaclllinen clinate, 2017. 8
Ronan Collobert, Samy Bengio, and Yoshua Bengio. A parallel mixture of SVMs for very large scale problems. Neural computation, 2002. 12
Andrea Dal Pozzolo. Adaptive machine learning for credit card fraud detection. 2015. 8, 18
Anna Veronika Dorogush, Vasily Ershov, and Andrey Gulin. Catboost: gradient boosting with categorical features support. arXiv preprint arXi::1810.1133 11366, 2018. 7
Finale Doshi- Velez and Been Kim. Considerations for evaluation and generalization in interpretable machine learning. Explainable and interpretable models in computer vision and machine learning, pp. 3-17, 2018. 11
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http://archiie. ics wwwp:tieeeeissi..ee..... uci edu/ ml/math- http:///popttteee/aaaaaeeaaaaaaa
8, 18
David Eigen, Marc' Aurclio Ranzato, and Πya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4914 2013. 5
Electricity. Uci-electricity dataset. https://archhis wwwpchiiieiii.ieee....m
www.eeiisss..ieeeeee.o......eeee www.uuii.uuuuaae aeddeed/////daasttt... 2014. 8
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models
with simple and efficient sparsity. arXiv preprint arXiv:2101.03961 2021. 2, 12, 23
Nicholas Frosst and Geoffrey Hinton. Distilling neural network into soft decision tree. arXiv preprint
arXiv:1711.09758 2017. 3, 7, 13
Matt W Gardner and SR Dorling, Artificial neural networks (the multilayer perceptron) review of applications in the atmospheric sciences. Atmospheric environment, 1998. 7
Alireza Ghods and Diane Cook. A survey of deep network techniques all classifiers can adopt. Data mining and knowledge discovery, 35(1):46 87, 2021.
Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of neural networks is fragile. In AAAT Conference on Artificial Intelligence, 2019. 11
Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for (abular data. arXio preprinl 21v:22106.11 1 11999, 2021.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. In Neural compulation, 1997. 8
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretability methods in deep neural networks. In NewrIPS, 2019.
Interpretml. Interpretmm//interpret Fit. interpretable models. explain blackbox machine learning,, Sep 2020.
URL https://githui: htgps/hww..oommeeeeeeeetttterrre 8
Ozan Irsoy, Olcay Taner Yildız, and Ethem Alpaydın. Soft decision trees. In ICPR, pp. 1819 1822. IEEE, 2012. 3, 13
Aya Abdelsalam Ismail, Mohamed Gunady, Héctor Corrada Bravo, and Soheil Feizi Benchmarking deep learning interpretability in time series predictions. arXiv preprint Xin::00000. 13924, 2020a. 11
14

---

Published in Transactions on Machine Learning Research (05/2023)
Aya Abdelsalam Ismail, Mahmudul Hasan, and Faisal Ishtiaq. Improving multimodal accuracy through modality pre-training and attention. arXiv preprint arXiv:201 1.06102, 2020b. 6
Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and GeofTrey E Hinton. Adaptive mixtures of local experts. Newral computation, 1991. 4, 12
Michael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the algorithm. Newral compudation, 1994. 12
Manu Joseph. Pytorch tabular: A framework for deep learning with tabular data. arXiv preprint arXiv:2104.13638 2021. 1
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. GightGBM: highly efficient gradient boosting decision tree. NIPS, 2017. 7
Solomon Kullback and Richard A Leibler. On information and sufficieccy. The annals of mathematical statistics, 1951.
Colin Lea, Michael Flynn, Rene Vidal, Austin Reiter, and Gregory Hager. Temporal convolutional networks for action sogmentation and detection. In CVPR, 2017.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXtv:2006.16663 2020. 12
Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E Gonzalez, and lon Stoica. Tune: A research platform for distributed model selection and training. arxiv preprinl arXix:1807.051 2018. 18
Bryan Lim, Serean O. Arik, Nicolas Locff, and Tomas Pfister. Temporal fusion transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting, 2021. 1
Zachary C Lipton. The doctor just won't accept that! arXiv preprint ar ar:i::1711.0000 2017.
Zachary C Lipton. The mythos of model interpretability. In Queue, 2018. 1
Scott Lundberg and Su-In Lee. A unified approach to interpreting model predictions. NewrIPS, 30, 2017.
6, 11
Haoran Luo, Fan Cheng, Heng Yu, and Yuqi Yi. Sdtr: Soft decision tree regressor for tabular data. IEFE Access, 9:55999 56011, 2021. 2, 3, 13
Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time scries forecasting. arXiv preprint arXim:1905.1 104,7,0777 2019. 2, 13
Sergei Popov, Stanislav Morozov, and Artem Babenko. Neural oblivious decision ensembles for deep learning on tabular data. arXiv preprint 1rXiv:1909.000 06312, 2019. 1, 13
Melanie F Pradier, Javier Zazo, Sonali Parbhoo, Roy H Perlis, Maurizio Zazzi, and Finale Doshi- Velez. Preferential mixture-of-cxpert Interpretable models that rely on human expertisc as much possible. In AMI Annual Symposium Proceedings, volume 2021, pp. 525. American Medical Informatics Association, 2021. 12
Isha Puri, Amit Dhurandhar, Tejaswini Pedapati, Karthikeyan Shanmugam, Dennis Wei, and Kush R Varshney. Cofrnets: Interpretable neural architecture inspired by continued fractions. NeurIPS, 2021.
Rossmann. Rossmann dataset.
2015. 7, 18
Babak Shahbaba and Radford Neal. Nonlinear models using dirichlet process mixtures. JMLR, 2009. 12
Noam Shazcer, Azalia, Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Lo, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The pparsely-gated mixture-of-xxperts layer. orXiv preprint MoX:::10110060 2017. 2, 5, 12, 23
15

---

Published in Transactions on Machine Learning Research (05/2023)
Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. Mesh-tensorflow: Deep learning for supercomputers. arXiv preprint wrXiv:1811.00056 2018. 12
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Tearning important features through propagating activation differences. In International Conference on Machine Learning, 2017. 11
Eyal Shulman and Lior Wolf. Meta decision trees for explainable recommendation systems. In Proceedings of the AAAT/ACM Conference on AI, Ethics, and Society, pp. 365-371, 2020. 13
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg, Smoothgrad: removing noise by adding noise. arxie preprint orXiv: 1 2017. 11
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In ICML, 2017.
11
Volker Tresp. Mixtures of gaussian processes. NIPS, 2001.
Oskar Triebe, Nikolay Laptev, and Ram Rajagopal. Ar-net: A simple auto-regressive neural network for time-series. Xiviv preprint arXiv:1911. 1914.1411,, 2019. 8
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017. 8
Ruofeng Wen, Kari Torkkola, Balakrishnan Narayanaswamy, and Dhruv Madeka. multi-horizon quantile recurrent forecaster. arXiv preprint arXi::1711.11053, 2018.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang, Informer: Beyond efficiont transformer for long sequence timo-scrics forccasting. In AAAI, 2021. 8, 18
16

---

Published in Transactions on Machine Learning Research (05/2023)
A H-IME Framework
IME
Inputs
Experts
Predictions
Experts
Expertz
error
Experta
Predictions
Experty
Predictionow
Expert
Assignment weights
Smoothness
Diversity
Targets
WINE
Hierarchical Assignment module
Hierarchical Accuracy Loss
Assignment module (Interpretabie/DNng
Accuracy Losset
ONN
Utilization
Assigument
Hierarchical Assignment weights
Targets
Prediction
Prediction
Figure 9: H-IME consists with DNN expert and IME.
B IME for classification
To adapt IME for classification problems, the model outputs the output class probabilities instead of regression value. Correspondingly, the prediction loss Lpred is changed from Eq. 2 to the following:
CEIY,
Eprea Aws X,X,Y) log Σn n
Am(X):
(8)
22m
where (F((,.(()) standard softmax cross entropy.
17

---

Published in Transactions on Machine Learning Research (05/2023)
C Experimental details
All experiments were ran on NVIDIA Tesla V100 GPU. We perform time-series data experiments in two settings multivariate and univariate forecasting. The multivariate setting involves multivariate imputs and a single output. The univariate setting involves univariate inputs and outputs, which are the target values described below. We use MSE as the evaluation metric.
C.1 Datasets
We perform 70/10/20 train/validation/tesst split for each dataset.
Tabular tasks without time component
Telecom Churn Prediction [Classification] Customer churn is the rate al which customers are lost, used by telecom companies to predict the number of customers that will leave a telecom service provider. Churn datasct (tel, 2018) consists of features such as services booked, account information, and demographics overall there are features and 7043 samples. The task is to classify a customer as churn or not. churn. Breast Cancer Prediction [Classification] The dataset based on MT. Breast Cancer Wisconsin dataset (Dua & Graff, 2017), consisting 30 real-valued features that are computed for each cell nucleus and 569 samples. The goal is to predict if the tissue is malignant or benign.
Credit Fraud Detection [Classification] The dataset (Dal Pozzolo, 2015) contains transactions made by credit cards. The dataset consists of 28 features obtained from PCA and 284,807 transactions. The task is to predict whether a given transaction is fraudulent or not. Note that this datasel is highly unbalanced with only 0.172% of samples labeled as fraud. FICO Score Prediction [Regression] The FICO score a proprictary credit score to determine creditworthiness for loans in the United States. The dataset (fic, 2018) is comprised of real-world credit applications made by customers and their assigned FICO Score, based on their credit report information. The dataset consists of 23 features and the goal is to predict scores based different features.
Tabular tasks with time component
We conduct experiments on the Rossmann Store Sales data (Rossmann), which has 30 features. Note that there time feature, but we do not have sequential encoding of features from many timesteps, thus the problem is treated as tabular. The dataset consists of samples from 1115 stores. The goal to predict daily product sales based on sales history and other factors, including promotions, competition, school and state holidays, seasonality, and locality.
Time-series forecasting tasks
The dataset measures the electricity consumption of 321 clients. We convert the dataset into hourly-level measurements and forecast the consumption of different clients over time. Climate² The dataset consists of 14 different quantities (air temperature, atmospheric pressure, humidity, wind direction, etc.), recorded every 10 minutes between 2009-2016. We convert the dataset into hourly-level measurements and forecast the hourly temperature over time.
Electricity'
ETT We conduct experiments ETT (Flectricity Transformer Temperature) (Zhou et al., 2021). This consists three datasets: two hourly-level datasets (ETTh) and one 15-minute-level dataset (ETTm), measuring six power load features and "oil lemperature", the chosen larget value for univariate forecasting.
C.2 Hyperparameters
All baselines were tuned using (Liaw et al., 2018) with at least 20 trials. Hyperparameter search grid used for each model available in Table 5. To avoid overfitting, dropout and early stopping were used. The best hyperparameters were chosen based on the validation datasct while the results reported in the table
Inttpe://archive.tt ics uci www..dd/al//sstseettttt. Phttps://wwx.kage hangg//e.ccmmmaarri///eeaaa ciintee
18

---

Published in Transactions on Machine Learning Research (05/2023)
are the test dataset. For all IME experiments set & for Luti- For (f, X), set 0.2. All models were trained using Adam optimizer, and the batch size and learning rate values were modified at each experiment.
Rossmann Hyperparameter used for IME are give in table 6. For baselines: CatBoost, LightGBM and XGooost we used parameters specified by benchmark 3, LR: The batch size is 512 and learning rate is .001. Soft DT (SDT): The batch size 512, learning rate is .0001 and the depth is MLP: The batch size 512, layers used, each containing 128 hidden units with learning rate .001. NAM: Each feature is modeled using MLP with 2 hidden layer each with 32 hidden units, with batch size 512 and learning rate .001.
Electricity For IME, we use batch size of 512 and sequence length 336; the remaining hyperparameters are available in Table 8, While hyperparameters for baseline are in Table 9.
Climate For IME, a batch size of 512 and sequence length 336 was used the remaining hyperparameters are available in Table 10. While for baselines detailed hyperparameters are available in Table 11.
LSTM
TCN
Transformer
Informer
Learning rate
0.00001 to 0.1
0.00001 to 0.01
0.00001 to 0.01
0.00001 to 0.01
Batch size
64, 64,128,255
64,128, 256
64,128, 256 128, 256,512 2 to 6
64, 24,
Hidden units
128, 256,512
128, 256,512 N/A
2048
Encoder layers
2 to 6
1 to 3
Decoder layers
N/A
N/A
2 to 6
1 to 3
Levels
N/A
to 10 5 to 15
N/A
N/A
Kernal
N/A
N/A
N/A
Attention heads
N/A
N/A
2,4,8
8 512
Embedding sizes
N/A
N/A
128, 256, 512
Table 5: Hyperparameter search grid used for different models
Number of
Learning rates
Model Hyperparameters
IME
experts
r
p
8
7
Linear Assign. LR Expert
20
.0001
.001
10
0
.1
1
Linear Assign. SDT Expert
20
.0001
.001
.1
0
.1
1
MLP Assign. LR Expert
20
.0001
.001
.1
0
.1
1
MLP Assign. SDT Expert
20
.0001
.001
.1
Table IME Hyperparameters for Rossmann dataset.
Number of
Learning rates
Hyperparameters
Dataset
Assigner
Expert
experts
P
7
8
X
Telecom Churn
LR
LR
2
0.001
0.0001
0
0
1
Breast. Cancer
LRR
LR
2
0.1
0.01
1
0
Credit
LR
LR
3
0.001
0.0001
0
0
FICO
LR
SDT
2
0.01
0.0005
20
0
Table 7: IME Hyperparameters for different tabular datasets.
19

---

Published in Transactions on Machine Learning Research (05/2023)
Number of
Learning rates
Model Hyperparameters
IME
experts
r
8
X
Linear Assigner LR. Expert
10
.0001
.001
10
.1
1
LSTM Assigner T.R Expert
10
.0001
.001
1
0
.1
1
Table 8: IME Hyperparameters for Electricity dataset.
Model
Forecast horizon
Batch size
Sequence length
Learning rate
24
512
336
.0001
ruto-segressive
48
512
336
.0001
168
512
336
.0001
24
512
336
.0001
LR
48
256
.0001
168
.0001
Forceast Horizon
Batch Size
Sequence Longth
Learning Rate
Hidden Units
Layers
24
512
168
.001
256
2
LSTM
48
256
96
.01
128
5
168
512
168
001
256
2
Forecast Horizon
Batch Size
Sequence Length
Learning Rate
Hidden Uuits
Kernel
Level
24
256
96
.0001
512
8
TCN
48
256
96
.0001
512
168
.0001
512
Forecast Horizon
Batch size
Sequence length
Learning rate
Hidden units
Encoder
Devoder
Heads
24
512
168
.001
256
3
5
4
Transformer
48
256
168
.01
256
3
5
4
168
512
168
.001
256
3
5
4
24
512
168
.0001
512
2
1
8
Informer
48
512
168
.0001
512
2
1
8
168
168
.0001
512
1
Table 9: Baseline Hyperparameters for Electricity dataset.
Number of
Learning rates
Model Hyperparametors
Features
IME
Experts
p
Linear Assigner LR Expert.
2
.002
.005
1
0
.1
Univariate
LSTM Assigner LR Expert
2
.001
.01
10
0
.1
Lincar Assigner LR Expert LSTM Assigner LR Expert
10
.0001
.001
1
.001
10
Multivariate
10
.0001
.001
1
0
10
Table 10: IME Hyperparameters for Climate dataset..
20

---

Published in Transactions on Machine Learning Research (05/2023)
Univariate
Model
Forecast horizon
Batch size
Sequence length
Learning rate
24
256
336
.001
Auto-regressve
48
256
336
.005
168
.005
24
256
336
.005
LR
48
256
336
.001
168
256
336
.005
Forecast horizon
Batch size
Sequence length
Learning rate
Hidden units
Layers
24
1024
24
.01
128
4
LSTM
48
256
240
0001
128
5
168
1024
24
-01
128
4
Forerast Horizon
Batch Size
Sequence Length
Learning Rate
Hidden Units
Kernel
Level
512
3
TCN
512
168
256
96
.0001
512
8
8
Forceast Horizon 24
Batch Size
Sequeuce Length
Learning Rate
Hidden Units
Encoder
Decoder
Heads
64
168
.0005
256
3
3
4
Transformer
48
64
96
.0001
512
2
4
8
168
128
336
.0005
256
4
4
8
24
512
720
.0001
512
2
1
8
Informer
48
512
720
.0001
512
2
1
8
168
720
.0001
512
Multivariate
Model
Forecast horizon
Batch size
Sequence length
Learning rate
24
256
168
.0005
Auto-regressive
48
256
168
.0005
168
256
336
.001
24
256
96
.001
LR.
48
256
.001
168
.0005
Forecast Horizon
Batch Size
Sequence Length
Learning
Hidden Units
Layers
24
512
168
.001
256
2
LSTM
48
512
168
.001
128
2
168
512
168
.001
256
2
Forevast horizon
Batch size
Sequence length
Learning rate
Hidden units
Kernel
Level
24
1024
96
.001
256
12
TCN
48
1024
96
.001
256
13
1
168
1024
.001
256
Forerast horizon
Batch size
Sequence Length
Learning
Hidden units
Encoder
Deroder
Heads
24
128
96
.0005
256
5
4
8
Transformer
48
128
336
.0005
128
4
4
8
168
128
168
.00005
256
2
3
4
24
512
168
.0001
512
3
2
8
Informer
48
512
96
.0001
512
2
1
8
168
512
336
.0001
512
2
Table Hyperparameters for Climate dataset.
21

---

Published in Transactions on Machine Learning Research (05/2023)
C.3 Additional regression experiments
For regression tasks FICO dataset, preformed partial interpretable variations of IMEs including S-IMEd, H-IME, and H-IMEa
Regression Results: From Table 12, we observe that for none of the inberentdy-inteeppelabbe models outperforms DNNs on FICO dataset, however, we partial interpretable models H-IME, can outperform DNN, with only 0.2% of samples assigned to the DNN.
RMSE
Models
(FICO dataset)
Regression
4.344
Simple Interpretable
DI'
4.900
NAMs
3.490
Inherently Interpretable
EBMs
3.512
S-IME,
3.370
S-IME,
3.380
Partial Interpretable
H-IME,
3.290
H-IMF
3.360
XGBoost 3.345
Non-Interpretabi (Black-Box)
DNNs
3.324
Table 12: Lower RMSEs are better. Bold indicates best results, while stalic is second best. We report results on a regression dataset (FICO) for understanding credit score predictions.
C.4 DNNs using past errors
Past errors considered useful information for the IME assignment module they can help the assignment module choose the best expert in the next time step. IME has a direct mechanism of benefiting from past errors by yielding better assignment. One natural question can be whether the past errors would also benefit DNNs they are simply input to the models for the downstream prediction task. To shed light onto this question, we experiment by using past errors as inputs to DNNs. The results on ETTml dataset are shown Table 13. We find that adding past errors can help the accuracy DNNs but even with past. errors, the performance significantly worse than that of IME. We also underline that IME does not use past errors for its experts while yiclding downstream predictions, but only for its assignment module. We note that explainability with past errors is more natural for the assignment module, while less so for the predictive experts.
Black Box Models
Black Box Models + Past Errors
IME
Forecast horizon
LSTM
Informer
Transformer
TCN
LSTM
Informer
Transformer
TCN
S-IME;
S-IME,
24
.017
.018
.017
.017
.015
.015
.017
.016
.012
.011
48
.029
.056
.036
.032
.032
.033
.033
.027
.021
.020
168
.161
.171
.137
.100
.113
.142
.123
.168
.047
.044
Table 13: The MSE baselines, baselines with past errors and S-IME for univariate ETTml time-series dataset at different forecasting horizons.
C.5 The impact of diversity loss
Multiple loss terms are employed in IME, as given in Eq. and their contributions can be controlled via
hyperparameters. In general, the benefit of some loss terms might be more prominent datasets. Tn
22

---

Published in Transactions on Machine Learning Research (05/2023)
Table 4, show the contribution of different IME components on the Rossmann dataset. For this particular dataset, Ldio is observed to have a minor effect. We find that adding Laio has more effect on time series datasets where the output consists multiple forecasts rather than single point prediction. In Table 14, we show the effect of Liv on the ETTml dataset it is observed to help reduce MSE for different forecasting horizons.
S-IME;
Forecast horizon
Without Laie
With Laiv
.014
.012
48
.023
.021
168
.055
.047
Table 14:
The MSE for S-IME; with and without Laio on univariate ETTml time-series dataset at different
forecasting horizons.
C.6
Hierarchical assignment differentiating between 'difficult' and 'easy' samples
H-IME assignments module can also identify 'difficult' samples (i.e., samples requiring DNN for an accurate prediction) and 'casy' samples (i.e., samples that can be predicted by simple interpretable model) offering insights the task. To verify this looked into the difference in accuracy between samples assigned to DNN versus an interpretable expert for the Rossmann dataset. Table 1 showed that the best-performing model was H-IME where 43.32% of samples were assigned to an interpretable expert, in Table 15 we compare the RMSE of samples assigned to DNN with those assigned to SDT. We find that for samples assigned to DNN, DNN accuracy is much better than a single SDT this shows that these samples are in fact 'difficult' requiring more complex models to get an accurate prodiction. For samples assigned to single interpretable model, we find that single SDT for those samples oulperforms DNN, confirming that those samples are 'easy" samples where model complexity is not required for high performance rather it may harm overall accuracy.
Sample Assignement
DNN
SDT
S-IME
H-IME
DNN
i.c., difficult samples
570.05
1292.12
842.27
518.97
Interpretable expert (SDT) i.e., easy samples
120.87
92.1
97.64
23.54
Table RMSE different H-IME assignment on Rossmann.
C.7 IME in comparison to other ME variants:
Sparsely-gated ME (Shazeer et al., 2017) assigns samples to a subset of experts and then combines the outputs of gates. Switch ME (Fedus al., 2021) assigns each sample to a single expert. For fair comparison, single expert is used at inference. Table 16 also shows that IME's assignment mechanism yields superior results.
ME variant
RMSE
S-IME;
1298.57
Sparscly-gated ME (Shazeer al., 2017)
1575.18
Switch ME (Fedus et. al., 2021)
2839.56
Table 16: IME in comparison to other ME variants Rossmann (metrics in RSME).
23

---

Published in Transactions on Machine Learning Research (05/2023)
D Interpretability Results
D.1 User study details:
The participants were given instructions shown in Fig. 10.
Then, a data-sample from Rossmann was provided as shown in Fig. 11 and the participant 8 were asked to answer questions based on two explanations.
Explanations from different models, and questions were shown to participant shown in Fig. 12 and Fig. 13. Finally, the participant asked which model he trusts more shown in Fig. 14.
Interpretability User Study Sample 1
The goal this user study to test explanation methods generated different models. The task to predict the model output given an explanation.
both explanations we present examples from the Rossmann dataset. retail dataset with the goal of predicting the daily sales a store. Store sales are influenced by many factors, including promotions, competition, school and state holidays, seasonality and locality. The dataset is available here
(https://docs, google. com http://pwwotttiii.ioaoaaaaaaaaaa beo0//edttussgsharrngg
Sales turnover any given day (this what you are predicting)
Store unique Id for each store
Customers the number of customers on given day
Open an indicator for whether the store was open: 0 = closed, 1 = open
StateHoliday indicates state holiday. Normally all stores, with few exceptions, are closed
holidays. Note that all schools closed public holidays and weekends.
public holiday, 2 Easter holiday, 3 Christmas, 0 None
School holiday indicates if the (Store, Date) was affected by the closure of public schools Store Type diffarentiates between 4 different
models: 1, 3,
Assortment describes an assortment level: basic, extra, extended Competition Distance - the distance in meters to the nearest competitor store Competition Open Since [Month/Year] gives the approximate year and month of the time nearest competitor was opened
Promo - indicates whether a store is running promo on that day
Promo2 Promo2 is continuing and consecutive promotion for some stores: store participating, store is participating
Promo2 Since [Year/Wee]] describes the year and calendar week when the store started participating in Promo2
Promo2 [month] describes Promo2 was running that month takes boolean value 0/1
Use the training data to answer the following questions (you can try to find the nearest example and sort values get general trends how input features affect the output).
Based on the training dataset and your understanding of the problem please answer the questions in the following sections. In each section, you will be given training example and previous input values (If any), then you the firet question, we aak you to guesa the correct output given the sample. Then assume that you given model explanation and asked to guess the model output given multiple explanations.
asked
questions
sample.
Figure 10: User study instructions
24

---

Published in Transactions on Machine Learning Research (05/2023)
Using the training data, predict the daily sales of the store in the sample below.
Features
Time Step
Store
Day Week
Customers
712
Open
Promo
State Holiday
School Holiday
Year
2015
Month
Day
Store Type
Assortment
Competition Distance
570
Competition Open Since Month
11
Competition Open Since Year
2007
Promo2
Promo2 Since Week
13
Promo2 Since Year
2010
Promo2 Jan
Promo2 Feb
0
Promo2 Mar
0
Promo2 Apr
Promo2 May
0
Promo2 Jun
0
Promo2 Jul
Promo2 Aug
0
Promo2 Sept
0
Promo2 Oct
1
Promo2 Nov
0
Promo2 Dec
0
Below two model explanations each from different model where each feature value represents the importance of a feature to the model prediction, positive values indicate that as the value of this feature increases the sales values increase, and vice-versa for the negative values. Use the explanations the questions below.
Figure 11: Data-sample from Rossmann dataset.
25

---

Published in Transactions on Machine Learning Research (05/2023)
Explanation from model
Explanation
(feature
importance
weights)
Features
Time Step 1
Store
-0.01
Day Of Week
-0.02
Customers
1.18
Open
0.00
Promo
0.07
State Holiday
-0.01
School Holiday
-0.01
Year
-0.17
Month
0.02
Day
0.01
Type
Assortment
0.00
Competition Distance
Competition Open Since Month
0.00
Competition Open Since Year
-0.01
Promo2
-0.04
Promo2 Since Week
0.00
Promo2 Since Year
-0.01
Promo2 Jan
0.02
Promo2 Feb
-0.10
Promo2 Mar
0.05
Promo2 Apr
Promo2 May
0.03
Promo2
-0.17
Promo2 Jul
-0.09
Promo2 Aug
-0.01
Promo2 Sept
-0.02
Promo2 Oct
-0.02
Promo2 Nov
-0.08
Promo2 Dec
0.05
Given the above input and explanation A shown above (not your own intuition) if the number of customers changes from 712 to 800 how will the model output change?
Increase
Decrease
No change
Figure 12: Explanation from Model A and sample question. Color code red to green representing positive value to negative value of explanation.
26

---

Published in Transactions on Machine Learning Research (05/2023)
Given the above input and explanation B (shown below) please predict model B's output.
Explanation
(feature
importance
weights)
Features
Time Step
Store
-014
Day Of Week
0.47
Customers
Open
0.13
Promo
0.10
State Holiday
0.02
School Holiday
Year
0.00
Month
0.00
0.03
Store Type
Assortment
-0.10
Competition Distance
-0.10
Competition Open Since Month
0.56
Competition Open Since Year
Promo2
-0.13
Promo2 Since Week
-0.08
Promo2 Since Year
Promo2 Jan
-0.06
Promo2 Feb
0.05
Promo2 Mar
0.09
Promo2 Apr
Promo2 May
0.03
Promo2 Jun
0.00
Promo2
-0.03
Promo2 Aug
Promo2 Sept
0.05
Promo2 Oct
-0.12
Promo2 Nov
0.01
Promo2 Dec
-0.03
Given the above input and explanation B shown above (not your own intuition) if the number of customers changes from 712 to 800 how will the model B output change?
Increase
Decrease
No change
Figure 13: Explanation from Model B and sample question. Color code is red to green representing positive value to negative value explanation.
27

---

Published in Transactions on Machine Learning Research (05/2023)
Which explanation method do trust more?
Explanation method A
Explanation method
Both
Neither
Figure 14: User study model trust question
28


Reviews:
[{"review_id": "5ooYd3jHvn", "reviewer": "Reviewer_mS9v", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper has several strengths, one of which is its ability to preserve accuracy while providing interpretability. The proposed framework, Interpretable Mixture of Experts (IME), achieves this by combining simple and interpretable models, such as linear regression. The paper also presents multiple options for IME to address interpretability needs of particular applications, including single-level and hierarchical assignment options. \n\nHowever, the paper's main weakness is its limited approach to enhancing interpretability. While the authors claim to introduce a new class of interpretable models, the use of linear regression for experts or assignment modules is not entirely novel, as mixture of experts have already been widely used in various fields. Furthermore, while the authors believe that using linear regression is preferable because of its interpretability, I am uncertain about why this would be superior to using deep neural networks (DNNs). The usefulness of knowing the trained parameters of linear regression does not seem very useful, and given the restricted capacity of linear regression, it seems that the proposed method may have limited applicability. Moreover, the paper fails to sufficiently demonstrate the benefits of enhancing interpretability in their experiments. Overall, while the paper presents the interpretable mixture of experts, further exploration and innovation in enhancing interpretability are needed to fully realize its potential.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes Interpretable Mixture of Experts (IME) for modeling tabular and time-seres data that combines multiple interpretable models (e.g., linear regression) in a mixture of experts. IME preserves accuracy, and provides useful interpretability capabilities. Moreover, the authors present multiple options for IME, including single-level assignment and hierarchical assignnment options. Through experiments, the paper shows that IME outperforms other interpretable architectures, and offers easier-to-interpret explanations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "Overall, a major drawback is the limited novelty of the proposed method. The results indicating that experts specialized in different data are not novel, and do not appear to be impactful. I believe that it would be preferable to introduce a methodology that can be interpreted, irrespective of the structure of experts and assignment modules, in order to support the claim of having an interpretable mixture of experts.", "broader_impact_concerns": "The proposed method appears to be applicable only in specific domains where linear regression performs well, such as time series or tabular data. Moreover, rather than introducing a new interpretation methodology, the paper employs a mixture of experts, which may not have a significant impact, since interpretable models are used as experts or assignment modules.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7IYhEKOy2d", "reviewer": "Reviewer_q2hs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The proposed IME framework yields high accuracy, comparable to black-box deep neural networks (DNNs) in many cases. Besides, IME offers useful interpretability capabilities, making it suitable for high-stakes decision making.\n2. IME is demonstrated to be more accurate than single interpretable models and perform comparably with existing state-of-the-art DNNs in accuracy. On most datasets, IME even outperforms black-box DNN baselines while providing faithful explanations.\n3. IME's explanations are found to be more faithful and trustworthy than commonly-used post-hoc explanations methods, according to a user study.\n\nWeaknesses:\n1. The experiments section covers both classification and regression tasks; however, the notion module in Section 2.1 states that this paper focuses on the regression task.\n2. Table 1 shows that MLP performs slightly better than LightGBM and XGBoost. However, in practical applications, LightGBM and XGBoost usually outperform DNN models for tabular datasets.\n3. It is promising that the authors have compared their proposed intrinsically interpretable model with post-hoc explanation methods such as SHAP. The results indicate that the proposed method significantly outperforms SHAP (e.g., IME is selected in 87% of cases compared to only 6.5% for SHAP). It would be helpful if the authors could provide further explanations for the comparison results. Additionally, it would be interesting to see a quantitative comparison of the interpretability performance between the proposed method and SHAP.\n4. It would be beneficial to include more explanation visualizations in the main document rather than in the appendix.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Recently XAI has received increasing attention from the community. Most current research focuses on post-hoc explanations. This article works on designing inherently interpretable models. It introduces a new interpretable modeling framework called Interpretable Mixture of Experts (IME) that is designed for tabular and time-series data and offers high accuracy comparable to DNNs while providing useful interpretability capabilities. IME consists of an assignment module and a mixture of experts, with each sample being assigned to a single expert for prediction. IME is demonstrated to be more accurate than single interpretable models and perform comparably with existing state-of-the-art DNNs in accuracy. Furthermore, IME's explanations are compared to commonly-used post-hoc explanations methods through a user study, and participants find IME's explanations more faithful and trustworthy.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "see the Strengths And Weaknesses section", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6jF244hRfF", "reviewer": "Reviewer_5L87", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n-  The IME framework provides a range of options that offer different predictive performance and interpretability characteristics.  Further, the authors do a good job of clearly and concisely describing these options.  \n\nWeaknesses (see next section for details):\n-   IME’s predictive performance results for time-series datasets may be strongly influenced by having access to information about “past-errors.”\n-  The claims/evidence in Section 4.4 could use some refinement/qualification to be better supported.  \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "\n\nThe authors propose an Interpretable Mixture of Experts (IME) framework that offers a variety of degrees of interpretability by using an assignment model (which could be interpretable or a black-box) to select which expert model (which could be one of a set of interpretable models or a black-box model) to use to make a prediction.  The authors evaluate IME's predictive performance on a variety of tabular and time-series datasets and run an ablation study of IME's design choices.  Further, the authors demonstrate IME's interpretability using a few examples and a user study. Overall, it appears that IME improves the Pareto frontier for interpretability and predictive performance.  \n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "\n\nMajor:\n-  Many of the experiments where IME has higher predictive performance than a black-box model come from time-series datasets, where IME was given information about \"past-errors.\"  Were the black-box models also given this information?  This seems like very useful information and, consequently, all models should have access to it in order to fairly compare their performance.  \n-  The User Study uses SHAP, which is a local feature attribution explanation (ie, it estimates the importance of a feature by imagining what would happen if it was “missing”), for a counterfactual task (ie, we want to know what would happen if a if a feature was “slightly changed”).  A local approximation-based explanation (eg, LIME) would have been a fairer comparison since it offers a similar type of explanation as IME with linear expert models.  \n\nModerate:\n-  When discussing simple/inherently interpretable/white-box models, there is always a question of \"how hard\" it is for a user to interpret the model based on the complexity of the architecture.  This is especially relevant when comparing the predictive performance of different model architectures.  As a result, it seems important to discuss the ways in which interpreting IME may be \"harder\" than some of the simple/inherently interpretable/white-box models that it is compared against.  For example, $\\text{S-IME}_i$ requires the user to interpret an $n$-way classifier and $n$ different expert models.\n-  While the synthetic datasets used for the “Identifying data distribution modes/incorrect model behavior/temporal regime changes” results make for clear exposition of these capabilities, XAI is rife with examples of methods working in very simple settings and then not working in more realistic settings.  As a result, the “real world implications” discussed at the end of these paragraphs seem under supported.\n-  The User Study is presented as evaluating the quality of IME’s explanations vs the quality of post-hoc explanations.  However, the results of this experiment may be biased because I think that (please correct me if I am wrong) IME is explaining itself while SHAP is explaining a standard black-box model.  As a result, if this counterfactual task is harder for that black-box model than it is for IME (as a predictive model), the experiment would be biased towards IME.  If this understanding is correct, then this should be clarified or the claim should be revised.  \n\nMinor:\n-  Please double check the bolding/italicization of the best results in the tables because there may be some errors.  For example, I think TCN has the best result for Climate 168.  \n-  Why isn't there an ablation for removing $L_{div}$?\n", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a framework for interpretable machine learning based on mixtures of experts. This enables them, for settings like time-series and tabular data, to maintain good accuracy---comparable to non-interpretable models---while also providing good explanations. 

All reviewers found a great deal to like, and I felt the same way. The proposed technique is clearly of interest and could be useful in a variety of settings. The authors provide a solid amount of evidence supporting their claims, including a user study. The authors have also improved the writing and positioning as a result of feedback from reviewers, resulting in a clear and well-written work.

The only drawbacks mentioned by the reviewers are limited novelty and potentially limited impact. The first of these is not a focus, so I will not attempt to evaluate it (my sense is that there are multiple innovations so even this bar would be cleared easily). The second concern is not a problem, as the reviewer notes that the proposed approach does impact important areas like time-series and tabular data.

Altogether this is solid work.
Decision: Accept (poster)