Intrinsic User-Centric Interpretability through Global Mixture of Experts
Authors: Vinitra Swamy
Syrielle Montariol
Julian Blackwell
Jibril Frej
Martin Jaggi
Tanja Käser
Venue: ICLR 2025 Poster (2025)
Keywords: interpretability
human-centric computing
mixture-of-experts
Primary area: interpretability and explainable AI
TL;DR: A family of intrinsically interpretable models for sparse and actionable interpretability, inspired by global mixture-of-experts.

Abstract: In human-centric settings like education or healthcare, model accuracy and model explainability are key factors for user adoption. Towards these two goals, intrinsically interpretable deep learning models have gained popularity, focusing on accurate predictions alongside faithful explanations. However, there exists a gap in the human-centeredness of these approaches, which often produce nuanced and complex explanations that are not easily actionable for downstream users. We present InterpretCC (interpretable conditional computation), a family of intrinsically interpretable neural networks at a unique point in the design space that optimizes for ease of human understanding and explanation faithfulness, while maintaining comparable performance to state-of-the-art models. InterpretCC achieves this through adaptive sparse activation of features before prediction, allowing the model to use a different, minimal set of features for each instance. We extend this idea into an interpretable, global mixture-of-experts (MoE) model that allows users to specify topics of interest, discretely separates the feature space for each data point into topical subnetworks, and adaptively and sparsely activates these topical subnetworks for prediction. We apply InterpretCC for text, time series and tabular data across several real-world datasets, demonstrating comparable performance with non-interpretable baselines and outperforming intrinsically interpretable baselines. Through a user study involving 56 teachers, InterpretCC explanations are found to have higher actionability and usefulness over other intrinsically interpretable approaches.

Published as a conference paper at ICLR 2025
INTRINSIC USER-CENTRIC INTERPRETABILITY THROUGH GLOBAL MIXTURE OF EXPERTS
Vinitra Swamy
Syrielle Montariol
Julian Blackwell EPFL
EPFL
EPFL
vinira.swamy@eppal @epfl.c
phelle.montrrio@@epl..h
juhan.blackwell@ppp.cco
Jibril Frej EPFL
Martin Jaggi
Tanja Käser
EPFL
EPFL
jibril.fej@eppl.ch
martin.jaggieepfl.ch
lanja.kaeser@epl..ch
ABSTRACT
In human-centric settings like education or healthcare, model accuracy and model explainability are key factors for user adoption. Towards these two goals, intrin- sically interpretable deep learning models have gained popularity, focusing on accurate predictions alongside faithful explanations. However, there exists gap in the humen-centeredness of these approaches, which often produce nuanced and complex explanations that are not easily actionable for downstream users. We present Interpret (interpretable conditional computation), a family of intrinsically interpretable neural networks unique point in the design space that optimizes for ease of human understanding and explanation faithfulness, while maintaining comparable performance
state-of-the-ar models. InterpretCC achieves this through adaptive sparse activation of features before prediction, al- lowing the model use different, minimal set of features for each instance, We extend this idea into an interpretable, global mixture-of-expers (MoE) model that allows users to specify topics of interest, discretely separates the feature space for each data point into topical subnetworks, and adaptively and sparsely acti- vates these topical subnetworks for prediction. We apply InterpretCC for text, time series and tabular data across several real-world datasets, demonstrat- ing comparable performance with non-interpretable baselines and outperforming intrinsically interpretable baselines. Through user study involving 56 teachers, Interp InterpretCo CC explanations are found to have higher actionability and usefulness over other intrinsically interpretable approaches.
INTRODUCTION
The rise in popularity of neural networks (NNs) over traditionally interpretable models has come with severe weakness: the lack of transparency of their predictions. Neural networks are considered black-box models due their high number of parameters and complex operations (Molnar 2020). Humans cannot understand how neural network decisions are made under the hood; this is a crucial limitation in sensitive applications such as education healthcare, where models' predictions might impact humans (Conati et al. 2018: Vellido 2020).
Towards transparency for neural networks, a range of explainable Al methods have emerged across two main axes: global vs. local explanations, and post-hoc vs. intrinsic explanations (Du et al. 2019). Global interpretability allows users to understand how the entire model operates by examining its structure and parameters, while local interpretability focuses on understanding the rationale behind an individual prediction (Molnar 2020). Most methods are post-hoc, where the explanation is extracted from model that has already been trained. In contrast, intrinsic methods directly incorporate interpretability into the model's structure.
Popular post-hoc methods include attribution methods like LIME Ribeiro et al. 2016) or SHAP (Lundberg and Lee 2017), counterfactual methods like CEM (Dhuranhhar et al.) 2018). pattern-based methods like PREMISE (Hedderich al.) 2022). Post-hoc methods require users to trust the explainer's 's approximation of the ground truth explanation (the underlying model's 's decision process), although they have been shown be systematically biased and inconsistent (Krishna et al.

---

Published as a conference paper at ICLR 2025
InterpretCC
InterpretCC Group Routing
Feature Gating
Goodel
Sab wriwoek
Select
Inpet
Mack Select
Trunccte
Predectons
Oalpol
Network
Hedel
Predectonn
Newwrk
Productioon
Winghted
San
Figure I: InterpretCC Architectures: Feature Gating (left, individual features); (i) All features are input into a discriminator network that outputs sparse feature activation mask; (ii) Only the features selected via the mask are passed to predictive network for the final prediction. Group Routing (right, pre-defined feature feature groups): (i) Features are statically assigned to distinct groups, with each feature routed to only one group; (ii) Features are input to a discriminator network, generating a sparse group activation mask; (iii) Predictions from activated sub-networks (selected via mask) are aggregated by a weighted sum to produce the final output.
2022, Swamy et al., 2022b). In-hoc explanations are subset of post-hoc methods that require access to model weights instead of treating the model as a black box (Molnar (020). For instance, Grad-CAM (Selvaraju et al., 2019) assesses the contribution of a component to the model's output, and TCAVIDTCAV (Kim et al. 2018 Ghorbani et al. 2019a) use user-defined concepts interpret neural network embeddings.
Recent intrinsically explainable model literature has focused example- based approaches, over- whelmingly for the image modality (e.g. B-cos networks (Böhle et al. 2022), PIP-Net (Nauta)
et al. 2023). ProtoPNet (Chen et al. 2019)) and less commonly in time senesseiees tabular, or text
modalities (Sawada and Nakamura 2022). Other approaches like NAM (Agarwal et al. 2021 I and
EBM (Nori al. (019) train a model for each input feature combination of features and output predictions using scores from these models, requiring a lot of subnetworks when the feature space is large. Research towards interpretable
cexture-of-experre models has highlighted hierarchical neural network structure with subnetworks, combining interpretable experts (i.e., decision trees) with NNs for partially interpretable points (Ismail et al., 2023), selectively activating experts (Li al., 2022). or extracting automated concepts over the input space for routing Youu et al. [023] Alvarez Melis and Jaakkola, 2018). All of these approaches, while expressive, are burdened with overly detailed explanations that limit human understandability and actionability.
In this paper, we therefore present user-centric intrinsically interpretable framework that achieves faithful local interpretability and provides sparse and actionable human-centric explanations, while maintaining comparable predictive performance to its black-box counterparts. To achieve these goals, we use conditional computation to craft interpretable neural pathways using two different architectures (see Figure based on routing through individual features or entire feature groups.
Our models enable statements like the following: "The student's regularity and video watching behavior were the only two aspects selected as important for the student's prediction of passing the course, and the model did not any other aspects to make this prediction". We refer to interpretability from the users' perspective, focusing on the model's local reasoning for a decision on specific data point, as opposed to global understanding of the model's 's internals. Our models characterized by sparse explicit routing, truncated feature spaces, and adaptivity per data point. These traits are important for buman-centric trustworthiness as they provide clear and concise instance-level explanations (Miller 2019 Swamy et al. 2023b).
With our family of InterpretCC models, we provide the following contributions:
[1] InterpretCC Feature Gating: A simple, interpretable NN architecture using gating mechanism sparsely activate specific features.
[2] InterpretCc Group Routing: An interpretable mixture-of architecture that uses numan-specinf group routing to separate the feature space and sparsely activate specific experts.
[3] An extension of intrinsic interpretability to multiple human-centrie modalities and domains. focusing on time-series (education), tabular (health, synthetic data), and text (sentiment, news)
'We do not focus on vision datasets since extracting concepts from vision has been well-studied by existing. modality- speeificc interpretability approaches, e.g. BBöhle et al. (2022). You et al. (2023): Donnelly et al. (2022)
Thomas (2023)
2

---

Published as a conference paper at ICLR 2025
[4] A novel XAI user study comparing teachers' preferences of interpretable-by by-blle model explanations towards designing educational interventions for struggling students.
Across experiments cight diverse datasets, show that InterpreICC models perform com- paratively to non-interpretable baselines (matching or exceeding 95% CIs in 15 of 16 comparisons) and outperform intrinsically interpretable baselines (ICC Feature Gating is 9.05% better on average than SENN Features and 3.27% better on average than NAM, while ICC Group Routing is on average 5.63% better than SENN Concepts). Moreover, participants of the user study preferred ICC explanations over baselines in terms of actionability, usefulness, conciseness and trustworthiness. We provide our code open source: https:/ com/ conn/epfl-nng /tee/eddiiiteeeteteeee
BACKGROUND
Architecture Foundations. Conditional Com- putation (CC) has become widely used to im-
Explasation
Meshell
Grusdlely
Reds
Spouls
Stage
prove the computationally expensive training and inference of large neural networks by acti- vating only parts of the network (Bengio al., 2013, 2016 Davis and Arel 2013). Inspired by the foundations laid out by CC, mixture-of- expert models have rapidly gained popularity for improving the efficiency of neural networks through activating different expert subnetworks at different layers. BASE layers (Lewis al.) 2021) direct each token to designated expert and Switch Transformers (Fedus et al. 2022b) use CC to select one out of 4 feedforwwrr net- works across each transformer layer, optimizing computational resources. Mixtral (Jiang al. 2024) a recent LLM using a mixture of experts to select 2 out of 8 expert networks at each layer, reducing the numbers of active parameters by factor of 4 compared to training, while allowing each token to have access to all the parameters.
LIME; SHAP
(Pai--Baa)
Roaas
Use al lapai latava
Approcazakas
Spaci
Explaniien
TCAY (uelsc
Checqee
Uter dell as orreps threagh cuaupls
Alggnt
cheoupt
Spare
medel inmmab
Toplls five
Use all iapai Iazace
Guaazned
Det prodie
Jaykan
SENN
Oxauge
Apveraa and anoey adactinn
Aligeed
cheoupts
Nelapiod
Paykis
prodec
NAM
Fosuue
Use all lapar faikees
Geaerked
Not Sttipeor
Tapbie
prodec
Interpeen
Roaas
Use all iapai Iasaca
Guanzked
Sparc
dea profec
Eaylis
(Featore Gaallg, Group Resting!
Ooogge
Use (LLL dolso
goops
Guansaed
Spanc
Eapkis
powtuc
Table I: Design Comparison: InterpretCC models are at the unique intersection of flexible ex- planation granularity (either features or concepts), guaranteed explanation faithfuleess to the model's decision process, optimal sparsity in the expla- nation, and explanations used in the prediction (stage). Basis describes the foundation of the ex- planation (e.g., user-defined concepts or raw fea- tures). A taxonomy can be found in Appendix A
With Interpret CC, we extend a similar routing idea with instance- -eetuneeedeeeeeent gating decisions towards an interpretability objective as opposed to only an efficiency or performance objective.
Interpretability Foundations. Explainability can be integrated into different stages of the modeling pipeline: post-hoc (after model training), in-hoc (requiring model weights), and intrinsic (interpretable by design) (Swamy et al., 2023b] MMonar 2020). Our positioning of the design of Interpret in comparison to popular approaches is described in Table [ Specifically, we categorize the approaches using their basis and the stage they are applied to in the pipeline as well as four key aspects for human- centric explanations: faithfulneess. the explanation reflects the model behavior with certainty {Lyu cct al. 2024 Dasgupta et al. 2022); ; sparsity, the model uses a minimal amount of features, optimizing for user understandability and actionability (Sun et al., 2024 Ayoobi al., 2023): y predictive stage, the explanation covers the entirety of what the model uses for prediction (Speith) [0022] (Schwalbe and
Finzel 2024); and granularity, the explanation is conveyed in terms of features or concepts (Miller
2019
Jain et al. 2020).
Post-hoc methods such LIME (Ribeiro al. 2016) or SHAP (Lundberg and Lee. 2017 approx- imate what the model finds important, and therefore cannot be considered faithful (cross-featue actions are often not described in the explanation). LIME and SHAP use the full input feature space and can be configured for sparseness, although they have been shown to choose a broad amount of features (Swamy et al. 2022b) using default settings. In-hoe interpretability approaches often require users to specify examples to define human- undrsttaaddt concepts (e.g. TCAV (Kim et al. 2018). DTCAV (Ghorbani et al., 2019a)), or use hybrid methods with both human-defined and automated concepts (Sawada and Nakamura 2022). In-hoc approaches have limited faithfulness due to the (lack of) completeness of the concepts and do not use explanations directly in the prediction. Inspired by these approaches, Interpret CC allows users to specify interpretable concepts that directly useful to them. However, we do not use examples, but instead allow users to specify a grouping over the feature space, achieving both sparsity and explanations used for prediction.
3

---

Published as a conference paper at ICLR 2025
Initial approaches have explored expert models for intrinsic interpretability. The Interpretable Mixture of Experts (IME) framework (Ismail et al. (2023) uses linear models alongside deep models to provide partially faithful explanations. LIMoE (Mustafa et al. (022)) focuses visual experts to identify concepts like textures and faces, enhancing interpretability in vision tasks. Similarly, the Sum-of- (SOP) model (You et al.] 2023) uses sparse feature groups emphasize the model's reliance on sub- sets of features for predictions. Approaches in extractive rationale methods and explain- seenntthen-pue -peedict methods (Jain al. 2020 Bastings et al., 2019 Yu al. 2019) produce intuitive text explanation guarantees with explanation selection before prediction, but are often not generalizable beyond that modality (see Appendix I.4). Few intrinsic approaches use expert knowledge define concepts directly, instead using prototype examples (Koh et al. 2020) or rules (Konstantinov and Utkin| 2024). The most relevant models to our work are Self-Explaining Neural Networks (SENN) (Alvarez Melis and Jaakkola 2018) and Neural Additive Models (NAM) (Agarwal et al. 202T). both neural models similar to ICC as opposed to Explainable Boosted Machines (EBM) which uses trees (Nori et al. 2019). SENN extracts concepts with prototypical examples and their relevances, but it lacks faithful- ness (it cannot explain what is not in a concept), sparsity (it explains all concepts), and concepts do not cover the entire feature space. NAM assigns model each feature and combines the outputs linearly, achieving faithfulness but not sparsity, as all features contribute. ICC differs by filtering the feature space instead of using all features (sparsity), using user-defined concepts instead of automated concepts (basis), and assigning each feature to a single group, making feature use explicit (predictive stage).
METHODOLOGY
Given an input x, the objective of our approach is to select a sparse subset of x that will be used to predict the output to solve the classification task. We propose two architectures:
Feature Gating: The approach only processes a subset of the features by applying a sparse mask M on the input before processing it by model f. The output is given by: f(M(x)).
Group Routing: A sparse mixture of models (Fedus et al.) 2022a) applied on huternlnteeppettable
groups of features where each expert is assigned to a group of features: Σi=1 G(z), (Mg(x)) where M.).); ) is sparse mask selecting only the features of group i, fi is the expert model associated with the i-th group, and G(z); is the output of the gating network for group i. If G(z); = 0, the entire group of features ignored.
3.1 FEATURE GATING
InterpretCC Feature Gating, shown in Fig. is the first step towards using CC paths for interpretability. The features are first passed through a discriminator network D to select which ones to use for computing the output. The Gumbel Softmax trick (Jang al., (017)) is applied on each dimension of D(x) to select features in a differentiable way (see Appendix G for more details). A feature j is activated (the associated value in the mask non-zero) if the Gumbel Softmax output exceeds a threshold т, a hyperparameter. This allows the model to adaptively select the number of features based on each instance, using fewer features for simpler cases and more for complex ones.
The output is computed using a model f on the masked input M(z). Since the explainability is at the feature level, using a black box model for f does not detract from the interpretability. Notably, ICC FG does not require human specification.
3.2 GROUP ROUTING
We build upon the instance-dependen gating architecture with feature groups. As displayed in Figure instead of selecting features individually, the mask is applied human interpretable groups of features. Doing so encourages cross-feature interactions while maintaining meaningful grouping for human users. To select the features belonging to group i, we use a binary mask MG(r)); that is computed using human-specifte rules. In section we detail our approach to compute Mg(r), for each dataset used in our experiments.
InterpretCC Group Routing is sparse mixture of experts utilizing gating network to assign weight G(z); to each group This process mirrors that of Feature Gating, starting with a discriminator network Dg with an input of all features and output of K dimensions (K is the number of groups). It then applies the Gumbel Softmax and a threshold T3 to each group. The model output a weighted sum of the output of each expert f; that only uses the features from the i-th group input. Using
4

---

Published as a conference paper at ICLR 2025
Feature-Rased Interpretability
Concept-Basod Interpeetability
Dataset
Non-interpettbbee Base Modale
NAM
Peatures
SENN
FRESH
InterpretCo
Peature Guting
Concepes
SENN
Interpret
Top Routing
Interpret CC Group Routing
DSP
82.81 =: 2.61
85. 20 0.64
71,70 4: 0.55
90.75 22 0.01
$1.50 2.26
83.08 1.10
84.90 2: 7.59
Education
Gen
HWTS
72:96- + 1.98
73.99 a 3.76
65
73.01
=: 2:13
4:00
68.63
57.00 2.09
3.75
Not Supponted
$2.89 =: 0.04
71.92 e 0.01
73.10 11.07
70.90 1.45
72.59
80-44
2
5.19
81.58 0.57
78.34 +: 0.55
VA
74.90 a 5.28
T1.39 8 3.38
74.37 1.11
77,80 :: 0.01
09,99
5.83
71.43 2: 1:11
72,08 à: 3.71
Health
Cancer
89.70
B8.77 7.31
80.52
621
Not Supp.
18.19
85.26 1,05
84,66 3.02
94.85 1.25
Text
SST
AG News
89.93 4:
Suppocted
Not Suppocted
82,05
38.73 à
85.72 5:31
Not
Suppored
87.25
92.58
91.75 1.56
90.35 Lo7
Synthetic
OpenXAI
67.85
1.31
83.67
1355
Not Supp.
90.83 1.93
89.47
Table 2: Interpretcc Performance (avg = std) on EDU (balanced accuracy), Text, Health, and Synthetic (accuracy) datasets compared to a non-interpretable baseline and four intrinsically interpretable baselines. All 95% CI overlap with the non-interpretable base module for values in black. Colored values indicate significantly higher (green) or lower (red) performance than the base module. The reported Group Routing results are the best performing ICC variations from Table 3
our sparsity criteria, we ensure that few groups are used to compute the output, making the Group Routing intrinsically interpretable at the group level, regardless of the types of models used as experts. Group Routing enables efficient inference without reducing the number of parameters available during training. During the training phase, employ soft masking, allowing all weights G(z);; to remain non-zero, thus granting the model access to every expert. This approach allows the model to leverage the full set of parameters during training, enhancing the training efficiency. At inference time we switch to using a hard mask, making the weights sparse and allowing for interpretability.
4 EXPERIMENTAL SETTINGS
We apply InterpretCC to five domains: education, news classification, sentiment classification, healthcare, and synthetic data covering Time Series, Text, and Tabular inputs; all for classification tasks. For Tabular features, the input a vector x E R". The mask in the Feature Gating is a sparse vector indicating which tabular feature to use and how important they are (if the weight is non-O) and the groups form partition over the features. For Text features, the input a sequence of N tokens: [t1, 12, f. The mask is sparse vector that indicates which token to use and each group consists of subset of the tokens. Finally, we consider Time Series of n features across T timesteps: We apply the same mask across all time steps for Interpret FG and GR.
E RaxT
EDU (time series, education domain). We predict student success in the early weeks of four massive open online courses (MOOCs), using students' clickstream data (see Table 5 in Appendix B for details about the courses). The raw elickstream input is transformed into weekly time-series features that have proven useful for student success prediction in previous literature (e.g. total video clicks, forum interactions). We select 45 input features used in multiple studies (Lallé and Conati 2020 Boroujeni al., 2016 Chen and Cui 2020 Marras al. 2021). For early prediction, we only use the first 40% of time steps as input.
Grouping: To derive human- interpretable concepts from these features, turn learning science literature. In routing by paper, we create 10 distinct feature subsets based on handcrafted initial input features from 10 papers, directing each specific expert subnetwork. For routing by pattern, we organize features according to five learning dimensions identified by (Asadi et al. 2023 Mcjia ct al. 2022):: effort, consistency, regularity, proactivity, control, and assessment-based (see Table for a detailed feature classification). Thirdly, routing by Large Language Model (LLM), uses GPT-4's capabilities to aid humans feature grouping (Achiam et al. 2023). GPT-4 is prompted as an 'expert learning scientist' to group the features into self-regulated behavior categories that are easy to understand, which are then used to separate the features for InterpretCC. More details are included in Appendix C
AG News and SST (text, news and review domains). For news categorization (AG News), we classify news into four categories ('World', 'Sports', 'Business', 'Sci/Tech') given a title and description of real-world article (Zhang et al. 2015). We use 36,000 training samples and 3,000 test samples evenly distributed across categories. For sentiment prediction (Stanford Sentiment Treebank, SST), use 11 11,855 sentences from movie reviews labeled by three annotators (Socher et al. 2013) and predict a binary sentiment from sentence fragment.
Grouping: The ICC routing model assigns words to subnetworks using the Dewey Decimal Code (DDC) hierarchy of topies for book classification create 10 subnetworks (see Table 7 Appendix
5

---

Published as a conference paper at ICLR 2025
C.2 for more details) (Satija 2013). Each word is encoded using SentenceBERT (Reimers and Gurevych 2019) and assigned to a subcategory (i.e. the word 'school' is assigned to the subcategory edducation under category 300 for 'social sciences') and routed to the appropriate parent network.
Breast Cancer (tabular, healthcare domain). The Wisconsin Breast Cancer dataset identifies cancerous tissue from fine needle aspirate (FNA) images, with 30 features (10 per cell nucleus) and diagnoses (Malignant: 1, Benign: 0) for 569 patients (Wolberg et al. 1995).
Grouping: For the grouping logic, we group each cell nucleus in a separate subnetwork, enabling features representing the same part of the tissue sample to be considered together.
Synthetic Dataset (tabular) We OpenXAI's synthetic dataset (Agarwal al. 2022), which includes ground truth labels and explanations, indicating the subset of features influencing each label. This dataset comprises of 5000 samples, 20 continuous features, and two classes. was created using the SynthGauss mechanism from five cluster neighborhoods (1000 points for each cluster), ensuring (1) feature independence, (2) unambiguous, well-separated local neighborhoods, and (3) an explanation for each instance.
three desirable properties for assessing explanations:
Grouping: We group the feature space by assigning each feature a cluster neighborhood based solely on the distribution of the training data. The average absolute value of each feature for each cluster is calculated, and the highest feature-cluster value determines the assignment.
EXPERIMENTAL RESULTS
Through the following three experiments, we demonstrate that our InterpretCC models do not compromise performance compared to black-bo. bbx models and provide explanations that are faithful as well as human-centered. InterpretCC is designed for data that has meaning for humans (i.e., interpretable features or meaningful raw data like text lab measurements); however, demonstrate it is also performant on a synthetic dataset with no interpretable features.
Experimental Setup. We run hyperparameter tuning and three different random seeds for each reported model (reproducibility details in Appendix F Since EDU MOOC courses have a low passing rate (below 30%), and thus the dataset has a heavy class imbalance, we use balanced accuracy for evaluation. The other datasets are more balanced (AG News, SST. Breast Cancer, Synthetic), hence we use accuracy as our evaluation metric. We perform an 80-10-10 vrain-vliddtiion-ess data split stratified on the output label, to conserve the class imbalance in each subset. In addition to InterpretCc Feature Gating and Group Routing, we also employ InterpretCo Top-K expert network solution with k=2 for group routing. This approach is similar to existing mixture-of-exppe approaches (Jiang et al. 2024 Li et al. 2022). except that their models make a layer-wise expert choice, which significantly reduces interpretability, while we make a global expert choice.
Base Prediction Module. We choose simple yet performant predictive modules reported in previous literature to isolate the difference in performance due to the interpretable architectures. For the EDU data, previous works use BiL.SMM on student behavioral data for best predictive performance (S (Swamy al. 2022b Marras et al. 2021). Thus, for comparative benchmarking, the most performant BiL.STM setting is used as a baseline model (Swamy et al.) 2022a). For the AG News and SST datasets, we use fine-tuned Dist i LBBER variations as baselines, also used in related works (Yang et al. 2019 HF Canonical Model Maintainers 2022). For the Breast Cancer dataset, we use fully connected
network as reported by
Interpretable Baselines. In addition to non- non-interpretable baselines, we compare ICC to three intrinsically interpretable methods: SENN (Alvarez Melis and Jaakkola [018)) NAM (Agarwal et al. 2021], and FRESH (Jain et al.) 2020). SENN generates concept-based explanations and is designed to learn "interpretable basis concepts" " parallel with the model optimization. Explanations consist of the concepts most similar to the input. NAM, General Additive Model (Hastic) (017). uses individual neural networks for each input feature to calculate feature weights. The model's output is the sum of these weights, and explanations Features architecture has three models working together, NAM has one subnetwork for each feature, and ICC FG has one model with two parts trained end-to-end. At inference, SENN and NAM also assign a score to each feature, which are then aggregated; ICC leverages cross-feature interactions with score per model. FRESH is extractive rationale architecture with three models: supp
given by displaying the feature weights. The SENN
https:://huggingfaccccco
sti http:////iiiitteeeeaaaaaaaaaaa htta

---

Published as a conference paper at ICLR 2025
Dataset
Baselime
InterpretCC Group Routing
100
Paper
Pottern
GPT-4
of of
foatures
DSP
82,81 + 2.61
82.37 + 6.27
82.29
+ 3.72
84,90 + 7.89
Geo
72.96 1 1.59
69.64 1 1.23
81.58
1 0.57
81.19 1: 1.53
HWTS
73.93 + 3.76
78.34
+ 0.98
72.34
+ 2.77
75.12
4.17
VA
74.90
1: 5.28
69.88
a 2.93
72.08
a 371
70.98
a: 2.77
AG News
SST
Heslth
EDU (VA)
EDU
Average
76.65 #3.31
75,56
#2.85
77,57
#2.69
78.05 44.01
(Goo)
Table 3: InterpretCC Group Routing Perfor- mance: balanced accuracy (average std) on routing strategies (paper, pattern, GPT-4) for the EDU datasets in comparison to the non-interpretable baseline.
Figure 2: InterpretCC Feature Gat- ing Sparsity: of features activated per data point across five representative datasets.
generates importance scores, exi selects a contiguous text mask, and pred applies this mask on the inputs to make predictions. Designed for text data, FRESH's ' contiguous explanations are unsuitable for modalities like tabular or time-series data, where feature proximity lacks relevance.
5.1 EXP. 1: INTERPRETCO DOES NOT COMPROMISE ON PERFORMANCE
Table shows the models' metries (average accuracy, standard deviations, and 95% confidence intervals) across three iterations of model training for all eight datasets for InterpretCo Feature Gating, Group Routing, and Top-K Routing. as well as baselines of NAM, SENN Features, SENN Concepts, and a non-interpretable model (reflecting post-hoc explainer performance). We do not report results for text features with NAM and SENN, since the architecture change required to do so would no longer respect the original model desigr instead we report text results for FRESH. Additional sensitivity and architecture experiments can be found in Appendix D These experiments show that while the performance of InterpretCC has overlapping 95% CIs while changing parameters, certain parameter settings have higher variability than others. For both education and health tasks, a of 10 and Guubell-oftman threshold of around 0.7 to 0.8 performant, sparse in activated features, and relatively stable
InterpretCC Feature Gating statistically significantly improves performance with respect the non-interpretable base prediction module for two EDÚ courses (DSP, HWTS) and the synthetic dataset. shows comparable performance on all other datasets (indicated by the overlapping CIs) except the Breast Cancer dataset, suggesting that a higher number of the available features is necessary for performant prediction. The intrinsically interpretable baselines SENN, NAM, and FRESH never outperform the base model, and always under under-perform or perform comparably to InterpretCC.
InterpretCc Group Routing outperforms (for the Geo course and Breast Cancer dataset) or performs comparably (95% CI overlap) to the fine-tuned, non-interpretable base module. It also consistently performs at least comparatively to SENN which, besides never outperforming the base model, has a relatively high variance. We further observe that the selected grouping method impacts performance (Table 33. We achieve 10% increase in performance compared to the base model when grouping using patterns GPT-4 for the Geo course. On average, over the four EDU courses, the automated LLM grouping and the pattern-basee buman-define grouping perform comparably, showing that using automated grouping methods does not mean compromising performance.
InterpretCC performs comparably to black-box models and outperforms intrinsically inter- pretable baselines across diverse benchmarks.
5.2 ExP. 2: INTERPRETCE PROVIDES FAITHFUL AND ESEENFRIENDLY EXPLANATIONS
Table 4 showcases the faithfulness of InterpretCC models in comparison to intrinsically inter- pretable models SENN and NAM, as well as three post-hoc explainers on top of a non-interpretable model (Integrated Gradients (IG) (Sundararajan et al.) 2017). LIME (Ribeiro
al.) 2016), and SHAP (Lundberg and Lee 2017)). We examine the relationship of the explanations to underlying data patterns from the synthetic dataset well to the ground truth of the underlying model (see Appendix [12 for detailed descriptions of the metrics). All 95% CI overlap in Ground Truth Alignment
'For NAM, one network (LLM) would be required word, as the words are distinct for each instance: it does not support text grouping. For SENN, LMs would need to embed each word, then simultaneously be trained represent concepts and pick representative words, with new metries for choosing prototypical examples.
7

---

Published as a conference paper at ICLR 2025
Model
Rehationship to Underlying Data Patierns
Relationshpp to Model
Ground Trun Aligamow (GTA)
Runk
Feature
Ground Trunh
Agreswoet (RA)
Agreement (FA)
Fuiiljuinezs (GTF)
4.11
60.00
87.99
always
TCC GR
89.51 0.56
44 (0 14,97
76
always 100
SENN Festures
85.83 2.56
27.99 9.80
88.01 9.79
always 100
SENN Concepts
65.
9:23
52,00 20.39
80:00 = 90
always 100
NAM
87.39 2.45
40,00 17.89
76.00 14.97
100
87.28 1.72
56.
14.97
91.99 16.00
LIME
84.75 1.83
14.97
SST
SHAP
83:47 +: 1.03
36,00 +: 8.00
52.00 20.39
100
Table 4: Synthetic faithfulness evaluation across eight models on explanations using metrics presented in OpenXAl (Agarwal et al. 2022). GTA, RA, and FA (K=5) refer the explanations' re relationship to underlying patterns in the synthetic data. GTF refers to bounds of the explanations' relationship to the model's s decision process. All 95% CIs overlap between the best per- forming intrinsically interpretable and non-interpretable model.
Figure 3: AG News and SST: # of ICC subnetwork activations (left) vs. avg. activation weights (right), grouped by subnetworks based on the Dewey Decimal Code.
(how close the explanation is the underlying synthetic data feature patterns, similarly (Liu et al. 202T), Rank Agreement (RA) and Feature Agreement (FA) from Agarwal et al. (2022), except for SENN Concepts performing statistically significantly worse than Interpret.co
Feature Gating in GTA. This indicates that ICC's identification of important features, and of their order of importance. close to the underlying data patterns and is either on par or better than both interpretable and non-interpretable models. In terms of predictive performance on the synthetic dataset (last row of Table 2 I CC holds the top three best performing models (89.47, 89.51, 90.83) and the ICC FG variation is significantly more performant than the non-interpretable base model, demonstrating the models' ability to capture signal on this dataset.
Our models provide concise and hence user-friendly explanations through sparse feature (group) activations (Miller 2019). Figure 2 shows the percentage of activated features for ICC Feature Gating. For EDU, only about 10% of the 45 features are activated with low variance. For Breast Cancer, 39.7% of the features are activated per data point. Unlike other datasets, text-based datasets have variable number of features (words). In AG News, with an average of 35 words per article, only a small percentage is activated, while in SST, which has shorter sentences (7 words on average), 59.8% of features are selected with high variance. This achieved sparsity, especially in EDU and AG News, highlights the most important features. This contrasts with post-hoc explainers, which tend to select broader range of features (e.g., LIME and SHAP for EDU (Swamy et al.) (0226)).
ICC Group Routing activates different subnetworks with different weights for each data point. Figure illustrates the number of activations and the average weight for each subnetwork for the text data sets (see Appendix H for detailed analysis on the Breast Cancer and EDU datasets). For AG News too). the average activation weight is similar across all subnetworks (min 0.10, max 0.21). However, some subnetworks are activated much more frequently (400 - Language: 18, 335 times). This indicates that most data points will be routed through the same subset of subnetworks, while the remaining subnetworks are important for specific data points only. SST (Figure 3 bottom) shows similar subnetwork activation patterns. However, in contrast to AG News, the distribution of average weights is not uniform: only three networks are activated with weights larger than 0.15. We suspect the high weights for subnetwork 400 in Figure reflect words that DDC has little relation to in the SentenceBERT embedding space.
Examples of EDU domain explanations are in Appendix E Figures 9 and 10 with additional AG News examples in Figure 15 We also compare ICC behavior across three grouping strategies for the DSP course (Appendix H.3.2) and analyze network sparsity at different prediction horizons H.3.II. Lastly, we show the variation in feature group selection across MOOCs, highlighting ICC's adaptability regardless of grouping method (Appendix H.3.3).
InterpretCC provides sparse and hence user-friendly explanations, while not compromising on explanation faithfulness.
5.3 EXP. 3: INTERPRETCC EXPLANATIONS ARE PREFERRED BY HUMANS
Setting, To validate the cerereeeteedssss of our approach, we conducted a user study comparing InterpretCC's explanations with the ones from other intrinsically interpretable methods, SENN
8

---

Published as a conference paper at ICLR 2025
and NAM. We focused on the education domain and time series input type, using the DSP course in the EDU dataset. We trained Intezpret Feature Gating and Group Routing (with nattern-aases feature grouping strategy, since it is heavily grounded on expert knowledge) as well as SENN and NAM and randomly selected four test samples (i.e., four students) for prediction. In designing the study, conducted four sequential pilots with eight learning scientists who were unaware of which method we presented in this paper. All explanations were simplified for non-technical audience and followed consistent templates.
We recruited 56 teachers using Prolific, (see Appendix E for detailed information about the participants' demographics and backgrounds (Figure 8 and the content of the study). We showed them each model's prediction of the student's 's success or failure along with its explanation. The explanations were given as short text and graph showing the features and concepts used by the model. Note that the choices we made for the presentation of the explanations might have an influence on the participants' perception of the explanations. Examples of how the explanations were presented and a discussion on this limitation can be found in Appendix B. Figures 9 10 and II With our study design, we aimed to highlight each framework's 's strengths without excessive post- processing. For instance, keeping only the top five features of NAM would be unfair, as other features also contribute to its predictions. Instead, we emphasized the top five features that each contribute positively, negatively, or not all to the prediction, highlighting its advantage (distinct insights into feature impact) over Interprett erreetCC while ensuring sparsity. We significantly post-processed the explanations of SENN and NAM provide them format understandable for a non-technical audience and iterated on the visualizations using a
design process (Cooley) (200)). We asked participants to compare these explanations according to five criteria (aligned with (Frej et al. 2024)), ), and to rank the criteria in terms of importance: Usefulness:
This explanation is useful to understand the prediction. Trustworthiness: This explanation lets me judge if should trust the model. Actionability: This explanation helps me know how to give feedback to the student. Completeness: This explanation has sufficient detail to understand why the prediction was made. Conciseness:
Every detail of this explanation is necessary.
NAM
SENN
ICC
QR
ICC
PG
Weight
Usefulness
3.25 20.
3,3
11:11
3.53
11:31
3,88 -0.4
0.28
Trutworthiness
3.28
3,64
10,92
3.36
0.23
Actiomisiitt
3.88 =0.6
3.37
Compleseness
3.18 =100
11.09
=1.07
Conciseness
3.13
613
3772
11.86
05
0.12
Global
3.2 2081
3,38
2085
3.41
10,58
3,78
-0.77
Figure 4: Model score for each user study criterion (average + std) and criteria weight according users' ranking. All scores range from (lowest) to 5 (highest).
Figure 5: Tukey's Honest Significant Difference (HSD) Test for each user study criterion and overall global satisfaction. ICC FG (with highest overall satisfaction in Table 4) is in blue.
Results. Using Friedman's Chi-Square test, we verify that the ranking of the criteria is consistent among the participants (p < 0.05). We report normalized criteria importance in Table 4 (last column). We observe that Usefulness, Trustworthiness and Actionability are consistently ranked the highest by the participants. We compute the Global satisfaction score as a weighted average of the criteria as a global satisfaction measure for each model (Table 4 last row). We additionally conduct pairwise Tukey's 's Honest Significant Difference (HSD) tests (Fig. 5 to determine the statistical significance of the differences between the means of user preferences per criteria.
We observe that TCC FG ranks the highest in the top 3 most important criteria as well as in global satisfaction. Furthermore, ICC GR ranks second in Usefulness and Actionability and first in Con- ciseness. Overall, InterpretCC models are favored over interpretable baselines in 4 out of criteria and in terms of global satisfaction. An ANOVA performed separately for each criterion as well as the global satisfaction measure (Table 9 Appendix E.3 indicates that there indeed a significant difference between the models for each criterion. Tukey's 's HSD tests confirm that ICC FG significantly outperforms NAM and SENN on all criteria but completeness (Fig.
Participants prefer Interpret explanations in terms of usefulness, trustworthiness, action- ability, and conciseness over other intrinsically interpretable models.
9

---

Published as a conference paper at ICLR 2025
6 DISCUSSION AND CONCLUSION
We proposed InterpretCC, family of intrinsically interpretable models that puts human under- standing at the forefront of the design. Through our experiments on feature gating and group routing ) models, we demonstrated that our modular architectures optimize for interpretabil- ity but do not compromise on performance. In a real-world setting, we showed that T rpret models are preferred over other intrinsically interpretable models in 4 of 5 explainability criteria.
InterpretCC is a locally intrinsic explanation framework that creates explanations that are specific to the input point and guaranteed to be faithful the model. In our architecture, the discriminator network is a black-box model. This is by design, to minimize explanation complexity. We believe the hierarchical prediction logic enabled by InterpretCC is the type of explanation that user wants: "Which concepts/features were used to make the prediction, and how important are they to the prediction (weighted sum)?" If the discriminator network was a glass-box model, would answer the question: "why were these concepts/features selected for the prediction?". The first explanation is directly actionable, while the second type of explanation is not. We therefore do not seek to answer this second question with our architecture. However, it would be possible for InterpretCC to have an interpretable discriminator network, as discussed further in Appendix I.3
ICC FG is best when individual features are important for actionable decisions based on the expla- nation. However, it can lose sparseness if all features are equally important: consider the extreme case where the prediction is sum over the full feature space and all features are independent. ICC GR requires more human effort than ICC FG and is, therefore, more suitable for scenarios with cross-feature dependencies and where broader concepts are more actionable than individual features.
ICC CR model's global mixture-of-expere design specializes subnetworks on subsets of features allows them to learn granular patterns (Appendix [.3]. Combining these specialists enhances prediction compared to a monolithic network that might miss underlying patterns (Table Expert- informed feature groupings help the model avoid overfitting correlations that do not generalize at inference time. For ICC FG, adaptive sparsity (a few features per instance) improves prediction quality by reducing noise and optimizing the praprettablityy-cccree tradeoff.
User-defined feature groups aim at deriving explanations useful to the user, but might compromise performance the user specifies grouping that carries minimal signal. Regardless of the grouping. InterpretCC optimizes for explanation actionability and understandability over performance. We believe that an accurate prediction at the cost of explanation usefulness is not worthwhile in an applied setting. It is possible that ICC explanations could be misleading, as concepts used in the explanation could lead to correlation that was not intended (Zheng et al., 202T Dacovi and Goldberg (02T). In these cases, we view ICC explanations through the lens of auditing model behavior ( (adaav et al., 2022), and encourage human intervention. Interpret user-centric advantages are highlighted when the input space is human-interpretabbe However, for domains that are hard to obtain expert knowledge, we envision ICC increasingly leveraging LLM-extracted features, reducing the necessity of human effort for human-centric explanations (Malberg et al. 2024 Baddour et al. 2024).
We acknowledge that the presentation of explanations in our user study (Section 5.3 has influence over our preliminary results on the user perception of Interpret.C. We conducted extensive iterations with cight pilot participants to mitigate study design bias. We note that any imbalance in wording is not necessarily in favor of our method; for instance, users found SENN's explanations more complete than Interpret (Fig. 5). The user study prioritizes the diversity of study participants and quality of responses over the number of samples evaluated; the task is mentally intensive and we found longer study can cause a drop in participant attentiveness. An extensive study over many different tasks and domains of expertise is necessary for generalizable conclusions.
For our text experiments, we fine-tune twenty Disti i BERT models as experts (ten for each task). For more complex tasks, for example requiring long context size, multi-step reasoning ability, or strong prior domain-specific knowledge, DistilBERT can be swapped with larger decoder models, either through fine-tuning or in in-context learning. Parameter-efficieen fine-tuning such as LoRA (Hu al. 2021 would allow fine-tuning even large LMs with limited computational cost, while in-context learning would use the same model instance for each feature and feature group.
Overall, encourage the machine learning community to design models for interpretability many different granularities and user-specified requirements. Interpret provides one such family of models as tradeoff' between human specification, explanation certainty, and performance.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGEMENTS
We kindly thank the Swiss State Secretariat for Education, Research and Innovation (SERT) for supporting this project, and Dr. Tanya Nazaretsky, Bahar Radhmehr, and Paola Mejia Domenzain for helpful discussions. This project has received funding from the European Union's 's Horizon 2020 Research and Innovation programme under grant agreement No. 101017915 (DIGIPREDICT).
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.0877 2023.
Abien Fred M Agarap. On breast cancer detection: application of machine learning algorithms on the wisconsin diagnostic dataset. Proceedings of the 2nd international conference on machine learning and soft computing, 2018.
Chirag Agarwal, Satyapriya Krishna, Eshika Saxena, Martin Pawelczyk, Nari Johnson, Isha Puri, Marinka Zitnik, and Himabindu Lakkaraju. Openxai: Towards a transparent evaluation of model explanations. NeurlPS, 2022.
Rishabh Agarwal, Levi Melnick, Nicholas Frosst, Xuezhou Zhang, Ben Lengerich, Rich Caruana, and Geoffrey E Hinton. Neural additive models: Interpretable machine learning with neural nets. NeurlPS, 2021.
Sajid Ali, Tamer Abuhmed, Shaker El-Sappagh, Khan Muhammad, Jose M Alonso-Moral, Roberto Confalonieri, Riccardo Guidotti, Javier Del Ser, Natalia Miaz-Rodríguez and Francisco Herrera. Explainable artificial intelligence (xai): What we know and what is left to attain trustworthy artificial intelligence. Information fusion, 99:101805, 2023.
David Alvarez Melis and Tommi Jaakkola. Towards robust interpretability with self-explainnng neural networks. NeurlPS, 2018
Mohammad Asadi, Vinitra Swamy, Jibril Frej. Julien Vignoud, Mirko Marras, and Tanja Käser. Ripple: Concept-based interpretation for time series models in education. In Proceedings of the AAAI Conference on Artificial Intelligence, 2023.
Hamed Ayoobi, Nico Potyka, and Francesca Toni. Sparx: Sparse argumentative explanations for neural networks. ECAI 2023, pages 149-156. IOS Press, 2023.
Moussa Baddour, Stéphane Paquelet, Paul Rollier, Marie De Tayrac, Olivier Dameron, and Thomas Labbé. Phenotypes extraction from text: Analysis and perspective in the Ilm era. In 2024 IEEE 12th International Conference on Intelligent Systems (IS), pages 1-8. IEEE, 2024.
Jasmijn Bastings, Wilker Aziz, and Ivan Titov. Interpretable neural predictions with differentiable binary variables. arXiv preprint 1905v: 2019.
Emmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup. Conditional computation in neural networks for faster models. In ICLR Workshop Track, 2016.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.344. 2013.
Moritz Böhle, Mario Fritz, and Bernt Schiele. B-cos networks: Alignment all we need for interpretabilitt. In CVPR, 2022.
Mina Shirvani Boroujeni, Kshitij Sharma, Lukasz Kidziński, Lorenzo Lucignano, and Pierre Dil- lenbourg. How to quantify student's regularity? In Katrien Verbert, Mike Sharples, and Tomaž Klobučar, editors, Adaptive and Adaptable Learning, 2016.
Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan Su. This looks like that: deep learning for interpretable image recognition. NewrlPS, 2019.
11

---

Published as a conference paper at ICLR 2025
Fu Chen and Ying Cui. Utilizing student time series behaviour in learning management systems for early prediction of course performance. Journal of Learning Analytics, 2020.
Andrea Cini, Ivan Marisca, Filippo Maria Bianchi, and Cesare Alippi. Scalable spatiotemporal graph neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 12267 2023.
Cristina Conati, Kaska Porayska-Pomi's Pomsta, and Manolis Mavrikis. Ai in education needs interpretable machine learning: Lessons from open learner modelling. arXiv preprint
2018.
Mike Cooley. Human-centered design. Information design, pages 59-81, 2000.
Sanjoy Dasgupta, Nave Frost, and Michal Moshkovitz. Framework for evaluating faithfulness of local explanations. In International Conference on Machine Learning, pages 4794-4815. PMLR, 2022.
Andrew Davis and Itamar Arel. Low-rank approximations for conditional feedforward computation in deep neural networks. arXiv preprint arXiv:1312.4461, 2013.
Amit Dhurandhar, Pin-Yo Yu Chen, Ronny Luss, Chun-Chen Tu, Paishun Ting, Karthikeyan Shanmugam, and Payel Das. Explanations based on the missing: Towards contrastive explanations with pertinent negatives. In NeurlPS, 2018.
Jon Donnelly, Alina Jade Barnett, and Chaofan Chen. Deformable protopnet: An interpretable image classifier using deformable prototypes. In CVPR, 2022.
Mengnan Du, Ninghao Liu, and Xia Hu. Techniques for interpretable machine learning, Communica- tions of the ACM, 63(1):68-77, 2019.
William Fedus, Jeff Dean, and Barret Zoph. A review of sparse expert models in deep learning. arXiv preprint arXiv:2209.0166 2022a.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research, 2022b.
Jibril Frej, Neel Shah, Marta Knezevic, Tanya Nazaretsky, and Tanja Käser. Finding paths for explainable mooc recommendation: learner perspective. In Learning Analytics and Knowledge Conference, 2024.
Amirata Ghorbani, James Wexler, James Zou, and Been Kim. Towards automatic concept-based explanations. NeurlPS, 2019a.
Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept-based explanations. Advances in neural information processing systems, 32, 2019b.
Trevor J Hastie. Generalized additive models. In Statistical models in S. Routledge, 2017.
Hussein Hazimeh. Zhe Zhao, Aakanksha Chowdhery, Maheswaran Sathiamoorthy, Yihua Chen, Rahul Mazumder, Lichan Hong, and Ed Chi. Dselect-k: Differentiable selection in the mixture of experts with applications to multi-task learning. Advances in Neural Information Processing Systems, 34:29335-29347 2021.
Michael A Hedderich, Jonas Fischer, Dietrich Klakow, and Jilles Vreeken. Label-descriptive patterns and their application to characterizing classification errors. In ICML, 2022.
HF Canonical Model Maintainers. dibilbeerbassuuncssceeeeennsssed 2022.
(revision bfdd146),
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint trXiv:2/06.09668 2021.
Aya Abdelsalam Ismail, Sercan O Arik, Jinsung Yoon, Ankur Taly, Soheil Feizi, and Tomas Pfister. Interpretable mixture of experts. Transactions Machine Learning Research, 2023.
12

---

Published as a conference paper at ICLR 2025
Robert A Jacobs. Bias/variance analyses of mixtures-of-eepprts architectures. Neural computation, (2):369 13369938,,11 1997.
Alon Jacovi and Yoav Goldberg. Aligning faithful interpretations with their social attribution. Transactions of the Association for Computational Linguistics, 9:294-3100 2021.
Sarthak Jain, Sarah Wiegreffe. Yuval Pinter, and Byron C Wallace. Learning to faithfully rationalize by construction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4459 4473, 2020.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizzttion with gumbel-softmax. In ICLR, 2017.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral experts. arXiv preprint arXiv:2401.0408B 2024.
Wenxin Jiang and Martin A Tanner. On the identifiability of mixtures-of-experts Neural Networks, 12(9):1253-122 1258, 1999.
Michael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6112111 1994.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory Sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). JCML, 2018.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International conference machine learning, pages PMLR, 2020.
Andrei Konstantinov and Lev V Utkin. Incorporating expert rules into neural networks in the framework of concept-based learning. arXiv preprint 4rXiv:2402.14726, 2024.
Satyapriya Krishna, Tessa Han, Alex Gu, Javin Pombra, Shahin Jabbari, Steven Wu, and Himabindu Lakkaraju. The disagreement problem in explainable machine learning: A practitioner's perspective. arXiv preprint arXiv:222.006602, 2022.
Sébastien Lallé and Cristina Conati. A data-driven student model to provide adaptive support during video watching across moocs. In Artificial Intelligence in Education, 2020.
Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. In International Conference Machine Learning, pages 6274. PMLR, 2021.
Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A Smith, and Luke Zettlemoyer. Branch-train- merge Embarrassingly parallel training of expert language models. Interpolate Workshop at NeurIPS, 2022.
Yang Liu, Sujay Khandagale, Colin White, and Willie Neiswanger. Synthetic benchmarks for scientific research in explainable machine learning. NeurlPS, 2021.
Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In NeurlPS, 2017.
Qing Lyu, Marianna Apidianaki, and Chris Ballison-Burche Towards faithful model explanation in nlp: A survey. Computational Linguistics, pages 1 1-67, 2024.
Simon Malberg, Edoardo Mosca, and Georg Groh. Felix: Automatic and interpretable feature engineering using llms. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 230-246. Springer, 2024.
Mirko Marras, Tuan Tu, Julien Vignoud, and Tanja Käser. Can feature predictive power generalize? benchmarking early predictors of student success across flipped and online courses. International Conference Educational Data Mining, 2021.
13

---

Published as a conference paper at ICLR 2025
Paola Mejia, Mirko Marras, Christian Giang, and Tanja Käser. Identifying and comparing multi- dimensional student profiles across flipped classrooms. In Artificial Intelligence in Education, 2022.
Tim Miller. Explanation artificial intelligence: Insights from the social sciences. Artificial Intelligence, 2019.
Christoph Molnar. Interpretuble machine learning. Lulu. com, 2020.
Basil Mustafa, Carlos Riquelme, Joan Puigcerver, Rodolphe Jenatton, and Neil Houlsby. Multimodal contrastive learning with limoe: the language-image mixture of experts. NeurIPS, 2022.
Meike Nauta, Jörg Schlötterer, Maurice van Keulen, and Christin Seifert. Pip-net: Patch-based intuitive prototypes for interpretable image classification. In CVPR, 2023.
Harsha Nori, Samuel Jenkins, Paul Koch, and Rich Caruana. Interpretml: A unified framework for machine learning interpretability. arXiv preprint arXiv:/909.09223 2019.
Juan Pinto and Luc Paquette. Towards a unified framework for evaluating explanations. arXiv preprint ar 2rxiv:2055.14011 2024.
Nils Reimers and Iryna Gurevych. Sentence-bert. Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908. 10084, 2019.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why Should Trustt You?": Explaining the predictions of any classifier. KDD, 2016.
Waddah Saced and Christian Omlin. Explainable ai (xai): A systematic survey of current challenges and future opportunities. Knowledge-Basse Based Systems, 263:110277 2023.
Mohinder Partap Satija. The theory and practice of the Dewey decimal classification system. Elsevier, 2013.
Yoshihide Sawada and Keigo Nakamura. Concept bottleneck model with additional unsupervised concepts. JEEE Access, 2022.
Gesina Schwalbe and Bettina Finzel. A comprehensive taxonomy for explainable artificial intelligence: a systematic survey of surveys on methods and concepts. Data Mining and Knowledge Discovery, 38(5):304333100 2024.
2024.
Mona L Scott. Dewey decimal classification. Libraries Unlimited, 1998.
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-CAM: Visual explanations from deep networks via gradient-based local- ization. International Journal of Computer Vision, 2019.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In NeurlPS, 2013.
Timo Speith. A review of taxonomies of explainable artificial intelligence (xai) methods. In Proceedings of the 2022 ACM conference on fairness, accountability, and transparency, pages 2239-2250, 2022.
Olivera Stojanović, Bastian Siegmann, Thomas Jarmer, Gordon Pipa, and Johannes Leugering. Bayesian hierarchical models can infer interpretable predictions of leaf area index from heteroge- neous datasets. Frontiers in Environmental Science, 9:7808 14, 2022.
Yiyang Sun, Zhi Chen, Vittorio Orlandi, Tong Wang, and Cynthia Rudin. Sparse and faithful explanations without sparse models. arXiv preprint 40X0v22402.09702, 2024.
Mukund Sundararajan, Ankur Taly, and Qigi Yan. Axiomatic attribution for deep networks. In International conference machine learning, pages 3319-3328. PMLR, 2017.
14

---

Published as a conference paper at ICLR 2025
Vinitra Swamy, Mirko Marras, and Tanja Käser. Meta transfer learning for early success prediction in moocs. In ACM Conference on Learning @ Scale, 2022a.
Vinitra Swamy, Bahar Radmehr, Natasa Krco, Mirko Marras, and Tanja Käser. Evaluating the explainers: Black-box explainable machine learning for student success prediction in MOOCs, In International Conference on Educational Data Mining, 2022b.
Vinitra Swamy, Sijia Du, Mirko Marras, and Tanja Kaser. Trusting the explainers: teacher validation of explainable artificial intelligence for course design. LAK23: 13th International Learning Analytics and Knowledge Conference, pages 345-356, 2023a.
Vinitra Swamy, Jibril Frej, and Tanja Käser. The future of human-centric explainable Artificial Intelligence (XAI) is not post-hoc explanations. arXiv preprint wxivv2337.003364 2023b.
Fel Thomas, Thibaut Boissin, Victor Boutin, Agustin Martin Picard, Paul Novello, Julien Colin, Drew Linsley, Tom Rousseau, Remi Cadene, Lore Goetschalckx, al. Unlocking feature visualization for deep network with magnitude constrained optimization. In NeurlPS, 2023.
Alfredo Vellido. The importance of interpretability and visualization in machine learning for applications in medicine and health care. Neural computing and applications, 32(24):18066 18083, 2020.
Thomas Verelst and Tinne Tuytelaars. Dynamic convolutions: Exploiting spatial sparsity for faster inference. In CVPR, 2020.
William Wolberg, Mangasarian Olvi, Nick Street, and W. Street. Breast Cancer Wisconsin (Diagnos- tic). UCI Machine Learning Repository, 1995.
Chhavi Yadav, Michal Moshkovitz, and Kamalika Chaudhuri. Xaudit: A theoretical look at auditing with explanations. arXiv preprint 1rXiv:2206.04740 2022.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc Le. Xinet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
Weiqiu You, Helen Qu, Marco Gatti, Bhuvnesh Jain, and Eric Wong. Sum-of-parts models: Faithful attributions for groups of features. arXiv preprint 0rXiv:2310.16316 2023.
Mo Yu, Shiyu Chang, Yang Zhang, and Tommi S Jaakkola. Rethinking cooperative rationalization: Introspective extraction and complement control. arXiv preprint arXiv: 13294, 2019.
Xiang Zhang, Marko Zeman, Theodoros Tsiligkaridis, and Marinka Zitnik. Graph-guice net- work for irregularly sampled multivariate time series. In International Conference on Learning Representations.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. NeurIPS, 2015.
Yiming Zheng, Serena Booth, Julie Shah, and Yilun Zhou The irrationality of neural rationale models. arXiv preprint arXiv:2110.07550 2021.
15


Reviews:
[{"review_id": "8qWPDPhDF3", "reviewer": "Reviewer_xwth", "summary": "The paper proposes InterpretCC (ICC), an intrinsic interpretable framework for prediction.\n\nInterpretCC has two modes:\n\n- InterpretCC Feature gating (ICC-FG): here the models rely on a sparse set of different features to make the decision for each instance.\n\n- InterpretCC Group routing (ICC-GR): The feature space is divided into subgroups, subnetworks are trained for each group and interpretCC activates different subnetworks for a given sample.\n\n## Model Architecture:\n\nAs described in figure 1, features (for feature gating) or group of features (group routing) are selected via a discriminator network (i.e the discriminator network predicts a mask) the masking is done using Gumbel softmax trick to keep it differentiable.\n\n- For the feature gating: The mask is multiplied by the features and passed to a predictive model to make the final output.\n\n- For the Group routing: The mask is used to select the group of features; each group has a subnetwork, and the final prediction is the weighted sum from the mask and the predictions of the subgroups. This can be viewed as a mixture of expert model where each subnetwork is an expert. Soft masking is used during training to efficiently train the subgroups while hard masking is used during inference. The paper investigates selecting groups in different ways, including handcrafted (user-defined) patterns and using LLMs for grouping.\n\n## Experiments\n\n### Data\n\nThe paper showed there approach on Time Series, Text, and Tabular data. For Tabular data the inputs are the features, for text the tokens are the features and for multivariate time series each input across a period of time is a feature i.e they apply the same mask across all time steps. They focused on classification problems.\n\n### Results\n\n- **Accuracy**  ICC was  compared with black box models and interpretable models like SENN and NAM across 8 datasets. Overall, InterpretCC Feature gating seems to outperform interpretable baselines. For non-text datasets ICC can also outperform DNN baselines. For breast cancer dataset, group routing appears to be extremely helpful. They also report that grouping using GPT is, on average, better than other methods, suggesting that using automated grouping methods does not mean compromising performance.\n\n- **Explanations**\n\n  - The paper used the synthetic dataset OpenXAI to evaluate the quality of explanations in comparison to ground-truth, all interpretable methods seem to align to groundtruth explanations in term of faithfulness, while ICC FG ranking of feature importance is higher than others.\n  - The paper shows that on all datasets, ICC has high feature sparsity, which makes it more user-friendly.\n\n- **User study**\n\n  -  Four samples were randomly selected (i.e., four students) for prediction from each model, 56 teachers were recruited, the teachers were showed them each model’s prediction of the student’s success or failure along with its explanation. Participants were asked to compare these explanations according to five criteria: usefulness, trustworthiness, actionability, completeness and conciseness. Here SENN, NAM, ICC FR and ICC GR were compared.\n\n  - Overall, ICC models are favored over baselines in 4 out of 5 criteria and in terms of global satisfaction from the domain experts.", "questions": "-For the group routing, is the input to the discriminator all features (from all groups) and the output just the number of groups? i.e., the discriminator doesn't exactly know which group the features belong to, correct? While each subnetwork only ever sees the features assigned to that subgroup? Please clarify...\n- In line 204, you mentioned the following \"feature j is activated (the associated value in the mask is non-zero) if the Gumbel Softmax output exceeds a threshold τ , a hyperparameter. This allows the model to adaptively select the number of\nfeatures based on each instance, using fewer features for simpler cases and more for complex ones\" how was $\\tau$ selected?, What is the effect of the changing $\\tau$ on the accuracy metrics?\n- How are SENN features different from InterpetCC feature gating? Is the only difference that InterpetCC has a Gumbel Softmax while SENN doesn't?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "## Originality -- high\n- ICC group routing is original.\n- The paper showed that grouping could be done by LLM, and it is as good as the handcrafted ones, showing that this approach can be used on a large scale without requiring extra labor.\n\n##  Quality -- high\n- The paper showed ICC can be used on different data types as they tested ICC  on 3 different data domains.\n- ICC was benchmarked against strong DNN baselines and good interpretable baselines across 8 datasets.\n- For interpretability, OpenXAI was used to show that the explanations produced by ICC match ground truth explanations (although since  ICC produces a mask used to select the input to the model, so this was expected).\n- The paper showed that explanations produced by ICC were sparse and, therefore, user-friendly.\n- Paper performed a user study that showed domain experts (here teachers) preferred explanations produced by ICC over other interpretable models.\n- Overall, the experiment section and the results are very thorough.\n\n## Clarity   -- high\n- The paper is well-written and easy to follow.", "weaknesses": "## Significance -- Medium \n- ICC feature gating is almost the same as SENN feature except with a sparse mask.\n- ICC group routing might be inefficient when the number of groups significantly increases, since the model complexity will increase as well.\n- ICC sparsity depends on the temperature of the Gumbel Softmax, but its effect was not investigated in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "77msb7zLa2", "reviewer": "Reviewer_csPd", "summary": "The authors propose two new neural network architectures that are designed to be intrinsically interpretable. The first uses a gating mechanism to select a sparse set of features, and the second uses a mixture-of-experts approach to select a sparse set of interpretable feature groups. In experiments on five datasets spanning three modalities, the authors show that their method achieves comparable (or better) performance compared to black-box, non-interpretable models and two intrinsically interpretable baselines. The authors also conduct a user study involving teachers and the task of predicting student performance, and they find that the explanations produced with their method are preferred compared to baselines.", "questions": "My primary concern with this paper is the lack of mention and comparison to existing work on interpretability methods that follow a \"select\" then \"predict\" pipeline. Why aren't these mentioned in the paper and why aren't they included as baselines?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. **Simple, intuitive, and novel idea for the design of intrinsically interpretable neural network architectures.** The authors present a novel idea for the design of neural network-based models that are intrinsically interpretable yet retain strong performance.\n2. **Thorough experimental analysis.** The authors evaluate their method on five datasets, spanning several different domains and modalities. They analyze both the performance of the model and the quality of the explanations it produces in terms of faithfulness, sparsity, and human satisfaction.\n3. **Results show that the proposed method produces useful explanations without sacrificing performance.** Across the five datasets, the proposed method performs comparably (or better) to black-box models with the same architectures in terms of predictive accuracy. The authors also show that the explanations produced by the proposed method tend to be sparse in terms of the percentage of features that are activated. The explanations also achieve high faithfulness scores on the synthetic data. Finally, the authors conduct a user study with 56 teachers, where they apply their method to the task of predicting student performance. They find that the study participants prefer the explanations produced by their method compared to baselines in terms of almost all of the criteria examined (e.g, usefulness, trustworthiness) as well as overall.\n4. **Writing clarity.** The paper is clearly written and easy to follow.", "weaknesses": "1. **Missing comparison/discussion of prior work on explain-then-predict / extractive rationale methods.** There is a substantial amount of existing work on intrinsically interpretable models that involve the same basic two steps proposed in this work: (1) select a subset of the input as the “explanation”/”rationale” and (2) use a model that sees only this explanation to make the final prediction. A lot of this has been done in the NLP space; see the discussion in Section 4.5.2 in [1], and the specific methods in [2]-[5]. Since these works take the same basic approach to producing explanations, I think they should be included as baselines in the evaluation. At the very least, the authors should mention this work in the related work section and justify why their work is sufficiently different such that an experimental comparison is not needed. As a related point, the authors say in their intro that prior work on intrinsically explainable models is “rare” for “text modalities”, and they say that one of their contributions is extending intrinsic interpretability methods to “modalities and domains” that are less common for this area, such as text. I’m not entirely convinced by this point, especially since the authors did not mention any existing intrinsically interpretable approaches for text data (e.g., [2]-[5]) in their related work section.\n2. **Primarily applicable to cases in which model inputs are composed of interpretable features.** The explanations produced by the proposed method take the form of a subset of model inputs (and in the group routing version, a subset with group labels). While this is human-understandable in the case in which model inputs are human-understandable (e.g., time-series features or words in a document), it is not clear that the explanations would be useful in cases where the model inputs are less structured/interpretable (e.g., pixels in an image, raw time-series data, text tokens). In many applications, the most performant models use raw/complex data as inputs as opposed to handcrafted features. Therefore, this seems to be a major limitation of the method. And it is not discussed in the paper. In addition, all experiments in the paper involve model inputs that consist of interpretable features (i.e., words, handcrafted times-series features, or image features). I would like to understand to what extent the method can be applied when the inputs are images, raw time-series, speech, text tokens, etc.\n3. **Doesn’t address the possibility that explanations produced with their method are misleading/unfaithful.** Although the authors claim that their method is guaranteed to be faithful, I don’t think this is actually the case. As pointed out in prior work (e.g., [6], [7]), “select-then-predict” methods of this nature can produce misleading explanations. For example, it could be the case that the predictive model looks for superficial patterns in the selected feature set (e.g., how many features are selected) rather than uses the features as a human would expect. The authors do not address this risk in their paper.\n4. **Limited analysis of impact of sparsity threshold.** In Section 6, the authors state that tuning the feature selection threshold “was key to achieving strong results.” I think the paper would be stronger if the authors included analysis of the impact of the threshold in the main text. There is some analysis in the appendix, but it appears that the experiments were only run with a single seed (there is no variance). In addition, it would be interesting to see the tradeoff between feature sparsity and performance (and how this is impacted by the choice of the threshold parameter).\n5. **User study has some limitations.** Overall, the user study appears well-executed and provides evidence of the utility of the authors proposed method. However, it does have some notable limitations. The most glaring is that the authors conducted the study on only four test samples from a single dataset. This sample is small and the task is specific, so it’s hard to understand how the findings would generalize beyond the specific cases examined. Further, as the authors acknowledge, it seems like the author’s decisions around how to visualize the explanations produced by each method could impact the results. \n\n[1] Lyu, Qing, Marianna Apidianaki, and Chris Callison-Burch. \"Towards faithful model explanation in nlp: A survey.\" Computational Linguistics (2024): 1-67.\n\n[2] Jain, Sarthak, et al. \"Learning to faithfully rationalize by construction.\" arXiv preprint arXiv:2005.00115 (2020).\n\n[3] Bastings, Jasmijn, Wilker Aziz, and Ivan Titov. \"Interpretable neural predictions with differentiable binary variables.\" arXiv preprint arXiv:1905.08160 (2019).\n\n[4] Yu, Mo, et al. \"Rethinking cooperative rationalization: Introspective extraction and complement control.\" arXiv preprint arXiv:1910.13294 (2019).\n\n[5] Lei, Tao, Regina Barzilay, and Tommi Jaakkola. \"Rationalizing neural predictions.\" arXiv preprint arXiv:1606.04155 (2016).\n\n[6] Jacovi, Alon, and Yoav Goldberg. \"Aligning faithful interpretations with their social attribution.\" Transactions of the Association for Computational Linguistics 9 (2021): 294-310.\n\n[7] Zheng, Yiming, et al. \"The irrationality of neural rationale models.\" arXiv preprint arXiv:2110.07550 (2021).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GxarghckGe", "reviewer": "Reviewer_CDui", "summary": "This paper proposed the intrinsic explanation framework called InterpretCC to combine feature gating and group routing based on the design space by considering user-centric features to enable actionable interpretation without sacrificing prediction performance compared with traditional prediction models. It conducted user evaluation and extended intrinsic explanation models to other less popular modalities, such as time series, tabular and text.", "questions": "- can you think about using graph model to extend the explanability from single feature based to feature interaction based.\n- in your evaluation, can you provide more details about features in EDU example, in Figure 10 in appendix, many features are similar, how these teachers could look at these features to measure whether \"“This student was predicted to pass the course because and only because of the student’s regularity and video watching behavior\".\n- can you please provide more details on how the user predefined features (or user preferred features) are considered in feature gating and group routing.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The unique strength of this paper is to put user-centric design into intrinsic explanation models, aiming to address the applicability of developed models in general domains. It is nice to see that efforts of pushing model design into user-centric design.\n\nIt extends the intrinsic explanation models from visions to other modalities, especially time series, text and tabular datasets. \n\nIt has conducted systematic evaluation of four domains with their baseline different explanation models and measured the interpretation based on several well-defined metrics. \n\nThe user evaluation of recruiting 56 teachers to judge the usefulness of the interpretation is a great plus.", "weaknesses": "- In real world scenarios, explanations are not just a set of features, rather than the interactions of a pair of features. Do you consider to identify the interactions of features in your interpretCC framemwork\n- In your user evaluation, as your method is providing local interpretation, as mentioned that interpretCC can recommend interpretation like \"“This student was predicted to pass the course because and only because of the student’s regularity and video watching behavior\". How can you prove such if and only if situation, because all prediction methods in interpretCC are association based, not causal based.\n- can you please provide more details on how to deal with the grouping of features (it is great to see that you are using LLM to group features) for MoE, but grouping features itself can be challenge as some features might belong to two different groups, which could lead to errors in feature gating, and MoE.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xyfuNLYI8X", "reviewer": "Reviewer_WueB", "summary": "This paper introduce a set of intrinsically (by design) interpretable machine learning methods, motivated by a global mixture of experts method. The goal of these methods (InterpretCC) focus on accurate predictions and aims to provide faithful explanations. The authors conduct experiments on a variety of tasks, including some user studies, and demonstrated the effectiveness of their approach.", "questions": "Please refers to my questions in the \"weakness section\". The general landscape of explainable machine learning requires a stronger and more scientific taxonomy / definitions of these terms, the authors are thus encourage to discuss / elaborate on this with respect to their proposed methods.\n\nI think this paper also lack a bit of theoretical analysis/justification (or at least tried to explain why the global mixture of experts method make sense and why it works for all these tasks). Could the authors try to discuss this?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. the motivation of this work is clear and solid, and I believe intrinsic explainable machine learning should definitely be an important direction of research with great importance\n2. good amount of experiments are conducted (including time series, text and tabular inputs) and the experimental results seems to be reasonable\n3. the proposed methods seems to be reasonable novel (based on a global mixture of experts, with non-overlapping group/ set selections)", "weaknesses": "1. there exist evidence of over-claiming (such as Table 1, method comparison; after reading the paper, I am not convinced that the proposed methods actually achieve \"faithfulness\", and maybe somewhat allow full \"coverage\") and definition of these terms are also not clear / or there is no solid definition (although briefly mentioned in background -- Interpretability Foundations)\n\n2. I am fully understand that explanation methods like InterpretCC are build based on expert knowledge or domain expertise (based on l231-296), are other intrinsic baseline methods adopt the same concepts/expert knowledge? would this really be a fair comparison otherwise? Also, for explainable machine learning, there is a trade-off between injecting how much domain knowledge v.s. black box methods (such as LIME or SHARP), could you add a discussion on if such domain knowledge does not exist (as this can often be the case in real-life scenarios)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)