Intrinsic User-Centric Interpretability through Global Mixture of Experts

## Abstract

Abstract Interpretability for neural networks is a trade-off between three key requirements: 1) faithfulness of the explanation ( i.e. , how perfectly it explains the prediction), 2) understandability of the explanation by humans, and 3) model performance. Most existing methods compromise one or more of these requirements; e.g. , post-hoc approaches provide limited faithfulness, automatically identified feature masks compromise understandability, and intrinsically interpretable methods such as decision trees limit model performance. These shortcomings are unacceptable for sensitive applications such as education and healthcare, which require trustworthy explanations, actionable interpretations, and accurate predictions. In this work, we present InterpretCC (interpretable conditional computation), a family of interpretable-by-design neural networks that guarantee human-centric interpretability, while maintaining comparable performance to state-of-the-art models by adaptively and sparsely activating features before prediction. We extend this idea into an interpretable, global mixture-of-experts (MoE) model that allows humans to specify topics of interest, discretely separates the feature space for each data point into topical subnetworks, and adaptively and sparsely activates these topical subnetworks for prediction. We apply variations of the InterpretCC architecture for text, time series and tabular data across several real-world benchmarks, demonstrating comparable performance with non-interpretable baselines, outperforming interpretable-by-design baselines, and showing higher actionability and usefulness according to a user study.

## 1 Introduction

In recent years, the steep rise in popularity of neural networks has come with a severe weakness: the lack of interpretability of their predictions. Neural networks are considered as black-box models due to their high number of parameters and complex operations. Therefore, humans cannot understand how the features impact the network decisions under the hood; this is a crucial limitation in sensitive applications such as education or healthcare, where models’ predictions might impact humans.

Interpretability is a relatively new field of research in the machine learning community. As a result of the increasing interest in providing explanations for black-box models, a range of popular methods have been proposed.
Most of these are post hoc, i.e. applied after model training. Popular approaches include local instance-based methods (e.g. LIME [], SHAP []), using adversarial examples for counterfactual explanations [], or global, pattern-based explanations like PREMISE []. Post-hoc methods require the user to trust the explainer’s approximation of the true explanation, which has been shown to be systematically biased and inconsistent [, ]. Much fewer works provide in-hoc explanations, by interpreting model gradients at inference time or customizing the training protocol. Grad-CAM [] for example assesses the contribution of a component to the model’s output. TCAV [] and DTCAV [] use user-defined concepts to interpret a neural network’s embeddings.

Finally, intrinsically interpretable models have mainly focused on example-based approaches, overwhelmingly for the image modality (e.g. B-cos networks [], PIP-Net [], ProtoPNet []) and rarely in time-series, tabular, or text modalities []. Very recent interpretable-by-design literature in mixture-of-experts models has highlighted a hierarchical neural network structure with subnetworks, combining interpretable experts (i.e. decision trees) with DNNs for partially interpretable points [], selectively activating experts [], or extracting automated concepts over the input space for routing []. However, the information used for the explanation is often overly complex and not tailored for the use case, limiting understandability and actionability.

In this paper, we therefore present a user-centered interpretable-by-design model that achieves guaranteed interpretability (faithfulness) and provides human-centric explanations, while maintaining comparable predictive performance to its black-box counterparts. To achieve these goals, we use conditional computation in neural networks to craft interpretable neural pathways using two different architectures (see Figure 1) based on routing through individual features or entire feature groups.

Figure: Figure 1: InterpretCC Architectures: Feature Gating (left) with individual features: (i) features are passed as input to a discriminator network which outputs a feature activation mask; (ii) features are passed individually as input to sub-network and sigmoid for prediction; (iii) activated features’ corresponding predictions are averaged for final output. Group Routing (right) with pre-defined feature groups: (i) features are passed through a discriminator network which outputs a route activation mask; (ii) features are split by group to be passed as input to corresponding expert sub-network for prediction; (iii) activated routes’ predictions are aggregated by a weighted sum for final output.
Refer to caption: x1.png

Our models enable a statement of the form: “This entry was predicted to be X because and only because it was assigned to human-interpretable categories A and B". We refer to interpretability from the users’ perspective, focusing on the model’s local reasoning for a decision on a specific data point, as opposed to a global understanding of the model’s internals. Our models are characterized by sparse explicit routing, truncated feature spaces, and adaptivity per data point. These traits are important for human-centric trustworthiness as they provide clear and concise instance-level explanations [, ].

With our family of InterpretCC models, we provide the following contributions:

- [1]
InterpretCC Feature Gating: A simple, interpretable-by-design NN architecture using a gating mechanism to sparsely activate specific features.
- [2]
InterpretCC Group Routing: An interpretable mixture-of-experts architecture that uses human-specified group routing to separate the feature space and sparsely activate specific experts.
- [3]
An extension to real-world, human-centric modalities and domains that are often ignored by interpretable-by-design deep learning approaches: time-series (education domain), tabular (health, synthetic data), and text (sentiment, news classification).(^1^11We do not use vision datasets since extracting concepts from vision has been well-studied by existing, modality-specific interpretability approaches, e.g. [, , , ].)
- [4]
A novel XAI user study comparing human preferences of interpretable-by-design model explanations towards designing educational interventions for struggling students.

Across experiments on eight datasets from various domains, we show that InterpretCC models perform comparatively to non-interpretable baselines, while outperforming interpretable-by-design baselines. Moreover, participants of the user study preferred InterpretCC explanations over baselines in terms of actionability, usefulness, conciseness and trustworthiness.
We provide our code and experiments open source: [https://github.com/epfl-ml4ed/interpretcc/](https://github.com/epfl-ml4ed/interpretcc/).

## 2 Background

**Table 1: Method Comparison: InterpretCC models are at the unique intersection of explanation faithfulness, sparsity, coverage, flexible granularity, and user-defined basis.**
|  | Explanation |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Method | Granularity | Basis | Faithfulness | Sparsity | Coverage |
| LIME, SHAP<br>(Post-Hoc) | Feature | Use all input features | Approximation | Not sparse | Partial |
| TCAV<br>(In-Hoc) | Concept | User defines concepts<br>through examples | Aligned<br>with concepts | Sparse | Partial |
|  | Feature | Use all input features | Guaranteed | Not sparse | Full |
| SENN | Concept | Automated concept<br>selection integrated | Aligned<br>with concepts | Not sparse | Partial |
| NAM | Feature | Use all input features | Guaranteed | Not sparse | Full |
|  | Feature | Use all input features | Guaranteed | Sparse | Full |
| InterpretCC<br>(Feature Gating,<br>Group Routing) | Concept | External definition of group<br>of features as concepts | Guaranteed | Sparse | Full |

Architecture Foundations. Conditional Computation (CC) has become widely used to improve the computationally expensive training and inference of large neural networks by activating only parts of the network [, ]. Bengio et al. [] outline how block dropout conditional computation policies can be optimized using reinforcement learning.
Inspired by the foundations laid out by CC, mixture-of-expert models have rapidly gained popularity for improving the efficiency of neural networks.
BASE layers [] direct each token to a designated expert and Switch Transformers [] use CC to select one out of 4 feedforward networks across each transformer layer, optimizing computational resources.
Mixtral [] is a recent LLM using a mixture of experts to select 2 out of 8 expert networks at each layer, reducing the numbers of active parameters by a factor of 4 compared to training, while allowing each token to have access to all the parameters. With the InterpretCC models, we extend a similar routing idea with instance-dependent gating decisions towards an interpretability objective as opposed to only an efficiency or performance objective.

Interpretability Foundations. Explainability can be integrated into different stages of the modeling pipeline: post-hoc (after model training), in-hoc (e.g., interpreting model gradients), and intrinsic (interpretable by design). Our positioning of InterpretCC in comparison to the most popular approaches is described in Table 1. Specifically, we categorize the approaches using their basis and the stage they are applied to in the pipeline as well as four key requirements for human-centric explanations [, ]: faithfulness (the explanation reflects the model behavior with certainty), sparsity (the model uses a minimal amount of features to ease actionability and understandability), coverage (the explanation covers the entirety of what the model uses for prediction), and flexible granularity, where the key features or the feature grouping can be user-defined.

Post-hoc methods such as LIME [] or SHAP [] approximate what the model finds important, and therefore cannot be faithful or complete. Other in-hoc interpretability approaches use human-understandable concepts (e.g. TCAV [] and DTCAV []), or hybrid methods using both human-defined and unsupervised concepts []. Such example-based approaches (e.g., TCAV) provide only partial coverage and limited faithfulness. Inspired by these approaches, InterpretCC allows users to specify interpretable concepts that are directly useful to them. However, we do not use examples, but instead allow users to specify groupings over the feature space, achieving guaranteed interpretability and full coverage.

Initial approaches have also explored expert models for intrinsic interpretability. The Interpretable Mixture of Experts (IME) framework [] proposes different tradeoffs to using a mixture of experts for interpretability, showcasing how linear models can be used as experts to have faithful explanations. LIMoE [] specializes visual experts to identify concepts such as textures and faces, enriching interpretability for vision. Also for vision, interpretability based on groups is explored by the Sum-of-Parts (SOP) model [], where the prediction process involves making sparse groups of features, highlighting the model’s dependence on subsets of features for its decisions. The most relevant intrinsic models to our approach are Self-Explaining Neural Networks (SENN) [] and Neural Additive Models (NAM) []. SENN automatically extracts concepts described by protypical examples alongside concept relevances to be combined for a prediction, again limiting faithfulness and coverage. NAM employs models for each feature in the dataset which outputs a score that is linearly combined for the prediction, making it non-sparse. InterpretCC differentiates from these models by filtering the feature space (sparsity), using user-specified concepts (basis), and not supporting overlapping in feature groups (coverage).

## 3 Methodology

Given an input $x$, the objective of our approach is to select a sparse subset of $x$ that will be used to predict the output to solve the classification task. We propose two architectures:

Feature Gating: The approach only processes a subset of the features by applying a sparse mask $\mathcal{M}$ on the input $x$ before processing it by a model $f$. The output is given by: $f(\mathcal{M}(x))$.

Group Routing: A sparse mixture of models [] applied on human-interpretable groups of features where each expert is assigned to a group of features:

$$ $\sum_{i=1}^{K}G(x)_{i}\cdot f_{i}(\mathcal{M}_{\mathcal{G}}(x)_{i}),$ (1) $$

where $\mathcal{M}_{\mathcal{G}}(.)_{i}$ is a sparse mask selecting only the features of group $i$, $f_{i}$ is the expert model associated with the $i$-th group, and $G(x)_{i}$ is the output of the gating network for group $i$. If $G(x)_{i}=0$, the entire group of features is ignored.

### 3.1 Feature Gating

The Feature Gating architecture shown in Fig. 1 is the first step towards using CC paths for interpretability. The features are first passed through a discriminator network $D$ to select which ones to use for computing the output.
The Gumbel Softmax trick [] is applied on each dimension of $D(x)$ to select features in a differentiable way (see Appendix F for more details). A feature $j$ is activated (the associated value in the mask is non-zero) if the Gumbel Softmax output exceeds a threshold $\tau$, a hyperparameter. This allows the model to adaptively select the number of features based on each instance, using fewer features for simpler cases and more for complex ones.

The output is computed using a model $f$ on the masked input $\mathcal{M}(x)$. Since the explainability is at the feature level, using a black box model for $f$ does not detract from the interpretability.

### 3.2 Group Routing

We build upon the instance-dependent gating architecture with feature groups.
As displayed in Figure 1, instead of selecting features individually, the mask is applied to human interpretable groups of features.
Doing so encourages cross-feature interactions while maintaining a meaningful grouping for human users.
To select the features belonging to group $i$, we use a binary mask $\mathcal{M}_{\mathcal{G}}(x)_{i}$ that is computed using human-specified rules.
In section 4, we detail our approach to compute $\mathcal{M}_{\mathcal{G}}(x)_{i}$ for each dataset used in our experiments.

Group Routing is a sparse mixture of experts utilizing a gating network to assign a weight $G(x)_{i}$ to each group.
This process mirrors that of Feature Gating, starting with a discriminator network $D_{\mathcal{G}}$ whose output has $K$ dimensions ($K$ begin the number of groups). It then applies the Gumbel Softmax and a threshold $\tau_{\mathcal{G}}$ to each group. As indicated in equation 1, the output of the model is a weighted sum of the output of each expert $f_{i}$ that only uses the features from the $i$-th group as input.
Using our sparsity criteria, we ensure that few groups are used to compute the output, making the Group Routing intrinsically interpretable at the group level, regardless of the types of models used as experts.
Group Routing also exhibits several traits in efficient inference without compromising the number of parameters the model can use at training.

During the training phase, we employ soft masking, allowing all weights $G(x)_{i}$ to remain non-zero, thus granting the model access to every expert. This approach allows the model to leverage the full set of parameters during training, enhancing the training efficiency. At inference time we switch to using a hard mask, making the weights sparse and allowing for interpretability and efficiency.

## 4 Experimental Settings

We apply InterpretCC to four domains: education, news/sentiment classification, healthcare, and synthetic data covering Time Series, Text, and Tabular inputs; all for classification tasks.
For Tabular features, the input is a vector $x\in\mathds{R}^{n}$. The mask in the Feature Gating is a sparse vector indicating which tabular feature to use and how important they are (if the weight is non-0) and the groups form a partition over the features. For Text features, the input is a sequence of $N$ tokens: $x=[t_{1},t_{2},\cdots,t_{N}]$. The mask is a sparse vector that indicates which token to use and each group consists of a subset of the tokens. Finally, we consider Time Series of $n$ features across $T$ timesteps: $x\in\mathds{R}^{n\times T}$. In this case, we apply the same mask across all time steps for both Feature Gating and Groups.

EDU (time series, education domain). We predict student success in the early weeks of four massive open online courses (MOOCs), using students’ clickstream data (see Table 6 in Appendix A for details about the courses). The raw clickstream input is transformed into weekly time-series features that have proven useful for student success prediction in previous literature (e.g. total video clicks, forum interactions). We select 45 input features used in multiple studies [, , , ]. For early prediction, we only use the first $40\%$ of time steps as input.

Grouping: To derive human-interpretable concepts from these features, we turn to learning science literature. In routing by paper, we create 10 distinct feature subsets based on handcrafted initial input features from 10 papers, directing each to a specific expert subnetwork.
For routing by pattern, we organize features according to five learning dimensions identified by [, ]: effort, consistency, regularity, proactivity, control, and assessment-based (see Table 8 for a detailed feature classification).
Thirdly, routing by Large Language Model (LLM), uses GPT-4’s capabilities to aid humans in feature grouping []. GPT-4 is prompted as an ’expert learning scientist’ to group the features into self-regulated behavior categories that are easy to understand, which are then used to separate the features for InterpretCC. More details are included in Appendix B.1.

AG News and SST (text). For news categorization (AG News), we classify news into four categories (‘World’, ‘Sports’, ‘Business’, ‘Sci/Tech’) given a title and description of a real-world article []. We use 36,000 training samples and 3,000 test samples evenly distributed across categories. For sentiment prediction (Stanford Sentiment Treebank, SST), we use 11,855 sentences labeled by three annotators[] and predict a binary sentiment from a sentence fragment.

Grouping: The InterpretCC routing model assigns words to subnetworks using the Dewey Decimal Code (DDC) hierarchy of topics for book classification to create 10 subnetworks (see Table 8, Appendix B.2 for more details) []. Each word is encoded using SentenceBERT [] and assigned to a subcategory (i.e. the word ‘school’ is assigned to the subcategory ‘education’ under category 300 for ‘social sciences’) and routed to the appropriate parent network.

Breast Cancer (tabular, healthcare domain). The Wisconsin Breast Cancer dataset aims to identify cancerous tissue from an image of a fine needle aspirate (FNA) of a breast mass []. It has 30 features (10 from each of 3 cell nuclei) and diagnoses (Malignant: 1, Benign: 0) for 569 patients.

Grouping: For the grouping logic, we group each cell nucleus in a separate subnetwork, enabling features representing the same part of the tissue sample to be considered together.

Synthetic Dataset (tabular) We use OpenXAI’s synthetic dataset [], which includes ground truth labels and explanations, indicating the subset of features influencing each label. This dataset comprises 5000 samples, 20 continuous features, and two classes. It was created using the SynthGauss mechanism from 5 cluster neighborhoods (1000 points for each cluster), which ensures three desirable properties for assessing explanations: (1) feature independence, (2) unambiguous and well-separated local neighborhoods, and (3) a unique ground truth explanation for each local neighborhood.

Grouping: We group the feature space by assigning each feature to a cluster neighborhood based solely on the distribution of the training data. The average absolute value of each feature for each cluster is calculated, and the highest feature-cluster value determines the assignment.

## 5 Experimental Results

Through the following three experiments, we demonstrate that our InterpretCC models do not compromise performance compared to black-box models and provide explanations that are faithful as well as human-centered.

Experimental Setup. We run hyperparameter tuning and three different random seeds for each reported model (reproducibility details in Appendix E). Since EDU MOOC courses have a low passing rate (below 30%), and thus the dataset has a heavy class imbalance, we use balanced accuracy for evaluation. The other datasets are more balanced (AG News, SST, Breast Cancer, synthetic), hence we use accuracy as our evaluation metric. We perform an 80-10-10 train-validation-test data split stratified on the output label, to conserve the class imbalance in each subset. In addition to InterpretCC Feature Gating and Group Routing, we also employ a InterpretCC Top-K expert network solution with k=2 for group routing. This approach is similar to existing mixture-of-expert approaches [, ], except that their models make a layer-wise expert choice, which significantly reduces interpretability, while we make a global expert choice.

Base Prediction Module. We choose simple yet performant predictive modules reported in previous literature to isolate the difference in performance due to the interpretable architectures. For the EDU data, previous works uses BiLSTMs on student behavioral data for best predictive performance [, ]. Thus, for comparative benchmarking, the most performant BiLSTM setting is used as a baseline model []. For the AG News and SST datasets, we use fine-tuned DistilBERT(^2^22[https://huggingface.co/distilbert/distilbert-base-uncased](https://huggingface.co/distilbert/distilbert-base-uncased)) variations as baselines, also used in related works [, ]. For the Breast Cancer dataset, we use a fully connected network as reported in [].

Interpretable Baselines.
In addition to non-interpretable baselines, we compare ICC to two interpretable-by-design methods: SENN [] and NAM [], adaptable to any base architecture.
SENN generates concept-based explanations and is designed to learn "interpretable basis concepts" in parallel with the model optimization. Explanations consist of the concepts most similar to the input.
NAM, a General Additive Model [], uses individual neural networks for each input feature to calculate feature weights. The model’s output is the sum of these weights, and explanations are given by displaying the feature weights.

### 5.1 Exp. 1: InterpretCC does not compromise on performance

Table 2 shows the models’ metrics (average accuracy, standard deviations, and $95\%$ confidence intervals (CIs)) across three iterations of model training for all eight datasets.

**Table 2: InterpretCC Performance (avg $\pm$ std) on EDU (balanced accuracy), Text, Health, and Synthetic (accuracy) datasets compared to a non-interpretable baseline and three interpretable-by-design baselines. All 95% CI overlap with the non-interpretable base module for values in black. Colored values indicate significantly higher (green) or lower (red) performance than the base module. The reported Group Routing results are the best performing InterpretCC variations from Table 3.**
|  |  |  | Feature-Based Interpretability | Concept-Based Interpretability |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Dataset | Non-interpretable<br>Base Module | NAM | SENN<br>Features | InterpretCC<br>Feature Gating | SENN<br>Concepts | InterpretCC<br>Top K Routing | InterpretCC<br>Group Routing |
|  | DSP | 82.81 $\pm$ 2.61 | 85.20 $\pm$ 0.64 | 71.70 $\pm$ 0.95 | 90.75 $\pm$ 0.01 | 81.50 $\pm$ 2.26 | 83.08 $\pm$ 1.10 | 84.90 $\pm$ 7.59 |
|  | Geo | 72.96 $\pm$ 1.59 | 65.12 $\pm$ 4.07 | 57.90 $\pm$ 2.69 | 71.92 $\pm$ 0.01 | 70.90 $\pm$ 2.45 | 80.44 $\pm$ 3.19 | 81.58 $\pm$ 0.57 |
|  | HWTS | 73.93 $\pm$ 3.76 | 73.11 $\pm$ 2.13 | 68.63 $\pm$ 3.78 | 82.89 $\pm$ 0.04 | 75.10 $\pm$ 11.67 | 72.59 $\pm$ 2.84 | 78.34 $\pm$ 0.95 |
| Education | VA | 74.90 $\pm$ 5.28 | 71.39 $\pm$ 3.38 | 74.37 $\pm$ 1.11 | 77.80 $\pm$ 0.01 | 69.99 $\pm$ 8.83 | 71.43 $\pm$ 1.11 | 72.08 $\pm$ 3.71 |
| Health | B. Cancer | 89.70 $\pm$ 1.05 | 88.77 $\pm$ 7.31 | 80.52 $\pm$ 6.21 | 78.19 $\pm$ 3.54 | 85.26 $\pm$ 1.03 | 84.66 $\pm$ 3.02 | 94.85 $\pm$ 1.25 |
|  | AG News | 89.93 $\pm$ 3.32 |  |  | 85.72 $\pm$ 5.31 |  | 87.25 $\pm$ 2.48 | 90.35 $\pm$ 1.07 |
| Text | SST | 91.12 $\pm$ 2.03 | Not<br>Supported | Not<br>Supported | 88.21 $\pm$ 3.41 | Not<br>Supported | 92.98 $\pm$ 0.88 | 91.75 $\pm$ 1.86 |
| Synthetic | OpenXAI | 86.67 $\pm$ 0.31 | 87.85 $\pm$ 1.31 | 83.67 $\pm$ 1.86 | 89.51 $\pm$ 0.51 | 84.67 $\pm$ 4.04 | 90.83 $\pm$ 1.93 | 89.47 $\pm$ 2.89 |

InterpretCC Feature Gating improves performance with respect to the non-interpretable base prediction module for two EDU courses (DSP, HWTS) and the synthetic dataset. It shows comparable performance on all other datasets (indicated by the overlapping CIs) except the Breast Cancer dataset, suggesting that a higher number of the available features is necessary for performant prediction. The interpretable-by-design baselines SENN and NAM never outperform the base model, and always under-perform or perform comparably to InterpretCC.(^3^33We do not report results for text features with NAM and SENN, since having a network for every word oreach word as a concept would be infeasible.)

InterpretCC Group Routing outperforms (for Geo course and Breast Cancer dataset) or performs comparably ($95\%$ CI overlap) to the fine-tuned, non-interpretable base module. It also consistently performs at least comparatively to SENN which, besides never outperforming the base model, has a relatively high variance.
We further observe that the selected grouping method impacts performance (Table 3). We achieve a $10\%$ increase in performance compared to the base model when grouping using patterns or GPT-4 for the Geo course. On average, over the four EDU courses, the automated LLM grouping and the pattern-based human-defined grouping perform comparably, showing that using automated grouping methods does not mean compromising on performance.

Figure: Table 3: InterpretCC Group Routing Performance: balanced accuracy (average $\pm$ std) on variations of routing strategies (paper, pattern, GPT-4) for the EDU datasets in comparison to the non-interpretable baseline.
Refer to caption: x2.png

InterpretCC performs comparable to black-box models and outperforms interpretable baselines.

### 5.2 Exp. 2: InterpretCC provides faithful and user-friendly explanations

Table 4 showcases the faithfulness of InterpretCC models in comparison to SENN and NAM on ground truth explanations from the synthetic dataset. All 95% CI overlap in ground truth accuracy (how close the explanation is to ground truth feature importance, similarly to []) and Pairwise Rank Agreement (PRA) [], except for SENN Concepts performing statistically significantly worse than InterpretCC Feature Gating in accuracy and InterpretCC Group Routing on PRA. This indicates that ICC’s identification of important features, and of their order of importance, is close to the ground truth and better than all baselines.

Besides being faithful, our models enable concise and hence user-friendly explanations by sparse feature (group) activations [, ].
Figure 2 shows the percentage of activated features per data point for ICC Feature Gating. For EDU, only $\sim$ 10% of the $45$ features are activated, with low standard deviation. For Breast Cancer, more features seem to be needed to make the prediction: $39.7\%$ of the features are activated per data point.
Contrarily to other datasets, for text-based data, the number of available features (words) varies per data point. AG News consists of news articles (average number of words per sample: $35$), and only a small percentage of words is activated for each article. SST contains much shorter sentences ($7$ words on average), with high variance, leading to a higher percentage of features selected per data point ($59.8\%$) as well as a high standard deviation.
The achieved sparsity, concisely indicating the most important features in the original data set (especially for EDU and AG News), is a major advantage of our feature gating approach. In comparison, post-hoc explainers often select a wider range of important features (e.g., in previous work [] on EDU, LIME and SHAP [, ] indicated broad importance over all input features).

ICC Group Routing activates different subnetworks with different weights for each data point. Figure 4 illustrates the number of activations and the average weight for each subnetwork for the text data sets (see Appendix G for detailed analysis on the Breast Cancer and EDU datasets). For AG News (Figure 4 top), the average activation weight is similar across all subnetworks (min $0.10$, max $0.21$). However, some subnetworks are activated much more frequently (400 - Language: $18,335$ times). This indicates that most data points will be routed through the same subset of subnetworks, while the remaining subnetworks are important for specific data points only. SST (Figure 4 bottom) shows similar subnetwork activation patterns. However, in contrast to AG News, the distribution of average weights is not uniform: only three networks are activated with weights larger than $0.15$.

Figure: Figure 3: AG News and SST: # of subnetwork activations (left) vs. average activation weights (right), grouped by subnetworks based on the Dewey Decimal Code.
Refer to caption: x3.png

Figure 4 illustrates two entries of AG News and corresponding ICC explanations. In the top example (Perseid meteor shower), the words ‘stars’, ‘meteor’, and ‘SPACE’ are routed to the Pure Science (500) subnetwork with a $50\%$ activation weight, resulting in the correct prediction of ‘Sci/Tech’ category. Likewise, for the bottom article, both the Technology and Arts subnetworks are highly weighted, resulting in the correct prediction of the ‘Business’ category. Interestingly, subnetwork Language (400) is also activated. We suspect the high weights showcased for 400 in Figure 4 are representative of words the DDC does not have a close relation to in SentenceBERT embedding space. Examples of explanations for the EDU domain for all models can be found in Appendix D, Figures 6 and 7. We also compare ICC behavior across the three grouping strategies for the DSP course (Appendix G.2.2) and analyze the network sparsity at different prediction horizons (G.2.1). Finally, we show the variation of the selection of feature groups across all MOOCs, indicating the adaptability of ICC in selecting relevant subnetworks whatever the grouping method (G.2.3).

Figure: Table 4: Synthetic faithfulness evaluation across five models on ground truth explanations using PRA and accuracy.

InterpretCC provides sparse and hence user-friendly explanations, while not compromising on explanation faithfulness.

### 5.3 Exp. 3: InterpretCC explanations are preferred by humans

Setting. To validate the user-centeredness of our approach, we conducted a user study comparing InterpretCC’s explanations with the ones from other interpretable-by-design methods, SENN and NAM. We focused on the education domain and time series input type, using the DSP course in the EDU dataset. We trained InterpretCC Feature Gating and Group Routing (with pattern-based feature grouping strategy, since it is heavily grounded on expert knowledge) as well as SENN and NAM and randomly selected four test samples (i.e., four students) for prediction.

We recruited 56 teachers using Prolific(^4^44[www.prolific.com/](www.prolific.com/)), (see Appendix D for detailed information about the participants’ demographics and backgrounds (Figure 5) and the content of the study). We showed them each model’s prediction of the student’s success or failure along with its explanation. The explanations were given as a short text and a graph showing the features and concepts used by the model. Note that the choices we made for the presentation of the explanations might have an influence on the participants’ perception of the explanations. Examples of how the explanations were presented and a discussion on this limitation can be found in Appendix D, Figures 6, 7, 9 and 8. We asked participants to compare these explanations according to five criteria (aligned with []), and to rank the criteria in terms of importance.

- [1]
Usefulness: This explanation is useful to understand the prediction.
- [2]
Trustworthiness: This explanation lets me judge if I should trust the model.
- [3]
Actionability: This explanation helps me know how to give feedback to the student.
- [4]
Completeness: This explanation has sufficient detail to understand why the prediction was made.
- [5]
Conciseness: Every detail of this explanation is necessary.

Results. Using Friedman’s Chi-Square test, we verify that the ranking of the criteria is consistent among the participants ($p<0.05$). We report normalized criteria importance in Table 5 (last column). We observe that Usefulness, Trustworthiness and Actionability are consistently ranked the highest by the participants. We compute the Global satisfaction score as a weighted average of the criteria as a global satisfaction measure for each model (Table 5, last row).

We observe that ICC FG ranks the highest in the top 3 most important criteria as well as in global satisfaction. Furthermore, ICC GR ranks second in Usefulness and Actionability and first in Conciseness. Overall, InterpretCC models are favored over interpretable baselines in 4 out of 5 criteria and in terms of global satisfaction. An ANOVA performed separately for each criterion as well as the global satisfaction measure (Table 10, Appendix D.3), indicates that there is indeed a significant difference between the models for each criterion. Post-hoc pairwise Tukey’s Honest Significant Difference (HSD) tests confirm that ICC FG significantly outperforms NAM and SENN on all criteria but completeness
(see Figure 11, Appendix D.3 for details).

Participants prefer InterpretCC explanations in terms of usefulness, trustworthiness, actionability, and conciseness over interpretable baselines.

## 6 Discussion and Conclusion

We propose InterpretCC, a family of interpretable-by-design models that puts interpretability and human understanding at the forefront of the design. Through our experiments with feature gating models and interpretable group routing (mixture-of-expert) models, we demonstrate that our optimally interpretable modular architectures do not compromise on performance. In a real-world setting, we show that InterpretCC models are preferred over other intrinsically interpretable models in 4 of 5 explainability criteria.

Limitations. InterpretCC Feature Gating is best when individual features are important for actionable decisions based on the explanation. However, it can lose sparseness if all features are equally important: consider the extreme case where the prediction is a sum over the full feature space and all features are independent. While this is rare in real-world datasets, it is a critical design consideration. InterpretCC Group Routing requires more human effort than Feature Gating and is, therefore, more adapted to scenarios with large feature spaces, where broader concepts are more relevant than individual features. For both InterpretCC models, we find that tuning the learning rate and feature threshold parameter was especially key to achieving strong results.

User-defined feature groups aim at deriving explanations useful to the user, but might compromise performance if the user specifies a grouping that carries minimal signal. This is by design: regardless of the grouping, InterpretCC optimizes for explanation actionability and understandability over performance. We believe that an accurate prediction at the cost of explanation usefulness is not worth it in an applied setting. Moreover, we have shown how the feature groups can be defined automatically using LLMs to reduce human effort, but still provide human-centric explanations.

We acknowledge that the presentation of explanations in our user study (Section 5.3) has influence over our preliminary results on user perception.
We had to significantly post-process the explanations of SENN and NAM to provide them in a format understandable for a non-technical audience. An extensive study over many different prediction tasks and participants’ domains of expertise is necessary for generalizable conclusions.

Future Work. For the text datasets, we fine-tune DistilBERT models to solve the task given groups of features. For more complex tasks, for example requiring very long context size, multi-step reasoning ability, or strong prior domain-specific knowledge, DistilBERT can be swapped with large decoder models (LLMs), either through fine-tuning or in-context learning. Parameter-efficient fine-tuning such as LoRA would allow fine-tuning even large LMs with limited computational cost, while in-context learning would use the same model instance for each feature and feature group.
Overall, we encourage the machine learning community to design models for interpretability at many different granularities and user-specified requirements. InterpretCC provides one such family of models as a tradeoff between human specification, explanation certainty, and performance.

## Impact Statement

This paper introduces work aimed at advancing the field of human-centered machine learning, addressing the critical balance between interpretability and model performance in neural networks. We demonstrate that these aspects do not necessarily need to be a trade-off, particularly in fields where AI’s can influence decisions that can have effects on people’s lives, such as finance, healthcare, and education.
Focusing on the applications presented in this paper, in healthcare, the ability to understand and trust AI-driven recommendations can greatly improve care delivery. In education, the application of interpretable models reduces the risks of unintended biases, help users to identify actionable items, and increases trust.

A key impacts is regulatory compliance, where guidelines and regulations around AI usage are increasing. Models like InterpretCC can help industries and organizations meet these legal standards by increasing the transparency of their AI-based decision-making tools.

ICC enhances trust and human-centered actionability in predictive tools by clearly listing the features impacting predictions. This empowers decision-makers to take informed actions. Different contexts and applications lead to different interpretability needs; different levels of granularity, and different sets or groups of features. ICC Group Routing allows users to define their own set of feature, adapted to their needs. Moreover, by identifying a finite set of key features, ICC may reduce spurious predictions and allow users to better understand the causes of unexpected model behavior. Overall, an interpretable model like ICC allows users to assess the model’s robustness more easily and faithfully.

Our tool also offers the potential to identify and mitigate biased behaviors within models, particularly if discriminatory predictions arise from certain feature uses. As different contexts may require varying levels of interpretability, the Group Routing feature of ICC allows users to define and adjust feature sets according to their specific needs, ensuring flexibility and relevance in diverse applications.

Comparable in societal and technological impact to other interpretable-by-design methods, such as those discussed by Agarwal et al. in their 2021 study, ICC represents a significant step forward in making AI more accessible and understandable for users and practitioners alike.

However, we acknowledge that despite the versatility of ICC across various data types, domains, and datasets demonstrated in this work, we cannot guarantee that it will always perform on par with or better than baseline models. It is important to note that the model weights we release are not safe for direct use in production tasks, such as detecting breast cancer, without further validation and adaptation.
We encourage other researchers and practitioners to adopt explainable AI methods.

## Appendix A Additional Details on Datasets

Here, we provide additional statistics regarding each dataset used in our study. In particular, we highlight their availability details and terms of use.

EDU. We predict student success during the early weeks of four massive open online courses (MOOCs), using students’ clickstream data (see Table 6 for details about the courses). Contrary to the other 4 datasets, this dataset is kept private for student privacy reasons, as required by HREC 058-2020/10.09.2020 and HREC 096-2020/09.04.2022.

**Table 6: Course Details and Statistics. Topic abbreviations: Math: Mathematics; NS: Natural Science; CS: Computer Science; SS: Social Science; Arch: Architecture; Bus: Economics and Business.**
| Title | Identifier | Topic | Level | Language | No. Weeks | No. Students | Passing Rate [%] |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Digital Signal Processing | DSP | CS | MSc | English | 10 | 4,012 | 23.1 |
| Éléments de Géomatique | Geo | Math | BSc | French | 11 | 452 | 45.1 |
| Household Water Treatment and Storage | HWTS | NS | BSc | French | 5 | 2,438 | 47.2 |
| Villes Africaines | VA | SS | BSc | En/Fr | 12 | 5,643 | 9.9 |

AG News is a news classification dataset, where given a title and description of a real-world article, it has to be classified into one of the four categories: ‘World’, ‘Sports’, ‘Business’, ‘Sci/Tech’ [].
It is freely available at the following location: [http://groups.di.unipi.it/~gulli/AG_corpus_of_news_articles.html](http://groups.di.unipi.it/~gulli/AG_corpus_of_news_articles.html), but only for non-commercial use.

SST. The Stanford Sentiment Treebank dataset aims at predicting the sentiment from a sentence fragment sourced from a movie review. The dataset is freely available here: [https://huggingface.co/datasets/sst](https://huggingface.co/datasets/sst).
This popular benchmark is an extension of the Movie Review Database (MRD) []. It includes two sets of labels: one for binary sentiment classification and one for multiclass. We use binary classification to demonstrate a different setting than the multiclass classification of AG News.

Breast Cancer. The Wisconsin Breast Cancer dataset attempts to identify the presence of cancerous tissue from an image of a fine needle aspirate (FNA) of a breast mass []. This dataset is freely available here: [https://archive.ics.uci.edu/dataset/17/breast+cancer+wisconsin+diagnostic](https://archive.ics.uci.edu/dataset/17/breast+cancer+wisconsin+diagnostic), and is licensed under a Creative Commons Attribution 4.0 International (CC BY 4.0) license.

Synthetic Dataset. We generate 5000 points of OpenXAI’s synthetic dataset [] through the class generate_gaussians with 5 cluster centers, which includes both ground truth labels and ground truth explanations, and is available here: [https://github.com/AI4LIFE-GROUP/OpenXAI](https://github.com/AI4LIFE-GROUP/OpenXAI) (under MIT license).

## Appendix B InterpretCC Group Routing Schema

In the following, we discuss the exact grouping schematic employed on the 4 EDU datasets (including three schemes) and the 2 text datasets.

**Table 7: EDU Routing by Pattern uses learning dimensions from [] to create interpretable feature groupings.**
| Dimensions | Corresponding measures | Student patterns |
| --- | --- | --- |
| Effort | Total time online<br>Total video clicks | Higher intensity<br>Lower intensity |
| Consistency | Mean session duration<br>Relative time online<br>Relative video clicks | Uniform<br>First half<br>Second half |
| Regularity | Periodicity of week day<br>Periodicity of week hour<br>Periodicity of day hour | Higher peaks<br>Lower peaks |
| Proactivity | Content anticipation<br>Delay in lecture view | Anticipated<br>Delayed |
| Control | Fraction time spent (video)<br>Pause action frequency<br>Average change rate | Higher intensity<br>Lower intensity |
| Assessment | Competency strength<br>Student shape | Higher intensity<br>Lower intensity |

### B.1 Feature Grouping Schema for EDU Datasets

The data is composed of student clickstream data from four different MOOCs. The raw clickstream input is transformed into 45 weekly time-series features that have proven useful for student success prediction in previous literature (e.g. total video clicks, forum interactions).

Routing by pattern. We organize features according to four learning dimensions identified by [] and detailed in Table 8—effort, consistency, regularity, proactivity, control, and assessment—based on [], with a focus on these dimensions in an extended experiment.

Routing by GPT-4. To leverage LLMs for automated feature groupings, we use the following prompt alongside the feature descriptions from the appendix table of Swamy et. al [].

Prompt: You are a talented learning scientist, specializing in self-regulated learning behavior. You want to group the following features together into smaller groups of logical concepts based on educational theory with headings. What are your feature groupings?

Response: Certainly, based on educational theory and self-regulated learning behavior, I would group the provided features into the following logical categories:

- •
Temporal Engagement:
–
DelayLecture
–
RegPeakTimeDayHour
–
RegPeriodicityDayHour
- •
Study Patterns:
–
NumberOfSessions
–
AvgTimeSessions
–
TotalTimeSessions
–
StdTimeSessions
–
StdTimeBetweenSessions
- •
Video Engagement:
–
AvgWatchedWeeklyProp
–
StdWatchedWeeklyProp
–
AvgReplayedWeeklyProp
–
StdReplayedWeeklyProp
–
AvgInterruptedWeeklyProp
–
StdInterruptedWeeklyProp
- •
Competency and Learning Progress:
–
CompetencyStrength
–
CompetencyAlignment
–
CompetencyAnticipation
–
ContentAlignment
–
ContentAnticipation

- •
Interaction Patterns:
–
TotalClicks
–
TotalClicksProblem
–
TotalClicksVideo
–
TotalClicksWeekday
–
TotalClicksWeekend
–
TotalTimeProblem
–
TotalTimeVideo
–
TotalClicksVideoLoad
–
TotalClicksVideo
–
FrequencyEventVideo
–
FrequencyEventLoad
–
FrequencyEventPlay
–
FrequencyEventPause
–
FrequencyEventStop
–
FrequencyEventSeekBackward
–
FrequencyEventSeekForward
–
FrequencyEventSpeedChange
–
AvgSeekLength
–
StdSeekLength
–
AvgPauseDuration
–
StdPauseDuration
–
AvgTimeSpeedingUp
–
StdTimeSpeedingUp
- •
Assessment Behavior:
–
StudentSpeed
–
StudentShape

Each grouping represents a different aspect of self-regulated learning behavior, focusing on how students engage with learning resources, interact with content, demonstrate competency, and approach assessments. This categorization aligns with principles of self-regulated learning and can help in analyzing and understanding students’ behaviors and strategies within an educational context.

Routing by Paper. In routing by paper, we create 4 distinct feature subsets based on handcrafted initial input features from 4 learning science research papers highlighted as predictive for MOOCs in []. We filter each feature set to a expert subnetwork for the research paper the set originated from [, , , ].

### B.2 Feature Grouping Schema for Text Datasets

For news categorization (AG News) and sentiment prediction (SST) feature grouping, we assign words to subnetworks. For this, we use the Dewey Decimal Code (DDC) for librarians and its hierarchy of topics for book classification to create 10 subnetworks, as showcased by topic in Table 8 []. Each word is assigned to a subcategory (i.e. the word ‘school’ is assigned to the subcategory ‘education’ under category 300 for ‘social sciences’) and routed to the appropriate parent network. The decision to use the DDC was to use subnetworks that were standardized, pervasive in daily life and clearly human-understandable. To conduct this assignment, we utilize SentenceBERT to encode the subtopics for each DDC heading (i.e. all of 010, 020, 030, etc. for the category 000) []. The choice of SentenceBERT is motivated towards capturing the broader context of multi-word category headings in a lightweight model. During training and inference, we again use SentenceBERT to encode each word in the input instance, then assign each word to the most similar DDC category in embedding space with cosine similarity.

## Appendix C Sparsity Criterion Experiments

For the feature gating architecture to further improve interpretability, we would like the network to learn sparse feature activations. That is, for a given input $x$ we would like to reduce the number of features that affect the model prediction. To achieve this we apply regularization to the generated feature mask.

One natural choice to enforce sparse feature activations is to apply $L_{1}$-norm regularization to the feature mask, which penalizes a high number of nonzero elements. Another choice is to use annealed regularization as presented by Verelst and Tuytelaars [], which might aid the model to first work through a prediction optimization phase that is not confounded by any additional error terms before moving towards a sparsity-enforcing phase.

We experiment with annealing L1 and L1 regularization across four courses, and find that traditional L1 regularization is more stable (at least in the time-series setting). The Baseline BiLSTM results are not reported as confidence intervals here as they are directly sourced from a recent benchmarking paper by Swamy et al., with confirmed similarity by Asadi et al. [, ]. We reproduce this benchmark above with similar values in Table 9.

**Table 9: Annealing L1 regularization in comparison with L1 regularization across EDU datasets.**
| EDU<br>Dataset | Baseline | InterpretCC<br>Feature Gating |  |
| --- | --- | --- | --- |
| 40% EP | BiLSTM | Annealing | L1 |
| DSP | 82 | 87.76 +/- 3.12 | 90.75 +/- 0.01 |
| Geo | 76.2 | 81.13 +/- 5.39 | 71.92 +/- 0.01 |
| HWTS | 72 | 77.58 +/- 0.01 | 82.89 +/- 0.04 |
| VA | 73.8 | 84.81 +/- 0.01 | 77.80 +/- 0.01 |

## Appendix D User Study

In this section, we discuss the details of the user study presented in Section 5.3, discussing the design, content, and additional analyses of the results (including an ANOVA and Tukey HSD tests).
We designed the study over four rounds of pilots, with 8 individuals from diverse backgrounds, continuously updating the study design following their feedback. The survey has been approved by the the Human Research Ethics Committee (HREC) under application number HREC 065-2022/27.09.2022.

### D.1 Design of the Study

We recruit 56 participants using Prolific,(^5^55[www.prolific.com/](www.prolific.com/)), selecting the ones who identified their current profession as a teacher and who have at least a bachelor’s degree. Our target participants have expertise in teaching, as they would be well-suited to understand both the educational context of the study and the consequences of black-box models for student outcomes. During the study, we ask the participants whether they have ever taken or prepared material for an online course (MOOC), their level of education, and what level they are teaching at (from primary school to graduate school). Detailed demographics distribution can be found in Figure 5. The sample of participants is gender-balanced, and about half of them have taken or participated in creating a MOOC. The median completion time is 22 minutes, and the average reward per hour is £14.55.

Figure: Figure 5: Demographics of teachers that participated in the user study.
Refer to caption: x5.png

At the beginning of the study, the participants are shown the following welcome message and disclaimer:

[
backgroundcolor=green!10,
linecolor=black,
linewidth=1pt,
innertopmargin=10pt,
innerbottommargin=10pt,
innerrightmargin=10pt,
innerleftmargin=10pt,
roundcorner=10pt,
frametitlebackgroundcolor=gray!30,
frametitlealignment=]
Dear participant,

Thank you for participating in our study on model explanations. We are very grateful for your participation and your invaluable insight. Please read this Explanatory Statement in full before moving forward. If you would like further information regarding any aspect of this project, please contact us via the email address provided below.

We are a group of researchers from the ML4ED Laboratory at EPFL dedicated to improving education through technology. The goal of this study is to evaluate different explanations to help a student improve their performance in an online course.

- This survey has been approved by the EPFL Human Research Ethics Committee (HREC) under application number HREC 065-2022/27.09.2022. HREC reviews research proposals involving human participants to ensure that they are ethically acceptable.

- All the personal information will be kept confidential and anonymized. Only demographic information is being recorded and will only be reported as aggregate in a way that prevents identification of any individual participant. You can freely withdraw at any time and any collected data you provided so far will be destroyed.

- All data will be collected and stored safely and reported in an anonymous form, in accordance with the Swiss Federal law on data protection (“Loi fédérale sur la protection des données” – RS 235.1).

- Only anonymized or aggregated data may be used in follow-up research (subject to ethics approval), and made available to other researchers for further analysis and for verification of the conclusions reached by the research team.

- Only the principal investigator and the aforementioned researchers have access to the original data under strict confidentiality. Results from the project may be published in conference papers and/or journal articles. In any case, no personal data will be published (only aggregated, anonymous and/or anonymized data will be published).

- Personal data of participants will be stored for 5 years from the date of collection. During this time, participants have the right to access their data and request information about the processing of their personal data. In order to exercise this right, you need to contact the Principal Investigator.

[
backgroundcolor=green!10,
linecolor=black,
linewidth=1pt,
innertopmargin=10pt,
innerbottommargin=10pt,
innerrightmargin=10pt,
innerleftmargin=10pt,
roundcorner=10pt,
frametitlebackgroundcolor=gray!30,
frametitlealignment=]

By participating in this survey, you agree that your data can be used for scientific purposes.

In the following study, you will be asked to compare explanations for approximately 35 minutes. Please ensure that you have enough time to finish the study correctly. Unfinished or only partially answered studies will not be considered as taken part.

We ask you to approach the questions and exercises with seriousness and to complete them to the best of your ability. We will subsequently check questionnaires for seriousness and will have to discard questionnaires that do not meet this requirement.

Thank you for your help. If you encounter any problem with the survey, or if you want to give extra feedback, or receive additional information, feel free to contact us (vinitra.swamy@epfl.ch).

### D.2 Content of the study

First, we explain the setting of the study to the participants with the following introductory message:

[backgroundcolor=orange!10, linecolor=black, linewidth=1pt, innertopmargin=10pt, innerbottommargin=10pt, innerrightmargin=10pt, innerleftmargin=10pt, roundcorner=10pt]
You are a teaching assistant helping with a Massive Open Online Course (MOOC). This course is taught at the Master’s level with quizzes and video lectures taking place over 10 weeks. Since it’s a difficult course with a low passing rate (23.1%), the teaching team wants to help students who are not doing well to perform better in the course by giving them personalized assistance, and encourage students who are already performing well to continue.

To do this, we have models to predict student success or failure using various weekly behavior features, such as number of video clicks or how accurately questions are answered on the weekly quizzes. If potential failure is predicted early (in our case, in the first 4 weeks of the course), we can use the explanation of the prediction to give additional support (i.e. specific tutoring or assignment reminders) to help the student pass the course.

We train four interpretable machine learning models. Each model predicts a student’s performance at the end of the course, in the form of “pass” or “fail”, but also gives us which factors contribute to student success or failure. We want to compare these explanations according to several criteria:

- •
Usefulness: This explanation is useful to understand the prediction.
- •
Trustworthiness: This explanation lets me judge if I should trust the model.
- •
Actionability: This explanation helps me know how to give feedback to the student.
- •
Completeness: This explanation has sufficient detail to understand why the prediction was made.
- •
Conciseness: Every detail of this explanation is necessary.

We randomly sample 4 students from the test set. Among the 4 selected students, 3 failed the class and one passed. For each student, we predict their success or failure with each model and generate an explanation. We provide them to the participants along with the models’ prediction of the student’s success or failure. The ground truth (student’s performance) and the models’ performance are not provided to the participants so that we do not bias their assessment.

The content of the explanation obtained by each method differs greatly. We simplify the explanations and render them in textual and graph format to make them as easy to understand as possible to a human. For InterpretCC explanations, we provide the full list of single features / feature groups used by the models. For Feature Gating, we show the evolution of the student’s behavior across weeks for each of the features used (see Figure 6). For Group Routing, we compute a generic score for each concept used by the model, by averaging the normalized behavior features that compose the concept. We show the evolution of that concept measure across weeks for the student. We also provide the definition of the concept and of the features that compose it (see Figure 7). For SENN, we select the top 5 groups of students, that we call concepts. We showcase all 180 feature-weeks for each concept along with their importance in that concept, highlighting the salient ones. We also provide the importance of each concept for the model’s prediction (see Figure 9). Finally, NAM assigns feature importance to all 180 feature-weeks used to make the prediction. We select the 5 feature-weeks found to have highest positive impact, 5 feature weeks with the lowest impact, and the 5 feature-weeks with highest negative impact. We show their importance in a barplot (see Figure 8).

Note that the choice we made for the presentation of the explanations might have an influence on the participants’ perception of the explanations. In an ideal setting, we would provide a very detailed description of how each model uses the features and how the explanation is derived, so that the participants can fairly assess the explanation’s quality. However, in a realistic setting, the user facing the explanation might not have the time or prior knowledge necessary to understand these elements. To tackle that trade-off between ensuring thoroughness and accessibility of the content of each model’s explanation, we opted for a balanced approach in presenting each explanation with a simple graph and an explanatory text that we kept as short as possible.

For each new sample (student taking the course), we provide the list of 4 model explanations in random order. We ask participants to compare these explanations according to five criteria using likert scales. A screenshot of the answer section is shown in Figure 10. We include a practice question to train the participants in how to answer the study and filter inattentive experts. We excluded from the analysis 5 participants who failed to answer correctly to half of the practice questions.

Figure: Figure 6: Explanation visualizations given to the participants for the InterpretCC Feature Gating method, for one student.
Refer to caption: x6.png

Along with the InterpretCC FG graphs, we provide the following explanatory text:
{graybox}
This student is predicted to fail the course. The model found the following 2 features to be the most predictive for this student, and only used these features to make the prediction:

- •
Quiz Speed on Attempts: The average time passed between two consecutive attempts for the same quiz.
- •
Total Time spent on Problems: The total (cumulative) time that a student has spent on problem events.

The plot shows the evolution of the student’s behavior for each feature across the 4 weeks.

Figure: Figure 7: Graph given to the participants for the InterpretCC Group Routing with pattern-based matching method, for one student.
Refer to caption: x7.png

Along with the InterpretCC GR graph, we provide the following explanatory text:
{graybox}
This student is predicted to fail the course. For this student, the model decided to only use 1 group(s) of features to make the prediction as this was the one(s) it found most important. The groups of features were designed by expert literature in learning sciences:

- •
Concept: PROACTIVITY

Definition: Proactivity measures the extent to which students are on time or ahead of the schedule, as engagement in pre-class activities has shown to be associated with exam performance.

- •
It includes the following features: The number of videos covered by the student from those that are in subsequent weeks and The average delay in viewing video lectures after they are released to students.

Figure: Figure 8: Graph given to the participants for the NAM method, for one student.
Refer to caption: x8.png

Along with the NAM graph, we provide the following explanatory text:
{graybox}
This student is predicted to fail the course. The model used all 180 feature-weeks (45 features from 4 weeks) to make the prediction. It has assigned a level of importance for each feature-week, showing how much it impacts the predictions, independently of the student’s behavior.
Out of the 180 feature-weeks, the plot shows the 5 feature-weeks found to have the highest positive impact, 5 feature weeks with the lowest impact, and the 5 feature-weeks with the highest negative impact. For example, Quiz Max Grade in Few Attempts in week 0 has an importance score of 3.48.

Figure: Figure 9: Graph given to the participants for the SENN method, for one student.
Refer to caption: x9.png

Along with the SENN graph, we provide the following explanatory text:
{graybox}
This student is predicted to fail the course. The model used all 180 feature-weeks (45 features from 4 weeks) to make the prediction. It groups them into 5 concepts automatically and assigned a score to each concept. Each concept can be interpreted as a group of features that are important for the prediction. The plot shows the importance of each feature-week for each concept, from high positive importance (blue) to high negative importance (red).

Here is the list ordered by absolute value of importance:

- •
Concept 3 (-0.21)
- •
Concept 2 (0.08)
- •
Concept 1 (-0.01)
- •
Concept 5 (-0.01)
- •
Concept 4 (0.01)

Figure: Figure 10: Format of the Likert scale question asked for each criterion and explanation.
Refer to caption: extracted/5629285/Images/user_study/qualtrics_question.png

### D.3 Significance Testing for User Study Results

We perform an ANOVA to determine the effect of the model and the sample (the student) on the score given by the participants, for each criterion and on average.
Table 10 shows the p-values testing the significance of the effect of the model and sample on the participants’ scores for each criterion. It can be interpreted the following way. In the first row, if the p-value is lower than the significance level (0.05), then there is a statistically significant difference in scores across the different models for that criterion. In the second row, if the p-value is below 0.05, there is a statistically significant difference in scores across the different students. Finally, a p-value lower than 0.05 in the final row shows that the effect of the model on the scores depends on the student.
According to the table, the model has always a significant impact of the value given to each criterion.
Then, we apply Tukey’s Honest Significant Difference (HSD) Test to determine, for each pair of explanation, if their scores are significantly different (Figure 11).

**Table 10: ANOVA results for the user study.**
|  | Usefulness | Trustworthiness | Actionability | Completeness | Conciseness | Global |
| --- | --- | --- | --- | --- | --- | --- |
| Model | 0.000 | 0.001 | 0.000 | 0.000 | 0.000 | 0.000 |
| Student | 0.004 | 0.018 | 0.144 | 0.003 | 0.075 | 0.075 |
| Model:Student | 0.143 | 0.058 | 0.063 | 0.000 | 0.171 | 0.171 |

Figure: Figure 11: Significance test using Tukey’s Honest Significant Difference (HSD) Test. It indicates which pairs of models have significantly different means. We highlight the top model on average (Global satisfaction, in the last plot), ICC FG (InterpretCC Feature Gating) in blue, and the models that are significantly worse according to each criterion in red. Example of interpretation: for the criterion Usefulness, ICC FG is scored significantly better than SENN and NAM, but is not significantly better than ICC GR.
Refer to caption: x10.png

## Appendix E Reproducibility and Compute Details

All EDU, Health, and Synthetic experiments were run on a single NVIDIA A100 GPU with 32 GB Memory, each model taking approximately 20 minutes (or often less) to train. Text experiments for InterpretCC took approximately an hour to train. SENN Concepts took the longest time of all models, with each model running within 3-4 hours.

All interpretable models reported in 2 and 3 has been hyperparameter tuned over the following parameters with early-stopping (where applicable):

- •
learning rate: 1e-3, 1e-4, 2e-5, 1e-5
- •
layer size: 16, 32, 64
- •
number of concepts (only for SENN): 5, 6, 7
- •
batch size: 8, 16, 32, 64)
- •
gumbel softmax threshold (only for InterpretCC): 0.1, 0.3, 0.5, 0.7

Each Feature-Based model was run for 100 epochs with early stopping, and each Concept-Based model was run for 20 epochs with early stopping. Other details related to preprocessing and thresholds are included directly in our repository.

## Appendix F Gumbel Softmax trick and its application to InterpretCC

To make the feature gating and routing architectures compatible with backpropagation, we need to make the masks differentiable. These discrete decisions can be trained end-to-end using the Gumbel Softmax trick []. This method adapts soft decisions into hard ones while enabling backpropagation, i.e. provides a simple way to draw samples from a categorical distribution.

Given a categorical distribution with class probabilities $\pi=[\pi_{1}\,\pi_{2}\,...\,\pi_{N}]$, one can draw discrete samples $z$ as follows:

$$ $z=\textsc{OneHot}\left(\arg\max_{i}[g_{i}+\log\pi_{i}]\right)$ $$

where $g_{1}...g_{N}$ are i.i.d. samples drawn from the Gumbel$(0,1)$ distribution. Then, the softmax function is used as a differentiable approximation to $\arg\max$ to generate a $N$-dimensional sample vector $y$ such that

$$ $y_{i}=\frac{\exp((\log(\pi_{i})+g_{i})/\tau)}{\sum_{j=1}^{N}\exp((\log(\pi_{j} )+g_{j})/\tau)}\quad\text{for }i=1,...,N$ $$

where $\tau$ is a softmax temperature parameter that is fixed at $\tau=1$ for experiments in this project.

Notice that for the gating mechanism, an independent sample is drawn for each ‘gate’ instead of for each datapoint in routing. For example in feature gating, for each feature $i$, a soft-decision $a_{i}\in(-\infty,+\infty)$ is outputted by the discriminator layers. The probability $\pi_{1}$ that the feature should be activated as well as the complement probability $\pi_{2}$ (feature is not activated) can then be computed by using the sigmoid function:

$$ $\pi_{1}=\sigma(a_{i})\qquad\pi_{2}=1-\pi_{1}=1-\sigma(a_{i})$ $$

The corresponding (1-dimensional) sample $y$ for each $i$ can thus be reduced to

$$ $y=\sigma\left(\frac{a_{i}+g_{1}-g_{2}}{\tau}\right)$ $$

In other words, the discriminator layers from Fig. 1 actually feed into an adapted Gumbel Sigmoid where $\sigma_{i}$ is the corresponding $y$ sample as described above.

For routing, the discriminator layers actually output the route logits to a Gumbel Softmax, which constructs the categorical sample vector (of dimension equal to the number of routes and $i$-th entry $y_{i}$ defined as above).

Finally, we can use a straight-through estimator during training. In other words, binary (or hard/quantized) samples are then used for the forward pass while gradients are obtained from the soft samples for backpropagation. This means that, given soft decisions $\sigma_{i}$, architectures that use a mask $M=[m_{1}\,...\,m_{N}]$ with $m_{i}=\mathds{1}_{\{\sigma_{i}\geq 0.5\}}$ differ in value during the forward and backward pass:

$$ $m_{i}=\begin{cases}\mathds{1}_{\{\sigma_{i}\geq 0.5\}}\qquad\text{(forward pass)},\\ \sigma_{i}\qquad\qquad\quad\text{(backward pass)}\end{cases}$ $$

## Appendix G Additional Group Routing Experiments

We conduct additional analyses for the high impact, real-world applications in Health and EDU.

### G.1 Grouping Analysis for Breast Cancer Dataset

Figure: Figure 12: Breast Cancer: Number of activations (left) vs. average activation weight (right) per cell.
Refer to caption: x11.png

For the Breast Cancer data set, the subnetworks grouping features from Cell $1$ and Cell $2$ are activated much more frequently than the third subnetwork (see Fig. 12). Furthermore, Cell $3$ also gets activated with higher weights than the other two cells (Cell $1$: $0.25$, Cell $2$: $0.40$, Cell $3$: $0.0.70$). Smoothness and texture of the tissue images were the most important features across cells.

### G.2 Grouping Analysis for EDU Datasets

We conduct sparsity and group routing paradigm analyses on a representative course of the EDU datasets (DSP). We additionally experiment with grouping by paper and pattern over all courses, including two additional MOOC courses with low predictive performance (Structures and Ventures) featured in [].

#### G.2.1 Sparsity Analysis on DSP

We conduct a sparsity analysis on the course used for the user study, DSP, and featured in several research works in the learning science community [, , ]. In line with [], we evaluate the sparsity at both the 40% and 60% prediction setting, representing early prediction at 4 and 6 weeks of the course respectively.

**Table 11: InterpretCC Feature Gating comparison of performance metrics between different sparsification methods next to their average and standard deviation of number of activated features for both 40% and 60% early success prediction settings; baseline benchmarks also provided for contrast.**
| Setting | Model | Performance<br>ACC<br>BAC<br>AUC | Activated Features<br>$\;\;\;\mu\;\;\;$<br>$\,\,\;\sigma\;\,\,$ |
| --- | --- | --- | --- |
| 40% | Baseline<br>L1 (1e-5)<br>L1 (1e-4)<br>Annealed MSE<br>Truncated AMSE | 0.835<br>0.653<br>0.858<br>0.790<br>0.711<br>0.817<br>0.763<br>0.748<br>0.784<br>0.768<br>0.770<br>0.823<br>0.805<br>0.743<br>0.823 | 97<br>0<br>8.54<br>0.58<br>2.20<br>1.64<br>13.26<br>3.72<br>5.95<br>1.03 |
| 60% | Baseline<br>L1 (1e-5)<br>L1 (1e-4)<br>Annealed L1 (1e-5)<br>Annealed MSE<br>Truncated AMSE | 0.944<br>0.925<br>0.982<br>0.914<br>0.935<br>0.963<br>0.914<br>0.917<br>0.957<br>0.910<br>0.927<br>0.957<br>0.892<br>0.927<br>0.952<br>0.787<br>0.788<br>0.839 | 97<br>0<br>37.60<br>3.97<br>31.38<br>4.45<br>38.36<br>3.61<br>21.45<br>3.23<br>-<br>- |

An annealed mean-squared regularization proved most effective, although it activated more features on average than $L_{1}$-norm regularization which more effectively reduced the feature space while achieving desirable balanced accuracy performance in this setting. By truncating the initial feature space to only the activated features and using the same architecture, performance is almost maintained even though the average number of activated features per datapoint is more than halved (from around 13 to 6). However, this method was not as effective for the 60% setting. Truncating the feature space largely reduces predictive capability (e.g. almost a 0.15 drop in balanced accuracy). L1 regularization in this case proved best. Using an annealed regularization did not significantly improve or change model performance as well.

#### G.2.2 Multiple Grouping Paradigms on DSP

Figure: Figure 13: EDU: Number of subnetwork activations (left) and Gumbel Softmax activation weights (right) across different groupings (GPT-4, Paper, Pattern) for course DSP 1 of the EDU domain.
Refer to caption: x12.png

To illustrate the influence of different feature groupings, we conduct a deep dive for course DSP 1 of the EDU domain. Figure 13 illustrates the number of subnetwork activations and corresponding weights for three different groupings.

For the first two groupings (GPT-4, Paper), the subnetwork activations (number of times the route was activated) closely mirror the Gumbel Sigmoid adaptive weighting, indicating that a few networks are activated with high weights for prediction. In the group by GPT-4 setting, we see behaviors of competency, interaction patterns, and assessment frequently activated for student pass-fail predictions. Although ‘interaction patterns’ is the largest category (most number of features chosen by GPT-4), it still comes second to competency (focusing on student achievement). In the group by paper setting, we see a clear preference for Marras et al. with over $17,500$ students predicted using this network (dark orange) and high weight given to the predictions from the network (light orange).

In contrast, in the third grouping (Pattern), we see a differentiation between the number of activations (dark orange) and the weight of the activations (light orange). Notably, the patterns of ‘Effort’, ‘Proactivity’, ‘Consistency’ and ‘Control’ all have higher than $50\%$ weight when they are activated, which means they contribute a lot to the overall prediction when chosen.

#### G.2.3 Grouping by paper and pattern over all courses

Figure: Figure 14: EDU analysis of group by paper routing averaged over all six courses for each paper grouping. The y-axis represents the proportion of points for which the subnetwork is activated.
Refer to caption: x13.png

In Figure 14, we see InterpretCC routing by research paper (grouping the features based on the paper they were proposed in). The Marras et al. and Chen Cui feature sets have clearly been identified as important over the majority of courses, echoing findings in other learning science literature using BiLSTM and random forest architectures [, , ]. The large standard deviations in the box-plots indicate that for at least some courses (in this case Structures and Venture), Chen Cui and Marras were not found significantly important. Notably, the same courses that have low accuracies on routing in 2 are those that have low scores on the two most popular feature sets, showing a consensus among performant InterpretCC models and a validation of the identification of importance.

Figure: Figure 15: EDU Weighting distribution across subnetworks using group by pattern for 6 courses.
Refer to caption: x14.png

In Fig. 15, we see a widely varying distribution of patterns selected across courses, showcasing the ability of InterpretCC to adaptively select subnetwork weights depending on the dataset.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: -For the group routing, is the input to the discriminator all features (from all groups) and the output just the number of groups? i.e., the discriminator doesn't exactly know which group the features belong to, correct? While each subnetwork only ever sees the features assigned to that subgroup? Please clarify...
- In line 204, you mentioned the following "feature j is activated (the associated value in the mask is non-zero) if the Gumbel Softmax output exceeds a threshold τ , a hyperparameter. This allows the model to adaptively select the number of
features based on each instance, using fewer features for simpler cases and more for complex ones" how was $\tau$ selected?, What is the effect of the changing $\tau$ on the accuracy metrics?
- How are SENN features different from InterpetCC feature gating? Is the only difference that InterpetCC has a Gumbel Softmax while SENN doesn't?
soundness: 3
strengths: ## Originality -- high
- ICC group routing is original.
- The paper showed that grouping could be done by LLM, and it is as good as the handcrafted ones, showing that this approach can be used on a large scale without requiring extra labor.

##  Quality -- high
- The paper showed ICC can be used on different data types as they tested ICC  on 3 different data domains.
- ICC was benchmarked against strong DNN baselines and good interpretable baselines across 8 datasets.
- For interpretability, OpenXAI was used to show that the explanations produced by ICC match ground truth explanations (although since  ICC produces a mask used to select the input to the model, so this was expected).
- The paper showed that explanations produced by ICC were sparse and, therefore, user-friendly.
- Paper performed a user study that showed domain experts (here teachers) preferred explanations produced by ICC over other interpretable models.
- Overall, the experiment section and the results are very thorough.

## Clarity   -- high
- The paper is well-written and easy to follow.
summary: The paper proposes InterpretCC (ICC), an intrinsic interpretable framework for prediction.

InterpretCC has two modes:

- InterpretCC Feature gating (ICC-FG): here the models rely on a sparse set of different features to make the decision for each instance.

- InterpretCC Group routing (ICC-GR): The feature space is divided into subgroups, subnetworks are trained for each group and interpretCC activates different subnetworks for a given sample.

## Model Architecture:

As described in figure 1, features (for feature gating) or group of features (group routing) are selected via a discriminator network (i.e the discriminator network predicts a mask) the masking is done using Gumbel softmax trick to keep it differentiable.

- For the feature gating: The mask is multiplied by the features and passed to a predictive model to make the final output.

- For the Group routing: The mask is used to select the group of features; each group has a subnetwork, and the final prediction is the weighted sum from the mask and the predictions of the subgroups. This can be viewed as a mixture of expert model where each subnetwork is an expert. Soft masking is used during training to efficiently train the subgroups while hard masking is used during inference. The paper investigates selecting groups in different ways, including handcrafted (user-defined) patterns and using LLMs for grouping.

## Experiments

### Data

The paper showed there approach on Time Series, Text, and Tabular data. For Tabular data the inputs are the features, for text the tokens are the features and for multivariate time series each input across a period of time is a feature i.e they apply the same mask across all time steps. They focused on classification problems.

### Results

- **Accuracy**  ICC was  compared with black box models and interpretable models like SENN and NAM across 8 datasets. Overall, InterpretCC Feature gating seems to outperform interpretable baselines. For non-text datasets ICC can also outperform DNN baselines. For breast cancer dataset, group routing appears to be extremely helpful. They also report that grouping using GPT is, on average, better than other methods, suggesting that using automated grouping methods does not mean compromising performance.

- **Explanations**

  - The paper used the synthetic dataset OpenXAI to evaluate the quality of explanations in comparison to ground-truth, all interpretable methods seem to align to groundtruth explanations in term of faithfulness, while ICC FG ranking of feature importance is higher than others.
  - The paper shows that on all datasets, ICC has high feature sparsity, which makes it more user-friendly.

- **User study**

  -  Four samples were randomly selected (i.e., four students) for prediction from each model, 56 teachers were recruited, the teachers were showed them each model’s prediction of the student’s success or failure along with its explanation. Participants were asked to compare these explanations according to five criteria: usefulness, trustworthiness, actionability, completeness and conciseness. Here SENN, NAM, ICC FR and ICC GR were compared.

  - Overall, ICC models are favored over baselines in 4 out of 5 criteria and in terms of global satisfaction from the domain experts.
weaknesses: ## Significance -- Medium 
- ICC feature gating is almost the same as SENN feature except with a sparse mask.
- ICC group routing might be inefficient when the number of groups significantly increases, since the model complexity will increase as well.
- ICC sparsity depends on the temperature of the Gumbel Softmax, but its effect was not investigated in the paper.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 4
questions: My primary concern with this paper is the lack of mention and comparison to existing work on interpretability methods that follow a "select" then "predict" pipeline. Why aren't these mentioned in the paper and why aren't they included as baselines?
soundness: 3
strengths: 1. **Simple, intuitive, and novel idea for the design of intrinsically interpretable neural network architectures.** The authors present a novel idea for the design of neural network-based models that are intrinsically interpretable yet retain strong performance.
2. **Thorough experimental analysis.** The authors evaluate their method on five datasets, spanning several different domains and modalities. They analyze both the performance of the model and the quality of the explanations it produces in terms of faithfulness, sparsity, and human satisfaction.
3. **Results show that the proposed method produces useful explanations without sacrificing performance.** Across the five datasets, the proposed method performs comparably (or better) to black-box models with the same architectures in terms of predictive accuracy. The authors also show that the explanations produced by the proposed method tend to be sparse in terms of the percentage of features that are activated. The explanations also achieve high faithfulness scores on the synthetic data. Finally, the authors conduct a user study with 56 teachers, where they apply their method to the task of predicting student performance. They find that the study participants prefer the explanations produced by their method compared to baselines in terms of almost all of the criteria examined (e.g, usefulness, trustworthiness) as well as overall.
4. **Writing clarity.** The paper is clearly written and easy to follow.
summary: The authors propose two new neural network architectures that are designed to be intrinsically interpretable. The first uses a gating mechanism to select a sparse set of features, and the second uses a mixture-of-experts approach to select a sparse set of interpretable feature groups. In experiments on five datasets spanning three modalities, the authors show that their method achieves comparable (or better) performance compared to black-box, non-interpretable models and two intrinsically interpretable baselines. The authors also conduct a user study involving teachers and the task of predicting student performance, and they find that the explanations produced with their method are preferred compared to baselines.
weaknesses: 1. **Missing comparison/discussion of prior work on explain-then-predict / extractive rationale methods.** There is a substantial amount of existing work on intrinsically interpretable models that involve the same basic two steps proposed in this work: (1) select a subset of the input as the “explanation”/”rationale” and (2) use a model that sees only this explanation to make the final prediction. A lot of this has been done in the NLP space; see the discussion in Section 4.5.2 in [1], and the specific methods in [2]-[5]. Since these works take the same basic approach to producing explanations, I think they should be included as baselines in the evaluation. At the very least, the authors should mention this work in the related work section and justify why their work is sufficiently different such that an experimental comparison is not needed. As a related point, the authors say in their intro that prior work on intrinsically explainable models is “rare” for “text modalities”, and they say that one of their contributions is extending intrinsic interpretability methods to “modalities and domains” that are less common for this area, such as text. I’m not entirely convinced by this point, especially since the authors did not mention any existing intrinsically interpretable approaches for text data (e.g., [2]-[5]) in their related work section.
2. **Primarily applicable to cases in which model inputs are composed of interpretable features.** The explanations produced by the proposed method take the form of a subset of model inputs (and in the group routing version, a subset with group labels). While this is human-understandable in the case in which model inputs are human-understandable (e.g., time-series features or words in a document), it is not clear that the explanations would be useful in cases where the model inputs are less structured/interpretable (e.g., pixels in an image, raw time-series data, text tokens). In many applications, the most performant models use raw/complex data as inputs as opposed to handcrafted features. Therefore, this seems to be a major limitation of the method. And it is not discussed in the paper. In addition, all experiments in the paper involve model inputs that consist of interpretable features (i.e., words, handcrafted times-series features, or image features). I would like to understand to what extent the method can be applied when the inputs are images, raw time-series, speech, text tokens, etc.
3. **Doesn’t address the possibility that explanations produced with their method are misleading/unfaithful.** Although the authors claim that their method is guaranteed to be faithful, I don’t think this is actually the case. As pointed out in prior work (e.g., [6], [7]), “select-then-predict” methods of this nature can produce misleading explanations. For example, it could be the case that the predictive model looks for superficial patterns in the selected feature set (e.g., how many features are selected) rather than uses the features as a human would expect. The authors do not address this risk in their paper.
4. **Limited analysis of impact of sparsity threshold.** In Section 6, the authors state that tuning the feature selection threshold “was key to achieving strong results.” I think the paper would be stronger if the authors included analysis of the impact of the threshold in the main text. There is some analysis in the appendix, but it appears that the experiments were only run with a single seed (there is no variance). In addition, it would be interesting to see the tradeoff between feature sparsity and performance (and how this is impacted by the choice of the threshold parameter).
5. **User study has some limitations.** Overall, the user study appears well-executed and provides evidence of the utility of the authors proposed method. However, it does have some notable limitations. The most glaring is that the authors conducted the study on only four test samples from a single dataset. This sample is small and the task is specific, so it’s hard to understand how the findings would generalize beyond the specific cases examined. Further, as the authors acknowledge, it seems like the author’s decisions around how to visualize the explanations produced by each method could impact the results. 

[1] Lyu, Qing, Marianna Apidianaki, and Chris Callison-Burch. "Towards faithful model explanation in nlp: A survey." Computational Linguistics (2024): 1-67.

[2] Jain, Sarthak, et al. "Learning to faithfully rationalize by construction." arXiv preprint arXiv:2005.00115 (2020).

[3] Bastings, Jasmijn, Wilker Aziz, and Ivan Titov. "Interpretable neural predictions with differentiable binary variables." arXiv preprint arXiv:1905.08160 (2019).

[4] Yu, Mo, et al. "Rethinking cooperative rationalization: Introspective extraction and complement control." arXiv preprint arXiv:1910.13294 (2019).

[5] Lei, Tao, Regina Barzilay, and Tommi Jaakkola. "Rationalizing neural predictions." arXiv preprint arXiv:1606.04155 (2016).

[6] Jacovi, Alon, and Yoav Goldberg. "Aligning faithful interpretations with their social attribution." Transactions of the Association for Computational Linguistics 9 (2021): 294-310.

[7] Zheng, Yiming, et al. "The irrationality of neural rationale models." arXiv preprint arXiv:2110.07550 (2021).

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: - can you think about using graph model to extend the explanability from single feature based to feature interaction based.
- in your evaluation, can you provide more details about features in EDU example, in Figure 10 in appendix, many features are similar, how these teachers could look at these features to measure whether "“This student was predicted to pass the course because and only because of the student’s regularity and video watching behavior".
- can you please provide more details on how the user predefined features (or user preferred features) are considered in feature gating and group routing.
soundness: 3
strengths: The unique strength of this paper is to put user-centric design into intrinsic explanation models, aiming to address the applicability of developed models in general domains. It is nice to see that efforts of pushing model design into user-centric design.

It extends the intrinsic explanation models from visions to other modalities, especially time series, text and tabular datasets. 

It has conducted systematic evaluation of four domains with their baseline different explanation models and measured the interpretation based on several well-defined metrics. 

The user evaluation of recruiting 56 teachers to judge the usefulness of the interpretation is a great plus.
summary: This paper proposed the intrinsic explanation framework called InterpretCC to combine feature gating and group routing based on the design space by considering user-centric features to enable actionable interpretation without sacrificing prediction performance compared with traditional prediction models. It conducted user evaluation and extended intrinsic explanation models to other less popular modalities, such as time series, tabular and text.
weaknesses: - In real world scenarios, explanations are not just a set of features, rather than the interactions of a pair of features. Do you consider to identify the interactions of features in your interpretCC framemwork
- In your user evaluation, as your method is providing local interpretation, as mentioned that interpretCC can recommend interpretation like "“This student was predicted to pass the course because and only because of the student’s regularity and video watching behavior". How can you prove such if and only if situation, because all prediction methods in interpretCC are association based, not causal based.
- can you please provide more details on how to deal with the grouping of features (it is great to see that you are using LLM to group features) for MoE, but grouping features itself can be challenge as some features might belong to two different groups, which could lead to errors in feature gating, and MoE.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Please refers to my questions in the "weakness section". The general landscape of explainable machine learning requires a stronger and more scientific taxonomy / definitions of these terms, the authors are thus encourage to discuss / elaborate on this with respect to their proposed methods.

I think this paper also lack a bit of theoretical analysis/justification (or at least tried to explain why the global mixture of experts method make sense and why it works for all these tasks). Could the authors try to discuss this?
soundness: 2
strengths: 1. the motivation of this work is clear and solid, and I believe intrinsic explainable machine learning should definitely be an important direction of research with great importance
2. good amount of experiments are conducted (including time series, text and tabular inputs) and the experimental results seems to be reasonable
3. the proposed methods seems to be reasonable novel (based on a global mixture of experts, with non-overlapping group/ set selections)
summary: This paper introduce a set of intrinsically (by design) interpretable machine learning methods, motivated by a global mixture of experts method. The goal of these methods (InterpretCC) focus on accurate predictions and aims to provide faithful explanations. The authors conduct experiments on a variety of tasks, including some user studies, and demonstrated the effectiveness of their approach.
weaknesses: 1. there exist evidence of over-claiming (such as Table 1, method comparison; after reading the paper, I am not convinced that the proposed methods actually achieve "faithfulness", and maybe somewhat allow full "coverage") and definition of these terms are also not clear / or there is no solid definition (although briefly mentioned in background -- Interpretability Foundations)

2. I am fully understand that explanation methods like InterpretCC are build based on expert knowledge or domain expertise (based on l231-296), are other intrinsic baseline methods adopt the same concepts/expert knowledge? would this really be a fair comparison otherwise? Also, for explainable machine learning, there is a trade-off between injecting how much domain knowledge v.s. black box methods (such as LIME or SHARP), could you add a discussion on if such domain knowledge does not exist (as this can often be the case in real-life scenarios)?

Meta-review: This paper presents a novel intrinsically interpretable Mixture of Experts (MoE) framework that utilizes two neural networks: one for feature gating and the other for group routing. This approach offers interpretability by highlighting the specific features selected from the overall feature set.  The experimental results demonstrate that the proposed method achieves comparable performance to non-interpretable models while providing interpretable outputs.

The paper is well-written and easy to understand. The proposed method exhibits novelty, and the empirical results consistently support the authors' claims.  Importantly, the authors address the crucial challenge of model interpretability.

However, the type of explanation provided – "selected features" – may not always be highly interpretable or actionable. Additionally, the requirement for predefined feature subsets could pose challenges in certain domains.

Despite these limitations, the authors tackle an important problem with a novel approach supported by strong empirical results. Therefore, I recommend accepting this paper.

Decision: Accept (Poster)